An eye fundus image classification method and system based on a dual-path neural network and a dynamic weight adjustment fusion module

Abstract

The application relates to an eye fundus image classification method and system based on a double-path neural network and a dynamic weight adjustment fusion module, and belongs to the technical field of medical image processing. An original eye fundus image is acquired, and the eye fundus image is pretreated; the pretreated image is input into a feature extraction module for processing, the feature extraction module comprises a local path and a global path, a local path module extracts disease detail information by using a convolutional neural network; a global path module extracts disease contour information by using a visual transformer; the images processed through the local path and the global path are fused through a dynamic weight adjustment fusion module; and the fused images are subjected to a full connection layer and an activation function Softmax layer of a classifier to realize grading of the eye fundus images. The application combines the advantages of neural networks and image processing technologies, intelligently allocates weights by using a dynamic weight adjustment fusion module, optimizes a feature integration process, and improves the accuracy and efficiency of eye fundus image grading.

Description

TECHNICAL FIELD

[0001] The application relates to an eye fundus image classification method and system based on a dual-path neural network and a dynamic weight adjustment fusion module, and belongs to the technical field of medical image processing. BACKGROUND

[0002] In computer-aided diagnosis, automatic grading of glaucoma disease is an important task, which is crucial for early diagnosis and treatment. The grading of glaucoma disease is based on the number, size and type of retinal lesions in the fundus photograph, which can be divided into four levels (0-3) from no lesion to the most severe lesion. Glaucoma disease diagnosis based on fundus photographs is mainly based on the differences in these information, unlike simple binary classification, glaucoma disease grading requires algorithms to more effectively identify the subtle differences in these information in fundus photographs. Therefore, developing a deep learning model that can accurately and efficiently assist in glaucoma disease grading has important clinical value.

[0003] Currently, deep learning is mainly used in the research of glaucoma disease image recognition, and the local connection and weight sharing of convolutional neural network can only extract local semantic information of the image. In addition, the maximum pooling in the convolutional neural network only retains the most prominent information in the set range, resulting in a large amount of information loss and reducing the spatial resolution. On the other hand, the visual transformer can learn the position information between image contour information, which can to some extent make up for the defects of convolutional neural network, but the existing visual transformer network has limited feature extraction ability due to its shallow depth.

[0004] In order to overcome these challenges, a dual-path network structure combining the advantages of convolutional neural network and visual transformer, namely DSTNet, is proposed. The network dynamically allocates weights through a dynamic weight adjustment fusion module to optimize the feature integration process, in order to achieve better performance. This dual-path network structure combining convolutional neural network and visual transformer has shown its effectiveness and reliability in other fields, but has not been widely used in the grading task of glaucoma disease fundus images.

[0005] Early detection and diagnosis of glaucoma disease and efficient treatment planning are the key to prevent patients from losing vision. Due to the special nature of fundus images, there are often many noises in the images. These noises not only affect the visual perception, but also interfere with the results of image classification. In order to improve the consistency, accuracy and efficiency of diagnosis, deep neural network-based glaucoma disease fundus image classification algorithms have received more and more attention. However, there are still some problems in the current deep neural network-based glaucoma disease fundus image classification methods, such as large amount of calculation, low image processing quality of the model, and most of them are simple binary classification of glaucoma disease fundus images rather than detailed classification, which limits the further improvement of classification performance. At the same time, the accuracy of glaucoma disease fundus image classification has an important influence on the development of clinical treatment strategies, so the detailed classification of glaucoma disease fundus image is particularly important. In view of this image classification problem, a glaucoma disease fundus image classification method based on dual-path convolutional neural network-transformer network and dynamic weight adjustment fusion module is proposed. SUMMARY

[0006] In order to solve the challenges of the existing glaucoma disease fundus image classification method in accuracy and efficiency, the present application provides a fundus image classification method based on dual-path neural network and dynamic weight adjustment fusion module, which improves the accuracy and efficiency of fundus image classification.

[0007] The technical scheme of the present application is: a fundus image classification method based on dual-path neural network and dynamic weight adjustment fusion module, the method comprises:

[0008] Step 1: obtain the original fundus image and pre-process the fundus image, including standardizing the image to a uniform size and performing data enhancement such as rotation and flipping to improve the generalization ability of the model;

[0009] Step 2: input the pre-processed image into the local path module, which uses convolutional neural network (DSNet) to extract disease detail information, especially microaneurysms, hemorrhage, exudates and neovascularization;

[0010] Step 3: input the pre-processed image into the global path module, which uses visual transformer (Swin Transformer) to extract disease contour information and capture overall image information;

[0011] Step 4: fuse the images processed by the local path and the global path through the dynamic weight adjustment fusion module, which can dynamically allocate weights according to the importance of detail information and contour information, and optimize the feature integration process;

[0012] Step5: The fused image is classified by the full connection layer and the Softmax layer of the classifier to realize the grading of the fundus image, i.e., four-level classification (0-3) from no to the most severe.

[0013] Further, Step1 crops and scales the fundus original image to an appropriate size, normalizes the data before inputting the local path and the global path, and performs data augmentation in different data augmentation manners, including rotation at different angles and horizontal flip; specifically, Step1 includes:

[0014] Step1.1, the original fundus image is cropped with the lesion as the center, and the cropped image is uniformly adjusted to a resolution of 224x224 to ensure that the input image size is consistent;

[0015] Step1.2, the cropped image data is normalized; the calculation formula of normalization is as follows:

[0016]

[0017] wherein X is an original feature value, X normalized is a normalized feature value, μ is a mean value, and σ is a standard deviation; normalization can effectively improve the convergence speed and accuracy of the model;

[0018] Step1.3, in order to increase the diversity of data and prevent model overfitting, the data is augmented by rotation (15°, 45°, 90°) and horizontal flip.

[0019] Further, Step2 includes:

[0020] Step2.1, the input image is subjected to a series of convolutional layers and pooling layers to extract detailed features; a batch normalization and an activation function are followed after each convolutional layer to prevent overfitting and introduce nonlinearity;

[0021] Step2.2, the extracted detailed features are subjected to the full-dimensional dynamic convolution layer (ODConv) of the convolutional neural network DSNet to enhance the perception and adaptability of the detailed information;

[0022] wherein the full-dimensional dynamic convolution operation in the convolutional neural network is represented as:

[0023] y=(α s ⊙α c ⊙α f ⊙α w ⊙W)*x

[0024] wherein x is an input feature map, y is an output feature map, w is a convolution kernel, and α s , αc , alpha f , alpha w respectively represent the attention weights along the spatial dimensions, input channels, output channels and the number of convolution kernels; these attention weights are calculated by the input feature map x, and multiplied with the convolution kernel W through element-wise multiplication (⊙), realizing the dynamic weighting of the convolution kernel;

[0025] Step2.3, the image after feature enhancement is integrated through the feature fusion layer, and the features of different levels are integrated to obtain more rich feature representation.

[0026] The local path module in Step2 is composed of a DSNet, which is improved based on the StarNet structure and includes an all-dimensional dynamic convolution (ODConv) technology. By introducing the all-dimensional dynamic convolution (ODConv) technology, the ability of feature extraction is further enhanced.

[0027] The convolutional neural network module is composed of multiple convolutional layers and pooling layers; the processed fundus image is input into the constructed DSNet convolutional neural network module to extract the local deep semantic information of the image;

[0028] The Step3 includes:

[0029] Step3.1, the input image is subjected to the multi-layer perceptron (MLP) and layer normalization (LN) operation of the global path module to provide weights for the primary Transformer layer;

[0030] Step3.2, the window multi-head self-attention mechanism (W-MSA) and the shift window multi-head self-attention mechanism (SW-MSA) are used to extract the disease contour information of the image;

[0031] Wherein, the window multi-head self-attention mechanism (W-MSA) and the shift window multi-head self-attention mechanism (SW-MSA) are respectively represented as:

[0032] Omega (MSA) = 4hwC 2 + 2 (hw) 2 C

[0033] Omega (W-MSA) = 4hwC 2 + 2M 2 hwC

[0034] Wherein, h represents the height of the feature map, w represents the width of the feature map, C represents the depth of the feature map, and M represents the size of each window;

[0035] Step3.3, the disease contour information is further extracted and integrated through the iterative Transformer layer.

[0036] In Step3.2, the disease contour information of the image is extracted using the window multi-head self-attention mechanism and the shift window multi-head self-attention mechanism, which includes:

[0037] Initial feature division: the image is divided into fixed-size windows, and self-attention is calculated independently in each window. By introducing a shift window operation, feature interaction between windows is achieved;

[0038] In Step3.3, the disease contour information is further extracted and integrated through iterative Transformer layers, which includes:

[0039] Feature integration and output: the features of each window are further processed through layer normalization and multi-layer perception, and finally the contour information map is output.

[0040] The global path module is composed of a visual transformer Swin Transformer, which includes a multi-head self-attention mechanism and layer normalization. The visual transformer focuses on contour information extraction of the image. The processed fundus image is input into the Swin Transformer module. The Swin Transformer module can efficiently capture global information of the image through hierarchical structure and shift window multi-head self-attention mechanism, to extract global spatial information.

[0041] The processed fundus image is processed by multiple Transformer blocks in the Swin Transformer, each block containing a multi-head self-attention mechanism and a feedforward neural network. The intra-window self-attention mechanism and inter-window shift operation are used to effectively capture global and detailed information of the image. Through hierarchical structure design, the size of the feature map is gradually reduced, while the dimension of the feature is increased, to extract higher-level contour information.

[0042] Further, in Step4, the dynamic weight adjustment fusion module dynamically allocates weights according to the importance of the features through channel attention and spatial attention mechanisms, to achieve adaptive fusion of the features. Specifically, it includes:

[0043] Step4.1, the features extracted by the path of the convolutional neural network and the path of the visual transformer are processed through attention mechanisms and multi-scale convolution, respectively, to enhance the expression ability of the features;

[0044] Step4.2, the weight predictor dynamically allocates fusion weights according to the features of the two paths;

[0045] Step4.3, finally, the local and contour information are weighted and summed according to the weights, to obtain a comprehensive feature map that integrates local and global information.

[0046] Further, the Step4.1 includes:

[0047] Step4.1.1, detail information processing: enhancing the detail information through multi-scale convolution and channel attention mechanism;

[0048] Step4.1.2, contour information processing: processing the contour information through multi-scale convolution and spatial attention mechanism, for strengthening the feature expression of key regions.

[0049] Further, in the Step4.2:

[0050] The weight predictor operation in the dynamic weight adjustment fusion module is represented as:

[0051]

[0052] Where e l and e g are the weight prediction values of the detail information and the contour information, respectively, and α l and α g are the weights normalized by the softmax function; e i is the weight prediction value of the input feature.

[0053] The fundus image input local path and global path, the local and contour information extracted by the DSNet module and the Swin Transformer module are input into the dynamic weight adjustment fusion module for feature fusion. The dynamic weight adjustment fusion module uses multi-scale convolution, channel attention mechanism, spatial attention mechanism and weight prediction technology to dynamically allocate weights according to the importance of the features, realize adaptive fusion of the features, and deeply mine and efficiently integrate the features.

[0054] Further, the Step5 includes:

[0055] Step5.1, flattening the fused feature map and mapping the features through a fully connected layer;

[0056] Step5.2, converting the output of the fully connected layer into a probability distribution through a softmax activation function to obtain the probability score of each fundus image classification;

[0057] Step5.3, selecting the class with the highest probability score as the final fundus image classification result.

[0058] The fused feature map is input into a classification module, the classification module is composed of two full connection layers and a Softmax activation function.

[0059] The application further provides a glaucoma disease diagnosis and evaluation system based on a dual-path neural network and a dynamic weight adjustment fusion module.

[0060] The application further provides an electronic device, including a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the above-mentioned glaucoma disease diagnosis and evaluation system based on a dual-path neural network and a dynamic weight adjustment fusion module.

[0061] The application has the following advantages:

[0062] 1. The application combines the advantages of convolutional neural networks and visual transformers and is applied to the grading of glaucoma disease fundus images. The combination of the two network structures enables the classification model to extract a large amount of semantic feature information and spatial position information from the images, thereby improving the classification accuracy. The convolutional neural network path focuses on extracting detailed information, especially the details of lesions such as microaneurysms, hemorrhages, exudates and neovascularization, while the visual transformer path utilizes its powerful contour information extraction capability to capture overall image information. The combination of the two significantly improves the accuracy of glaucoma disease fundus image grading.

[0063] 2. In glaucoma disease grading, fundus images are often affected by noise and other factors, which interfere with the recognition and classification of lesions. The application uses full-dimensional dynamic convolution (ODConv) in the convolutional neural network path, which can adaptively adjust the convolution kernel weight, enhance the depth perception and adaptability of detailed information, and thus reduce the interference of noise. In addition, the introduction of the visual transformer path enables the network to more comprehensively capture the global contour information of the image, which is crucial for accurate diagnosis.

[0064] 3. Given that existing deep learning networks have a large amount of computation when processing medical images and have high requirements for image quality, the application constructs a lightweight DSTNet network based on the characteristics of fundus images, optimizes the feature fusion strategy and controls the parameter quantity of the network. While improving the classification accuracy, the training parameters are reduced, enabling the DSTNet to be efficiently applied to the small number of local hospital data sets, thereby realizing the grading task of glaucoma disease fundus images.

[0065] 4. The application also designs a dynamic weight adjustment fusion module, which can dynamically allocate weights according to the importance of detail information and contour information, and optimize the feature integration process. This adaptive fusion strategy makes DSTNet more flexible and effective when processing fundus images with different lesion characteristics.

[0066] 5. Experimental results show that the performance of the application on the local hospital dataset is better than that of existing glaucoma disease fundus image grading methods, verifying the technical breakthrough and clinical application potential of the application in glaucoma disease fundus image grading, providing a new and efficient solution for assisting early diagnosis and treatment, and helping to improve the vision protection and quality of life of glaucoma patients. BRIEF DESCRIPTION OF DRAWINGS

[0067] Figure 1 is a flow structure schematic diagram of the method of the application;

[0068] Figure 2 is a convolutional neural network DSNet structure diagram of the method of the application;

[0069] Figure 3 is a DS Block structure diagram of the application;

[0070] Figure 4 is a dynamic weight adjustment fusion module structure diagram of the application. DETAILED DESCRIPTION

[0071] Example 1: As shown, a fundus image classification method based on a dual-path neural network and a dynamic weight adjustment fusion module, the method comprising: Figures 1-3

[0072] Step 1: Obtain the original fundus image and pre-process the fundus image, including standardizing the image to a uniform size and performing data enhancement such as rotation and flipping to improve the generalization ability of the model;

[0073] Further, the Step 1 crops and scales the original fundus image to an appropriate size, normalizes the data before inputting the local path and global path, and performs data augmentation according to different data enhancement methods, including rotation at different angles and horizontal flipping; specifically, the Step 1 comprises:

[0074] Step 1.1, crop the original fundus image with the lesion as the center, and uniformly adjust the cropped image to a resolution of 224x224 to ensure consistent input image size;

[0075] Step 1.2, normalize the cropped image data; the normalization calculation formula is as follows: ​

[0076]

[0077] where X is the original feature value, X normalized is the normalized feature value, μ is the mean, σ is the standard deviation, and normalization can effectively improve the convergence speed and accuracy of the model;

[0078] Step 1.3, in order to increase the diversity of data and prevent model overfitting, rotation (15°, 45°, 90°) and horizontal flip are used to augment the data.

[0079] Step 2: input the preprocessed image into the local path module, and the local path module uses convolutional neural network (DSNet) to extract disease detail information, especially microaneurysm, hemorrhage, exudate and neovascularization, etc.

[0080] Further, the Step 2 comprises:

[0081] Step 2.1, the input image is extracted through a series of convolutional layers and pooling layers, and each convolutional layer is followed by batch normalization and activation function to prevent overfitting and introduce nonlinearity;

[0082] Step 2.2, the extracted detail features are enhanced through the full-dimensional dynamic convolution layer (ODConv) of the convolutional neural network DSNet to enhance the perception and adaptability of the detail information;

[0083] where the full-dimensional dynamic convolution operation in the convolutional neural network is represented as:

[0084] y=(α s ⊙α c ⊙α f ⊙α w ⊙W)*x

[0085] where x is the input feature map, y is the output feature map, W is the convolution kernel, and α s , α c , α f , α w represent the attention weights along the spatial dimension, input channel, output channel and convolution kernel number respectively; these attention weights are calculated through the input feature map x, and multiplied with the convolution kernel W through element-wise multiplication (⊙), realizing the dynamic weighting of the convolution kernel;

[0086] Step 2.3, the feature enhanced image is input into the feature fusion layer to integrate the features at different levels to obtain more rich feature representation.

[0087] The local path module in Step2 is composed of a DSNet, which is improved based on a StarNet structure and includes an all-dimensional dynamic convolution (ODConv) technology. By introducing the all-dimensional dynamic convolution (ODConv) technology, the ability of feature extraction is further enhanced.

[0088] The convolutional neural network module is composed of multiple convolutional layers and pooling layers; the processed fundus image is input into the constructed DSNet convolutional neural network module to extract the local deep semantic information of the image.

[0089] Step3: input the preprocessed image into the global path module, and the global path module uses a visual transformer (Swin Transformer) to extract the disease contour information and capture the overall information of the image.

[0090] The Step3 includes:

[0091] Step3.1, the input image is subjected to a multi-layer perceptron (MLP) and a layer normalization (LN) operation of the global path module to provide weights for the primary Transformer layer.

[0092] Step3.2, the window multi-head self-attention mechanism (W-MSA) and the shift window multi-head self-attention mechanism (SW-MSA) are used to extract the disease contour information of the image.

[0093] Wherein, the window multi-head self-attention mechanism (W-MSA) and the shift window multi-head self-attention mechanism (SW-MSA) are respectively represented as:

[0094] Ω(MSA)=4hwC 2 +2(hw) 2 C

[0095] Ω(W-MSA)=4hwC 2 +2M 2 hwC

[0096] Wherein, h represents the height of the feature map, w represents the width of the feature map, C represents the depth of the feature map, and M represents the size of each window.

[0097] Step3.3, the disease contour information is further extracted and integrated through the iterative Transformer layer.

[0098] Wherein, in Step3.2, the window multi-head self-attention mechanism and the shift window multi-head self-attention mechanism are used to extract the disease contour information of the image, including:

[0099] Initial feature division: the image is divided into fixed-size windows, and self-attention is calculated independently within each window. By introducing a shift window operation, feature interaction between windows is achieved.

[0100] In Step3.3, the disease contour information is further extracted and integrated through iterative Transformer layers, including:

[0101] Feature integration and output: the features of each window are further processed through layer normalization and multi-layer perception, and the contour information map is finally output.

[0102] The global path module is composed of a visual transformer Swin Transformer, including a multi-head self-attention mechanism and layer normalization. The visual transformer focuses on the extraction of contour information of the image. The processed fundus image is input into the Swin Transformer module. The Swin Transformer module can efficiently capture the global information of the image through hierarchical structure and shift window multi-head self-attention mechanism, to extract global spatial information.

[0103] The processed fundus image is processed through multiple Transformer blocks in the Swin Transformer, each block containing a multi-head self-attention mechanism and a feedforward neural network. The intra-window self-attention mechanism and inter-window shift operation are used to effectively capture the global and detailed information of the image. Through hierarchical structure design, the size of the feature map is gradually reduced, while the dimension of the feature is increased, to extract higher-level contour information.

[0104] Step4: The images processed by the local path and the global path are fused through a dynamic weight adjustment fusion module. This module can dynamically allocate weights according to the importance of detailed information and contour information, optimizing the feature integration process.

[0105] Further, in Step4, the dynamic weight adjustment fusion module dynamically allocates weights according to the importance of features through channel attention and spatial attention mechanisms, achieving adaptive fusion of features. Specifically, it includes:

[0106] Step4.1, the features extracted by the convolutional neural network path and the visual transformer path are processed through attention mechanisms and multi-scale convolution respectively, to enhance the expression ability of the features.

[0107] Step4.2, the weight predictor dynamically allocates fusion weights according to the features of the two paths.

[0108] Step4.3, finally, the local and contour information are weighted and summed according to the weights, to obtain a comprehensive feature map that integrates local and global information.

[0109] Further, the Step4.1 comprises:

[0110] Step4.1.1, detail information processing: enhancing the detail information through multi-scale convolution and channel attention mechanism;

[0111] Step4.1.2, contour information processing: processing the contour information through multi-scale convolution and spatial attention mechanism, for strengthening the feature expression of key regions.

[0112] Further, in the Step4.2:

[0113] The weight predictor operation in the dynamic weight adjustment fusion module is represented as:

[0114]

[0115] Wherein, e l and e g are the weight prediction values of the detail information and the contour information, respectively, and α l and α g are the weights normalized by the softmax function; e i is the weight prediction value of the input feature.

[0116] The fundus image input local path and global path, the local and contour information extracted by the DSNet module and the Swin Transformer module are input into the dynamic weight adjustment fusion module for feature fusion. The dynamic weight adjustment fusion module uses multi-scale convolution, channel attention mechanism, spatial attention mechanism and weight prediction technology to dynamically allocate weights according to the importance of the features, realize adaptive fusion of the features, and deeply mine and efficiently integrate the features.

[0117] Step5: The fused image is passed through the full connection layer and the activation function Softmax layer of the classifier to realize the grading of the fundus image.

[0118] Further, the Step5 comprises:

[0119] Step5.1, flatten the fused feature map and map the features through the full connection layer;

[0120] Step5.2, convert the output of the full connection layer into a probability distribution through the softmax activation function to obtain the probability score of each fundus image grading;

[0121] Step5.3, select the class with the highest probability score as the final fundus image grading result.

[0122] The fused feature map is input into a classification module, and the classification module is composed of two full connection layers and a Softmax activation function.

[0123] The application further provides a glaucoma disease diagnosis and evaluation system based on a dual-path neural network and a dynamic weight adjustment fusion module, which comprises:

[0124] An image preprocessing module is used to acquire an original fundus image and pre-process the fundus image;

[0125] A detail information extraction module is used to input the pre-processed image into a local path module, and the local path module extracts disease detail information by using a convolutional neural network;

[0126] A disease contour information extraction module is used to input the pre-processed image into a global path module, and the global path module extracts disease contour information by using a visual transformer;

[0127] A fusion module is used to fuse the images processed by the local path and the global path through a dynamic weight adjustment fusion module;

[0128] A fundus image classification module is used to realize the grading of the fundus image by the full connection layer and the activation function Softmax layer of the classifier after the fusion of the images.

[0129] The application further provides an electronic device comprising a memory, a processor and a computer program stored in the memory and capable of running on the processor, and the processor realizes the fundus image classification method based on the dual-path neural network and the dynamic weight adjustment fusion module when executing the program.

[0130] To verify the effectiveness of the method, the performance of the method was evaluated on a glaucoma disease fundus image dataset provided by a medical university affiliated hospital. The dataset used by the present application is from the glaucoma disease fundus image provided by a medical university affiliated hospital, which includes 492 normal fundus images, 297 early glaucoma images, 173 medium glaucoma fundus images and 306 late glaucoma disease fundus images. For each ultrasound image, the ultrasound record and pathological report are sorted by professional physicians. In this process, the training set and the test set are randomly divided in the ratio of 8:2. The algorithm of the present application is developed under the Pytorch framework, and is trained on an NVIDIA GeForce RTX3060 graphics card. The Adam optimization algorithm and the cross-entropy loss function are used in the training process, and the hyperparameters are set as follows: the maximum Epoch number is 200, the learning rate is 0.0005, and the batch size is 32.

[0131] The present application adopts accuracy Accuracy, precision Precision, balanced F-score F1 accuracy (ACC), area under the curve (AUC), quadratic weighted kappa measure (QWK) and F1-score to evaluate the classification performance of the method in glaucoma disease image recognition. The formula of each index is as follows:

[0132]

[0133] Wherein, TP (True Positive) is the positive sample predicted as positive class by the model, FN (False Negative) is the positive sample predicted as negative class by the model, TN (True Negative) is the negative sample predicted as negative class by the model, and FP (False Positive) is the negative sample predicted as positive class by the model.

[0134] In order to verify the effectiveness of the present application, two groups of experiments were carried out. The comparative test of improved DSNet and traditional convolutional neural network; The comparative test with the existing glaucoma disease diagnosis method model.

[0135] Table 1 is the objective evaluation comparison of different methods on the test set of hospital dataset

[0136] Model Parameters ACC QWK F1 AUC VGG16 134.28 69.94 77.53 69.61 88.73 ResNet50 23.51 70.70 78.38 70.60 89.38 DenseNet121 6.95 70.95 95.16 70.56 89.17 EfficientNetB0 4.01 69.69 79.65 69.48 89.65 InceptionV3 21.79 73.48 80.02 73.36 90.68 MobileNetv2 2.23 71.21 77.68 71.24 90.24 The present invention 7.95 74.49 79.78 74.50 91.67 DSTNet - 80.30 80.97 80.36 93.08

[0137] Compared with traditional convolutional neural network networks such as VGG16, ResNet50, InceptionV3 and the like, the DSNet still realizes excellent classification accuracy while greatly reducing the parameter amount. In addition, compared with traditional lightweight networks such as EfficientNetB0 and MobileNetV2, the DSNet also achieves excellent performance while keeping similar parameter amounts. The DSTNet method proposed in the application is superior to other methods in various index comparisons.

[0138] The specific embodiments of the application are described in detail above with reference to the accompanying drawings, but the application is not limited to the above-described embodiments, and various changes can be made within the knowledge of those skilled in the art without departing from the purpose of the application.

Claims

1. A fundus image classification method based on a dual-path neural network and a dynamic weight adjustment fusion module, characterized by: The method comprises: Step 1: Obtain the original fundus image and preprocess the fundus image; Step 2: The pre-processed image is input into the local path module, which uses a convolutional neural network to extract disease details. Step 3: The pre-processed image is input into the global path module, which uses the visual transformer to extract the disease contour information; Step 4: The images passing through the local path and the global path are fused through the dynamic weight adjustment fusion module; Step 5: Pass the fused image through the fully connected layer of the classifier and the activation function Softmax layer to achieve fundus image classification; Step 3 includes: Step 3.1, pass the input image through the multi-layer perceptron and layer normalization operation of the global path module; Step 3.2: Use the window multi-head self-attention mechanism and the shifted window multi-head self-attention mechanism to extract the disease contour information of the image; Among them, the window multi-head self-attention mechanism and the shift window multi-head self-attention mechanism are respectively expressed as: Ω(MSA)=4hwC 2 +2(hw) 2 C Ω(W-MSA)=4hwC 2 +2M 2 hwC Where h represents the height of the feature map, w represents the width of the feature map, C represents the depth of the feature map, and M represents the size of each window; Step 3.3: Further extract and integrate disease profile information through iterative Transformer layers; Among them, in Step 3.2, the use of the window multi-head self-attention mechanism and the shifted window multi-head self-attention mechanism to extract the disease contour information of the image includes: Initial feature partitioning: The image is divided into fixed-size windows, and self-attention is calculated independently within each window. By introducing a shift window operation, feature interaction between windows is achieved; In Step 3.3, the iterative Transformer layer further extracts and integrates the disease profile information, including: Feature integration and output: The features of each window are further processed through layer normalization and multi-layer perceptron, and finally the contour information map is output.

2. The fundus image classification method based on a dual-path neural network and a dynamic weight adjustment fusion module according to claim 1, characterized in that: Step 1 includes: Step 1.

1. Crop the original fundus image with the lesion as the center and adjust the cropped image to a uniform resolution of 224×224; Step 1.2: Normalize the cropped image data. The normalization formula is as follows: Among them, X is the original eigenvalue, X normalized is the normalized eigenvalue, μ is the mean, and σ is the standard deviation; Step 1.3: Amplify the data by rotating and horizontally flipping.

3. The fundus image classification method based on a dual-path neural network and a dynamic weight adjustment fusion module according to claim 1, characterized in that: Step 2 includes: Step 2.1, the input image is passed through a series of convolutional layers and pooling layers to extract detailed features; Step 2.2: Enhance the extracted detail features through the full-dimensional dynamic convolution layer of the convolutional neural network; Among them, the full-dimensional dynamic convolution operation in the convolutional neural network is expressed as: y=(a s ⊙a c ⊙a f ⊙a w ⊙W*x Among them, x is the input feature map, y is the output feature map, W is the convolution kernel, α s , α c , α f , α w Represent the attention weights along the spatial size, input channels, output channels, and number of convolution kernels respectively; Step 2.3: Pass the feature-enhanced image through the feature fusion layer to integrate features at different levels.

4. The fundus image classification method based on a dual-path neural network and a dynamic weight adjustment fusion module according to claim 1, characterized in that: In Step 4, the dynamic weight adjustment fusion module dynamically allocates weights according to the importance of features through channel attention and spatial attention mechanisms to achieve adaptive fusion of features; Specifically include: Step 4.1: Process the features extracted from the convolutional neural network path and the visual transformer path through the attention mechanism and multi-scale convolution respectively; Step 4.2: Use the weight predictor to dynamically assign fusion weights based on the characteristics of the two paths; Step 4.3: Finally, the local and contour information are weighted and summed to obtain a comprehensive feature map that integrates local and global information.

5. The fundus image classification method based on a dual-path neural network and a dynamic weight adjustment fusion module according to claim 1, characterized in that: Step 4.1 includes: Step 4.1.1, detail information processing: enhance detail information through multi-scale convolution and channel attention mechanism; Step 4.1.2, contour information processing: The contour information is processed through multi-scale convolution and spatial attention mechanism to enhance the feature expression of key areas.

6. The fundus image classification method based on a dual-path neural network and a dynamic weight adjustment fusion module according to claim 1, characterized in that: In Step 4.2: The weight predictor operation in the dynamic weight adjustment fusion module is expressed as: Among them, e l and e g are the weighted prediction values ​​of detail information and contour information, α l and α g is the weight normalized by the softmax function; e i Refers to the weighted predicted value of the input feature.

7. The fundus image classification method based on a dual-path neural network and a dynamic weight adjustment fusion module according to claim 1, characterized in that: Step 5 includes: Step 5.1, flatten the fused feature map and perform feature mapping through the fully connected layer; Step 5.2: Convert the output of the fully connected layer into a probability distribution through the softmax activation function to obtain the probability score of each fundus image grade; Step 5.

3. Select the category with the highest probability score as the final fundus image grading result.

8. A glaucoma disease diagnosis and evaluation system based on a dual-path neural network and a dynamic weight adjustment fusion module, characterized in that: include: A module for executing a fundus image classification method based on a dual-path neural network and a dynamic weight adjustment fusion module as described in any one of claims 1 to 4.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it implements the fundus image classification method based on a dual-path neural network and a dynamic weight adjustment fusion module as described in any one of claims 1 to 4.

Images