Eye fundus image classification method and system based on dual-path neural network and dynamic weight adjustment fusion module

Abstract

The invention relates to a fundus image classification method and system based on a dual-path neural network and a dynamic weight adjustment fusion module, and belongs to the technical field of medical image processing. Obtaining an original fundus image, and preprocessing the fundus image; the preprocessed image is input into a feature extraction module for processing, the feature extraction module comprises a local path and a global path, and the local path module extracts disease detail information by using a convolutional neural network; the global path module extracts disease contour information by using a visual converter; fusing the images passing through the local path and the global path through a dynamic weight adjustment fusion module; and the fundus images are graded through a full connection layer of a classifier and an activation function Softmax layer of the fused images. According to the method, the advantages of the neural network and the image processing technology are combined, the dynamic weight adjustment fusion module is utilized to intelligently distribute the weight, the feature integration process is optimized, and the accuracy and efficiency of eye fundus image grading are improved.

Description

Technical Field

[0001] The invention relates to a fundus image classification method and system based on a dual-path neural network and a dynamic weight adjustment fusion module, and belongs to the technical field of medical image processing. Background Art

[0002] In computer-aided diagnosis, the automatic classification of glaucoma disease is an important task, which is crucial for early diagnosis and treatment. Glaucoma disease classification can be divided into four levels (0-3) from no lesions to the most severe lesions according to the number, size and type of retinal lesions in fundus photographs. The diagnosis of glaucoma disease based on fundus photographs is mainly based on the differences in this information. Unlike simple binary classification, glaucoma disease classification requires the algorithm to be able to more effectively identify the subtle differences in this information in fundus photographs. Therefore, it is of great clinical value to develop a deep learning model that can accurately and efficiently assist in the classification of glaucoma disease.

[0003] At present, deep learning research on glaucoma disease image recognition mainly uses convolutional neural networks. However, the local connectivity and weight sharing of convolutional neural networks result in only extracting local semantic information of the image, and the maximum pooling in the convolutional neural network only retains the most prominent information within the set range, resulting in a large amount of information loss and reduced spatial resolution. On the other hand, visual transformers can learn the position information between image contour information, which can make up for the defects of convolutional neural networks to a certain extent, but the existing visual transformer network is shallow and has limited feature extraction capabilities.

[0004] To overcome these challenges, a dual-path network structure combining the advantages of convolutional neural networks and visual transformers, namely DSTNet, is proposed. The network dynamically allocates weights through a dynamic weight adjustment fusion module to optimize the feature integration process in order to achieve better performance. This dual-path network structure combining convolutional neural networks and visual transformers has demonstrated its effectiveness and reliability in other fields, but has not yet been widely used in the task of grading fundus images of glaucoma disease.

[0005] Early detection and diagnosis of glaucoma disease and formulation of efficient treatment plans are the key to prevent patients from losing their vision. Due to the special properties of fundus images, there are often a lot of noise in the images. These noises not only affect visual perception, but also interfere with the results of image classification. In order to improve the consistency, accuracy and efficiency of diagnosis, the classification algorithm of fundus images of glaucoma disease based on deep neural networks has received more and more attention. However, there are still some problems in the classification methods of fundus images of glaucoma disease based on deep neural networks, such as large amount of calculation, low image processing quality of the model, and most of them are simple binary classification of fundus images of glaucoma disease without detailed classification, which limits the further improvement of classification performance. At the same time, the accuracy of grading fundus images of glaucoma disease has an important impact on the formulation of clinical treatment strategies, so the detailed grading of fundus images of glaucoma disease is particularly important. In view of this image classification problem, a grading method of fundus images of glaucoma disease based on dual-path convolutional neural network-transformer network and dynamic weight adjustment fusion module is proposed. Summary of the invention

[0006] In order to solve the challenges of accuracy and efficiency in existing glaucoma fundus image grading methods, the present invention provides a fundus image classification method based on a dual-path neural network and a dynamic weight adjustment fusion module, which is used to improve the accuracy and efficiency of fundus image grading.

[0007] The technical solution of the present invention is: a fundus image classification method based on a dual-path neural network and a dynamic weight adjustment fusion module, the method comprising:

[0008] Step 1: Obtain the original fundus image and preprocess the fundus image, including standardizing the image to a uniform size and performing data enhancement, such as rotation and flipping, to improve the generalization ability of the model;

[0009] Step 2: The preprocessed image is input into the local path module, which uses a convolutional neural network (DSNet) to extract disease details, especially microaneurysms, hemorrhage, exudates, and neovascularization.

[0010] Step 3: Input the preprocessed image into the global path module, which uses the visual transformer (Swin Transformer) to extract the disease contour information and capture the overall image information;

[0011] Step 4: The images that have passed through the local path and the global path are fused through a dynamic weight adjustment fusion module. This module can dynamically assign weights according to the importance of detail information and contour information to optimize the feature integration process.

[0012] Step 5: The fused image is passed through the fully connected layer of the classifier and the activation function Softmax layer to achieve the classification of the fundus image, and is classified into four levels (0-3) from none to the most severe.

[0013] Furthermore, the Step 1 crops and scales the original fundus image to an appropriate size, normalizes the data before inputting the local path and the global path, and amplifies the data according to different data enhancement methods, including rotation at different angles and horizontal flipping; specifically, the Step 1 includes:

[0014] Step 1.1, crop the original fundus image with the lesion as the center, and adjust the cropped image to a uniform resolution of 224×224 to ensure the consistency of the input image size;

[0015] Step 1.2, normalize the cropped image data; the normalization calculation formula is as follows:

[0016]

[0017] Among them, X is the original eigenvalue, X normalized is the normalized eigenvalue, μ is the mean, and σ is the standard deviation. Normalization can effectively improve the convergence speed and accuracy of the model;

[0018] Step 1.3: In order to increase the diversity of the data and prevent the model from overfitting, the data is augmented by rotation (15°, 45°, 90°) and horizontal flipping.

[0019] Furthermore, the Step 2 includes:

[0020] Step 2.1, the input image is passed through a series of convolutional layers and pooling layers to extract detailed features. Each convolutional layer is followed by batch normalization and activation function to prevent overfitting and introduce nonlinearity;

[0021] Step 2.2, the extracted detail features are used through the full-dimensional dynamic convolution layer (ODConv) of the convolutional neural network DSNet to enhance the perception and adaptability of the detail information;

[0022] Among them, the full-dimensional dynamic convolution operation in the convolutional neural network is expressed as:

[0023] y=(α s ⊙α c ⊙α f ⊙α w ⊙W)*x

[0024] Among them, x is the input feature map, y is the output feature map, w is the convolution kernel, α s , αc , α f , α w Represent the attention weights along the spatial size, input channel, output channel and number of convolution kernels respectively; these attention weights are calculated through the input feature map x and multiplied with the convolution kernel W by element-wise multiplication (⊙), thus realizing the dynamic weighting of the convolution kernel;

[0025] Step 2.3: Pass the feature-enhanced image through the feature fusion layer to integrate features at different levels to obtain richer feature representation.

[0026] The local path module in the Step 2 is composed of DSNet. The DSNet module is improved based on the StarNet structure and includes the full-dimensional dynamic convolution (ODConv) technology. By introducing the full-dimensional dynamic convolution (ODConv) technology, the feature extraction capability is further enhanced.

[0027] The convolutional neural network module consists of multiple convolutional layers and pooling layers. The processed fundus image is input into the constructed DSNet convolutional neural network module to extract the local depth semantic information of the image.

[0028] The Step 3 includes:

[0029] Step 3.1, the input image is passed through the multi-layer perceptron (MLP) and layer normalization (LN) operations of the global path module to provide weights for the primary Transformer layer;

[0030] Step 3.2, use the window multi-head self-attention mechanism (W-MSA) and the shifted window multi-head self-attention mechanism (SW-MSA) to extract the disease contour information of the image;

[0031] Among them, the window multi-head self-attention mechanism (W-MSA) and the shifted window multi-head self-attention mechanism (SW-MSA) are respectively expressed as:

[0032] Ω(MSA)=4hwC 2 +2(hw) 2 C

[0033] Ω(W-MSA)=4hwC 2 +2M 2 HkDJ

[0034] Among them, h represents the height of the feature map, w represents the width of the feature map, C represents the depth of the feature map, and M represents the size of each window;

[0035] Step 3.3, further extract and integrate disease profile information through iterative Transformer layers.

[0036] Among them, in Step 3.2, the disease contour information of the image is extracted using the window multi-head self-attention mechanism and the shift window multi-head self-attention mechanism, including:

[0037] Initial feature division: The image is divided into windows of fixed size, and self-attention is calculated independently in each window. The shift window operation is introduced to achieve feature interaction between windows.

[0038] In Step 3.3, the disease profile information is further extracted and integrated through the iterative Transformer layer, including:

[0039] Feature integration and output: The features of each window are further processed through layer normalization and multi-layer perceptron, and finally the contour information map is output.

[0040] The global path module is composed of a visual transformer Swin Transformer, including a multi-head self-attention mechanism and layer normalization; the visual transformer focuses on extracting the contour information of the image, and inputs the processed fundus image into the Swin Transformer module. The Swin Transformer module can efficiently capture the global information of the image through a hierarchical structure and a shift window multi-head self-attention mechanism to extract global spatial information.

[0041] The processed fundus image is processed by multiple Transformer blocks in Swin Transformer, each of which contains a multi-head self-attention mechanism and a feed-forward neural network. The self-attention mechanism within the window and the shift operation between windows are used to effectively capture the global and detailed information of the image. Through hierarchical structural design, the size of the feature map is gradually reduced, while the dimension of the feature is increased to extract higher-level contour information.

[0042] Furthermore, in the Step 4, the dynamic weight adjustment fusion module dynamically allocates weights according to the importance of features through the channel attention and spatial attention mechanisms to achieve adaptive fusion of features; specifically, it includes:

[0043] Step 4.1, the features extracted from the path of the convolutional neural network and the path of the visual transformer are processed through the attention mechanism and multi-scale convolution respectively to enhance the expressiveness of the features;

[0044] Step 4.2, use the weight predictor to dynamically assign fusion weights based on the features of the two paths;

[0045] Step 4.3, finally, the local and contour information are weighted and summed to obtain a comprehensive feature map that integrates local and global information.

[0046] Furthermore, the Step 4.1 includes:

[0047] Step 4.1.1, detail information processing: enhance the detail information through multi-scale convolution and channel attention mechanism;

[0048] Step 4.1.2, contour information processing: The contour information is processed through multi-scale convolution and spatial attention mechanism to enhance the feature expression of key areas.

[0049] Furthermore, in Step 4.2:

[0050] The weight predictor operation in the dynamic weight adjustment fusion module is expressed as:

[0051]

[0052] Among them, e l and e g are the weighted prediction values ​​of detail information and contour information, α l and α g is the weight normalized by the softmax function; e i Refers to the weighted predicted value of the input feature;

[0053] The local path and global path of the fundus image are input, and the local and contour information extracted by the DSNet module and the Swin Transformer module are input into the dynamic weight adjustment fusion module for feature fusion. The dynamic weight adjustment fusion module uses multi-scale convolution, channel attention mechanism, spatial attention mechanism and weight prediction technology to dynamically assign weights according to the importance of features, realize adaptive fusion of features, and deeply mine and efficiently integrate features.

[0054] Furthermore, the Step 5 includes:

[0055] Step 5.1, flatten the fused feature map and perform feature mapping through the fully connected layer;

[0056] Step 5.2, convert the output of the fully connected layer into a probability distribution through the softmax activation function to obtain the probability score of each fundus image classification;

[0057] Step 5.3. Select the category with the highest probability score as the final fundus image grading result.

[0058] The fused feature map is input into the classification module, which consists of two fully connected layers and a Softmax activation function. After being processed by the fully connected layer of the classification module, the probability score of the fundus image of glaucoma disease is obtained through the activation function Softmax layer, and finally the category with the highest score is taken as the classification result of the fundus image of glaucoma disease, realizing the automatic classification of the fundus image of glaucoma disease.

[0059] The present invention also provides a glaucoma disease diagnosis and evaluation system based on a dual-path neural network and a dynamic weight adjustment fusion module, comprising: a module for executing the above-mentioned fundus image classification method based on a dual-path neural network and a dynamic weight adjustment fusion module.

[0060] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for classifying fundus images based on a dual-path neural network and a dynamic weight adjustment fusion module is implemented.

[0061] The beneficial effects of the present invention are:

[0062] 1. The present invention combines the advantages of convolutional neural networks and visual transformers and is applied to the classification of fundus images of glaucoma diseases. By fusing the two network structures, the classification model can extract a large amount of semantic feature information and spatial position information in the image, thereby improving the classification accuracy. The convolutional neural network path focuses on extracting detail information, especially lesion details such as microaneurysms, hemorrhage, exudates, and new blood vessels, while the visual transformer path uses its powerful contour information extraction capability to capture the overall image information. The combination of the two significantly improves the accuracy of grading fundus images of glaucoma diseases.

[0063] 2. In glaucoma disease grading, fundus images are often affected by multiple factors such as noise, which interferes with the identification and classification of lesions. The present invention uses full-dimensional dynamic convolution (ODConv) in the convolutional neural network path. This technology can adaptively adjust the convolution kernel weights, enhance the depth perception and adaptability of detail information, and thus reduce the interference of noise. In addition, the introduction of the visual transformer path enables the network to more comprehensively capture the global contour information of the image, which is crucial for accurate diagnosis.

[0064] 3. In view of the large amount of computation required by existing deep learning networks when processing medical images and the high requirements for image quality, the present invention constructs a lightweight DSTNet network based on the characteristics of fundus images, optimizes the feature fusion strategy, and controls the number of network parameters. While improving the classification accuracy, it reduces the training parameters, allowing DSTNet to be efficiently applied to a small number of local hospital data sets to achieve the task of grading fundus images of glaucoma diseases.

[0065] 4. The present invention also designs a dynamic weight adjustment fusion module, which can dynamically assign weights according to the importance of detail information and contour information to optimize the feature integration process. This adaptive fusion strategy makes DSTNet more flexible and effective in processing fundus images with different lesion characteristics.

[0066] 5. Experimental results show that the performance of the present invention on the local hospital dataset is better than the existing glaucoma fundus image grading method, which verifies the technological breakthrough and clinical application potential of the present invention in glaucoma fundus image grading, and provides a new and efficient solution for assisting early diagnosis and treatment, which helps to improve the vision protection and quality of life of glaucoma patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is a schematic diagram of the process structure of the method of the present invention;

[0068] Figure 2 It is a structural diagram of the convolutional neural network DSNet of the method of the present invention;

[0069] Figure 3 It is the structural diagram of DS Block of the present invention;

[0070] Figure 4 This is a structural diagram of the dynamic weight adjustment fusion module of the present invention. DETAILED DESCRIPTION

[0071] Example 1: Figure 1-Figure 3 As shown, a fundus image classification method based on a dual-path neural network and a dynamic weight adjustment fusion module comprises:

[0072] Step 1: Obtain the original fundus image and preprocess the fundus image, including standardizing the image to a uniform size and performing data enhancement, such as rotation and flipping, to improve the generalization ability of the model;

[0073] Furthermore, the Step 1 crops and scales the original fundus image to an appropriate size, normalizes the data before inputting the local path and the global path, and amplifies the data according to different data enhancement methods, including rotation at different angles and horizontal flipping; specifically, the Step 1 includes:

[0074] Step 1.1, crop the original fundus image with the lesion as the center, and adjust the cropped image to a uniform resolution of 224×224 to ensure the consistency of the input image size;

[0075] Step 1.2, normalize the cropped image data; the normalization calculation formula is as follows:

[0076]

[0077] Among them, X is the original eigenvalue, X normalized is the normalized eigenvalue, μ is the mean, and σ is the standard deviation. Normalization can effectively improve the convergence speed and accuracy of the model;

[0078] Step 1.3: In order to increase the diversity of the data and prevent the model from overfitting, the data is augmented by rotation (15°, 45°, 90°) and horizontal flipping.

[0079] Step 2: The preprocessed image is input into the local path module, which uses a convolutional neural network (DSNet) to extract disease details, especially microaneurysms, hemorrhage, exudates, and neovascularization.

[0080] Furthermore, the Step 2 includes:

[0081] Step 2.1, the input image is passed through a series of convolutional layers and pooling layers to extract detailed features. Each convolutional layer is followed by batch normalization and activation function to prevent overfitting and introduce nonlinearity;

[0082] Step 2.2, the extracted detail features are used through the full-dimensional dynamic convolution layer (ODConv) of the convolutional neural network DSNet to enhance the perception and adaptability of the detail information;

[0083] Among them, the full-dimensional dynamic convolution operation in the convolutional neural network is expressed as:

[0084] y=(α s ⊙α c ⊙α f ⊙α w ⊙W)*x

[0085] Among them, x is the input feature map, y is the output feature map, W is the convolution kernel, α s , α c , α f , α w Represent the attention weights along the spatial size, input channel, output channel and number of convolution kernels respectively; these attention weights are calculated through the input feature map x and multiplied with the convolution kernel W by element-wise multiplication (⊙), thus realizing the dynamic weighting of the convolution kernel;

[0086] Step 2.3: Pass the feature-enhanced image through the feature fusion layer to integrate features at different levels to obtain richer feature representation.

[0087] The local path module in the Step 2 is composed of DSNet. The DSNet module is improved based on the StarNet structure and includes the full-dimensional dynamic convolution (ODConv) technology. By introducing the full-dimensional dynamic convolution (ODConv) technology, the feature extraction capability is further enhanced.

[0088] The convolutional neural network module consists of multiple convolutional layers and pooling layers. The processed fundus image is input into the constructed DSNet convolutional neural network module to extract the local depth semantic information of the image.

[0089] Step 3: Input the preprocessed image into the global path module, which uses the visual transformer (Swin Transformer) to extract the disease contour information and capture the overall image information;

[0090] The Step 3 includes:

[0091] Step 3.1, the input image is passed through the multi-layer perceptron (MLP) and layer normalization (LN) operations of the global path module to provide weights for the primary Transformer layer;

[0092] Step 3.2, use the window multi-head self-attention mechanism (W-MSA) and the shifted window multi-head self-attention mechanism (SW-MSA) to extract the disease contour information of the image;

[0093] Among them, the window multi-head self-attention mechanism (W-MSA) and the shifted window multi-head self-attention mechanism (SW-MSA) are respectively expressed as:

[0094] Ω(MSA)=4hwC 2 +2(hw) 2 C

[0095] Ω(W-MSA)=4hwC 2 +2M 2 HkDJ

[0096] Among them, h represents the height of the feature map, w represents the width of the feature map, C represents the depth of the feature map, and M represents the size of each window;

[0097] Step 3.3, further extract and integrate disease profile information through iterative Transformer layers.

[0098] Among them, in Step 3.2, the disease contour information of the image is extracted using the window multi-head self-attention mechanism and the shift window multi-head self-attention mechanism, including:

[0099] Initial feature division: The image is divided into windows of fixed size, and self-attention is calculated independently in each window. The shift window operation is introduced to achieve feature interaction between windows.

[0100] In Step 3.3, the disease profile information is further extracted and integrated through the iterative Transformer layer, including:

[0101] Feature integration and output: The features of each window are further processed through layer normalization and multi-layer perceptron, and finally the contour information map is output.

[0102] The global path module is composed of a visual transformer Swin Transformer, including a multi-head self-attention mechanism and layer normalization; the visual transformer focuses on extracting the contour information of the image, and inputs the processed fundus image into the Swin Transformer module. The Swin Transformer module can efficiently capture the global information of the image through a hierarchical structure and a shift window multi-head self-attention mechanism to extract global spatial information.

[0103] The processed fundus image is processed by multiple Transformer blocks in Swin Transformer, each of which contains a multi-head self-attention mechanism and a feed-forward neural network. The self-attention mechanism within the window and the shift operation between windows are used to effectively capture the global and detailed information of the image. Through hierarchical structural design, the size of the feature map is gradually reduced, while the dimension of the feature is increased to extract higher-level contour information.

[0104] Step 4: The images that have passed through the local path and the global path are fused through a dynamic weight adjustment fusion module. This module can dynamically assign weights according to the importance of detail information and contour information to optimize the feature integration process.

[0105] Furthermore, in the Step 4, the dynamic weight adjustment fusion module dynamically allocates weights according to the importance of features through the channel attention and spatial attention mechanisms to achieve adaptive fusion of features; specifically, it includes:

[0106] Step 4.1, the features extracted from the path of the convolutional neural network and the path of the visual transformer are processed through the attention mechanism and multi-scale convolution respectively to enhance the expressiveness of the features;

[0107] Step 4.2, use the weight predictor to dynamically assign fusion weights based on the features of the two paths;

[0108] Step 4.3, finally, the local and contour information are weighted and summed to obtain a comprehensive feature map that integrates local and global information.

[0109] Furthermore, the Step 4.1 includes:

[0110] Step 4.1.1, detail information processing: enhance the detail information through multi-scale convolution and channel attention mechanism;

[0111] Step 4.1.2, contour information processing: The contour information is processed through multi-scale convolution and spatial attention mechanism to enhance the feature expression of key areas.

[0112] Furthermore, in Step 4.2:

[0113] The weight predictor operation in the dynamic weight adjustment fusion module is expressed as:

[0114]

[0115] Among them, e l and e g are the weighted prediction values ​​of detail information and contour information, α l and α g is the weight normalized by the softmax function; e i Refers to the weighted predicted value of the input feature;

[0116] The local path and global path of the fundus image are input, and the local and contour information extracted by the DSNet module and the Swin Transformer module are input into the dynamic weight adjustment fusion module for feature fusion. The dynamic weight adjustment fusion module uses multi-scale convolution, channel attention mechanism, spatial attention mechanism and weight prediction technology to dynamically assign weights according to the importance of features, realize adaptive fusion of features, and deeply mine and efficiently integrate features.

[0117] Step 5: The fused image is passed through the fully connected layer of the classifier and the activation function Softmax layer to achieve fundus image classification.

[0118] Furthermore, the Step 5 includes:

[0119] Step 5.1, flatten the fused feature map and perform feature mapping through the fully connected layer;

[0120] Step 5.2, convert the output of the fully connected layer into a probability distribution through the softmax activation function to obtain the probability score of each fundus image classification;

[0121] Step 5.3. Select the category with the highest probability score as the final fundus image grading result.

[0122] The fused feature map is input into the classification module, which consists of two fully connected layers and a Softmax activation function. After being processed by the fully connected layer of the classification module, the probability score of the fundus image of glaucoma disease is obtained through the activation function Softmax layer. For example, the fundus image of glaucoma disease is classified into four levels (0-3) from none to the most severe, and finally the category with the highest score is obtained as the classification result of the fundus image of glaucoma disease, realizing the automatic classification of the fundus image of glaucoma disease.

[0123] The present invention also provides a glaucoma disease diagnosis and evaluation system based on a dual-path neural network and a dynamic weight adjustment fusion module, comprising:

[0124] An image preprocessing module, used for acquiring the original fundus image and preprocessing the fundus image;

[0125] A detail information extraction module is used to input the preprocessed image into the local path module, and the local path module uses a convolutional neural network to extract the disease detail information;

[0126] A module for extracting disease contour information is used to input the preprocessed image into a global path module, and the global path module extracts disease contour information using a visual transformer;

[0127] A fusion module, used for fusing images passing through the local path and the global path through a dynamic weight adjustment fusion module;

[0128] The fundus image classification module is used to classify the fundus image by passing the fused image through the fully connected layer of the classifier and the activation function Softmax layer.

[0129] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for classifying fundus images based on a dual-path neural network and a dynamic weight adjustment fusion module is implemented.

[0130] To verify the effectiveness of the method of the present invention, the performance of the method of the present invention was evaluated on a glaucoma fundus image dataset provided by a medical university affiliated hospital. The dataset used in the present invention comes from glaucoma fundus images provided by a medical university affiliated hospital, including 492 normal fundus images, 297 early glaucoma images, 173 mid-term glaucoma fundus images, and 306 late-stage glaucoma fundus images. For each ultrasound image, ultrasound records and pathology reports are sorted out by professional physicians. In this process, the present invention randomly divides the training set and the test set in a ratio of 8:2. The algorithm of the present invention was developed under the Pytorch framework and trained on an NVIDIA GeForce RTX3060 graphics card. The Adam optimization algorithm and the cross entropy loss function were used in the training process, and the hyperparameters were set as follows: the maximum number of epochs was 200, the learning rate was 0.0005, and the batch size was 32.

[0131] The present invention uses accuracy, precision, balanced F-score F1 accuracy (ACC), area under the curve (AUC), quadratic weighted kappa metric (QWK) and F1-score to evaluate the classification performance of the method in glaucoma disease image recognition. The formula for each indicator is as follows:

[0132]

[0133] Among them, TP (True Positive) is the positive sample predicted by the model as the positive class, FN (False Negative) is the positive sample predicted by the model as the negative class, TN (True Negative) is the negative sample predicted by the model as the negative class, and FP (False Positive) is the negative sample predicted by the model as the positive class.

[0134] In order to verify the effectiveness of the present invention, two groups of experiments were conducted: a comparative experiment between the improved DSNet and the traditional convolutional neural network; and a comparative experiment with the existing glaucoma disease diagnosis method model.

[0135] Table 1 shows the objective evaluation comparison of different methods on the hospital dataset test set

[0136] Model Parameters ACC QUR F1 AUC VGG16 134.28 69.94 77.53 69.61 88.73 ResNet50 23.51 70.70 78.38 70.60 89.38 DenseNet121 6.95 70.95 95.16 70.56 89.17 EfficientNetB0 4.01 69.69 79.65 69.48 89.65 InceptionV3 21.79 73.48 80.02 73.36 90.68 MobileNetv2 2.23 71.21 77.68 71.24 90.24 The present invention 7.95 74.49 79.78 74.50 91.67 DSTNet - 80.30 80.97 80.36 93.08

[0137] Compared with traditional convolutional neural networks such as VGG16, ResNet50, and InceptionV3, DSNet has achieved excellent classification accuracy while significantly reducing the number of parameters. In addition, compared with traditional lightweight networks such as EfficientNetB0 and MobileNetV2, DSNet has also achieved excellent performance while maintaining a similar number of parameters. The DSTNet method proposed in this invention is superior to other methods in comparison of various indicators.

[0138] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A fundus image classification method based on a dual-path neural network and a dynamic weight adjustment fusion module, characterized in that: The method comprises: Step 1: Obtain the original fundus image and preprocess the fundus image; Step 2: Input the preprocessed image into the local path module, which uses a convolutional neural network to extract disease details; Step 3: Input the preprocessed image into the global path module, which uses the visual transformer to extract the disease contour information; Step 4: The images passing through the local path and the global path are fused through a dynamic weight adjustment fusion module; Step 5: The fused image is passed through the fully connected layer of the classifier and the activation function Softmax layer to achieve fundus image classification.

2. The fundus image classification method based on a dual-path neural network and a dynamic weight adjustment fusion module according to claim 1, characterized in that: The Step 1 includes: Step 1.1, crop the original fundus image with the lesion as the center, and adjust the cropped image to a uniform resolution of 224×224; Step 1.2, normalize the cropped image data; the normalization calculation formula is as follows: Among them, X is the original eigenvalue, X normalized is the normalized eigenvalue, μ is the mean, and σ is the standard deviation; Step 1.3: Amplify the data by rotation and horizontal flipping.

3. The fundus image classification method based on dual-path neural network and dynamic weight adjustment fusion module according to claim 1 is characterized in that: The Step 2 includes: Step 2.1, extract detail features from the input image through a series of convolutional layers and pooling layers; Step 2.2, enhance the extracted detail features through the full-dimensional dynamic convolution layer of the convolutional neural network; Among them, the full-dimensional dynamic convolution operation in the convolutional neural network is expressed as: y=(a s ⊙a c ⊙a f ⊙a w ⊙W*x Among them, x is the input feature map, y is the output feature map, w is the convolution kernel, α s , α c , α f , α w Represent the attention weights along the spatial size, input channel, output channel and number of convolution kernels respectively; Step 2.3: Pass the feature-enhanced image through the feature fusion layer to integrate features at different levels.

4. The fundus image classification method based on dual-path neural network and dynamic weight adjustment fusion module according to claim 1 is characterized in that: The Step 3 includes: Step 3.1, pass the input image through the multi-layer perceptron and layer normalization operation of the global path module; Step 3.2, use the window multi-head self-attention mechanism and the shift window multi-head self-attention mechanism to extract the disease contour information of the image; Among them, the window multi-head self-attention mechanism and the shift window multi-head self-attention mechanism are respectively expressed as: Ω(MSA)=4hwC 2 +2(hw) 2 C Ω(W-MSA)=4hwC 2 +2M 2 hwC Among them, h represents the height of the feature map, w represents the width of the feature map, C represents the depth of the feature map, and M represents the size of each window; Step 3.3, further extract and integrate disease profile information through iterative Transformer layers. Among them, in Step 3.2, the disease contour information of the image is extracted using the window multi-head self-attention mechanism and the shift window multi-head self-attention mechanism, including: Initial feature division: The image is divided into windows of fixed size, and self-attention is calculated independently in each window. The shift window operation is introduced to achieve feature interaction between windows. In Step 3.3, the disease profile information is further extracted and integrated through the iterative Transformer layer, including: Feature integration and output: The features of each window are further processed through layer normalization and multi-layer perceptron, and finally the contour information map is output.

5. The fundus image classification method based on dual-path neural network and dynamic weight adjustment fusion module according to claim 1, characterized in that: In the Step 4, the dynamic weight adjustment fusion module dynamically allocates weights according to the importance of features through channel attention and spatial attention mechanisms to achieve adaptive fusion of features; Specifically include: Step 4.1, the features extracted from the path of the convolutional neural network and the path of the visual transformer are processed through the attention mechanism and multi-scale convolution respectively; Step 4.2, use the weight predictor to dynamically assign fusion weights based on the features of the two paths; Step 4.3, finally, the local and contour information are weighted and summed to obtain a comprehensive feature map that integrates local and global information.

6. The fundus image classification method based on dual-path neural network and dynamic weight adjustment fusion module according to claim 1 is characterized in that: The Step 4.1 includes: Step 4.1.1, detail information processing: enhance the detail information through multi-scale convolution and channel attention mechanism; Step 4.1.2, contour information processing: The contour information is processed through multi-scale convolution and spatial attention mechanism to enhance the feature expression of key areas.

7. The fundus image classification method based on dual-path neural network and dynamic weight adjustment fusion module according to claim 1 is characterized in that: In Step 4.2: The weight predictor operation in the dynamic weight adjustment fusion module is expressed as: Among them, e l and e g are the weighted prediction values ​​of detail information and contour information, α l and α g is the weight normalized by the softmax function; e i Refers to the weighted predicted value of the input feature.

8. The fundus image classification method based on dual-path neural network and dynamic weight adjustment fusion module according to claim 1, characterized in that: The Step 5 includes: Step 5.1, flatten the fused feature map and perform feature mapping through the fully connected layer; Step 5.2, convert the output of the fully connected layer into a probability distribution through the softmax activation function to obtain the probability score of each fundus image classification; Step 5.

3. Select the category with the highest probability score as the final fundus image grading result.

9. A glaucoma disease diagnosis and evaluation system based on a dual-path neural network and a dynamic weight adjustment fusion module, characterized in that: include: A module for executing a fundus image classification method based on a dual-path neural network and a dynamic weight adjustment fusion module as described in any one of claims 1 to 6.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it implements a fundus image classification method based on a dual-path neural network and a dynamic weight adjustment fusion module as described in any one of claims 1 to 6.

Images