Fundus image classification method and device, electronic equipment and storage medium

Abstract

The application provides an eye fundus image classification method and device, an electronic device and a storage medium, and belongs to the technical field of medical image processing. The eye fundus image is obtained, multi-scale feature extraction is performed on the eye fundus image, a plurality of eye fundus feature maps of different scales are obtained, attention feature extraction is performed on the eye fundus feature map of each scale, eye fundus attention features are obtained, feature splicing is performed on each eye fundus attention feature, multi-scale eye fundus fusion features are obtained, and multi-lesion classification is performed on the eye fundus image according to the multi-scale eye fundus fusion features. The accuracy of multi-lesion classification of the eye fundus image can be improved.

Description

Technical Field

[0001] This application relates to the field of medical image processing technology, and in particular to a method and apparatus for classifying fundus images, an electronic device, and a storage medium. Background Technology

[0002] Fundus images are crucial for diagnosing the presence of fundus diseases in patients. In related technologies, image classification models based on convolutional networks often use the extracted feature maps for fundus image classification. Different fundus diseases present with varying lesion sizes and extents in fundus images. Directly utilizing a single extracted feature map can miss crucial fundus information, leading to reduced accuracy in classifying fundus images when a single model simultaneously identifies multiple eye diseases. Therefore, improving the accuracy of classifying multiple lesions in fundus images has become an urgent problem to be solved. Summary of the Invention

[0003] The main objective of this application is to provide a fundus image classification method, fundus image classification device, electronic device and storage medium, which aim to improve the accuracy of classifying multiple lesions in fundus images.

[0004] To achieve the above objectives, a first aspect of this application proposes a fundus image classification method, the method comprising:

[0005] Acquire fundus images;

[0006] Multi-scale feature extraction is performed on the fundus image to obtain multiple fundus feature maps at different scales;

[0007] For each scale of the fundus feature map, attention features are extracted from the fundus feature map to obtain fundus attention features;

[0008] Each of the fundus attention features is spliced ​​together to obtain multi-scale fundus fusion features;

[0009] The fundus images are classified into multiple lesions based on the multi-scale fundus fusion features.

[0010] In some embodiments, for each scale of the fundus feature map, attention feature extraction is performed on the fundus feature map to obtain fundus attention features, including:

[0011] For each scale of the fundus feature map, channel attention features are extracted from the fundus feature map to obtain the target fundus channel attention features;

[0012] Spatial attention features are extracted from the target fundus channel attention features to obtain the target fundus spatial attention features;

[0013] The fundus attention features are obtained by combining the target fundus channel attention features and the target fundus spatial attention features.

[0014] In some embodiments, for each scale of the fundus feature map, channel attention feature extraction is performed on the fundus feature map to obtain target fundus channel attention features, including:

[0015] For each scale of the fundus feature map, channel attention features are extracted from the fundus feature map to obtain initial fundus channel attention features;

[0016] The initial fundus channel attention features and the fundus feature map are combined to obtain the target fundus channel attention features.

[0017] In some embodiments, for each scale of the fundus feature map, channel attention feature extraction is performed on the fundus feature map to obtain initial fundus channel attention features, including:

[0018] For each scale of the fundus feature map, the fundus feature map is averaged and pooled to obtain the first fundus pooling feature;

[0019] Max pooling is performed on the fundus feature map to obtain the second fundus pooling feature;

[0020] The first fundus pooling feature is convolved to obtain the first fundus convolution feature;

[0021] The second fundus pooling feature is convolved to obtain the second fundus convolution feature;

[0022] The initial fundus channel attention features are determined based on the first fundus convolution features and the second fundus convolution features.

[0023] In some embodiments, the step of extracting spatial attention features from the target fundus channel attention features to obtain target fundus spatial attention features includes:

[0024] Max pooling is performed on the attention features of the target fundus channel to obtain the first channel pooling features;

[0025] The target fundus channel attention features are averaged and pooled to obtain the second channel pooling features;

[0026] From the channel dimension, the first channel pooling feature and the second channel pooling feature are concatenated to obtain the channel concatenated feature;

[0027] The channel splicing features are convolutionally processed to obtain the target fundus spatial attention features.

[0028] In some embodiments, the multi-scale feature extraction of the fundus image to obtain multiple fundus feature maps at different scales includes:

[0029] The fundus image is subjected to shallow feature extraction to obtain shallow image features;

[0030] Multi-scale bottleneck feature extraction is performed on the shallow image features to obtain fundus feature maps of multiple different scales.

[0031] In some embodiments, prior to acquiring the fundus image, the fundus image classification method further includes:

[0032] Obtain a sample fundus image and at least one sample lesion label from the sample fundus image;

[0033] The sample fundus images are input into a preset model to classify fundus lesions and obtain at least one predicted lesion label;

[0034] The target loss data is obtained by calculating the cross-loss based on each sample lesion label and each predicted lesion label.

[0035] The model parameters of the preset model are adjusted based on the target loss data to obtain a multilesion classification model for fundus images.

[0036] To achieve the above objectives, a second aspect of this application provides a fundus image classification apparatus, the apparatus comprising:

[0037] The acquisition module is used to acquire fundus images;

[0038] A multi-scale feature extraction module is used to extract multi-scale features from the fundus image to obtain multiple fundus feature maps at different scales.

[0039] The attention feature extraction module is used to extract attention features from the fundus feature map for each scale to obtain fundus attention features;

[0040] The feature stitching module is used to stitch together multiple fundus attention features to obtain multi-scale fundus fusion features.

[0041] The fundus image classification module is used to classify multiple lesions in the fundus image based on the multi-scale fundus fusion features.

[0042] To achieve the above objectives, a third aspect of the present application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the fundus image classification method of the first aspect described above.

[0043] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the fundus image classification method of the first aspect described above.

[0044] The fundus image classification method, device, electronic device, and computer-readable storage medium proposed in this application acquire fundus images and perform multi-scale feature extraction to obtain multiple fundus feature maps at different scales. This fully extracts the feature information of the fundus images, avoiding the omission of key feature information that plays a crucial role in lesion classification, thereby improving the localization and recognition capabilities of lesions of different sizes. To extract the key feature information that plays a crucial role in the identification of lesions of different sizes and avoid interference from irrelevant information, attention features are extracted from the fundus feature maps at each scale to obtain fundus attention features. Each fundus attention feature is then concatenated to obtain multi-scale fundus fusion features, which comprehensively utilize fundus attention features at different scales for the detection of multiple fundus lesions, avoiding the omission of key feature information. Classifying multiple fundus lesions based on multi-scale fundus fusion features improves the accuracy of fundus image multi-lesion classification compared to methods that only use a single feature map. Attached Figure Description

[0045] Figure 1 This is a flowchart of the fundus image classification method provided in the embodiments of this application;

[0046] Figure 2 yes Figure 1 The flowchart of step S120 in the middle;

[0047] Figure 3a This is a schematic diagram of multi-scale feature extraction provided in an embodiment of this application;

[0048] Figure 3b This is another schematic diagram of multi-scale feature extraction provided in the embodiments of this application;

[0049] Figure 3c This is a schematic diagram of the bottleneck layer structure provided in an embodiment of this application;

[0050] Figure 3d This is another schematic diagram of the bottleneck layer provided in the embodiments of this application;

[0051] Figure 4 yes Figure 1 The flowchart of step S130 in the process;

[0052] Figure 5 This is a schematic diagram of attention feature extraction provided in an embodiment of this application;

[0053] Figure 6 yes Figure 4 The flowchart of step S410 in the middle;

[0054] Figure 7 yes Figure 6 The flowchart of step S610 in the text;

[0055] Figure 8 This is a schematic diagram of channel attention feature extraction provided in an embodiment of this application;

[0056] Figure 9 yes Figure 4 The flowchart of step S420 in the middle;

[0057] Figure 10 This is a schematic diagram of spatial attention feature extraction provided in an embodiment of this application;

[0058] Figure 11 This is a schematic diagram of fundus image classification provided in an embodiment of this application;

[0059] Figure 12 This is a flowchart of the training process of the fundus image multilesion classification model provided in the embodiments of this application;

[0060] Figure 13 This is a schematic diagram of the structure of the fundus image classification device provided in the embodiments of this application;

[0061] Figure 14 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0063] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0065] Fundus imaging is one of the most important bases for diagnosing the presence of fundus diseases in patients. Currently, there are a large number of patients suffering from various types and degrees of eye diseases. Typically, only a specialist doctor can accurately determine the presence of fundus diseases based on fundus images. Due to the large number of patients and the limited number of ophthalmologists, many patients do not receive timely diagnosis and treatment.

[0066] In recent years, thanks to the improved performance of Graphics Processing Units (GPUs), the accumulation of big data, and advancements in artificial intelligence (AI) technology, a large number of AI-based applications have emerged, serving everyday life. Meanwhile, medical professionals, by labeling medical data and using AI technology to train models based on the labeled data, have enabled some models to serve as screening tools for ophthalmic diseases, such as diabetic retinopathy.

[0067] However, when a single model is required to simultaneously determine whether multiple eye diseases are present in a fundus image, its performance is often unsatisfactory. This is mainly because currently popular image classification models based on convolutional networks primarily use the final feature map for classification. Different ophthalmic diseases present with lesions of varying sizes and extents in fundus images. Directly utilizing a single feature map can easily overlook some crucial information, leading to poor model performance and thus reducing the accuracy of classifying multiple lesions in fundus images.

[0068] Based on this, embodiments of this application provide a fundus image classification method, a fundus image classification device, an electronic device, and a computer-readable storage medium, aiming to improve the accuracy of classifying multiple lesions in fundus images.

[0069] The fundus image classification method, fundus image classification device, electronic device and computer-readable storage medium provided in the embodiments of this application are specifically described through the following embodiments. First, the fundus image classification method in the embodiments of this application is described.

[0070] The fundus image classification method provided in this application relates to the field of medical image processing technology. The fundus image classification method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the fundus image classification method, but is not limited to the above forms.

[0071] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0072] Figure 1 This is an optional flowchart of the fundus image classification method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S110 to S150.

[0073] Step S110: Obtain fundus image;

[0074] Step S120: Multi-scale feature extraction is performed on the fundus image to obtain multiple fundus feature maps at different scales;

[0075] Step S130: For each scale of fundus feature map, extract attention features from the fundus feature map to obtain fundus attention features;

[0076] Step S140: Perform feature stitching on each fundus attention feature to obtain multi-scale fundus fusion features;

[0077] Step S150: Classify multiple lesions in fundus images based on multi-scale fundus fusion features.

[0078] In step S110 of some embodiments, the fundus image is an image of the fundus region acquired through techniques such as fundus photography. Fundus images can provide important information about eye health and diseases. Fundus lesions can be detected by observing the structure of the retina, the morphology and distribution of blood vessels, pigmentary changes in the macular region, edema, cupping, and the shape and color of the optic disc in the fundus image. Fundus images can be obtained by taking pictures of the fundus region with a fundus camera, or by acquiring fundus images from a fundus image dataset. The fundus image dataset can be the ODIR5K dataset.

[0079] Different fundus diseases present with varying lesion sizes and extents in fundus images. To simultaneously detect lesions from multiple fundus diseases such as glaucoma, cataracts, and diabetic retinopathy, a more comprehensive fundus image representation applicable to multi-lesion detection is needed. This application uses feature maps output from multiple stages of a convolutional network-based image classification model to determine multi-scale features. It is understood that as the number of stages increases, the receptive field of the output feature map at that stage becomes larger, meaning the feature map represents a wider range, thus enabling the detection of larger targets in the fundus image. The convolutional network-based image classification model can be ResNet50 or other networks with similar stage output feature maps. Specifically, ResNet50 is used to extract multi-scale features from the fundus image. The unique residual structure of ResNet50 directly learns the differential features of different fundus structures and regions in the input fundus image, capturing deeper feature representations and obtaining multiple fundus feature maps at different scales. This avoids the loss of this information due to the stacking of network layers, improving the model's ability to perceive targets at different scales. Simultaneously, this residual structure helps improve gradient propagation. Multiple fundus feature maps at different scales include both global features representing overall information and local features representing detailed information. Scale refers to the size or spatial hierarchy of objects in the feature map. The process of multi-scale feature extraction using ResNet50 is as follows.

[0080] Please see Figure 2 In some embodiments, step S120 may include, but is not limited to, steps S210 to S220:

[0081] Step S210: Perform shallow feature extraction on the fundus image to obtain shallow image features;

[0082] Step S220: Multi-scale bottleneck feature extraction is performed on the shallow image features to obtain fundus feature maps of different scales.

[0083] In step S210 of some embodiments, such as Figure 3aAs shown, the input passes through five stages of ResNet50 to obtain the output. The input is a fundus image, and the output is the fundus feature map of the last stage. These five stages are Stage 0, Stage 1, Stage 2, Stage 3, and Stage 4. The features output by Stage 0 are the shallow image features. Shallow image features are quite similar to the fundus image, containing more pixel information, but their semantics are lower and they contain more noise. Shallow image features can be fine-grained information such as texture, color, shape, and edges.

[0084] like Figure 3b As shown, in stage 0, the fundus image is processed by convolutional layers to obtain convolutional features; the convolutional features are then batch normalized to obtain normalized fundus features; the normalized fundus features are activated using an activation function to obtain activated features; and the activated features are then max-pooled to obtain shallow image features. In stage 0, the input size is (3, 224, 224), where 3 represents the number of channels in the input features, and (224, 224) represents the width and height of the input features. The output size is (64, 56, 56), where 64 represents the number of channels in the output features, and (56, 56) represents the width and height of the output features. During convolution, the kernel size is 7×7, the number of kernels is 64, and the kernel stride is 2. During max-pooling, the kernel size is 3×3, and the kernel stride is 2. The activation function is ReLU.

[0085] In step S220 of some embodiments, as the network depth increases, the resolution of the feature maps gradually decreases. The feature maps gradually transition from low-level features such as edges and corners to high-level features such as objects and scenes, resulting in richer semantic information and containing more abstract and global features. Simultaneously, deeper features exhibit better invariance to changes in the position and shape of tissue structures in fundus images, making the model more robust and better able to handle complex fundus lesion classifications. Therefore, this embodiment uses the output feature maps of the last three stages of a convolutional network-based image classification model as multi-scale features to obtain multiple fundus feature maps at different scales. If the convolutional network-based image classification model is ResNet50, then the multiple fundus feature maps at different scales include the output feature maps of stages 2, 3, and 4. Fundus feature maps are deep image features, containing abstract semantic information. Fundus feature maps can reflect fundus information such as the morphology and distribution of blood vessels, the morphology of the macular region, and fundus exudates. It should be noted that in ResNet50, as the number of stages increases, the number of channels in the output feature map increases, while the width and height of the output feature map decrease, resulting in a larger receptive field for the output feature map, thus enabling the detection of targets at different scales.

[0086] ResNet50 introduces a bottleneck layer in the residual structure. For example... Figure 3bAs shown, in stage 1, bottleneck features are extracted from shallow image features using one bottleneck layer 1 and two bottleneck layers 2 to obtain the first-stage bottleneck features; in stage 2, bottleneck features are extracted from the first-stage bottleneck features using one bottleneck layer 1 and three bottleneck layers 2 to obtain the second-stage bottleneck features; in stage 3, bottleneck features are extracted from the second-stage bottleneck features using one bottleneck layer 1 and five bottleneck layers 2 to obtain the third-stage bottleneck features; in stage 4, bottleneck features are extracted from the third-stage bottleneck features using one bottleneck layer 1 and two bottleneck layers 2 to obtain the fourth-stage bottleneck features; the second-stage bottleneck features, the third-stage bottleneck features, and the fourth-stage bottleneck features are used as fundus feature maps of multiple different scales.

[0087] Bottleneck layer 1 has a different number of input channels and output channels, while bottleneck layer 2 has the same number of input channels and output channels. For example... Figure 3c As shown, bottleneck layer 1 includes four variable parameters: C, W, C1, and S. C represents the number of input channels, W represents the input size, C1 represents the number of output channels, and S represents the convolution stride. Bottleneck layer 1 consists of two branches, left and right. The input features are processed by three convolutional blocks in the left branch to obtain the output features of the left branch, and the input features are processed by one convolutional block in the right branch to obtain the output features of the right branch. The output features of the left and right branches are fused, and the fused features are activated by the ReLU activation function to obtain the output features of bottleneck layer 1. The number of channels in the input features is C, and the size of the input features is (W, W). The first convolutional block of the left branch includes a convolutional layer with a kernel size of 1×1, a kernel count of C1, and a stride of S, a batch normalization layer, and a ReLU activation layer. The second convolutional block of the left branch includes a convolutional layer with a kernel size of 3×3, a kernel count of C1, and a stride of 1, a batch normalization layer, and a ReLU activation layer. The third convolutional block in the left branch includes a convolutional layer with a kernel size of 1×1, a kernel count of C1*4, a stride of 1, and a batch normalization layer. The convolutional block in the right branch includes a convolutional layer with a kernel size of 1×1, a kernel count of C1*4, a stride of S, and a batch normalization layer. The output feature of bottleneck layer 1 has C1*4 channels and a feature size of (W / S, W / S). Figure 3b As shown, in stage 1, bottleneck features are extracted from the shallow image features of shape (64,56,56) through bottleneck layer 1 with variable parameters (64,56,64,1), and the output feature map has shape (256,56,56). Here, 256 is obtained by the third variable parameter 64*4, and 56 is obtained by 56 / 1.

[0088] like Figure 3dAs shown, bottleneck layer 2 includes two variable parameters, C and W, where C represents the number of input channels and W represents the input size. The input features are processed through three convolutional blocks to obtain convolutional output features. The convolutional output features and input features are then fused, and the fused features are activated using the ReLU activation function to obtain the output features of bottleneck layer 2. The number of channels in the input features is C, and the size of the input features is (W, W). The first convolutional block includes a convolutional layer with a kernel size of 1×1, a kernel count of C / 4, and a stride of 1, a batch normalization layer, and a ReLU activation layer. The second convolutional block includes a convolutional layer with a kernel size of 3×3, a kernel count of C / 4, and a stride of 1, a batch normalization layer, and a ReLU activation layer. The third convolutional block includes a convolutional layer with a kernel size of 1×1, a kernel count of C, and a stride of 1, and a batch normalization layer. The number of channels in the output features of bottleneck layer 2 is C, and the feature size is (W, W). Figure 3b As shown, in stage 1, bottleneck feature extraction is performed on the features with shape (256,56,56) through bottleneck layer 2 with variable parameters (256,56), and the output feature map with shape (256,56,56) is generated.

[0089] Steps S210 to S220, by progressively extracting and enhancing features, ensure that the features output at each stage contain semantic information at different scales. This semantic information at different scales is then used to identify lesions of different sizes and ranges in fundus images, thereby improving the accuracy of fundus lesion classification. Simultaneously, by extracting bottleneck features, the computational load of feature extraction is significantly reduced, improving the efficiency of fundus lesion classification while maintaining accuracy, enabling real-time detection of multiple fundus lesions.

[0090] To improve the ability to identify targets at different scales in fundus images, it is necessary to use fundus feature maps at multiple scales output by a convolutional network-based image classification model. A convolutional block attention module extracts attention features from each scale of the fundus feature map to identify and differentiate key features for different fundus lesions, while ignoring irrelevant features, thus obtaining fundus attention features and improving the accuracy and efficiency of multi-lesion classification. The convolutional block attention module consists of a channel attention module and a spatial attention module connected in series. These modules focus on different dimensions: the channel attention module extracts channel attention features, which characterize the channel dimensions important for fundus lesion detection; the spatial attention module extracts spatial attention features, which characterize the spatial dimensions important for fundus lesion detection. The input features can generally be represented as B*C*H*W, where B is the batch size, C is the number of channels, and H*W is the two-dimensional feature. The channel attention module operates on the C dimension, while the spatial attention module operates on the H*W dimension. Attention mechanisms such as channel attention and spatial attention are used to allow the model to focus on the importance of different dimensions, selecting the more important features from a large number of features. Through continuous filtering, better features are retained, while also increasing the richness of features. The specific operation of the convolutional block attention module is as follows.

[0091] Please see Figure 4 In some embodiments, step S130 may include, but is not limited to, steps S410 to S430:

[0092] Step S410: For each scale of fundus feature map, channel attention features are extracted from the fundus feature map to obtain the target fundus channel attention features.

[0093] Step S420: Extract spatial attention features from the target fundus channel attention features to obtain the target fundus spatial attention features;

[0094] Step S430: Combine the target fundus channel attention features and the target fundus spatial attention features to obtain fundus attention features.

[0095] In step S410 of some embodiments, a channel attention module is constructed. For the fundus feature maps of the last three stages, the channel attention module extracts channel attention features from the fundus feature map of each stage, assigning higher weights to important channel features and lower weights to irrelevant channel features, in order to focus on channel features important for fundus lesion detection, thereby obtaining the target fundus channel attention features corresponding to the fundus feature map of that stage. The target fundus channel attention features are features of channel dimensions important for fundus lesion detection, such as vascular morphology and vascular color changes.

[0096] In step S420 of some embodiments, a spatial attention module is constructed. This module extracts spatial attention features from the target fundus channel attention features, assigning higher weights to important spatial locations and lower weights to irrelevant spatial locations. This focuses on the lesion region features of the fundus image while ignoring irrelevant features of the background region, resulting in the target fundus spatial attention features. These target fundus spatial attention features are used to characterize the features of fundus lesion regions, such as the location of hemorrhages and microaneurysms.

[0097] In step S430 of some embodiments, the channel attention module and the spatial attention module are connected in series as follows: Figure 5 As shown, the target fundus channel attention features and the target fundus spatial attention features are multiplied in alignment to obtain the output of the convolutional block attention module for the fundus feature map. This output is used as the fundus attention feature. The fundus feature map and the fundus attention feature have the same feature dimensions. The fundus attention feature consists of salient feature information related to fundus lesion detection, such as the morphological and positional features of the optic cup, optic disc, and retinal vessels.

[0098] Steps S410 to S430 above combine channel attention with spatial attention, which can simultaneously focus on important features in the channel dimension and important spatial regions in the spatial dimension, making the feature information richer and focusing on feature information that is important for the detection of multiple lesions, thereby improving the accuracy and efficiency of multiple lesion classification.

[0099] Please see Figure 6 In some embodiments, step S410 may include, but is not limited to, steps S610 to S620:

[0100] Step S610: For each scale of fundus feature map, perform channel attention feature extraction on the fundus feature map to obtain the initial fundus channel attention features;

[0101] Step S620: Combine the initial fundus channel attention features and fundus feature map to obtain the target fundus channel attention features.

[0102] In step S610 of some embodiments, such as Figure 5 As shown, channel attention features are extracted from the fundus feature map at each scale using the channel attention module, resulting in the initial fundus channel attention features corresponding to that scale. These initial fundus channel attention features are the output features of the channel attention module and are used to characterize the fundus features of important channels in the fundus feature map.

[0103] In step S620 of some embodiments, in order to avoid the loss of some feature information during channel attention feature extraction, the initial fundus channel attention features and the fundus feature map are multiplied in place to capture the complex relationship between the initial fundus channel attention features and the fundus feature map, and to increase the diversity of features to obtain the target fundus channel attention features, so as to ensure the sufficiency and comprehensiveness of feature extraction, improve the expressive power of features, and thus improve the performance of fundus multilesion detection.

[0104] Steps S610 to S620 above utilize channel attention to focus on the features of important channels and perform feature cross-referencing with fundus feature maps to ensure the comprehensiveness and sufficiency of feature expression, thereby improving the ability to distinguish and identify different fundus lesions.

[0105] Please see Figure 7 In some embodiments, step S610 may include, but is not limited to, steps S710 to S750:

[0106] Step S710: For each scale of fundus feature map, average pooling is performed on the fundus feature map to obtain the first fundus pooling feature.

[0107] Step S720: Max pooling is performed on the fundus feature map to obtain the second fundus pooling feature;

[0108] Step S730: Perform convolution processing on the pooling features of the first fundus to obtain the convolutional features of the first fundus;

[0109] Step S740: Perform convolution processing on the pooling features of the second fundus to obtain the convolutional features of the second fundus;

[0110] Step S750: Determine the initial fundus channel attention features based on the first fundus convolution features and the second fundus convolution features.

[0111] In step S710 of some embodiments, Figure 8The structure of the channel attention module is illustrated. This module comprises an adaptive 2D average pooling layer and an adaptive 2D max pooling layer in parallel, two convolutional layers, and two activation layers. The two activation layers employ the ReLU activation function and the sigmoid activation function, respectively. It's understood that the adaptive pooling layer does not use a fixed-size pooling window; instead, it adaptively adjusts the size of the pooling window based on the size of the input feature map. This better handles input feature maps of different sizes while maintaining the spatial resolution of the output feature map, preserving its spatial relationships and detailed information. A fundus feature map with dimensions B*C*W*H, after passing through the above pooling layers, will yield a feature map with dimensions B*C*1*1. Here, B is the batch size of the feature map, C is the number of channels in the feature map, W is the width of the feature map, and H is the height of the feature map.

[0112] To extract channel attention features, for each scale of the fundus feature map, an adaptive two-dimensional average pooling layer is used to perform two-dimensional average pooling in the width and height directions to downsample the fundus feature map, compressing the space to 1*1 while preserving channel information, thus obtaining the first fundus pooling feature. Specifically, the size of the pooling window is calculated based on the size of the fundus feature map and a preset output size, which is equal to the ratio of the fundus feature map size to the preset output size. The preset output size is the size of the output feature map. The fundus feature map is divided into multiple regions in the spatial dimension using the pooling window. The pixel mean of each region is calculated to obtain the first fundus pooling feature. The first fundus pooling feature is the average representation of the overall features of the fundus feature map, and its size is equal to the preset output size.

[0113] In step S720 of some embodiments, average pooling focuses more on global information, while max pooling focuses more on local information. This combination of average and max pooling strategies allows for simultaneous attention to both global and local information in the fundus image, improving the ability to identify lesions in the fundus image. For each scale of fundus feature map, an adaptive two-dimensional max pooling layer is used to perform two-dimensional max pooling in the width and height directions to downsample the fundus feature map, obtaining a second fundus pooling feature. The second fundus pooling feature has the same feature size as the first fundus pooling feature. Specifically, the size of the pooling window is calculated based on the size of the fundus feature map and a preset output size, where the pooling window size is equal to the ratio of the fundus feature map size to the preset output size, and the preset output size is the size of the output feature map. The fundus feature map is divided into multiple regions in the spatial dimension using the pooling window; the maximum pixel value of each region is calculated to obtain the second fundus pooling feature. The second fundus pooling feature is a salient feature representation of the fundus feature map. The size of the second fundus pooling feature is equal to the preset output size. Both the first and second fundus pooling features contain channel information from the fundus feature map.

[0114] In step S730 of some embodiments, the first fundus pooling feature is dimensionality reduced by the first convolutional layer, reducing the number of feature channels of the first fundus pooling feature from C to C / 16, and the dimensionality-reduced first fundus convolutional feature is activated by the ReLU activation function to obtain the first fundus convolutional feature. The kernel size of the first convolutional layer is 1.

[0115] In step S740 of some embodiments, the second fundus pooling feature is upscaled by a second convolutional layer, increasing the number of feature channels of the second fundus pooling feature from C / 16 to C, to obtain the second fundus convolutional feature.

[0116] In step S750 of some embodiments, the output feature maps of the two parallel pooling branches are passed through two convolutional layers and the ReLU activation function, and then the two output feature maps are superimposed. That is, the first fundus convolutional feature and the second fundus convolutional feature are added in the same position, and the superimposed feature map is activated by the sigmoid activation function to obtain the initial fundus channel attention feature output by the channel attention module.

[0117] Steps S710 to S750 above utilize average pooling and max pooling to extract global and local feature information from the fundus image, respectively. This combined global and local feature information improves the comprehensiveness of channel attention feature extraction. A first convolution process is performed on the global and local feature information to remove redundant or irrelevant information. A second convolution process is then performed on the global and local feature information to increase the dimensionality of the feature data and ensure feature richness.

[0118] Please see Figure 9 In some embodiments, step S420 may include, but is not limited to, steps S910 to S940:

[0119] Step S910: Max pooling is performed on the attention features of the target fundus channel to obtain the pooling features of the first channel;

[0120] Step S920: Average pooling is performed on the attention features of the target fundus channel to obtain the pooling features of the second channel;

[0121] Step S930: From the channel dimension, the first channel pooling feature and the second channel pooling feature are concatenated to obtain the channel concatenated feature;

[0122] Step S940: Perform convolution processing on the channel splicing features to obtain the target fundus spatial attention features.

[0123] In step S910 of some embodiments, the convolutional block attention module includes a spatial attention module in addition to the channel attention module described above. The spatial attention module is structured as follows: Figure 10 As shown, the spatial attention module includes an average pooling layer, a max pooling layer, a convolutional layer, and an activation layer, with the activation layer using the sigmoid activation function. The average and max pooling layers can be adaptive or fixed-size pooling layers. A feature map with dimensions B*C*W*H will be transformed into a feature map with dimensions B*1*W*H after passing through the above pooling layers. It can be understood that spatial dimension pooling focuses on the spatial regions of the fundus image, while channel dimension pooling focuses on the channel information of the feature map. To extract spatial attention features, the target fundus channel attention features are max-pooled in the channel dimension using a max pooling layer, compressing the channel dimension to 1 while preserving spatial information, resulting in the first channel pooling feature. The first channel pooling feature is a salient feature representation of the fundus image. Specifically, the target fundus channel attention features are divided into multiple regions using a pooling window in the channel dimension, and the maximum pixel value of each region is calculated to obtain the first channel pooling feature.

[0124] In step S920 of some embodiments, the target fundus channel attention features are average-pooled using an average pooling layer. The target fundus channel attention features are divided into multiple regions along the channel dimension using a pooling window. The average pixel value of each region is calculated to obtain the second channel pooling feature. The second channel pooling feature is an average feature representation of the fundus image. Both the first and second channel pooling features contain spatial region information of the fundus image, such as the arrangement of various fundus structures and the location of a specific fundus structure.

[0125] In step S930 of some embodiments, in order to obtain a comprehensive feature representation and improve the ability to identify lesions of different sizes, the first channel pooling feature and the second channel pooling feature are concatenated along the channel dimension to obtain the channel concatenated feature. Both the first and second channel pooling features are feature maps of dimension B*1*W*H, and the channel concatenated feature is a feature map of dimension B*2*W*H.

[0126] In step S940 of some embodiments, in order to extract higher-level semantic information and avoid interference from irrelevant information in lesion classification, after performing a convolution operation on the channel splicing features through a convolutional layer, the convolutional features are activated using a sigmoid activation function to obtain the output feature map of the spatial attention module. This output feature map is used as the target fundus spatial attention feature. The target fundus spatial attention feature has the same size as the target fundus channel attention feature. It should be noted that the number of output channels of the convolutional layer is 1, and the kernel size is 7.

[0127] Steps S910 to S940 above utilize spatial attention mechanisms to enable the model to focus on key areas of various retinal structures such as blood vessels, optic disc, and macula, thereby using the feature information of these key areas to detect retinal lesions and improve the accuracy of retinal lesion classification.

[0128] In step S140 of some embodiments, an adaptive two-dimensional average pooling layer is used to perform average pooling processing on the fundus attention features output by the convolutional block attention module in both the width and height directions. Multiple fundus attention features after average pooling are then concatenated along the channel dimension to obtain multi-scale fundus fusion features. These multi-scale fundus fusion features are the output of the multi-scale feature fusion module and are obtained by concatenating multiple fundus attention features. Figure 1 One-to-one correspondence.

[0129] like Figure 11 As shown, a multi-scale feature fusion module is constructed by using the output feature maps of a convolutional network-based image classification model at multiple stages and overlaying them with convolutional block attention modules. The multi-scale feature fusion module first processes the output feature maps from the last three stages using the convolutional block attention module. Then, it performs adaptive two-dimensional average pooling operations on the width and height dimensions of each of the three processed feature maps. Finally, it concatenates the three pooled feature maps along the channel dimension to obtain the output feature map of the multi-scale feature fusion module.

[0130] In step S150 of some embodiments, such as Figure 11As shown, adding a multi-scale feature fusion module to a convolutional network-based image classification model yields a multi-scale feature fusion-based image classification model. This model is a multi-label classification model with high universality and can be used as a classification model for multiple fundus lesions in fundus images for detection. Specifically, the multi-scale fundus fusion features are input into a fully connected layer for classifying multiple fundus lesions, thus obtaining the categories of fundus lesions. The number of input channels of the fully connected layer is the number of feature channels of the multi-scale fundus fusion features, and the number of output channels of the fully connected layer is the number of fundus lesion categories. The categories of fundus lesions can be one or more of the following fundus diseases: diabetic retinopathy, age-related macular degeneration, glaucoma, cataracts, etc.

[0131] For different convolutional network-based image classification networks, such as ResNet50, the number of channels in the output feature maps of stages 2, 3, and 4 are 512, 1024, and 2048, respectively. When using the ODIR5K dataset described above with 5 multi-labels, the number of input channels in the fully connected layer is 512 + 1024 + 2048, which is 3584, and the number of output channels is 5.

[0132] Please see Figure 12 In some embodiments, the training process of the fundus image multilesion classification model may include, but is not limited to, steps S1210 to S1240:

[0133] Step S1210: Obtain the sample fundus image and at least one sample lesion label from the sample fundus image;

[0134] Step S1220: Input the sample fundus image into the preset model to classify fundus lesions and obtain at least one predicted lesion label;

[0135] Step S1230: Calculate the cross-loss based on the lesion label of each sample and the predicted lesion label to obtain the target loss data;

[0136] Step S1240: Adjust the model parameters of the preset model according to the target loss data to obtain a multilesion classification model for fundus images.

[0137] In step S1210 of some embodiments, a multi-label classification dataset of fundus images is obtained. The dataset is then denoised and filtered according to specific rules, and training and testing data are separated to train a multi-label classification model of fundus images based on multi-scale feature fusion. The training data is used for model training, and the testing data is used for model performance testing. Fundus images from the training data are used as sample fundus images. Each sample fundus image has at least one sample lesion label, which characterizes the category of fundus lesions in the sample fundus image.

[0138] The ODIR5K dataset, containing 10,000 fundus images, is a suitable multi-label dataset. All data has been anonymized, with each image labeled with a specific disease, including diabetic retinopathy, age-related macular degeneration, glaucoma, cataracts, myopia, hypertensive retinopathy, and other diseases. The ODIR5K dataset can be denoised and filtered according to the disease classification task. If the classification task only requires detecting diabetic retinopathy, age-related macular degeneration, glaucoma, cataracts, and myopia, then any fundus image showing only the label of hypertensive retinopathy or another disease will be removed. That is, if a fundus image is only labeled "hypertensive retinopathy" or only "other diseases," it will be filtered out. If a fundus image is labeled with both "cataracts" and "hypertensive retinopathy," it will not be filtered out, but only the "cataract" label will be retained. After data filtering, the dataset retained five major category labels: diabetic retinopathy, age-related macular degeneration, glaucoma, cataracts, and myopia, ultimately yielding 7989 fundus images. 7167 fundus images were used as training data, and 822 fundus images were used as test data. It is noteworthy that the patients corresponding to the fundus images in the training data and the patients corresponding to the fundus images in the test data do not overlap to avoid data leakage.

[0139] After dividing the training and testing data, the label information in the training and testing data is transformed. This application performs label transformation on fundus images in the ODIR5K dataset. If the image contains labels such as diabetic retinopathy, age-related macular degeneration, glaucoma, cataract, and myopia, the corresponding label for the fundus image is "1,1,1,1,1"; if the fundus image does not contain labels for the above five eye diseases, the corresponding label is "0,0,0,0,0"; if the image only contains the myopia label, the corresponding label is "0,0,0,0,1", and so on. Through this method, text labels can be converted to numeric labels, facilitating model training and testing.

[0140] This invention trains a multi-lesion classification model for fundus images on a workbench equipped with an NVIDIA-RTX3090 GPU. The programming language is Python, and the PyTorch deep learning framework is used. During training, the batch size is set to 64, the optimizer is AdamW, and the training consists of 50 epochs with an initial learning rate of 10. -4 The learning rate is adjusted using cosine annealing, and the termination learning rate is 10. -7 .

[0141] Furthermore, to improve the training effect and testing efficiency of the model, fundus image data augmentation is performed during the training phase, and fundus image scaling and cropping are performed during the testing phase. In this invention, after reading the fundus image during the model training phase, the short side of the fundus image is scaled to 256 while maintaining the aspect ratio. Next, the scaled fundus image is randomly cropped to create image blocks with a height and width of 224x224. Finally, the image blocks are randomly horizontally flipped, normalized, and converted into tensors. For example, tensor conversion can be implemented using the transforms module in PyTorch.

[0142] After reading the fundus image during the testing phase, the short side of the fundus image is scaled to 256 while maintaining the aspect ratio. Next, the scaled fundus image is center-cropped to create a 224x224 image block. Finally, the image block is normalized and converted into a tensor.

[0143] In step S1220 of some embodiments, the preset model is a model constructed from a multi-scale feature fusion module, an image classification model based on a convolutional network, and a fully connected layer. Sample fundus images or tensors converted from sample fundus images are input into the preset model for fundus lesion classification to obtain at least one predicted lesion label.

[0144] In step S1230 of some embodiments, a loss function for multi-label classification of fundus images is constructed, using the binary cross-entropy loss function (BCE). Based on the binary cross-entropy loss function, the cross-loss is calculated between the sample lesion label and each predicted lesion label. The binary cross-entropy loss function is as follows:

[0145]

[0146] Where N is the number of tags in a multi-label system, i represents the i-th tag, and y i p(y) represents the true information of the i-th label, i.e., the lesion label of the sample. i ) indicates that the output belongs to y i The predicted probability of a label is the predicted lesion label.

[0147] Assuming each batch during the training phase contains M fundus images, the target loss data for each batch is represented as follows:

[0148]

[0149] Among them, L bce_iThis represents the BCE loss value of the i-th fundus image in each batch during the training phase.

[0150] In step S1240 of some embodiments, the model parameters of the preset model are adjusted according to the target loss data, and the model with the smallest target loss data is used as the final multi-label classification model to obtain the fundus image multi-lesion classification model. After training the fundus image multi-lesion classification model using training data, the model is tested using test data, and the model with the smallest target loss data on the test set is used as the final fundus image multi-lesion classification model.

[0151] This application uses mean average precision (mAP), a commonly used evaluation metric in image multi-label classification, to evaluate the performance of our method on the ODIR5K dataset compared to other methods. Specific performance data is shown in Table 1.

[0152]

[0153] Table 1

[0154] As can be seen from Table 1, the performance indicators of the basic models were all improved after adding the multi-scale feature fusion module. After adding the multi-scale feature fusion module to the six models, the average gain of the mAP index was 1.31%.

[0155] Through the above steps S1210 to S1240, the training of the fundus image multilesion classification model can be completed, so as to use the fundus image multilesion classification model to detect fundus multilesions, thereby improving the accuracy of fundus multilesion detection.

[0156] Please see Figure 13 This application also provides a fundus image classification device that can implement the above-described fundus image classification method. The fundus image classification device includes:

[0157] The acquisition module 1310 is used to acquire fundus images;

[0158] The multi-scale feature extraction module 1320 is used to extract multi-scale features from fundus images to obtain fundus feature maps at multiple different scales.

[0159] The attention feature extraction module 1330 is used to extract attention features from the fundus feature map for each scale to obtain fundus attention features;

[0160] The feature stitching module 1340 is used to stitch together multiple fundus attention features to obtain multi-scale fundus fusion features.

[0161] The fundus image classification module 1350 is used to classify multiple lesions in fundus images based on multi-scale fundus fusion features.

[0162] The specific implementation of this fundus image classification device is basically the same as the specific implementation of the fundus image classification method described above, and will not be repeated here.

[0163] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described fundus image classification method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0164] Please see Figure 14 , Figure 14 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0165] The processor 1410 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0166] The memory 1420 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1420 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1420 and called and executed by the processor 1410 using the fundus image classification method of the embodiments of this application.

[0167] The input / output interface 1430 is used to implement information input and output;

[0168] The communication interface 1440 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0169] Bus 1450 transmits information between various components of the device (e.g., processor 1410, memory 1420, input / output interface 1430, and communication interface 1440);

[0170] The processor 1410, memory 1420, input / output interface 1430 and communication interface 1440 are connected to each other within the device via bus 1450.

[0171] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described fundus image classification method.

[0172] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0173] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0174] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0175] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0176] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0177] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0178] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0179] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. The coupling or direct coupling or communication connection between the shown or discussed units may be through some interfaces, or indirect coupling or communication connection between the apparatus or units, and may be electrical, mechanical, or other forms.

[0180] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0181] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0182] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0183] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A fundus image classification method, characterized in that, The method includes: Acquire fundus images; Multi-scale feature extraction is performed on the fundus image to obtain multiple fundus feature maps at different scales. The multiple fundus feature maps at different scales include the output feature maps of the last three stages of ResNet50, namely stage 2, stage 3 and stage 4. The number of channels of the output feature maps of stage 2, stage 3 and stage 4 are 512, 1024 and 2048, respectively. For each scale of the fundus feature map, attention features are extracted from the fundus feature map using a convolutional block attention module to obtain fundus attention features. The convolutional block attention module is composed of a channel attention module and a spatial attention module connected in series. The channel attention module and the spatial attention module focus on different dimensions. The channel attention module is used to extract channel attention features, which are used to characterize the channel dimension features for fundus lesion detection. The spatial attention module is used to extract spatial attention features, which are used to characterize the spatial dimension features for fundus lesion detection. Each fundus attention feature is subjected to two-dimensional average pooling in the width and height dimensions by a two-dimensional average pooling layer. The multiple fundus attention features after average pooling are then spliced ​​together in the channel dimension to obtain multi-scale fundus fusion features. The fundus images are classified into multiple lesions based on the multi-scale fundus fusion features. For each scale of the fundus feature map, attention features are extracted from the fundus feature map using a convolutional block attention module to obtain fundus attention features, including: For each scale of the fundus feature map, the channel attention module extracts channel attention features from the fundus feature map to obtain target fundus channel attention features; the spatial attention module extracts spatial attention features from the target fundus channel attention features to obtain target fundus spatial attention features; and the target fundus channel attention features and the target fundus spatial attention features are combined to obtain the fundus attention features. For each scale of the fundus feature map, the channel attention module extracts channel attention features from the fundus feature map to obtain target fundus channel attention features, including: For each scale of the fundus feature map, the channel attention module is used to extract channel attention features from the fundus feature map to obtain initial fundus channel attention features; the initial fundus channel attention features and the fundus feature map are combined to obtain the target fundus channel attention features; The step of performing multi-scale feature extraction on the fundus image to obtain multiple fundus feature maps at different scales includes: The fundus image is subjected to shallow feature extraction to obtain shallow image features; the shallow image features are subjected to multi-scale bottleneck feature extraction to obtain fundus feature maps of multiple different scales; Prior to acquiring the fundus image, the fundus image classification method further includes: Obtain a sample fundus image and at least one sample lesion label from the sample fundus image; input the sample fundus image into a preset model for fundus lesion classification to obtain at least one predicted lesion label; perform cross-loss calculation based on each sample lesion label and each predicted lesion label to obtain target loss data; adjust the model parameters of the preset model based on the target loss data to obtain a fundus image multi-lesion classification model.

2. The fundus image classification method according to claim 1, characterized in that, For each scale of the fundus feature map, channel attention features are extracted from the fundus feature map to obtain initial fundus channel attention features, including: For each scale of the fundus feature map, the fundus feature map is averaged and pooled to obtain the first fundus pooling feature; Max pooling is performed on the fundus feature map to obtain the second fundus pooling feature; The first fundus pooling feature is convolved to obtain the first fundus convolution feature; The second fundus pooling feature is convolved to obtain the second fundus convolution feature; The initial fundus channel attention features are determined based on the first fundus convolution features and the second fundus convolution features.

3. The fundus image classification method according to claim 1, characterized in that, The step of extracting spatial attention features from the target fundus channel attention features using the spatial attention module to obtain target fundus spatial attention features includes: Max pooling is performed on the attention features of the target fundus channel to obtain the first channel pooling features; The target fundus channel attention features are averaged and pooled to obtain the second channel pooling features; From the channel dimension, the first channel pooling feature and the second channel pooling feature are concatenated to obtain the channel concatenated feature; The channel splicing features are convolutionally processed to obtain the target fundus spatial attention features.

4. A fundus image classification device, characterized in that, The device includes: The acquisition module is used to acquire fundus images; The multi-scale feature extraction module is used to extract multi-scale features from the fundus image to obtain multiple fundus feature maps at different scales. The multiple fundus feature maps at different scales include the output feature maps of the last three stages of ResNet50, namely stage 2, stage 3 and stage 4. The number of channels of the output feature maps of stage 2, stage 3 and stage 4 are 512, 1024 and 2048, respectively. An attention feature extraction module is used to extract attention features from the fundus feature map at each scale using a convolutional block attention module to obtain fundus attention features. The convolutional block attention module is composed of a channel attention module and a spatial attention module connected in series. The channel attention module and the spatial attention module focus on different dimensions. The channel attention module is used to extract channel attention features, which are used to characterize the channel dimension features for fundus lesion detection. The spatial attention module is used to extract spatial attention features, which are used to characterize the spatial dimension features for fundus lesion detection. The feature stitching module is used to perform two-dimensional average pooling on multiple fundus attention features in the width and height dimensions through a two-dimensional average pooling layer, and to stitch the multiple fundus attention features after average pooling in the channel dimension to obtain multi-scale fundus fusion features. The fundus image classification module is used to classify multiple lesions in the fundus image based on the multi-scale fundus fusion features. For each scale of the fundus feature map, attention features are extracted from the fundus feature map using a convolutional block attention module to obtain fundus attention features, including: For each scale of the fundus feature map, the channel attention module extracts channel attention features from the fundus feature map to obtain target fundus channel attention features; the spatial attention module extracts spatial attention features from the target fundus channel attention features to obtain target fundus spatial attention features; and the target fundus channel attention features and the target fundus spatial attention features are combined to obtain the fundus attention features. For each scale of the fundus feature map, the channel attention module extracts channel attention features from the fundus feature map to obtain target fundus channel attention features, including: For each scale of the fundus feature map, the channel attention module is used to extract channel attention features from the fundus feature map to obtain initial fundus channel attention features; the initial fundus channel attention features and the fundus feature map are combined to obtain the target fundus channel attention features; The step of performing multi-scale feature extraction on the fundus image to obtain multiple fundus feature maps at different scales includes: The fundus image is subjected to shallow feature extraction to obtain shallow image features; the shallow image features are subjected to multi-scale bottleneck feature extraction to obtain fundus feature maps of multiple different scales; Before acquiring the fundus image, the fundus image classification device is further configured to: Obtain a sample fundus image and at least one sample lesion label from the sample fundus image; input the sample fundus image into a preset model for fundus lesion classification to obtain at least one predicted lesion label; perform cross-loss calculation based on each sample lesion label and each predicted lesion label to obtain target loss data; adjust the model parameters of the preset model based on the target loss data to obtain a fundus image multi-lesion classification model.

5. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the fundus image classification method according to any one of claims 1 to 3.

6. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the fundus image classification method according to any one of claims 1 to 3.

Images