Lightweight classification model and classification method for multiple types of fundus lesions

Abstract

The present invention discloses a lightweight classification model and classification method for multiple types of fundus lesions. The classification model includes three parts: a backbone network L-Resnet, a feature fusion module FFM, and classification output. The backbone network consists of a CBR (Conv+BN+Relu) module and four convolutions at different stages, which are responsible for reducing model parameters, increasing network depth, and strengthening the extraction of deep semantic information from OCT images; the feature fusion module FFM consists of an upsampling module and a fast convolutional spatial pyramid module (CSPPF), which is responsible for extracting shallow network detail information and deep semantic information; the classification output consists of average pooling, concatenation, and two fully connected layers, which are responsible for outputting the classification results of fundus lesions. The present invention can significantly reduce the number of model parameters and can achieve high-precision classification of multiple fundus lesions.

Description

Technical Field

[0001] The present invention belongs to the technical field of image classification, and relates to a lightweight classification model and method, equipment and storage medium for multiple types of fundus lesions. Background Art

[0002] Conducting fundus disease examinations can effectively prevent and treat fundus lesions, provide treatment plans as early as possible, and avoid the progression of eye diseases and subsequent loss of vision and labor, which is of great significance to social development. Currently, the diagnosis and treatment of eye diseases are mostly based on fundus images, and Optical Coherence Tomography (OCT), as a non-contact, non-invasive ophthalmic imaging diagnostic technology, is a commonly used imaging acquisition method for fundus diseases. Ophthalmology is a discipline that focuses on imaging diagnosis, which is consistent with intelligent algorithms that require large amounts of data for training. The use of intelligent algorithms provides a new solution for screening fundus lesions.

[0003] With the rapid development of deep learning in the field of imaging, the use of deep learning methods to process fundus OCT images has also become a hot research direction in medical imaging in recent years. The attention mechanism can effectively enhance the extraction of existing features. Fang et al. used the lesion-aware convolutional network (LACNN) to generate a soft attention map in OCT images and merged the attention map into the classification network to improve classification accuracy. For details, see "FANG L, WANG C, LI S, et al. Attention tolesion: Lesion-aware convolutional neural network for retinal optical coherence tomography image classification [J]. IEEE transactions on medical imaging, 2019, 38(8): 1959-1970". Liu et al. proposed a one-stage attention-based retinal OCT image classification and segmentation method, which achieved simultaneous classification and segmentation. For details, see "LIU X, BAI Y, CAO J, et al. Joint disease classification and lesion segmentation via one-stage attention-based convolutional neural network in OCT images[J]. Biomedical Signal Processing and Control, 2022, 71: 103087". Mishra et al. proposed a deep network based on perturbation composite attention to extract context-aware diagnostic features to improve classification accuracy. For details, see "MISHRA SS, MANDAL B, PUHAN NB. Perturbed composite attention model for macular optical coherencetomography image classification[J]. IEEE Transactions on Artificial Intelligence, 2021, 3(4): 625-635".

[0004] Multi-scale convolutional structures can extract features of different sizes to enhance the learning ability of the network. Das et al. used a deep multi-scale convolutional neural network to encode multi-scale lesion features, combined them for classification, and used a joint multi-loss optimization strategy to enhance the network learning ability. For details, see "DAS V, DANDAPAT S, BORA P K. Automated classification of retinal OCT images using a deep multi-scale fusion CNN[J]. IEEE Sensors Journal, 2021, 21(20): 23256-23265". Anju et al. used multi-scale convolutional layers to generate local structures of various sizes and proposed a multi-scale convolutional network. For details, see "THOMAS A, HARIKRISHNAN PM, KRISHAN AK, et al. A novel multiscale convolutional neural network based age-related macular degeneration detection using OCT images[J]. Biomedical Signal Processing and Control, 2021, 67: 102538". Paima et al. proposed a multi-scale neural network that enhanced classification performance by capturing inter-scale variations and using a feature fusion strategy of fast convolution blocks. For details, see "SOTOUDEH-PAIMAS, JODEIRI A, HAJIZADEH F, et al. Multi-scale convolutional neural network for automated AMD classification using retinal OCT images[J]. Computers in biology and medicine, 2022, 144: 105368".

[0005] Using transfer learning in different network combinations can also improve classification accuracy. Mesut et al. used nine transfer learning models to classify each lesion type and used the slime mold algorithm (SMA) to select the best network for single-target classification from the models for output. M,ERGEN B, V.Use of dominant activations obtained by processing OCT images with the CNNs and slime moldmethod in retinal disease detection[J].Biocybernetics and BiomedicalEngineering, 2022,42(2):646-666".

[0006] The inventors have found that the existing deep learning-based OCT image classification effect is not good, mainly due to the following deficiencies: (1) The addition of the attention mechanism strengthens the extraction of existing features and enriches the local lesion feature information, but it is difficult to extract lesions with small target features; (2) Different forms of multi-scale networks cannot solve the problem of edge texture information being lost during network transmission; (3) The combination of existing networks will lead to an excessive number of algorithm model parameters and a slowdown in network operation. How to enhance the ability to extract small target features unique to different lesion characteristics, while strengthening feature extraction, reducing the true loss of lesion characteristics, and reducing the number of model parameters is a problem that needs to be solved at present. Summary of the Invention

[0007] In order to solve the above problems, the present invention provides a lightweight classification model for multiple types of fundus lesions. While reducing the number of model parameters, it can increase the network's classification accuracy of fundus OCT images, and can solve the shortcomings of the existing technology such as slow running speed and low classification accuracy.

[0008] The second object of the present invention is to provide a lightweight method for classifying multiple types of fundus lesions.

[0009] A third object of the present invention is to provide an electronic device.

[0010] A fourth object of the present invention is to provide a computer storage medium.

[0011] The technical solution adopted by the present invention is a lightweight classification model for multiple types of fundus lesions, including a backbone network L-Resnet, a feature fusion module FFM and classification output.

[0012] The backbone network consists of a CBR (Conv+BN+Relu) module and four convolution stages, which is responsible for reducing model parameters, increasing network depth, and enhancing the extraction of deep semantic information from OCT images.

[0013] The feature fusion module (FFM), consisting of an upsample module and a fast convolutional spatial pyramid module (CSPPF), is responsible for extracting shallow network detail information and deep semantic information. Upsampling is used to enhance the fusion of shallow and deep network information and strengthen the classification of lesion OCT images.

[0014] The classification output is composed of average pooling, concatenation, and two fully connected layers, and is responsible for outputting the classification results of fundus lesions.

[0015] Furthermore, the CBR module of the backbone network L-Resnet is responsible for channel expansion of the original image and enriching the image feature expression;

[0016] The channel expansion uses 64 convolution kernels of size 3×3 to extract features from the input tensor and generate a new tensor at the same time. After the convolution operation, the size of the tensor is halved;

[0017] The convolution of different stages of the backbone network L-Resnet is divided by the size of the output feature tensor. The input OCT image size is 224×224, and the output size after processing by the CBR module is 112×112.

[0018] The output tensor sizes of the convolutions in the four different stages are 56×56, 28×28, 14×14, and 7×7, respectively. The number of multi-scale convolution blocks MultiBlock used in each stage is 2, 2, 6, and 2, respectively. The number of convolution kernels used in each convolution in each stage is 64, 128, 192, and 256, respectively.

[0019] Furthermore, the multi-scale convolution block MultiBlock is composed of depthwise convolutions with convolution kernels of different sizes. Four convolution kernels of different sizes are used to extract lesion features of different sizes in the image. The four different sub-tensors obtained are concatenated to form a complete tensor. Skip connections are used to link the input and output of the same convolution operation, enabling the convolution operation to learn useful features and eliminate network degradation.

[0020] Each convolution kernel of the depthwise convolution is responsible for one channel, and one channel is convolved by only one convolution kernel. The number of feature channels generated in this process is the same as the number of input channels.

[0021] The different sub-tensors are concatenated into a tensor according to the channel dimension through the Concat operation;

[0022] The jump connection passes the input tensor directly to the convolution output, so that the tensor after the input tensor kernel convolution is combined;

[0023] The multi-scale convolution block MultiBlock uses four convolution kernels of different sizes in the same convolution operation to complete the convolution operation. During the convolution operation, the input tensor is divided into four equal parts, and convolution kernels of sizes 1, 3, 5, and 7 are used to perform convolution operations on the four input sub-tensors respectively. Convolution kernels of different sizes are used to extract lesion texture information and lesion edge information of different sizes. The new feature tensors obtained by different convolution kernels are shuffled through channels to enhance information flow and feature expression capabilities. Finally, they are activated using an activation function and passed to the next layer.

[0024] The network degradation is the phenomenon that the network training accuracy decreases during the network deepening process;

[0025] The channel shuffling disrupts a set of feature tensors and reorders them to form a new feature tensor.

[0026] Furthermore, the FFM module upsamples deep features through the Upsample module and extracts features of different sizes again through the CSPPF module, enriching the expression of lesion geometric information and lesion spatial information in the OCT image in the network, and transferring features of different specificities in the OCT image extracted from the shallow layer to the deep layer, realizing the fusion of shallow texture information and deep semantic information;

[0027] The upsampling of the feature fusion module (FFM) first expands the channel dimension of the input tensor to 256 through a 1×1 convolution, and then uses bilinear interpolation to expand the feature size to keep it consistent with the feature size of the previous layer.

[0028] The fast convolutional spatial pyramid CSPPF of the feature fusion module (FFM) first expands the channel through a 1×1 convolution, and then uses three cascaded 3×3 convolution kernels to achieve the receptive field that can be achieved with convolution kernel sizes of 3, 5, and 7. The obtained output tensor is spliced according to the channel dimension, and after the channels are shuffled, the 1×1 convolution kernel is used again to adjust the channels, limiting the output tensor channels to 256.

[0029] A lightweight classification method for multiple types of fundus lesions is carried out according to the following steps:

[0030] S1. Preprocess the fundus lesion image to convert the image size to 224×224. Then, pass the processed fundus lesion image through the CBR module and sequentially feed it into four different stages of convolution modules composed of multi-scale convolution to enhance the network's ability to extract information and strengthen the extraction of semantic information, and then output it.

[0031] S2, the feature fusion module uses the upsampling module and the fast convolutional pyramid module to extract lesion features again, fuses the deep and shallow lesion features, and outputs three feature tensors of different sizes;

[0032] S3. Fusion the output of the backbone network and the output of the feature fusion module to obtain the classification results of fundus lesions.

[0033] Furthermore, the image size conversion of S1 and the feature extraction performed by convolution at different stages are performed according to the following formula, and the outputs of different convolutions are passed to the feature fusion module.

[0034] X 224 =R e (Ι i )

[0035]

[0036]

[0037]

[0038]

[0039]

[0040]

[0041]

[0042]

[0043]

[0044]

[0045] Among them Ι i is the input image; R e Set the image size; X 224 After preprocessing, the feature tensor size is 224×224; is a convolution operation with a kernel size of 3×3 and 64 kernels; Β is a batch normalization operation; R is the activation function (ReLU); s is the step size of the convolution operation, is a new tensor of size 112×112 and 64 channels obtained after passing through the CBR convolution module; C m (x) represents the multi-scale convolution layer function; x is the input tensor; C is a tensor partitioning operation that divides the tensor into 4 sub-tensors; n×nis a convolution kernel of size n×n (n=1, 3, 5, 7), which convolves the sub-tensors obtained above respectively; F is the channel dimension splicing function; S is the channel shuffling function; M is the multi-scale convolution block function; loop is the number of convolution operations; add is the tensor addition flag, which directly outputs the tensor when it is 0, and outputs it after adding the input and output tensors when it is 1; The first stage convolution output size is 56×56, the intermediate tensor of 64 channels, F 56 The final output size of the first stage is a 56×56 tensor; The output size of the second stage convolution is 28×28, with 128 channels of intermediate tensor, F 28 The final output size of the second stage is a 28×28 tensor; The output size of the third stage convolution is 14×14, with 192 channels of intermediate tensor, F 14 The final output size of the third stage is a 14×14 tensor; The output size of the fourth stage convolution is 7×7, the intermediate tensor of 256 channels, and F7 is the final output size of the fourth stage is 7×7 tensor;

[0046] Furthermore, the feature fusion module of S2 extracts deep and shallow layer information and fuses the extracted information according to the following formula:

[0047]

[0048]

[0049]

[0050] Among them F i is the output result of the first, second and third stage convolution; F j is the output of the second, third, and fourth stage convolutions; {·} represents the cascade operation; F is the channel concatenation function; S is the channel shuffling; C CBR is the CBR convolution module; C 3×3 is a convolution operation with a convolution kernel size of 3×3; L interpolation It is the upsampling module operation; The tensor obtained by the fast convolution pyramid module; is the tensor after upsampling output; f i The sum of the tensors obtained after upsampling and fast convolution pyramid module; i = {56, 28, 14}; j = {28, 14, 7};

[0051] Furthermore, the feature fusion and result output of S3 are performed according to the following formula:

[0052] X all =F(P avgpool (F7), P avgpool (f 14 ), P avgpool (f 28 ), P avgpool (f 56 ))

[0053] out=C softmax F c (F c (X all ,c=512),c=k n )

[0054] where X all is the total output after the average pooling; F is the feature concatenation function; P avgpool is the average pooling operation; F7 is the feature tensor of the backbone network output size 7; f 14 The feature fusion module outputs a feature tensor of size 14; f 28 The feature fusion module outputs a feature tensor of size 28; f 56 The feature fusion module outputs a feature tensor of size 56; F c is a fully connected convolution operation; k n is the number of categories; C softmax is the softmax classifier; out is the final classification category.

[0055] An electronic device that implements image classification according to the above method;

[0056] A computer storage medium storing at least one program instruction, wherein the at least one program instruction is loaded and executed by a processor to implement the above-mentioned image classification method

[0057] The beneficial effects of the present invention are:

[0058] In response to the problems of unclear lesion features, different lesion sizes, and different lesion locations in fundus lesion OCT images, the present invention proposes a robust quantitative classification model and classification method MB-CNN for multiple types of fundus lesions. It reduces the number of model parameters by reducing the use of convolution kernels in the backbone network, while deepening the network depth and strengthening the extraction of image semantic information by the deep layer of the network. The multi-scale convolution module MultiBlock uses convolution kernels of different sizes to extract features of different sizes, extract OCT fundus lesion features of different sizes, enrich feature expression, and strengthen the extraction of small lesion features. The feature fusion module performs upsampling through the Upsample module, and re-extracts features of different sizes through the CSPPF module through a convolution cascade operation of three convolution kernels with a size of 3×3, fusing the detail information extracted by the shallow layer of the network with the semantic information extracted by the deep layer, thereby strengthening the lesion feature extraction and feature expression capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0060] Figure 1 It is a structural diagram of the classification model of an embodiment of the present invention.

[0061] Figure 2 Schematic diagram of the backbone network of the classification model according to an embodiment of the present invention.

[0062] Figure 3 Schematic diagram of a multi-scale convolution module of a classification model according to an embodiment of the present invention.

[0063] Figure 4 Schematic diagram of the Upsample module of the classification model according to an embodiment of the present invention.

[0064] Figure 5 Schematic diagram of the CSPPF module of the classification model according to an embodiment of the present invention.

[0065] Figure 6 is an OCT image heat map of the classification model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0066] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0067] Example 1,

[0068] A lightweight classification model for multiple types of fundus lesions, the structure of which is as follows Figure 1 As shown, it includes the backbone network L-Resnet, the feature fusion module FFM and the classification output;

[0069] The backbone network first uses a CBR convolution module with a convolution kernel size of 3×3 to expand the feature channel, and then uses four different convolution stages composed of stacked multi-scale convolution modules. The series connection of convolution modules can enhance the network's ability to extract deep semantic information. Figure 2 As shown in the figure, in the four different convolution stages, the tensor sizes are 56×56, 28×28, 14×14, and 7×7, and the number of tensor channels are 64, 128, 192, and 256, respectively.

[0070] like Figure 3 As shown, the multi-scale convolution module consists of a multi-scale convolutional layer, a CBR convolution with a kernel size of 1×1, and skip connections. The CBR module first uses a 1×1 convolution operation to adjust the output channels, enhancing the exchange of feature information between channels within the feature tensor. The multi-scale convolutional layer first uses a tensor split operation to divide the tensor into four sub-tensors. Four depthwise convolutions of sizes 1×1, 3×3, 5×5, and 7×7 are then applied to each of the four sub-tensors. This allows the network to extract features of different sizes, ensuring redundancy while enhancing feature extraction. The four sub-tensors are then fused in the channel dimension using a concat operation. Batch normalization is applied to facilitate the next convolution operation. The corresponding positions of the resulting feature tensor are fixed. To maximize the likelihood that the sub-tensors will be convolved with kernels of different sizes in the next convolution, channels are shuffled before feature activation, enriching the feature representation. Finally, a skip connection is used to add the input and output, allowing the convolution to extract new and useful feature information as much as possible without losing the original features. At the same time, the use of depthwise convolution can significantly reduce the number of parameters.

[0071] Figure 4As shown in the figure, the Upsample module consists of a 1×1 CBR convolution module and upsampling using linear interpolation. The CBR convolution module uniformly expands the channels to 256, while the linear interpolation operation interpolates according to the tensor size of the previous convolution stage, allowing it to fully preserve the extracted feature information.

[0072] Figure 5 As shown in the figure, the CSPPF module uses a convolution cascade mode. First, a CBR module is used to expand the channel to 256, and then three convolutions with a convolution kernel size of 3×3 are cascaded. It can achieve the receptive field that can be achieved by convolution kernel sizes of 3×3, 5×5, and 7×7, and realize feature re-extraction.

[0073] Combine Figure 1 、 Figure 4 、 Figure 5 As shown in the figure, the feature fusion module can re-extract the features of the results output by the two adjacent convolution stages, fuse the texture information extracted from the shallow layer and the semantic information extracted from the deep layer, and then directly transmit them to the last layer, reducing the loss of useful information.

[0074] Figure 1 As shown in the figure, the four tensors obtained by the backbone network and feature fusion module are compressed into a 1×1×256 tensor using the average pooling operation. After being concatenated according to the channel dimension using concat, they are passed through two fully connected layers and classified and output using the softmax classifier.

[0075] The design of the network model based on deep learning is based on the application background. In image classification, the network model pays more attention to the analysis and understanding of the image content. In the model design, it is more inclined to extract the feature information of the target, such as the contour and texture, and gradually convert the extracted feature information into semantic information that can represent the overall information of the target through convolution. Therefore, strengthening the feature extraction ability of the network and reducing the loss of useful information are the keys to accurate classification. In order to fully extract image feature information, minimize the loss of useful information, and reduce the number of network model parameters, the embodiment of the present invention proposes a lightweight classification model and method for multiple types of fundus lesions. The method uses preprocessing to compress the image size to 224×224, and then enters and exits the image into the backbone network. The backbone network first uses a 3×3 CBR convolution operation to expand the tensor channels from 3 to 64, and then gradually converts the image information into semantic information through four stages of convolution operations, thereby preparing for classification. In the four different convolution stages, the tensor sizes are 56×56, 28×28, 14×14, and 7×7, respectively, and the number of tensor channels is 64, 128, 192, and 256, respectively. The multi-scale convolution module consists of a multi-scale convolutional layer, a CBR convolution with a kernel size of 1×1, and skip connections. The CBR module first uses a 1×1 convolution operation to adjust the output channels, enhancing the exchange of feature information between channels within the feature tensor. The multi-scale convolutional layer first uses a tensor split operation to divide the tensor into four sub-tensors. Four depthwise convolutions of sizes 1×1, 3×3, 5×5, and 7×7 are then applied to each of the four sub-tensors. This allows the network to extract features of different sizes, ensuring redundancy while enhancing feature extraction. The four sub-tensors are then fused in the channel dimension using a concat operation. Batch normalization is used to facilitate the next convolution operation. The corresponding positions of the resulting feature tensor are fixed. To maximize the chance that the subtensor can be convolved with a kernel of a different size in the next convolution, the channels are shuffled before feature activation, enriching the feature representation. Finally, a skip connection is used to sum the input and output, enabling the convolution to extract as much new and useful feature information as possible while preserving existing features. The use of depthwise convolution significantly reduces the number of parameters. The Upsample module consists of a 1×1 CBR convolution module and upsampling using linear interpolation. The CBR convolution module uniformly expands the channels to 256, while the linear interpolation operation is performed based on the tensor size of the previous convolution stage. This fully preserves the extracted feature information. The CSPPF module uses a convolution cascade pattern, first using a CBR module to expand the channels to 256, then cascading three convolutions with kernel sizes of 3×3. This achieves the receptive field achieved by kernel sizes of 3×3, 5×5, and 7×7, respectively, enabling feature re-extraction.The four tensors obtained by the backbone network and feature fusion module are compressed into a 1×1×256 tensor using the average pooling operation. They are then concatenated according to the channel dimension using concat and passed through two fully connected layers before being classified and output using the softmax classifier.

[0076] Example 2,

[0077] A lightweight classification method for multiple types of fundus lesions is characterized by following the steps below:

[0078] S1. Preprocess the fundus lesion image to convert the image size to 224×224. Then, pass the processed fundus lesion image through the CBR module and sequentially feed it into four different stages of convolution modules composed of multi-scale convolution to enhance the network's ability to extract information and strengthen the extraction of semantic information, and then output it.

[0079] S2, the feature fusion module uses the upsampling module and the fast convolutional pyramid module to extract lesion features again, fuses the deep and shallow lesion features, and outputs three feature tensors of different sizes;

[0080] S3. Fusion the output of the backbone network and the output of the feature fusion module to obtain the classification results of fundus lesions.

[0081] Furthermore, the image size conversion of S1 and the feature extraction performed by convolution at different stages are performed according to the following formula, and the outputs of different convolutions are passed to the feature fusion module.

[0082] X 224 =R e (Ι i )

[0083]

[0084]

[0085]

[0086]

[0087]

[0088]

[0089]

[0090]

[0091]

[0092]

[0093] Among them Ι i is the input image; R e Set the image size; X 224 After preprocessing, the feature tensor size is 224×224; is a convolution operation with a kernel size of 3×3 and 64 kernels; Β is a batch normalization operation; R is the activation function (ReLU); s is the step size of the convolution operation, is a new tensor of size 112×112 and 64 channels obtained after passing through the CBR convolution module; C m (x) represents the multi-scale convolution layer function; x is the input tensor; C is a tensor partitioning operation that divides the tensor into 4 sub-tensors; n×n is a convolution kernel of size n×n (n=1, 3, 5, 7), which convolves the sub-tensors obtained above respectively; F is the channel dimension splicing function; S is the channel shuffling function; M is the multi-scale convolution block function; loop is the number of convolution operations; add is the tensor addition flag, which directly outputs the tensor when it is 0, and outputs it after adding the input and output tensors when it is 1; The first stage convolution output size is 56×56, the intermediate tensor of 64 channels, F 56 The final output size of the first stage is a 56×56 tensor; The output size of the second stage convolution is 28×28, with 128 channels of intermediate tensor, F 28 The final output size of the second stage is a 28×28 tensor; The output size of the third stage convolution is 14×14, with 192 channels of intermediate tensor, F 14 The final output size of the third stage is a 14×14 tensor; The output size of the fourth stage convolution is 7×7, the intermediate tensor of 256 channels, and F7 is the final output size of the fourth stage is 7×7 tensor;

[0094] Furthermore, the feature fusion module of S2 extracts deep and shallow layer information and fuses the extracted information according to the following formula:

[0095]

[0096]

[0097]

[0098] Among them F iis the output result of the first, second and third stage convolution; F j is the output of the second, third, and fourth stage convolutions; {·} represents the cascade operation; F is the channel concatenation function; S is the channel shuffling; C CBR is the CBR convolution module; C 3×3 is a convolution operation with a convolution kernel size of 3×3; L interpolation It is the upsampling module operation; The tensor obtained by the fast convolution pyramid module; is the tensor after upsampling output; f i The sum of the tensors obtained after upsampling and fast convolution pyramid module; i = {56, 28, 14}; j = {28, 14, 7};

[0099] Furthermore, the feature fusion and result output of S3 are performed according to the following formula:

[0100] X all =F(P avgpool (F7), P avgpool (f 14 ), P avgpool (f 28 ), P avgpool (f 56 ))

[0101] out=C softmax F c (F c (X all ,c=512),c=k n )

[0102] where X all is the total output after the average pooling; F is the feature concatenation function; P avgpool is the average pooling operation; F7 is the feature tensor of the backbone network output size 7; f 14 The feature fusion module outputs a feature tensor of size 14; f 28 The feature fusion module outputs a feature tensor of size 28; f 56 The feature fusion module outputs a feature tensor of size 56; F c is a fully connected convolution operation; k n is the number of categories; C softmax is the softmax classifier; out is the final classification category.

[0103] The ultimate goal of the classification network is to achieve each image in the dataset i can accurately identify the corresponding type k nFor a given training data set {(Ι1:k1), (Ι2:k2)····Ι i :k n}, establish I by the following method i With k n The corresponding relationship between , so that the training loss loss is minimized:

[0104]

[0105] Among them, loss is the overall loss; m is the number of training images; k i is the true category label corresponding to the i-th image; a i is the predicted category label of the i-th image output by softmax;

[0106] From the loss function, we can see that when the prediction accuracy of the network model is higher, its training loss value is smaller, and the minimum loss value is used to find the optimal parameters of the network model.

[0107] In order to verify the effectiveness of the lightweight classification model and method for multiple types of fundus lesions, the UCSD dataset, Duke dataset, and NEH dataset were selected for testing. The above three datasets are all OCT image datasets. Fang's algorithm (FANG L, WANG C, LI S, et al. Attention to lesion: Lesion-aware convolutional neural network for retinal optical coherence tomography image classification [J]. IEEE transactions on medical imaging, 2019, 38(8): 1959-1970); Liu's method (LIUX, BAI Y, CAO J, et al. Joint disease classification and lesion segmentation via one-stage attention-based convolutional neural network in OCT images[J].Biomedical Signal Processing and Control,2022,71:103087); Mishra's method (MISHRAS S,MANDAL B,PUHAN N B.Perturbed composite attention model for macular optical coherence tomography image classification[J].IEEE Transactions onArtificial Intelligence,2021,3(4):625-635); Das's method (DAS V,DANDAPAT S,BORA PK.Automated classification of retinal OCT images using a deep multi-scalefusion CNN[J]. IEEE Sensors Journal, 2021, 21(20):23256-23265); Anju’s method (THOMASA, HARIKRISHNAN PM, KRISHAN AK, et al.A novel multiscale convolutional neural network based age-related macular degeneration detection using OCT images[J].Biomedical Signal Processing and Control,2021,67:102538); Paima's method (SOTOUDEH-PAIMA S,JODEIRI A,HAJIZADEH F,et al.Multi-scale convolutional neural network for automated AMD classification using retinal OCT images[J].Computers in biology and medicine,2022,144:105368); Mesut’s method (. M,ERGEN B, V. Use of dominant activations obtained by processing OCT images with the CNNs and slime mold method in retinal disease detection [J]. Biocybernetics and Biomedical Engineering, 2022, 42 (2): 646-666) and the experimental results of the present invention were compared and analyzed for objective data.

[0108] Table 1 Comparison of different methods on the UCSD dataset

[0109]

[0110] Table 2 Comparison of different methods on the Duke dataset

[0111]

[0112]

[0113] Table 3 Comparison of different methods on the NEH dataset

[0114]

[0115] As shown in Tables 1-3, the method implemented in this paper (MB-CNN) achieves the highest overall accuracy on all three datasets, with overall accuracy improvements of 0.36% to 7.1% compared to the comparison methods. Furthermore, the number of parameters is only 1.35M.

[0116] Drusen lesions are small and have unclear features, often resulting in low recognition accuracy. As shown in Table 1, the method implemented in this paper achieves a 92.24% recognition accuracy for Drusen, an improvement of 22.24%, 22.44%, and 4.51% over Fang, Liu, and Das, respectively. It also achieved 91.7% on the NEH dataset.

[0117] The higher the overall accuracy, the more accurate the classification results. Tables 1-3 clearly indicate the test results under the same indicators. Therefore, the method implemented by the present invention significantly improves the classification accuracy and can more accurately classify OCT lesion images. Figure 6 The heat map in the image classification process is shown, that is, the area of interest of the image. It can be seen that Figure 6 As shown in (a-c), the model can accurately focus on the lesion areas of different diseases. Figure 6 (d) shows that for a normal fundus, the focus is mainly on the fovea area.

[0118] If the image classification method described in the embodiment of the present invention is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the existing technology, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the image classification method described in the embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk.

[0119] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included in the scope of protection of the present invention.

Claims

1. A lightweight classification model for multiple types of fundus lesions, characterized by: Includes the backbone network L-Resnet, feature fusion module FFM and classification output; The backbone network consists of a CBR (Conv+BN+Relu) module and four convolution stages at different stages, which is responsible for reducing model parameters, increasing network depth, and strengthening the extraction of deep semantic information from OCT images. The CBR module of the backbone network L-Resnet is responsible for channel expansion of the original image and enriching the image feature expression. The channel expansion uses 64 convolution kernels of size 3×3 to extract features from the input tensor and generate a new tensor at the same time. After the convolution operation, the size of the tensor is halved; The convolution of different stages of the backbone network L-Resnet is divided by the size of the output feature tensor. The input OCT image size is 224×224, and the output size after processing by the CBR module is 112×112. The output tensor sizes of the convolutions in the four different stages are 56×56, 28×28, 14×14, and 7×7, respectively. The number of multi-scale convolution blocks MultiBlock used in each stage is 2, 2, 6, and 2, respectively. The number of convolution kernels used in each convolution in each stage is 64, 128, 192, and 256, respectively. The multi-scale convolution block MultiBlock is composed of depthwise convolutions with convolution kernels of different sizes. Four convolution kernels of different sizes are used to extract lesion features of different sizes in the image. The four different sub-tensors obtained are concatenated to form a complete tensor. Skip connections are used to link the input and output of the same convolution operation, so that the convolution operation can learn useful features and eliminate network degradation. The feature fusion module (FFM), consisting of an upsample module and a fast convolutional spatial pyramid module (CSPPF), is responsible for extracting shallow network detail information and deep semantic information. Upsampling is used to enhance the fusion of shallow and deep network information and strengthen the classification of lesion OCT images. The FFM module upsamples deep features through the Upsample module and extracts features of different sizes again through the CSPPF module, enriching the expression of lesion geometric information and lesion spatial information in the OCT image in the network, and transferring features of different specificities in the OCT image extracted from the shallow layer to the deep layer, realizing the fusion of shallow texture information and deep semantic information; The classification output is composed of average pooling, concatenation, and two fully connected layers, and is responsible for outputting the classification results of fundus lesions.

2. The lightweight classification model for multiple types of fundus lesions according to claim 1 is characterized in that: Each convolution kernel of the depthwise convolution is responsible for one channel, and one channel is convolved by only one convolution kernel. The number of feature channels generated in this process is the same as the number of input channels. The different sub-tensors are concatenated into a tensor according to the channel dimension through the Concat operation; The jump connection passes the input tensor directly to the convolution output, so that the tensor after the input tensor kernel convolution is combined; The multi-scale convolution block MultiBlock uses four convolution kernels of different sizes in the same convolution operation to complete the convolution operation. During the convolution operation, the input tensor is divided into four equal parts, and convolution kernels of sizes 1, 3, 5, and 7 are used to perform convolution operations on the four input sub-tensors respectively. Convolution kernels of different sizes are used to extract lesion texture information and lesion edge information of different sizes. The new feature tensors obtained by different convolution kernels are shuffled through channels to enhance information flow and feature expression capabilities. Finally, they are activated using an activation function and passed to the next layer. The network degradation is the phenomenon that the network training accuracy decreases during the network deepening process; The channel shuffling disrupts a set of feature tensors and reorders them to form a new feature tensor.

3. The lightweight classification model for multiple types of fundus lesions according to claim 1 is characterized in that: The upsampling of the feature fusion module (FFM) first expands the channel dimension of the input tensor to 256 through a 1×1 convolution, and then uses bilinear interpolation to expand the feature size to keep it consistent with the feature size of the previous layer. The fast convolutional spatial pyramid CSPPF of the feature fusion module (FFM) first expands the channel through a 1×1 convolution, and then uses three cascaded 3×3 convolution kernels to achieve the receptive field that can be achieved with convolution kernel sizes of 3, 5, and 7. The obtained output tensor is spliced according to the channel dimension, and after the channels are shuffled, the 1×1 convolution kernel is used again to adjust the channels, limiting the output tensor channels to 256.

4. A lightweight classification method for multiple types of fundus lesions, implemented based on the lightweight classification model according to any one of claims 1 to 3, characterized in that: Follow these steps: S1. Preprocess the fundus lesion image to convert the image size to 224×224. Then, pass the processed fundus lesion image through the CBR module and sequentially feed it into four different stages of convolution modules composed of multi-scale convolution to enhance the network's ability to extract information and strengthen the extraction of semantic information, and then output it. S2, the feature fusion module uses the upsampling module and the fast convolutional pyramid module to extract lesion features again, fuses the deep and shallow lesion features, and outputs three feature tensors of different sizes; S3. Fusion the output of the backbone network and the output of the feature fusion module to obtain the classification results of fundus lesions.

5. The lightweight classification method for multiple types of fundus lesions according to claim 4, characterized in that: The image size conversion of S1 and the feature extraction performed by convolution at different stages are performed according to the following formula, and the output of different convolutions is passed to the feature fusion module. X 224 =R e (I i ) Among them Ι i is the input image; R e Set the image size; X 224 After preprocessing, the feature tensor size is 224×224; is a convolution operation with a kernel size of 3×3 and 64 kernels; Β is a batch normalization operation; R is the activation function (ReLU); s is the step size of the convolution operation, is a new tensor of size 112×112 and 64 channels obtained after passing through the CBR convolution module; C m (x) represents the multi-scale convolution layer function; x is the input tensor; C is a tensor partitioning operation that divides the tensor into 4 sub-tensors; n×n is a convolution kernel of size n×n (n=1, 3, 5, 7), which convolves the sub-tensors obtained above respectively; F is the channel dimension splicing function; S is the channel shuffling function; M is the multi-scale convolution block function; loop is the number of convolution operations; add is the tensor addition flag, which directly outputs the tensor when it is 0, and outputs it after adding the input and output tensors when it is 1; The first stage convolution output size is 56×56, the intermediate tensor of 64 channels, F 56 The final output size of the first stage is a 56×56 tensor; The output size of the second stage convolution is 28×28, with 128 channels of intermediate tensor, F 28 The final output size of the second stage is a 28×28 tensor; The output size of the third stage convolution is 14×14, with 192 channels of intermediate tensor, F 14 The final output size of the third stage is a 14×14 tensor; The output size of the fourth stage convolution is 7×7, the intermediate tensor of 256 channels, and F7 is the final output size of the fourth stage is 7×7 tensor.

6. The lightweight classification method for multiple types of fundus lesions according to claim 4, characterized in that: The feature fusion module of S2 extracts deep and shallow layer information and fuses the extracted information according to the following formula: Among them F i is the output result of the first, second and third stage convolution; F j is the output of the second, third, and fourth stage convolutions; {·} represents the cascade operation; F is the channel concatenation function; S is the channel shuffling; C CBR is the CBR convolution module; C 3×3 is a convolution operation with a convolution kernel size of 3×3; L interpolation It is the upsampling module operation; The tensor obtained by the fast convolution pyramid module; is the tensor after upsampling output; f i The sum of the tensors obtained after upsampling and fast convolution pyramid module; i = {56, 28, 14}; j = {28, 14, 7}.

7. The lightweight classification method for multiple types of fundus lesions according to claim 4, characterized in that: The feature fusion and result output of S3 are performed according to the following formula: X all =F(P avgpool (F7),P avgpool (f 14 ),P avgpool (f 28 ),P avgpool (f 56 )) out=C softmax F c (F c (X all ,c=512),c=k n ) where X all is the total output after the final average pooling; F is the feature splicing function; P avgpool is the average pooling operation; F7 is the feature tensor of the backbone network output size 7; f 14 The feature fusion module outputs a feature tensor of size 14; f 28 The feature fusion module outputs a feature tensor of size 28; f 56 The feature fusion module outputs a feature tensor of size 56; F c is a fully connected convolution operation; k n is the number of categories; C softmax is the softmax classifier; out is the final classification category.

8. An electronic device, characterized in that: The method according to any one of claims 4 to 7 is used to implement feature extraction and image classification.

9. A computer storage medium, characterized in that The storage medium stores at least one program instruction, and the at least one program instruction is loaded and executed by the processor to implement the lightweight classification method according to any one of claims 4 to 7.

Images