Lesion grading method, device, equipment and storage medium

Abstract

The present application discloses a method, apparatus, device, and storage medium for lesion grading, relating to the field of computer vision technology. The method includes: obtaining a retinal image to be processed; inputting the retinal image to be processed into a pre-trained diabetic retinopathy grading model to obtain a diabetic retinopathy grading result, where the diabetic retinopathy grading model is trained based on a multi-branch network architecture. The present application can improve the credibility of the diabetic retinopathy grading model.

Description

Technical Field

[0001] This application relates to the field of computer vision, and in particular to a lesion grading method, device, equipment, and storage medium. Background Art

[0002] There is a problem of uneven sample distribution in existing diabetic retinopathy grading samples. For example, there is a long-tail distribution phenomenon in the dataset, resulting in the existing diabetic retinopathy grading model being biased towards the head classes and ignoring the tail classes during training, and the credibility of the diabetic retinopathy grading model is not high.

[0003] The above content is only used to assist in understanding the technical solution of this application, and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of this application is to provide a lesion grading method, device, equipment, and storage medium, aiming to solve the technical problem of the low credibility of the current diabetic retinopathy grading model.

[0005] To achieve the above purpose, this application proposes a lesion grading method, and the method includes:

[0006] Obtain a to-be-processed retinal image;

[0007] Input the to-be-processed retinal image into a pre-trained diabetic retinopathy grading model to obtain a diabetic retinopathy grading result. The diabetic retinopathy grading model is trained based on a multi-branch network architecture.

[0008] In one embodiment, the diabetic retinopathy grading model includes a trained vision transformer, a feature extractor, and a multi-branch classifier. Before the step of inputting the to-be-processed retinal image into the pre-trained diabetic retinopathy grading model to obtain a diabetic retinopathy grading result, it includes:

[0009] Construct the feature extractor based on a convolutional neural network model with a multi-branch network structure, a multi-scale feature fusion module, and a multi-head self-attention mechanism.

[0010] In one embodiment, before the step of inputting the to-be-processed retinal image into the pre-trained diabetic retinopathy grading model to obtain a diabetic retinopathy grading result, it further includes:

[0011] Obtain unlabeled fundus image data;

[0012] Perform self-supervised learning on an initial vision transformer based on the unlabeled fundus image data to obtain a first vision transformer;

[0013] Obtain the target data set;

[0014] Input the target data set into the first vision transformer for supervised learning to obtain the trained vision transformer.

[0015] In one embodiment, the step of inputting the to-be-processed retinal image into the pre-trained diabetic retinopathy grading model to obtain the diabetic retinopathy grading result includes:

[0016] Input the to-be-processed retinal image into the feature extractor to obtain lesion features;

[0017] Input the lesion features into the multi-branch classifier to obtain a first grading result;

[0018] Input the to-be-processed retinal image into the vision transformer to generate weights;

[0019] Multiply the weights and the first grading result to obtain the diabetic retinopathy grading result.

[0020] In one embodiment, the step of inputting the to-be-processed retinal image into the vision transformer to generate weights includes:

[0021] Cut the to-be-processed retinal image into a plurality of image patches;

[0022] Map the plurality of image patches into one-dimensional vectors respectively;

[0023] Concatenate the one-dimensional vectors with randomly initialized classification tokens to obtain a concatenated vector;

[0024] Input the concatenated vector into the multi-head self-attention layer for processing to obtain a processing result;

[0025] Input the processing result into the linear layer to obtain the weights.

[0026] In one embodiment, the multi-branch classifier includes a plurality of classifiers, and the step of inputting the lesion features into the multi-branch classifier to obtain a first grading result includes:

[0027] Extract the lesion features through the improved loss functions in each classifier;

[0028] Combine the weights of each classifier and superimpose the lesion features extracted by each classifier to obtain the first grading result.

[0029] In one embodiment, the feature extractor includes five stages. The first three stages of the feature extractor are composed of the multi-scale feature fusion module, and the last two stages of the feature extractor are composed of the multi-head self-attention mechanism. A dual-branch convolutional neural network model is used as a downsampling module for downsampling between each stage, and the convolutional modules in adjacent downsampling modules adopt a multi-scale feature extraction structure.

[0030] In addition, to achieve the above object, the present application also proposes a lesion grading device, which includes:

[0031] A data acquisition module, configured to acquire a to-be-processed retinal image;

[0032] A data processing module, configured to input the to-be-processed retinal image into a pre-trained diabetic retinopathy grading model to obtain a diabetic retinopathy grading result. The diabetic retinopathy grading model is trained based on a multi-branch network architecture.

[0033] In addition, to achieve the above object, the present application also proposes a lesion grading device, which includes: a memory, a processor, and a computer program stored on the memory and executable on the processor. The computer program is configured to implement the steps of the lesion grading method as described above.

[0034] In addition, to achieve the above object, the present application also proposes a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium, and when the computer program is executed by a processor, it implements the steps of the lesion grading method as described above.

[0035] One or more technical solutions proposed by the present application have at least the following technical effects:

[0036] The present application inputs a to-be-processed retinal image into a pre-trained diabetic retinopathy grading model to obtain a diabetic retinopathy grading result. Among them, the diabetic retinopathy grading model is trained based on a multi-branch network architecture, which can enable the diabetic retinopathy grading model to dynamically adjust the weights of each branch according to different input features. This dynamic fusion strategy enables the model to more accurately focus on the features that have an important impact on classification, while suppressing the interference of irrelevant features on the result, and effectively coping with the problem of sample imbalance. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0038] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0039] Figure 1 It is a schematic flowchart provided for Embodiment 1 of the lesion grading method of the present application;

[0040] Figure 2 It is a schematic structural diagram of the feature extractor of the present application;

[0041] Figure 3 It is a schematic structural diagram of the first downsampling in the feature extractor of the present application;

[0042] Figure 4 It is a schematic structural diagram of the subsequent downsampling in the feature extractor of the present application;

[0043] Figure 5 It is a schematic structural diagram of the multi-scale feature fusion module in the feature extractor of the present application;

[0044] Figure 6 It is a schematic structural diagram of the spatial attention module in the multi-scale feature fusion module of the present application;

[0045] Figure 7 It is a schematic flowchart of the visual transformer processing images;

[0046] Figure 8 It is a schematic overall flowchart of the lesion grading method of the present application;

[0047] Figure 9 It is the sample distribution of the sample dataset APTOS2019;

[0048] Figure 10 It is a schematic diagram of the result predicted by only using the feature extractor to extract features;

[0049] Figure 11 It is a schematic diagram of the result predicted by using the feature extractor and the multi-branch classifier to extract features;

[0050] Figure 12 It is a schematic diagram of the ablation experiment results of multiple models on the sample dataset APTOS2019;

[0051] Figure 13 It is a schematic module structure diagram of the lesion grading device in the embodiment of the present application;

[0052] Figure 14 It is a schematic device structure diagram of the hardware operating environment involved in the lesion grading method in the embodiment of the present application.

[0053] The realization of the purpose, functional features and advantages of this application will be further described in conjunction with embodiments and with reference to the accompanying drawings. Specific embodiments

[0054] It should be understood that the specific embodiments described herein are only used to explain the technical solution of this application and are not used to limit this application.

[0055] To better understand the technical solution of this application, the following will be described in detail in conjunction with the drawings of the specification and specific embodiments.

[0056] The main solution of the embodiment of this application is: obtaining a to-be-processed retinal image; inputting the to-be-processed retinal image into a pre-trained diabetic retinopathy grading model to obtain a diabetic retinopathy grading result, and the diabetic retinopathy grading model is trained based on a multi-branch network architecture.

[0057] In this embodiment, for the convenience of description, the following will be described with the lesion grading system as the execution subject.

[0058] Since there is a problem of uneven sample distribution in the existing diabetic retinopathy grading samples, such as the long-tail distribution phenomenon in the dataset, the existing diabetic retinopathy grading models tend to be biased towards the head classes and ignore the tail classes during training, and the credibility of the diabetic retinopathy grading models is not high.

[0059] This application provides a solution. This application inputs the to-be-processed retinal image into a pre-trained diabetic retinopathy grading model to obtain a diabetic retinopathy grading result. Among them, training the diabetic retinopathy grading model based on a multi-branch network architecture can enable the diabetic retinopathy grading model to dynamically adjust the weights of each branch according to different input features. This dynamic fusion strategy enables the model to more accurately focus on the features that have an important impact on classification, while suppressing the interference of irrelevant features on the result, and effectively coping with the problem of sample imbalance.

[0060] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device, a lesion grading device, etc. that can implement the above functions, or an electronic system, a lesion grading system, etc. that can implement the above functions. The following will take the lesion grading system as an example to illustrate this embodiment and the following embodiments.

[0061] Based on this, the embodiment of this application provides a lesion grading method, referring to Figure 1 , Figure 1 which is the flowchart provided for the first embodiment of the lesion grading method of this application.

[0062] In this embodiment, the lesion grading method includes steps S60 to S70:

[0063] Step S60, obtaining a retinal image to be processed;

[0064] The retinal image to be processed is a retinal image.

[0065] Step S70 , inputting the retinal image to be processed into a pre-trained diabetic retinopathy grading model to obtain a diabetic retinopathy grading result, wherein the diabetic retinopathy grading model is trained based on a multi-branch network architecture.

[0066] As an embodiment, the grading results may include normal, mild lesion, moderate lesion, severe lesion and proliferative diabetic retinopathy.

[0067] The multi-branch network architecture allows the model to extract different types of features from the same input. Each branch can focus on different types of features, such as local details, global structure, or specific lesion characteristics. Furthermore, the multi-branch structure allows the network to process different features in parallel in different branches, which enhances the model's flexibility and processing power. Each branch can perform calculations independently, thereby improving overall computational efficiency. Since lesions in fundus images can appear in different sizes and morphologies, the model can process features of different scales through different branch networks.

[0068] This embodiment provides a lesion grading method, which inputs the retinal image to be processed into a pre-trained diabetic retinopathy grading model to obtain a diabetic retinopathy grading result. The diabetic retinopathy grading model is trained based on a multi-branch network architecture, which can enable the diabetic retinopathy grading model to dynamically adjust the weights of each branch according to the different input features. This dynamic fusion strategy enables the model to focus more accurately on features that have an important impact on classification, while suppressing the interference of irrelevant features on the results, effectively addressing the problem of sample imbalance.

[0069] Currently, there are problems in the detection of diabetic retinopathy, such as too few samples and unbalanced sample distribution. Most of the existing publicly available fundus image datasets have small data volumes, which cannot fully train deep learning models and demonstrate high prediction accuracy. Secondly, the long-tail distribution phenomenon in the dataset is serious, and the model tends to favor the head class and ignore the tail class during training, resulting in low model credibility.

[0070] In addition, the deep learning networks used in the graded detection of diabetic retinopathy can be roughly divided into two categories: CNN (convolutional neural network model) and Transformer (converter).

[0071] In the case of only supervised learning, the traditional convolutional neural network model has a relatively low number of parameters. However, it mostly relies on a fixed structure, making it difficult to adapt to feature extraction in different complex situations and insufficient in capturing subtle lesions in images, resulting in inaccurate classification results.

[0072] The traditional Transformer model has a large number of parameters and requires a large amount of data and computing resources during training. Therefore, it is difficult to adapt to the existing fundus image dataset when training from scratch.

[0073] Based on the characteristics of the Transformer model, there is a two-stage model training method: in the first stage, pre-training is carried out on a large number of fundus images, and in the second stage, end-to-end supervised learning is carried out on the target dataset.

[0074] This method effectively improves the accuracy of the model. However, it requires a large amount of data support and computing resource support. In addition, this method does not solve the problem of unbalanced sample distribution.

[0075] The present invention aims to reconstruct a feature extractor suitable for fundus images by combining the advantages of CNN and Transformer to improve the prediction accuracy. Secondly, it combines the pre-trained base model to enable the model to fairly learn the lesion features of each category and solve the problem of sample imbalance.

[0076] In the embodiments of the present application, first, a feature extractor constructed based on a convolutional neural network model is described. [[ID=X19]]

[0077] Based on Embodiment 1 of the present application, in Embodiment 2 of the present application, the same or similar content as that in Embodiment 1 above can be referred to the above introduction and will not be repeated hereinafter. On this basis, the diabetic retinopathy grading model includes a trained vision transformer, a feature extractor, and a multi-branch classifier. Before step S70 of inputting the to-be-processed retinal image into the pre-trained diabetic retinopathy grading model to obtain the diabetic retinopathy grading result, step S10 is further included:

[0078] Step S10, constructing the feature extractor based on a convolutional neural network model with a multi-branch network structure, a multi-scale feature fusion module, and a multi-head self-attention mechanism.

[0079] Among them, the full name of the multi-scale feature fusion module is Full Perception Feed-Forward Fusion, also known as the PFF module, which is used to extract the high-frequency features of fundus images.

[0080] Among them, the multi-head self-attention mechanism is also known as MSA and is used to generate global features with long-range dependencies.

[0081] Optionally, referring to Figure 2 , Figure 2 is a schematic structural diagram of the feature extractor of this application. The feature extractor includes five stages. The first three stages of the feature extractor are composed of the multi-scale feature fusion module, and the last two stages of the feature extractor are composed of the multi-head self-attention mechanism. A double-branch convolutional neural network model is used as the downsampling module for downsampling between stages, and the convolutional modules in adjacent downsampling modules adopt a multi-scale feature extraction structure.

[0082] Among them, the specific structure of the feature extractor includes Q1~Q10:

[0083] Q1, the first downsampling layer.

[0084] Q2, the multi-scale feature fusion module.

[0085] Q3, the subsequent downsampling layer.

[0086] Q4, the multi-scale feature fusion module.

[0087] Q5, the subsequent downsampling layer.

[0088] Q6, the multi-scale feature fusion module.

[0089] Q7, the subsequent downsampling layer.

[0090] Q8, the multi-head attention mechanism module.

[0091] Q9, the subsequent downsampling layer.

[0092] Q10, the multi-head attention mechanism module.

[0093] In Figure 2 , H, W, and C respectively represent the height, width, and channels of the image (both H and W are 224). L represents the number of modules in each stage. As can be seen from Table 1 below:

[0094] Table 1

[0095]

[0096] Among them, for the first downsampling, reference can be made to Figure 3 , Figure 3 is a schematic structural diagram of the first downsampling in the feature extractor of this application.

[0097] Among them, the steps of the first downsampling in the feature extractor are as described in A11~A1325:

[0098] A11, perform a convolution operation on the input image using a 3x3 convolutional kernel.

[0099] A12, perform normalization on the convolutional feature map, and apply the ReLU (Rectified Linear Unit) activation function to the normalized feature map.

[0100] A1311, perform a convolution operation using a 3x3 depthwise separable convolutional kernel.

[0101] A1312, perform normalization on the convolutional feature map, and apply the ReLU activation function to the normalized feature map. A1313, perform a convolution operation using a 3x3 convolutional kernel and set the stride to 2.

[0102] A1321, perform a convolution operation using a 3x3 convolutional kernel and set the stride to 2.

[0103] A1322, perform normalization on the convolutional feature map, and apply the ReLU activation function to the normalized feature map.

[0104] A1323, perform a convolution operation using a 3x3 convolutional kernel.

[0105] A1324, perform normalization on the convolutional feature map, and apply the ReLU activation function to the normalized feature map.

[0106] A1325, perform a convolution operation using a 1x1 convolutional kernel.

[0107] Among them, for subsequent downsampling, refer to Figure 4 , Figure 4 which is the schematic diagram of the subsequent downsampling structure in the feature extractor of this application.

[0108] Among them, the steps of the subsequent downsampling in the feature extractor are as described in A151~A154 and A161~A162.

[0109] A151, perform a convolution operation using a 1x1 convolutional kernel.

[0110] A152, perform normalization on the convolutional feature map, and apply the ReLU activation function to the normalized feature map.

[0111] A153, perform a convolution operation using a 3x3 convolutional kernel and set the stride to 2.

[0112] A154, perform a convolution operation using a 1x1 convolutional kernel.

[0113] A161, perform average pooling operation.

[0114] A162 performs a convolution operation using a 1x1 convolution kernel.

[0115] Among them, the inspiration for the multi-branch structure proposed in this application comes from ensemble learning. While this application hopes to obtain feature information of different categories, it also aims to avoid problems such as overly large model parameters, overfitting, and reduced inference speed.

[0116] Considering the overall scale of the model, this application uses separable convolution and average pooling to construct each branch. The specific structure of the branch is shown in formula (1):

[0117]

[0118] The superposition of the output values of each branch is the final prediction result.

[0119] The corresponding loss function of the branch is shown in formula (2):

[0120]

[0121] After one-hot encoding (one-hot coding) of the label, it is used as a weight and multiplied by the basic cross-entropy loss to suppress the loss calculation of other categories.

[0122] Among them, for the multi-scale feature fusion module, reference can be made to Figure 5 , Figure 5 is the schematic diagram of the structure of the multi-scale feature fusion module in the feature extractor of this application.

[0123] Among them, the multi-scale feature fusion module consists of a multi-scale convolution and a spatial attention module. That is, first, the input is subjected to multi-scale convolution to generate multi-scale maps. Secondly, the multi-scale maps and the original input are superimposed and enter the spatial attention module to generate weights. Finally, the weights are multiplied by the original input as the output of the multi-scale feature fusion.

[0124] Among them, the dashed line on the right side of the spatial attention module represents (1 - the output of the spatial attention module), and the solid line on the left side of the spatial attention module represents the normal output of the spatial attention module.

[0125] Among them, the generation of multi-scale maps by subjecting the input to multi-scale convolution is as shown in Figure 5 , and the input tensor is respectively subjected to convolution operations using convolution kernels of K0*K0, K1*K1, K2*K2, and K3*K3, and then the tensors generated by multi-scale feature extraction are combined.

[0126] Among them, K represents the convolution kernel size, G represents the padding pixel range, and the relationship between K and G is shown in Table 2 below:

[0127] Table 2

[0128]

[0129] Among them, C represents the number of channels, and its value should vary with the position of the module in the stage, as shown in Table 1.

[0130] Among them, F0-F3 represent the feature tensors generated by the four convolution kernels respectively.

[0131] Among them, for the spatial attention module, you can refer to Figure 6 , Figure 6 This is a schematic diagram of the structure of the spatial attention module in the multi-scale feature fusion module of this application.

[0132] Where X and X' represent the input and output of the spatial attention module, respectively.

[0133] Among them, H, W, and C represent the height, width, and channels of the image respectively (H and W are both 224). The value of C should vary with the position of the module in the stage, as shown in Table 1.

[0134] The input is globally pooled, then convolved with a 1x1 convolution kernel. The convolved feature map is normalized, the ReLU activation function is applied to the normalized feature map, and the convolution operation is performed again with a 1x1 convolution kernel to obtain a first output. The input is convolved with a 1x1 convolution kernel, the convolved feature map is normalized, the ReLU activation function is applied to the normalized feature map, and the convolution operation is performed again with a 1x1 convolution kernel to obtain a second output. A probability map is generated based on the first and second outputs.

[0135] To address the challenges of insufficient sample size and unbalanced sample distribution, this paper reconstructs a five-stage image feature extractor and a multi-branch classifier corresponding to the number of lesion categories in the dataset. It also introduces a foundational model to guide the weighting of each classifier. Compared to existing detection methods, this method excels in adapting to datasets with unbalanced sample distribution, facilitating feature extraction of subtle lesions, and improving model prediction accuracy.

[0136] The deep learning networks used in the graded detection of diabetic retinopathy can be roughly divided into two categories: CNN and Transformer.

[0137] Specifically, the convolutional neural network model converts low-level features into high-level features through multiple single-branch downsampling processes, and finally completes classification through a linear layer. During the downsampling process, subtle lesion features are easily overlooked, resulting in a decrease in the model's prediction accuracy.

[0138] For the structure of the Vision Transformer (ViT), reference can be made to Figure 7 , Figure 7 which is a schematic diagram of the process for the Vision Transformer to process images.

[0139] Among them, the Vision Transformer cuts the picture into multiple patches (image blocks), maps each patch into a one-dimensional vector, concatenates these one-dimensional vectors with a randomly initialized CLS token (classification token), and passes through a multi-head self-attention layer. Finally, classification is completed using the CLS token through a linear layer. When training on a large-scale dataset, the performance of ViT is higher than that of the convolutional neural network model. However, when training on a small-scale dataset, the effect of ViT is much lower than that of the convolutional neural network model.

[0140] Based on Embodiment 2 of the present application, in Embodiment 3 of the present application, for the same or similar content as that in Embodiment 2 above, reference can be made to the above introduction and will not be elaborated hereinafter. On this basis, before step S70, inputting the to-be-processed retinal image into the pre-trained diabetic retinopathy grading model to obtain the diabetic retinopathy grading result, steps S20 to S50 are further included:

[0141] Step S20, obtaining unlabeled fundus image data;

[0142] Step S30, performing self-supervised learning on the initial Vision Transformer based on the unlabeled fundus image data to obtain the first Vision Transformer;

[0143] Step S40, obtaining a target dataset;

[0144] Step S50, inputting the target dataset into the first Vision Transformer for supervised learning to obtain the trained Vision Transformer.

[0145] It should be noted that in order to avoid neglecting subtle lesions during downsampling, one method is to enlarge the size of the input fundus image. However, this method will increase the demand for computing resources and has little effect on improving the accuracy.

[0146] Considering that ViT performs better than CNN in large-scale datasets, there is a two-stage training method,

[0147] that is, first perform self-supervised learning on ViT using a large amount of unlabeled fundus image data (masking a part of the information of the input image and letting the model regenerate the complete image. In this way, it is not necessary to require the image to have labels; for fundus images, data annotation is an extremely complicated task, and self-supervised learning can avoid this problem),

[0148] Then, ViT is used for supervised learning on the target dataset. This approach requires substantial computational resources and time, and also demands the collection of a large number of fundus images (hundreds of thousands or even millions). The ViT trained in this way has significantly improved the prediction accuracy, but the problem of unbalanced distribution in the target dataset has not been resolved in the second stage.

[0149] This embodiment provides a lesion grading method. In this application, the to-be-processed retinal image is input into a pre-trained diabetic retinopathy grading model to obtain the diabetic retinopathy grading result. Among them, the diabetic retinopathy grading model is trained based on a multi-branch network architecture, which enables the diabetic retinopathy grading model to dynamically adjust the weights of each branch according to different input features. This dynamic fusion strategy enables the model to more precisely focus on the features that have an important impact on classification, while suppressing the interference of irrelevant features on the result, and effectively addressing the problem of sample imbalance.

[0150] Based on Embodiment 2 of this application, in Embodiment 4 of this application, the same or similar content as that in Embodiment 2 above can be referred to the above introduction and will not be elaborated hereinafter. On this basis, in step S70, inputting the to-be-processed retinal image into a pre-trained diabetic retinopathy grading model to obtain the diabetic retinopathy grading result includes steps S71 to S74:

[0151] Step S71, inputting the to-be-processed retinal image into the feature extractor to obtain lesion features;

[0152] It should be noted that the present invention finally proposes a framework for diabetic retinopathy grading: it consists of a feature extractor (HFEB) for extracting image features, a multi-branch classifier (MBFF), and a vision transformer (Vision Transformer) for guiding branch weights.

[0153] Among them, ensemble learning is a common method, that is, by combining the outputs of a picture in multiple models as the final output. This method also requires substantial computational resources, and it is also prone to overfitting during training, resulting in performance degradation.

[0154] The purpose of the present invention is first to construct a suitable feature extractor, which should aim to extract all lesion features in the fundus image. The traditional convolutional neural network model consists of multiple single-branch downsampling modules, and the convolutional modules between adjacent downsamplings often consist of single-branch convolutional layers. Considering the characteristics of obvious lesions and subtle lesions, the downsampling module of the feature extractor should consider multiple branch structures, and the convolutional modules of adjacent downsamplings should adopt a multi-scale feature extraction structure.

[0155] Step S72: Input the lesion features into the multi-branch classifier to obtain the first classification result;

[0156] Subsequently, the core mechanism of the Transformer model, multi-head self-attention, is used to integrate the lesion features extracted by the CNN, enabling the model to balance between extracting local features (such as tiny lesions) and capturing global information (such as overall structural changes). Through such a design, the sensitivity of the model to high-frequency information, edge details, and structural changes is enhanced, thereby achieving more efficient feature representation.

[0157] Step S73: Input the to-be-processed retinal image into the vision transformer to generate weights;

[0158] Secondly, the problem of unbalanced sample distribution is solved. Generally, the method to solve such problems is to resample the data, making the sample distribution balanced by oversampling the few-sample classes or undersampling the multi-sample classes. However, the former is prone to overfitting and cannot learn robust features, while the latter will cause underfitting and result in certain information loss. Integrating multiple datasets with the same task can also balance the sample distribution, but due to problems such as different annotation standards and image quality differences in different datasets, the generalization ability of the model will decline.

[0159] Step S74: Multiply the weights and the first classification result to obtain the diabetic retinopathy classification result.

[0160] The present invention proposes a multi-branch classifier module, aiming to improve the diabetic retinopathy classification performance without integrating external datasets, only training on the target dataset and without changing the distribution of the dataset. Each classifier in this module should use an improved loss function to extract the lesion features corresponding to the classes in the image, and combine the weights given by the basic model, and stack the features of each classifier as the final output.

[0161] Based on Embodiment 4 of the present application, in Embodiment 5 of the present application, the same or similar content as that in the above Embodiment 4 can be referred to the above introduction and will not be elaborated hereinafter. On this basis, Step S73: Input the to-be-processed retinal image into the vision transformer to generate weights further includes Steps S731 to S735:

[0162] Step S731: Cut the to-be-processed retinal image into several image patches;

[0163] Step S732: Map the several image patches into one-dimensional vectors respectively;

[0164] Step S733: Concatenate the one-dimensional vectors with randomly initialized classification tokens to obtain the concatenated vectors;

[0165] Step S734, input the spliced vector into the multi-head self-attention layer for processing to obtain a processing result;

[0166] Step S735, input the processing result into a linear layer to obtain the weight.

[0167] Among them, the vision transformer cuts the picture into multiple patches (image blocks), maps each patch into a one-dimensional vector, splices these one-dimensional vectors with a randomly initialized CLS token (classification token), and passes through the multi-head self-attention layer. Finally, classification is completed using the CLS token through the linear layer.

[0168] This embodiment provides a method for lesion grading. In this application, the to-be-processed retinal image is input into a pre-trained diabetic retinopathy grading model to obtain a diabetic retinopathy grading result. Among them, the diabetic retinopathy grading model is trained based on a multi-branch network architecture, which enables the diabetic retinopathy grading model to dynamically adjust the weights of each branch according to different input features. This dynamic fusion strategy enables the model to more accurately focus on the features that have an important impact on classification, while suppressing the interference of irrelevant features on the result, and effectively coping with the problem of sample imbalance.

[0169] Refer to Figure 8 , Figure 8 which is the overall process schematic diagram of the lesion grading method of this application.

[0170] As can be seen from Figure 8 , there are two processing paths for the to-be-processed retinal image in the framework. One is that after the image passes through the feature extractor, it is then classified by a multi-branch classifier; the other is that the to-be-processed retinal image directly passes through the vision transformer to generate weights, and multiplies with the output generated by the classifier to obtain the diabetic retinopathy grading result.

[0171] Among them, Figure 8 B in contains the data output by the multi-branch classifier, representing the features obtained by further processing the output of the feature extractor by the branches of the corresponding categories.

[0172] Among them, Figure 8 C in represents the loss value that needs to be calculated during the training of the diabetic retinopathy grading model and participates in backpropagation, and ω represents the prediction probability of the vision transformer for each category of the picture.

[0173] Among them, alpha and beta are used for weighting the loss function (alpha = 0.2, beta = 0.5).

[0174] The total loss calculated is: beta * (alpha * fusion loss + (1 - alpha) * branch loss) + (1 - beta) * weight loss.

[0175] This application is applied to the dataset APTOS2019 to prove the effectiveness of the structure proposed in this application.

[0176] It should be noted that APTOS2019 is an open dataset for diabetic retinopathy (DR) grading, which was released in the 2019 Kaggle competition. This dataset contains 3,662 labeled fundus images, each of which was taken by a retinal imaging device. The labels are divided into 5 grades: normal, mild, moderate, severe, and proliferative diabetic retinopathy (PDR). The number of samples for each label is as follows: there are 1,805 in the normal category, 370 in the mild category, 999 in the moderate category, 193 in the severe category, and 295 in the proliferative diabetic retinopathy category. As Figure 9 shown, Figure 9 Figure 3.1 shows the sample distribution of the sample dataset APTOS2019. Figure 9 The training set, validation set, and test set are divided in the ratio of 8:1:1.

[0177] Refer to Figure 10 and Figure 11 , Figure 10 Figure 3.2 shows the schematic diagram of the results predicted by only using the feature extractor to extract features. Figure 11 Figure 3.3 shows the schematic diagram of the results predicted by using the feature extractor and the multi-branch classifier to extract features. Among them, Figure 11 the E in Figure 11 represents Figure 10 the category with improved prediction accuracy compared to Figure 11 the D in Figure 11 represents Figure 10 the category with decreased prediction accuracy compared to

[0178] As Figure 10 and Figure 11 shown, it can be seen that under the action of only the multi-classifier, the accuracy of the tail classes increases significantly, while the prediction effect of the head classes decreases to a certain extent. This is because the same weight is given to each classifier during training.

[0179] In view of the above problems, the embodiments of this application are proposed.

[0180] Based on Embodiment 4 of the present application, in Embodiment 6 of the present application, for the same or similar content as in the above Embodiment 4, reference can be made to the above introduction and will not be elaborated hereinafter. On this basis, the multi-branch classifier includes several classifiers. Step S72 of inputting the lesion features into the multi-branch classifier to obtain the first grading result includes steps S721 to S722:

[0181] Step S721, extracting the lesion features through the improved loss function in each classifier;

[0182] Step S722, combining the weights of each classifier and superimposing the lesion features extracted by each classifier to obtain the first grading result.

[0183] Among them, in order to avoid reducing the prediction accuracy of the head classes while improving the prediction accuracy of the tail classes, it is possible to choose to perform weighted superposition on the features of each classifier, and the weights are derived from the Vision Transformer (ViT). The actual effects of the framework and the results of the ablation experiment are as Figure 12 shown Figure 12 which is a schematic diagram of the ablation experiment results of multiple models on the sample data set APTOS2019.

[0184] From Figure 12 it can be seen that the feature extractor + multi-branch classifier + Vision Transformer (using pre-trained parameters) structure adopted in the present application has higher accuracy, area under the curve, Kappa coefficient, and precision than other structures.

[0185] This embodiment provides a lesion grading method. By combining CNN with the multi-head self-attention mechanism, the HFEB module can not only efficiently extract local detail features but also capture global structural information. This innovative design improves the limitations of existing convolutional networks, enabling the model to automatically adapt and extract key features when facing complex retinal images, thereby reducing the probability of misidentification and improving the overall classification performance. Secondly, the multi-branch classifier structure is used to design multiple independent branches, and the weights of each branch are dynamically adjusted according to different input features. This dynamic fusion strategy enables the model to more accurately focus on the features that have an important impact on classification, while suppressing the interference of irrelevant features on the results, and effectively dealing with the problem of sample imbalance.

[0186] It should be noted that the above examples are only for understanding the present application and do not constitute a limitation on the lesion grading method of the present application. Based on this technical concept, more forms of simple transformations are within the protection scope of the present application.

[0187] The present application also provides a lesion grading device. Please refer to Figure 13 where the lesion grading device includes:

[0188] The data acquisition module 10 is used to acquire the to-be-processed retinal images;

[0189] The data processing module 20 is used to input the to-be-processed retinal images into a pre-trained diabetic retinopathy grading model to obtain a diabetic retinopathy grading result. The diabetic retinopathy grading model is trained based on a multi-branch network architecture.

[0190] The lesion grading device provided in this application adopts the lesion grading method in the above embodiment, and can solve the technical problem of low credibility of the diabetic retinopathy grading model. Compared with the prior art, the beneficial effects of the lesion grading device provided in this application are the same as those of the lesion grading method provided in the above embodiment, and other technical features in the lesion grading device are the same as the features disclosed in the method of the above embodiment, and will not be elaborated here.

[0191] This application provides a lesion grading device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the lesion grading method in the first embodiment above.

[0192] Next, refer to Figure 14 , which shows a schematic structural diagram of a lesion grading device suitable for implementing the embodiments of this application. The lesion grading device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistant), PADs (Portable Application Description: tablet computers), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 14 The shown lesion grading device is only an example and should not impose any limitation on the functions and usage scope of the embodiments of this application.

[0193] As Figure 14As shown, the lesion grading device may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 (ROM: Read Only Memory) or a program loaded from a storage device 1003 into a random access memory (RAM: Random Access Memory). In the RAM 1004, various programs and data required for the operation of the lesion grading device are also stored. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems may be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the lesion grading device to communicate with other devices wirelessly or wiredly to exchange data. Although the figure shows a lesion grading device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be alternatively implemented or had.

[0194] Particularly, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program codes for executing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network via the communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the methods of the embodiments disclosed in the present application are executed.

[0195] The lesion grading device provided by the present application adopts the lesion grading method in the above embodiments, and can solve the technical problem of low credibility of the diabetic retinopathy grading model. Compared with the prior art, the beneficial effects of the lesion grading device provided by the present application are the same as those of the lesion grading method provided by the above embodiments, and other technical features in the lesion grading device are the same as those disclosed in the method of the previous embodiment, and will not be elaborated herein.

[0196] It should be understood that each part disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in a suitable manner in any one or more embodiments or examples.

[0197] As described above, it is only the specific implementation manner of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art within the technical scope disclosed in this application can easily think of changes or substitutions, which should all be covered within the protection scope of this application. Therefore, the protection scope of this application shall be subject to the protection scope of the claims.

[0198] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the lesion grading method in the above embodiments.

[0199] The computer-readable storage medium provided by this application can be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM: Random Access Memory), read-only memory (ROM: Read Only Memory), erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this embodiment, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or combined with an instruction execution system, device, or device. The program code contained on the computer-readable storage medium can be transmitted by any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0200] The above computer-readable storage medium can be included in the lesion grading device; it can also exist separately without being assembled into the lesion grading device.

[0201] The above computer-readable storage medium carries one or more programs. When the one or more programs are executed by a lesion grading device, the lesion grading device is caused to: obtain a to-be-processed retinal image; input the to-be-processed retinal image into a pre-trained diabetic retinopathy grading model to obtain a diabetic retinopathy grading result, where the diabetic retinopathy grading model is trained based on a multi-branch network architecture.

[0202] Computer program code for performing the operations of the present application can be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, by using an Internet service provider to connect through the Internet).

[0203] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagram can represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0204] The modules described in the embodiments of the present application can be implemented in software or in hardware. In some cases, the name of the module does not constitute a limitation on the unit itself.

[0205] The readable storage medium provided by this application is a computer-readable storage medium. The computer-readable storage medium stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned lesion grading method, which can solve the technical problem of low credibility of the diabetic retinopathy grading model. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by this application are the same as those of the lesion grading method provided in the above embodiments, and will not be elaborated here.

[0206] The above are only partial embodiments of this application, and thus do not limit the patent scope of this application. Any equivalent structural transformation made under the technical concept of this application by using the content of the specification and drawings of this application, or any direct / indirect application in other related technical fields, is included in the patent protection scope of this application.

Claims

1. A lesion grading method, characterized in that: The method comprises: Get the retinal image to be processed; A feature extractor is constructed based on a convolutional neural network model with a multi-branch network structure, a multi-scale feature fusion module, and a multi-head self-attention mechanism. Inputting the retinal image to be processed into a pre-trained diabetic retinopathy grading model to obtain a diabetic retinopathy grading result, specifically comprising: Inputting the retinal image to be processed into the feature extractor to obtain lesion features; Inputting the lesion features into a multi-branch classifier to obtain a first classification result; Inputting the retinal image to be processed into the trained visual converter to generate weights; multiplying the weight and the first grading result to obtain the diabetic retinopathy grading result; The diabetic retinopathy grading model is obtained based on multi-branch network architecture training, and the diabetic retinopathy grading model includes the trained visual converter, the feature extractor and the multi-branch classifier; The step of inputting the retinal image to be processed into the visual converter and generating weights includes: Cutting the retinal image to be processed into a plurality of image blocks; Mapping the plurality of image blocks into one-dimensional vectors respectively; Concatenating the one-dimensional vector with a randomly initialized classification label to obtain a concatenated vector; Input the concatenated vector into the multi-head self-attention layer for processing to obtain a processing result; The processing result is input into the linear layer to obtain the weight.

2. The method according to claim 1, wherein Before the step of inputting the retinal image to be processed into a pre-trained diabetic retinopathy grading model to obtain a diabetic retinopathy grading result, the step further includes: Acquire unlabeled fundus image data; Performing self-supervised learning on the initial visual converter based on the unlabeled fundus image data to obtain a first visual converter; Get the target dataset; The target data set is input into the first visual converter for supervised learning to obtain the trained visual converter.

3. The method according to claim 1, wherein The multi-branch classifier includes several classifiers, and the step of inputting the lesion features into the multi-branch classifier to obtain a first classification result includes: Extracting lesion features through the improved loss function in each classifier; The lesion features extracted by each classifier are superimposed in combination with the weights of each classifier to obtain the first grading result.

4. The method according to claim 1, wherein The feature extractor includes five stages. The first three stages of the feature extractor are composed of the multi-scale feature fusion module, and the last two stages of the feature extractor are composed of the multi-head self-attention mechanism. A dual-branch convolutional neural network model is used as a downsampling module for downsampling between each stage, and the convolution modules in adjacent downsampling modules adopt a multi-scale feature extraction structure.

5. A lesion grading device, characterized in that: The device comprises: A data acquisition module, used to obtain the retinal image to be processed; A data processing module, configured to input the retinal image to be processed into a pre-trained diabetic retinopathy grading model to obtain a diabetic retinopathy grading result, wherein the diabetic retinopathy grading model is trained based on a multi-branch network architecture; The data processing module is also used to construct a feature extractor based on a convolutional neural network model with a multi-branch network structure, a multi-scale feature fusion module, and a multi-head self-attention mechanism; The data processing module is further configured to input the retinal image to be processed into the feature extractor to obtain lesion features; input the lesion features into a multi-branch classifier to obtain a first grading result; input the retinal image to be processed into a trained visual converter to generate weights; and multiply the weights by the first grading result to obtain the diabetic retinopathy grading result; the diabetic retinopathy grading model is obtained based on a multi-branch network architecture training, and the diabetic retinopathy grading model includes the trained visual converter, the feature extractor, and the multi-branch classifier; The data processing module is also used to cut the retinal image to be processed into several image blocks; map the several image blocks into one-dimensional vectors respectively; splice the one-dimensional vector with the randomly initialized classification label to obtain a spliced vector; input the spliced vector into the multi-head self-attention layer for processing to obtain a processing result; input the processing result into the linear layer to obtain the weight.

6. A lesion grading device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the lesion grading method according to any one of claims 1 to 4.

7. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the lesion grading method according to any one of claims 1 to 4 are implemented.

Images