Lesion grading method, device and equipment and storage medium

Abstract

The invention discloses a lesion grading method and device, equipment and a storage medium, and relates to the technical field of computer vision, and the method comprises the steps: obtaining a to-be-processed retina picture; the to-be-processed retina picture is input into a diabetic retinopathy grading model obtained through pre-training, a diabetic retinopathy grading result is obtained, the diabetic retinopathy grading model is obtained based on multi-branch network architecture training, and the reliability of the diabetic retinopathy grading model can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and in particular to a lesion grading method, device, equipment and storage medium. Background Art

[0002] There is an imbalanced sample distribution problem in the existing diabetic retinopathy grading samples. For example, there is a long-tail distribution phenomenon in the data set, which causes the existing diabetic retinopathy grading model to favor the head class and ignore the tail class during training, and the credibility of the diabetic retinopathy grading model is not high.

[0003] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of the present application is to provide a lesion grading method, device, equipment and storage medium, aiming to solve the technical problem that the current diabetic retinopathy grading model is not reliable.

[0005] To achieve the above objectives, the present application proposes a lesion grading method, the method comprising: Get the retinal image to be processed; The retinal image to be processed is input into a pre-trained diabetic retinopathy grading model to obtain a diabetic retinopathy grading result, wherein the diabetic retinopathy grading model is trained based on a multi-branch network architecture.

[0006] In one embodiment, the diabetic retinopathy grading model comprises a trained visual converter, a feature extractor and a multi-branch classifier, and the step of inputting the retinal image to be processed into the pre-trained diabetic retinopathy grading model to obtain the diabetic retinopathy grading result comprises: The feature extractor is constructed based on a convolutional neural network model with a multi-branch network structure, a multi-scale feature fusion module and a multi-head self-attention mechanism.

[0007] In one embodiment, before the step of inputting the retinal image to be processed into a pre-trained diabetic retinopathy grading model to obtain a diabetic retinopathy grading result, the step further includes: Acquire unlabeled fundus image data; Performing self-supervised learning on the initial visual converter based on the unlabeled fundus image data to obtain a first visual converter; Get the target dataset; The target data set is input into the first visual converter for supervised learning to obtain the trained visual converter.

[0008] In one embodiment, the step of inputting the retinal image to be processed into a pre-trained diabetic retinopathy grading model to obtain a diabetic retinopathy grading result comprises: Inputting the retinal image to be processed into the feature extractor to obtain lesion features; Inputting the lesion feature into the multi-branch classifier to obtain a first classification result; Inputting the retinal image to be processed into the visual converter to generate weights; The weight is multiplied by the first grading result to obtain the diabetic retinopathy grading result.

[0009] In one embodiment, the step of inputting the to-be-processed retinal image into the visual converter and generating weights comprises: Cutting the retinal image to be processed into a plurality of image blocks; Mapping the plurality of image blocks into one-dimensional vectors respectively; Concatenating the one-dimensional vector with a randomly initialized classification label to obtain a concatenated vector; Input the concatenated vector into the multi-head self-attention layer for processing to obtain a processing result; The processing result is input into the linear layer to obtain the weight.

[0010] In one embodiment, the multi-branch classifier includes several classifiers, and the step of inputting the lesion feature into the multi-branch classifier to obtain a first classification result includes: Extracting lesion features through the improved loss function in each of the classifiers; The lesion features extracted by each classifier are superimposed in combination with the weight of each classifier to obtain the first grading result.

[0011] In one embodiment, the feature extractor includes five stages, the first three stages of the feature extractor are composed of the multi-scale feature fusion module, the last two stages of the feature extractor are composed of the multi-head self-attention mechanism, and a dual-branch convolutional neural network model is used as a downsampling module for downsampling between each stage, and the convolution modules in adjacent downsampling modules adopt a multi-scale feature extraction structure.

[0012] In addition, to achieve the above-mentioned purpose, the present application also proposes a lesion grading device, the lesion grading device comprising: A data acquisition module, used to acquire the retinal image to be processed; The data processing module is used to input the retinal image to be processed into a pre-trained diabetic retinopathy grading model to obtain a diabetic retinopathy grading result, wherein the diabetic retinopathy grading model is trained based on a multi-branch network architecture.

[0013] In addition, to achieve the above-mentioned purpose, the present application also proposes a lesion grading device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the lesion grading method described above.

[0014] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the lesion grading method described above are implemented.

[0015] One or more technical solutions proposed in this application have at least the following technical effects: The present application inputs the retinal image to be processed into a pre-trained diabetic retinopathy grading model to obtain the diabetic retinopathy grading result. Among them, the diabetic retinopathy grading model is trained based on a multi-branch network architecture, which can enable the diabetic retinopathy grading model to dynamically adjust the weights of each branch according to different input features. This dynamic fusion strategy enables the model to focus more accurately on features that have an important impact on classification, while suppressing the interference of irrelevant features on the results, effectively dealing with the problem of sample imbalance. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0018] Figure 1 A schematic diagram of a process flow provided for Embodiment 1 of the lesion grading method of the present application; Figure 2 This is a schematic diagram of the structure of the feature extractor of this application; Figure 3 This is a schematic diagram of the structure of the first downsampling in the feature extractor of this application; Figure 4 This is a schematic diagram of the structure of subsequent downsampling in the feature extractor of this application; Figure 5 This is a schematic diagram of the structure of the multi-scale feature fusion module in the feature extractor of this application; Figure 6 This is a schematic diagram of the structure of the spatial attention module in the multi-scale feature fusion module of this application; Figure 7 Schematic diagram of the process of processing images for the visual converter; Figure 8 This is a schematic diagram of the overall process of the lesion grading method of this application; Fig. 9 The sample distribution of the sample dataset APTOS2019; Fig.10 This is a schematic diagram of the prediction results obtained by using only the feature extractor to extract features; Fig.11 A schematic diagram of the results obtained by using a feature extractor and a multi-branch classifier to extract features for prediction; Fig.12 Schematic diagram of ablation experiment results of multiple models on the sample dataset APTOS2019; Fig.13 This is a schematic diagram of the module structure of the lesion grading device according to an embodiment of the present application; Fig.14 Schematic diagram of the device structure of the hardware operating environment involved in the lesion grading method in the embodiment of the present application.

[0019] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0020] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0021] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0022] The main solution of the embodiment of the present application is: obtaining a retinal image to be processed; inputting the retinal image to be processed into a pre-trained diabetic retinopathy grading model to obtain a diabetic retinopathy grading result, wherein the diabetic retinopathy grading model is trained based on a multi-branch network architecture.

[0023] In this embodiment, for the convenience of description, the following description is made by taking the lesion grading system as the execution subject.

[0024] Due to the problem of unbalanced sample distribution in the existing diabetic retinopathy grading samples, such as the long-tail distribution phenomenon in the data set, the existing diabetic retinopathy grading model tends to favor the head class and ignores the tail class during training, and the credibility of the diabetic retinopathy grading model is not high.

[0025] The present application provides a solution, which inputs the retinal image to be processed into a pre-trained diabetic retinopathy grading model to obtain a diabetic retinopathy grading result. The diabetic retinopathy grading model is trained based on a multi-branch network architecture, which can enable the diabetic retinopathy grading model to dynamically adjust the weights of each branch according to different input features. This dynamic fusion strategy enables the model to focus more accurately on features that have an important impact on classification, while suppressing the interference of irrelevant features on the results, effectively addressing the problem of sample imbalance.

[0026] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a lesion grading device, etc., or an electronic system capable of realizing the above functions, a lesion grading system, etc. The following takes the lesion grading system as an example to illustrate this embodiment and the following embodiments.

[0027] Based on this, the present application embodiment provides a lesion grading method, referring to Figure 1 , Figure 1 A schematic diagram of the process flow provided for the first embodiment of the lesion grading method of the present application.

[0028] In this embodiment, the lesion grading method includes steps S60 to S70: Step S60, obtaining a retinal image to be processed; The retinal image to be processed is a retinal image.

[0029] Step S70, inputting the retinal image to be processed into a pre-trained diabetic retinopathy grading model to obtain a diabetic retinopathy grading result, wherein the diabetic retinopathy grading model is trained based on a multi-branch network architecture.

[0030] As an embodiment, the grading results may include normal, mild lesion, moderate lesion, severe lesion and proliferative diabetic retinopathy.

[0031] Among them, the multi-branch network architecture allows the model to extract different types of features from the same input. Each branch can focus on different types of features, such as local details, global structures, or specific lesion features. In addition, the multi-branch structure allows the network to process different features in parallel in different branches, which enhances the flexibility and processing power of the model. Each branch can be calculated independently, thereby improving the overall computational efficiency. Since lesions in fundus images may appear in different sizes and forms, the model can process features of different scales through different branch networks.

[0032] The present embodiment provides a lesion grading method, which inputs the retinal image to be processed into a pre-trained diabetic retinopathy grading model to obtain a diabetic retinopathy grading result, wherein the diabetic retinopathy grading model is trained based on a multi-branch network architecture, which can enable the diabetic retinopathy grading model to dynamically adjust the weights of each branch according to different input features. This dynamic fusion strategy enables the model to focus more accurately on features that have an important impact on classification, while suppressing the interference of irrelevant features on the results, effectively addressing the problem of sample imbalance.

[0033] Currently, there are problems such as too few samples and unbalanced sample distribution in the detection of diabetic retinopathy: most of the existing public fundus image datasets have small data volumes, which cannot fully train the deep learning model and show a high prediction accuracy; secondly, the long-tail distribution phenomenon in the dataset is serious, and the model tends to favor the head class and ignores the tail class during training, resulting in low credibility of the model.

[0034] In addition, the deep learning networks used in the graded detection of diabetic retinopathy can be roughly divided into two categories: CNN (convolutional neural network model) and Transformer (converter).

[0035] In the case of supervised learning only, the traditional convolutional neural network model has a low number of parameters, but it mostly relies on a fixed structure and is difficult to adapt to feature extraction in different complex situations. It is not enough to capture subtle lesions in the image, resulting in inaccurate classification results. However, traditional Transformer models have large parameters and require high data volumes and computing resources during training. Therefore, they are difficult to adapt to existing fundus image datasets when trained from scratch.

[0036] Based on the characteristics of the Transformer model, there is a two-stage model training method: the first stage is pre-training on a large number of fundus images, and the second stage is end-to-end supervised learning on the target dataset.

[0037] This method effectively improves the accuracy of the model, but it requires huge data and computing resources. In addition, this method does not solve the problem of unbalanced sample distribution.

[0038] The present invention aims to combine the advantages of CNN and Transformer to reconstruct a feature extractor suitable for fundus images to improve the prediction accuracy, and secondly, to combine the pre-trained basic model to enable the model to fairly learn the lesion characteristics of each category and solve the sample imbalance problem.

[0039] In the embodiments of the present application, a feature extractor constructed based on a convolutional neural network model is first described.

[0040] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above description, and will not be described in detail later. On this basis, the diabetic retinopathy grading model includes a trained visual converter, a feature extractor, and a multi-branch classifier. Step S70, inputting the retinal image to be processed into the pre-trained diabetic retinopathy grading model, before obtaining the diabetic retinopathy grading result, also includes step S10: Step S10, constructing the feature extractor based on a convolutional neural network model with a multi-branch network structure, a multi-scale feature fusion module and a multi-head self-attention mechanism.

[0041] Among them, the multi-scale feature fusion module is called Full Perception Feed-Forward Fusion, also known as PFF module, which is used to extract high-frequency features of fundus images.

[0042] Among them, the multi-head self-attention mechanism, also known as MSA, is used to generate global features of long-range dependencies.

[0043] Optionally, refer to Figure 2 , Figure 2 This is a structural diagram of the feature extractor of the present application, wherein the feature extractor includes five stages. The first three stages of the feature extractor are composed of the multi-scale feature fusion module, and the last two stages of the feature extractor are composed of the multi-head self-attention mechanism. A dual-branch convolutional neural network model is used as a downsampling module for downsampling between stages, and the convolution modules in adjacent downsampling modules adopt a multi-scale feature extraction structure.

[0044] Among them, the specific structure of the feature extractor includes Q1~Q10: Q1, the first downsampling layer.

[0045] Q2, multi-scale feature fusion module.

[0046] Q3, subsequent downsampling layer.

[0047] Q4, multi-scale feature fusion module.

[0048] Q5, subsequent downsampling layer.

[0049] Q6, multi-scale feature fusion module.

[0050] Q7, subsequent downsampling layer.

[0051] Q8, multi-head attention mechanism module.

[0052] Q9, subsequent downsampling layer.

[0053] Q10, multi-head attention mechanism module.

[0054] exist Figure 2 In the example, H, W, and C represent the height, width, and channels of the image (H and W are both 224). L represents the number of modules in each stage, as shown in Table 1 below: Table 1

[0055] Among them, for the first downsampling, you can refer to Figure 3 , Figure 3 This is a schematic diagram of the structure of the first downsampling in the feature extractor of this application.

[0056] Among them, the first downsampling step in the feature extractor is as described in A11~A1325: A11, uses a 3x3 convolution kernel to perform a convolution operation on the input image.

[0057] A12, normalizes the convolutional feature map and applies the ReLU (Rectified Linear Unit) activation function to the normalized feature map.

[0058] A1311, uses a 3x3 separable convolution kernel for convolution operation.

[0059] A1312, normalize the convolutional feature map and apply the ReLU activation function to the normalized feature map. A1313, use a 3x3 convolution kernel for convolution operation and set the stride to 2.

[0060] A1321, uses a 3x3 convolution kernel for convolution operation and sets the stride to 2.

[0061] A1322, normalizes the convolutional feature map and applies the ReLU activation function to the normalized feature map.

[0062] A1323, uses a 3x3 convolution kernel for convolution operation.

[0063] A1324, normalizes the convolutional feature map and applies the ReLU activation function to the normalized feature map.

[0064] A1325, uses a 1x1 convolution kernel for convolution operation.

[0065] For subsequent downsampling, please refer to Figure 4 , Figure 4 This is a schematic diagram of the structure of subsequent downsampling in the feature extractor of this application.

[0066] Among them, the subsequent downsampling steps in the feature extractor are as described in A151~A154 and A161~A162.

[0067] A151, uses a 1x1 convolution kernel for convolution operation.

[0068] A152, normalizes the convolutional feature map and applies the ReLU activation function to the normalized feature map.

[0069] A153, uses a 3x3 convolution kernel for convolution operation and sets the stride to 2.

[0070] A154, uses a 1x1 convolution kernel for convolution operation.

[0071] A161, performs average pooling operation.

[0072] A162, uses a 1x1 convolution kernel for convolution operation.

[0073] Among them, the inspiration for the multi-branch structure proposed in this application comes from integrated learning. This application hopes to obtain feature information of different categories while avoiding problems such as excessive model parameters, overfitting and reduced reasoning speed.

[0074] Considering the overall scale of the model, this application uses separable convolution and average pooling to construct each branch. The specific structure of the branch is shown in formula (1):

[0075] The sum of the output values ​​of each branch is the final prediction result.

[0076] The corresponding loss function of the branch is shown in formula (2):

[0077] The labels are one-hot encoded and used as weights, which are multiplied by the basic cross entropy loss to suppress the loss calculation of other categories.

[0078] Among them, for the multi-scale feature fusion module, you can refer to Figure 5 , Figure 5 This is a schematic diagram of the structure of the multi-scale feature fusion module in the feature extractor of this application.

[0079] Among them, the multi-scale feature fusion module is composed of a multi-scale convolution and a spatial attention module. That is, the input is first subjected to multi-scale convolution to generate a multi-scale graph, and then the multi-scale graph and the original input are superimposed and enter the spatial attention module to generate weights, and finally the weights are multiplied by the original input as the output of the multi-scale feature fusion.

[0080] Among them, the dotted line on the right side of the spatial attention module represents (1-the output of the spatial attention module), and the solid line on the left side of the spatial attention module represents the output of the normal spatial attention module.

[0081] Among them, the input is convolved at multiple scales to generate a multi-scale image such as Figure 5 As shown, the input tensor is convolved using convolution kernels of K0*K0, K1*K1, K2*K2, and K3*K3 respectively, and then the tensors generated by multi-scale feature extraction are merged.

[0082] Among them, K represents the convolution kernel size, G represents the filling pixel range, and the relationship between K and G is shown in Table 2 below: Table 2

[0083] Among them, C represents the number of channels, and its value should vary with the position of the module in the stage, as shown in Table 1.

[0084] Among them, F0-F3 represent the feature tensors generated by the four convolution kernels respectively.

[0085] Among them, for the spatial attention module, you can refer to Figure 6 , Figure 6 This is a schematic diagram of the structure of the spatial attention module in the multi-scale feature fusion module of this application.

[0086] Among them, X and X' represent the input and output of the spatial attention module respectively.

[0087] Among them, H, W, and C represent the height, width, and channels of the image respectively (H and W are both 224), and the value of C should vary with the position of the module in the stage, as shown in Table 1.

[0088] The input is subjected to a global pooling operation, and then a 1x1 convolution kernel is used to perform a convolution operation, the feature map after the convolution is normalized, the ReLU activation function is applied to the normalized feature map, and a 1x1 convolution kernel is used to perform a convolution operation again to obtain a first output result. The input is subjected to a convolution operation using a 1x1 convolution kernel, the feature map after the convolution is normalized, the ReLU activation function is applied to the normalized feature map, and a 1x1 convolution kernel is used to perform a convolution operation again to obtain a second output result. A probability map is generated based on the first output result and the second output result.

[0089] Based on the problem of too few samples and unbalanced sample distribution, the present invention reconstructs a five-stage image feature extractor, a multi-branch classifier corresponding to the number of lesion categories in the data set, and introduces a basic model to guide the weight of each classifier. Compared with existing detection methods, this method has excellent performance in adapting to data sets with unbalanced sample distribution, promoting feature extraction of subtle lesions, and improving model prediction accuracy.

[0090] The deep learning networks used in the graded detection of diabetic retinopathy can be roughly divided into two categories: CNN and Transformer.

[0091] Specifically, the convolutional neural network model converts low-level features into high-level features through multiple single-branch downsampling processes, and finally completes the classification through the linear layer. During the downsampling process, subtle lesion features are easily ignored, resulting in a decrease in the model's prediction accuracy. For the structure of Vision Transformer (ViT), please refer to Figure 7 , Figure 7 Schematic diagram of the process of processing images for the Vision Transformer.

[0092] Among them, the visual converter cuts the image into multiple patches (image blocks), maps each patch into a one-dimensional vector, concatenates these one-dimensional vectors with a randomly initialized CLS token (classification marker) and passes through a multi-head self-attention layer, and finally uses the CLS token to complete the classification through a linear layer. When trained on a large-scale dataset, ViT's performance is higher than that of the convolutional neural network model, but when trained on a small-scale dataset, ViT's effect is much lower than that of the convolutional neural network model.

[0093] Based on the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the second embodiment can be referred to the above description, and will not be described in detail later. On this basis, step S70, inputting the retinal image to be processed into the pre-trained diabetic retinopathy grading model, before obtaining the diabetic retinopathy grading result, also includes steps S20~S50: Step S20, obtaining unlabeled fundus image data; Step S30, performing self-supervised learning on the initial visual converter based on the unlabeled fundus image data to obtain a first visual converter; Step S40, obtaining a target data set; Step S50: input the target data set into the first visual converter for supervised learning to obtain the trained visual converter.

[0094] It should be noted that in order to avoid subtle lesions being ignored during downsampling, increasing the size of the input fundus image is one method, but this method will increase the demand for computing resources and will have little effect on improving accuracy.

[0095] Considering that ViT outperforms CNN in large-scale datasets, there is a two-stage training method. That is, we first use large-scale unlabeled fundus image data to perform self-supervised learning on ViT (masking part of the input image information and letting the model regenerate the complete image, which does not require the image to have a label; for fundus images, data labeling is an extremely complicated task, and self-supervised learning can avoid this problem). Then, ViT is used for supervised learning of the target dataset. This method requires huge computing resources and time, and also requires the collection of a large number of fundus images (hundreds of thousands or even millions). ViT trained in this way greatly improves the prediction accuracy, but the imbalanced distribution of the target dataset is not solved in the second stage.

[0096] The present embodiment provides a lesion grading method, which inputs the retinal image to be processed into a pre-trained diabetic retinopathy grading model to obtain a diabetic retinopathy grading result, wherein the diabetic retinopathy grading model is trained based on a multi-branch network architecture, which can enable the diabetic retinopathy grading model to dynamically adjust the weights of each branch according to different input features. This dynamic fusion strategy enables the model to focus more accurately on features that have an important impact on classification, while suppressing the interference of irrelevant features on the results, effectively addressing the problem of sample imbalance.

[0097] Based on the second embodiment of the present application, in the fourth embodiment of the present application, the same or similar contents as those in the second embodiment can be referred to the above description, and will not be described in detail later. On this basis, step S70, inputting the retinal image to be processed into the pre-trained diabetic retinopathy grading model, and obtaining the diabetic retinopathy grading result includes steps S71 to S74: Step S71, inputting the retinal image to be processed into the feature extractor to obtain lesion features; It should be noted that the present invention ultimately proposes a framework for grading diabetic retinopathy: it consists of a feature extractor (HFEB) for extracting image features, a multi-branch classifier (MBFF), and a vision transformer (Vision Transformer) for guiding branch weights.

[0098] Among them, ensemble learning is a common method, that is, combining the outputs of multiple models of an image as the final output. This method also requires huge computing resources, and it is easy to overfit during training, resulting in performance degradation.

[0099] The purpose of the present invention is first to construct a suitable feature extractor, which should aim to extract all lesion features in fundus images. The traditional convolutional neural network model consists of multiple single-branch downsampling modules, and the convolution module between two adjacent downsamplings is often composed of a single-branch convolution layer. Considering the characteristics of obvious lesions and subtle lesions, the downsampling module of the feature extractor should take into account multiple branch structures, and the convolution modules of adjacent downsamplings should adopt a multi-scale feature extraction structure.

[0100] Step S72, inputting the lesion feature into the multi-branch classifier to obtain a first classification result; Then, we use the core mechanism of the Transformer model, multi-head self-attention, to integrate the lesion features extracted by CNN, so that the model can achieve a balance between extracting local features (such as tiny lesions) and capturing global information (such as overall structural changes). Through this design, the model's sensitivity to high-frequency information, edge details, and structural changes is improved, thereby achieving more efficient feature representation.

[0101] Step S73, inputting the retinal image to be processed into the visual converter to generate weights; The second is to solve the problem of unbalanced sample distribution. Generally speaking, the solution to this problem is to resample the data, balancing the sample distribution by oversampling the few-sample category or undersampling the many-sample category. However, the former can easily cause overfitting and fail to learn robust features, while the latter can cause underfitting and result in certain information loss. Integrating multiple datasets with the same task can also balance the sample distribution, but since different datasets may have different labeling standards and image quality differences, the generalization of the model will decrease.

[0102] Step S74, multiplying the weight and the first grading result to obtain the diabetic retinopathy grading result.

[0103] The present invention proposes a multi-branch classifier module, which aims to improve the performance of diabetic retinopathy grading without integrating external datasets, training only on the target dataset and without changing the distribution of the dataset. Each classifier in the module should use the improved loss function to extract the lesion features of the corresponding category in the image, and combine the weights given to the classifier by the basic model, and superimpose the features of each classifier as the final output.

[0104] Based on the fourth embodiment of the present application, in the fifth embodiment of the present application, the same or similar contents as those in the fourth embodiment can be referred to the above description, and will not be repeated in the following. On this basis, step S73, inputting the retinal image to be processed into the visual converter, and generating weights also includes steps S731~S735: Step S731, cutting the retinal image to be processed into a plurality of image blocks; Step S732, mapping the plurality of image blocks into one-dimensional vectors respectively; Step S733, concatenating the one-dimensional vector with the randomly initialized classification label to obtain a concatenated vector; Step S734, inputting the concatenated vector into a multi-head self-attention layer for processing to obtain a processing result; Step S735, input the processing result into the linear layer to obtain the weight.

[0105] Among them, the visual converter cuts the image into multiple patches (image blocks), maps each patch into a one-dimensional vector, concatenates these one-dimensional vectors with a randomly initialized CLS token (classification marker) and passes them through a multi-head self-attention layer, and finally uses the CLS token to complete the classification through a linear layer.

[0106] The present embodiment provides a lesion grading method, which inputs the retinal image to be processed into a pre-trained diabetic retinopathy grading model to obtain a diabetic retinopathy grading result, wherein the diabetic retinopathy grading model is trained based on a multi-branch network architecture, which can enable the diabetic retinopathy grading model to dynamically adjust the weights of each branch according to different input features. This dynamic fusion strategy enables the model to focus more accurately on features that have an important impact on classification, while suppressing the interference of irrelevant features on the results, effectively addressing the problem of sample imbalance.

[0107] Reference Figure 8 , Figure 8 Schematic diagram of the overall process of the lesion grading method of this application.

[0108] Depend on Figure 8 It can be seen that there are two processing paths for the retinal image to be processed in the framework. One is that the image passes through the feature extractor and then is classified by a multi-branch classifier; the other is that the retinal image to be processed directly passes through the visual converter to generate weights, which are multiplied by the output generated by the classifier to obtain the diabetic retinopathy grading result.

[0109] in, Figure 8 B contains the data output by the multi-branch classifier, which represents the features obtained by further processing the output of the feature extractor by the branch of the corresponding category.

[0110] in, Figure 8 C in the figure indicates that the loss value needs to be calculated to participate in back propagation during the training of the diabetic retinopathy grading model, and ω represents the predicted probability of each category of the image by the visual converter.

[0111] Among them, alpha and beta are used to weight the loss function (alpha = 0.2, beta = 0.5).

[0112] The calculated total loss = beta * (alpha * fusion loss + (1-alpha) * branch loss) + (1-beta) * weight loss.

[0113] This application is applied to the dataset APTOS2019 to demonstrate the effectiveness of the structure proposed in this application.

[0114] It should be noted that APTOS2019 is a public dataset for grading diabetic retinopathy (DR), which was released in the 2019 Kaggle competition. The dataset contains 3,662 labeled fundus images, each of which is taken by a retinal imaging device. The labels are divided into five levels: normal, mild, moderate, severe, and proliferative diabetic retinopathy (proliferative DR). The number of samples for each label is: 1,805 for normal, 370 for mild, 999 for moderate, 193 for severe, and 295 for proliferative diabetic retinopathy. Fig. 9 As shown, Fig. 9 is the sample distribution of the sample data set APTOS2019, Fig. 9 The training set, validation set, and test set are divided into 8:1:1 ratios.

[0115] Reference Fig.10 and Fig.11 , Fig.10 This is a schematic diagram of the results obtained by using only the feature extractor to extract feature prediction. Fig.11 This is a schematic diagram of the results obtained by using a feature extractor and a multi-branch classifier to extract feature predictions, where: Fig.11 The E in it stands for Fig.11 Comparison Fig.10 The categories where the prediction accuracy is improved, Fig.11 The D in Fig.11 Comparison Fig.10 Classes where prediction accuracy decreases.

[0116] like Fig.10 and Fig.11 As shown, it can be seen that with the help of multiple classifiers alone, the accuracy of the tail class is significantly improved, while the prediction effect of the head class is reduced to a certain extent. This is because each classifier is given the same weight in training.

[0117] In view of the above problems, embodiments of the present application are proposed.

[0118] Based on the fourth embodiment of the present application, in the sixth embodiment of the present application, the same or similar contents as those in the fourth embodiment can be referred to the above description, and will not be described in detail later. On this basis, the multi-branch classifier includes several classifiers, and step S72, inputting the lesion features into the multi-branch classifier, and obtaining the first classification result includes steps S721~S722: Step S721, extracting lesion features through the improved loss function in each of the classifiers; Step S722, combining the weights of the classifiers, superimposing the lesion features extracted by the classifiers to obtain the first grading result.

[0119] In order to avoid reducing the prediction accuracy of the head class while improving the prediction accuracy of the tail class, we can choose to superimpose the features of each classifier with weights, and the weights come from the visual transformer (ViT). The actual effect of the framework and the ablation experiment results are shown in Figure 2. Fig.12 As shown, Fig.12 Schematic diagram of the ablation experiment results of multiple models on the sample dataset APTOS2019.

[0120] Depend on Fig.12 It can be seen that the feature extractor + multi-branch classifier + visual converter (using pre-trained parameters) structure adopted in this application has higher accuracy, area under the curve, Kappa coefficient, and precision than other structures.

[0121] This embodiment provides a lesion grading method. By combining CNN with a multi-head self-attention mechanism, the HFEB module can not only efficiently extract local detail features, but also capture global structural information. This innovative design improves the limitations of existing convolutional networks, allowing the model to automatically adapt and extract key features when faced with complex retinal images, thereby reducing the probability of misidentification and improving overall classification performance. Secondly, a multi-branch classifier structure is used to design multiple independent branches, and the weights of each branch are dynamically adjusted according to different input features. This dynamic fusion strategy enables the model to focus more accurately on features that have an important impact on classification, while suppressing the interference of irrelevant features on the results, effectively addressing the problem of sample imbalance.

[0122] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the lesion grading method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0123] This application also provides a lesion grading device, please refer to Fig.13 , the lesion grading device comprises: A data acquisition module 10 is used to acquire a retinal image to be processed; The data processing module 20 is used to input the retinal image to be processed into a pre-trained diabetic retinopathy grading model to obtain a diabetic retinopathy grading result, wherein the diabetic retinopathy grading model is trained based on a multi-branch network architecture.

[0124] The lesion grading device provided by the present application adopts the lesion grading method in the above embodiment, which can solve the technical problem that the credibility of the diabetic retinopathy grading model is not high. Compared with the prior art, the beneficial effects of the lesion grading device provided by the present application are the same as the beneficial effects of the lesion grading method provided by the above embodiment, and other technical features in the lesion grading device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0125] The present application provides a lesion grading device, which includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the lesion grading method in the above-mentioned embodiment 1.

[0126] Reference below Fig.14 , which shows a schematic diagram of the structure of a lesion grading device suitable for implementing the embodiment of the present application. The lesion grading device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.14 The lesion grading device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0127] like Fig.14As shown, the lesion grading device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 (ROM: Read Only Memory) or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory). In RAM1004, various programs and data required for the operation of the lesion grading device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the lesion grading device to communicate with other devices wirelessly or by wire to exchange data. Although the lesion grading device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0128] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0129] The lesion grading device provided by the present application adopts the lesion grading method in the above embodiment, which can solve the technical problem that the credibility of the diabetic retinopathy grading model is not high. Compared with the prior art, the beneficial effects of the lesion grading device provided by the present application are the same as the beneficial effects of the lesion grading method provided by the above embodiment, and other technical features in the lesion grading device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0130] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0131] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0132] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the lesion grading method in the above-mentioned embodiment.

[0133] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0134] The computer-readable storage medium may be included in the lesion grading device; or may exist independently without being assembled into the lesion grading device.

[0135] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the lesion grading device, the lesion grading device: obtains a retinal image to be processed; inputs the retinal image to be processed into a pre-trained diabetic retinopathy grading model to obtain a diabetic retinopathy grading result, and the diabetic retinopathy grading model is trained based on a multi-branch network architecture.

[0136] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0137] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0138] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0139] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned lesion grading method, and can solve the technical problem that the credibility of the diabetic retinopathy grading model is not high. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the lesion grading method provided in the above-mentioned embodiment, and will not be repeated here.

[0140] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A lesion grading method, characterized in that: The method comprises: Get the retinal image to be processed; The retinal image to be processed is input into a pre-trained diabetic retinopathy grading model to obtain a diabetic retinopathy grading result, wherein the diabetic retinopathy grading model is trained based on a multi-branch network architecture.

2. The method according to claim 1, characterized in that The diabetic retinopathy grading model comprises a trained visual converter, a feature extractor and a multi-branch classifier, and the step of inputting the retinal image to be processed into the pre-trained diabetic retinopathy grading model to obtain the diabetic retinopathy grading result comprises: The feature extractor is constructed based on a convolutional neural network model with a multi-branch network structure, a multi-scale feature fusion module and a multi-head self-attention mechanism.

3. The method according to claim 2, characterized in that Before the step of inputting the retinal image to be processed into a pre-trained diabetic retinopathy grading model to obtain a diabetic retinopathy grading result, the step further includes: Acquire unlabeled fundus image data; Performing self-supervised learning on the initial visual converter based on the unlabeled fundus image data to obtain a first visual converter; Get the target dataset; The target data set is input into the first visual converter for supervised learning to obtain the trained visual converter.

4. The method according to claim 2, characterized in that The step of inputting the retinal image to be processed into a pre-trained diabetic retinopathy grading model to obtain a diabetic retinopathy grading result comprises: Inputting the retinal image to be processed into the feature extractor to obtain lesion features; Inputting the lesion feature into the multi-branch classifier to obtain a first classification result; Inputting the retinal image to be processed into the visual converter to generate weights; The weight is multiplied by the first grading result to obtain the diabetic retinopathy grading result.

5. The method according to claim 4, characterized in that The step of inputting the retinal image to be processed into the visual converter and generating weights comprises: Cutting the retinal image to be processed into a plurality of image blocks; Mapping the plurality of image blocks into one-dimensional vectors respectively; Concatenating the one-dimensional vector with a randomly initialized classification label to obtain a concatenated vector; Input the concatenated vector into the multi-head self-attention layer for processing to obtain a processing result; The processing result is input into the linear layer to obtain the weight.

6. The method according to claim 4, characterized in that The multi-branch classifier includes several classifiers, and the step of inputting the lesion feature into the multi-branch classifier to obtain a first classification result includes: Extracting lesion features through the improved loss function in each of the classifiers; The lesion features extracted by each classifier are superimposed in combination with the weight of each classifier to obtain the first grading result.

7. The method according to claim 2, characterized in that The feature extractor includes five stages. The first three stages of the feature extractor are composed of the multi-scale feature fusion module, and the last two stages of the feature extractor are composed of the multi-head self-attention mechanism. A dual-branch convolutional neural network model is used as a downsampling module for downsampling between each stage, and the convolution modules in adjacent downsampling modules adopt a multi-scale feature extraction structure.

8. A lesion grading device, characterized in that: The device comprises: A data acquisition module, used to acquire the retinal image to be processed; The data processing module is used to input the retinal image to be processed into a pre-trained diabetic retinopathy grading model to obtain a diabetic retinopathy grading result, wherein the diabetic retinopathy grading model is trained based on a multi-branch network architecture.

9. A lesion grading device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the lesion grading method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the lesion grading method according to any one of claims 1 to 7 are implemented.

Images