Diabetic retinopathy lesion detection method based on adaptive multi-label recognition

Abstract

The present application belongs to the technical field of diabetic retinopathy lesion detection, and particularly relates to a diabetic retinopathy lesion detection method based on adaptive multi-label recognition, which comprises: acquiring multi-phase image data of FFA images of a patient; using a "16-tile input" method to combine and splice each group of 10 time-series FFA images; building a D-LAM model composed of a class-adaptive mapping module, a label-specific decoding module and a smooth focus loss function; using the processed FFA image data set to train and verify the model, and finally realizing automatic recognition of different types of lesions on the test set. The method significantly improves the recognition ability of micro-lesions such as microangioma, effectively handles the complex semantic correlation between multi-labels, and effectively assists doctors in identifying diabetic retinopathy, thereby improving the recognition efficiency and accuracy of diabetic retinal lesions.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of lesion detection of diabetic retinopathy, and particularly relates to a diabetic retinopathy lesion detection method based on adaptive multi-label recognition. BACKGROUND

[0002] Diabetic retinopathy (DR) is a common microvascular complication of diabetes, and its incidence significantly increases with the extension of the diabetes duration. The development of DR is accompanied by retinal microvascular lesions, including microaneurysms, hemorrhagic spots, hard exudates, soft exudates and other lesions. The appearance and changes of these features mark the progression of the disease. Therefore, early detection and accurate grading are crucial for preventing vision impairment. Fundus Fluorescein Angiography (FFA) is widely used for the diagnosis of DR, as it can clearly present the subtle features of retinal blood vessels and is considered as the gold standard for diagnosing DR.

[0003] However, the interpretation of FFA images highly depends on the experience and subjective judgment of doctors, and due to the complex and uneven distribution of lesions, misdiagnosis and missed diagnosis are prone to occur. In addition, the DR diagnosis process is time-consuming and laborious, and the traditional manual grading method is difficult to meet the clinical needs. Therefore, the automatic and intelligent DR grading method has gradually become a research hotspot. With the development of deep learning technology, significant progress has been made in automatic lesion recognition based on images. Existing methods mainly focus on the classification and detection of color fundus photographs, but the research on FFA sequence images is relatively less, and faces two major challenges: first, it is difficult to identify the tiny lesions (such as microaneurysms) in FFA sequence images; second, the complex correlation between multi-label lesions is difficult to be effectively handled by existing models. These problems restrict the application effect of automatic grading models, and it is urgent to improve the model architecture to improve the accuracy and robustness of lesion recognition. SUMMARY

[0004] In order to overcome the technical defects that the existing lesion processing model is difficult to identify the tiny lesions of FFA sequence images and difficult to handle the complex correlation between multi-label lesions, the present application provides a diabetic retinopathy lesion detection method based on adaptive multi-label recognition.

[0005] The present application provides a diabetic retinopathy lesion detection method based on adaptive multi-label recognition, comprising the following steps:

[0006] S1, data collection: using a fundus camera to collect multi-phase FFA image data of diabetic retinopathy patients, and labeling the lesion types in the images by an ophthalmologist;

[0007] S2, data preprocessing: for each group of 10 time sequence FFA images in step S1, a "16x16 grid input" method is used for single combination splicing, wherein the first 10 positions of the 4x4 grid are filled in time sequence, 6 images are randomly selected from the remaining images to fill the remaining positions of the 4x4 grid, and image cropping and normalization processing are performed;

[0008] S3, model construction: an adaptive multi-label recognition model, namely D-LAM model, is built, the D-LAM model comprises a class adaptive mapping module, a label-specific decoding module and a smooth focal loss function, wherein the class adaptive mapping module improves the recognition ability of the micro lesions through an adaptive response mechanism controlled by a temperature parameter; the transformer decoder is used to build the label-specific decoding module to process the label association; the smooth focal loss function is designed by combining label smoothing and focal loss to enhance the learning effect on difficult classification samples;

[0009] The step of controlling the adaptive response mechanism of the class adaptive mapping module through the temperature parameter is as follows:

[0010] S311, the normalized class response map score is calculated by the feature mapping generator, as shown in formula (1):

[0011] ,

[0012] In formula (1), head(F) represents the output generated by 1x1 convolution operation, is the L2 norm of the convolution layer weight, which is used for normalizing the output; the normalization process ensures the scale consistency of the class response map, which helps to improve the training effect of the model;

[0013] S312, the base response base_logit is calculated, which is the mean value of the class response map score in the spatial dimension, as shown in formula (2):

[0014] ,

[0015] In formula (2), H and W represent the height and width of the feature map respectively, score ij is the value of the i-th row and j-th column in the class response map score; the mean value operation provides a global response reference for the model, which helps to generate adaptive response subsequently;

[0016] S313, the adaptive response att_logit is controlled by the temperature parameter T, according to different values of the temperature parameter T, there are two cases of maximum pooling response and weighted summation response:

[0017] ,

[0018] S314, the base response base_logit and the adaptive response att_logit are combined by weighting to generate the final output response map M':

[0019] ,

[0020] In formula (4), λ is a hyperparameter for controlling the adaptive response weight, and heads represents the number of attention heads;

[0021] The processing steps of the label-specific decoding module are:

[0022] S321, extract high-dimensional features src and corresponding spatial position encodings pos from the input image F through the backbone network:

[0023] ,

[0024] S322, generate label embeddings and query vectors accordingly to guide label-specific feature extraction during the decoding process; then input the generated high-dimensional features src, spatial position encodings pos, initial query vectors tgt, and label query vectors query_input into the Transformer decoder to extract label-specific responses hs related to specific labels from image features:

[0025] ,

[0026] S323, group the label-specific responses hs output by the Transformer decoder, first intercept the responses of the last num_labels classes from the channel dimension, then take the mean in the spatial dimension to obtain the final response M for each label:

[0027] ;

[0028] The calculation steps of the smooth focal loss function are:

[0029] S331, perform smoothing processing on the target label:

[0030] ,

[0031] In formula (8), y represents the original target label, and smoothing is the smoothing coefficient, which is set to 0.1 by default;

[0032] S332, the comprehensive loss is composed of the label-smoothed weighted binary cross-entropy loss and the focal loss, and the calculation formula is:

[0033] ,

[0034] In formula (9), a is a balance coefficient, used to adjust the weight between the smooth loss and the focal loss.

[0035] S4, model application: using the FFA image set processed in step S2 to train and verify the D-LAM model, and finally realizing automatic identification of different types of lesions on the test set.

[0036] Preferably, in step S1, the image data acquisition is performed by two ophthalmologists using a Heidelberg retinal angiography device under the condition of a 55° field of view. Such operation can exclude subjects suffering from ischemic optic neuropathy, age-related macular degeneration, optic disc edema and other ocular fundus diseases.

[0037] Preferably, in step S1, the lesion type labeling is marked by three ophthalmologists without communication, and if the marking results of the images are consistent, the lesion category is determined; otherwise, images with inconsistent marking need to be discussed until a consistent marking opinion is reached.

[0038] Preferably, in step S3, in the final label prediction stage, the output results of the class adaptive mapping module and the label-specific decoding module are fused with a weight ratio of 3:2.

[0039] The technical scheme provided by the present application has the following technical effects compared with the prior art:

[0040] (I) The present application proposes a "16-square input" method, which sequentially combines multiple images of each group of fluorescent fundus angiography (FFA) examination. This method not only retains the time sequence dynamic characteristics of fundus angiography, but also enhances the feature expression of key time points through random supplement, effectively improving the comprehensiveness and accuracy of lesion identification. Compared with traditional single image input, this method can more fully utilize the information obtained during the examination process, significantly improving the identification efficiency of diabetic retinal lesions.

[0041] (II) The class adaptive mapping module designed in the present application can accurately capture the features of lesions of different scales through the adaptive response mechanism controlled by the temperature parameter and the normalized class response map. Especially for microhemangioma and other small lesions, this module can adaptively adjust the sensitivity of feature extraction, significantly improving the identification ability of subtle lesions.

[0042] (Three) The label-specific decoding module proposed in the application realizes accurate modeling of complex label association by adopting the Transformer architecture. Through the generation mechanism of label embedding and query vector, the module can effectively handle complex scenarios of multiple lesion co-occurrence and accurately identify the association between different types of lesions. This mechanism significantly improves the recognition accuracy of the model for multiple lesions, enabling the model to have stronger discrimination ability when dealing with complex cases.

[0043] (Four) The application innovatively designs a smooth focal loss function, which effectively solves the sample imbalance problem by combining the advantages of label smoothing and focal loss. This loss function not only improves the learning effect of the model on difficult-to-classify samples, but also enhances the generalization ability of the model.

[0044] (Five) The end-to-end adaptive multi-label recognition model of the application organically combines various innovative modules to build a complete lesion recognition system. This system can simultaneously process temporal information, spatial features, and label association, greatly improving the overall performance of the model. In actual clinical applications, the application can effectively assist doctors in identifying diabetic retinopathy, improving the efficiency and accuracy of diabetic retinopathy lesion recognition, and has important clinical application value.

[0045] (Six) The model of the application has good scalability and adaptability, not only suitable for diabetic retinopathy detection, but also can be applied to other types of medical image multi-label recognition tasks by appropriate adjustment. This versatility makes the application have broad application prospects. BRIEF DESCRIPTION OF DRAWINGS

[0046] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the application and, together with the specification, serve to explain the principles of the application.

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0048] Figure 1 The overall flowchart of the diabetic retinopathy lesion detection method based on adaptive multi-label recognition in some embodiments of the application;

[0049] Figure 2 The framework diagram of the adaptive multi-label recognition model in the diabetic retinopathy lesion detection method based on adaptive multi-label recognition in some embodiments of the application;

[0050] Figure 3This is a structural diagram of the category adaptive mapping module in the diabetic retinopathy lesion detection method based on adaptive multi-label recognition according to a certain embodiment of the present invention;

[0051] Figure 4 This is a structural diagram of the tag-specific decoding module in the diabetic retinopathy lesion detection method based on adaptive multi-tag recognition according to a certain embodiment of the present invention. Detailed Implementation

[0052] To better understand the above-mentioned objectives, features, and advantages of the present invention, the solutions of the present invention will be further described below. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other.

[0053] Many specific details are set forth in the following description in order to provide a full understanding of the invention, but the invention may also be practiced in other ways different from those described herein; obviously, the embodiments in the specification are only some embodiments of the invention, and not all embodiments.

[0054] The following is in conjunction with the appendix Figures 1 to 4 Specific embodiments of the present invention will be described in detail below.

[0055] In one embodiment, such as Figure 1 As shown, a method for detecting diabetic retinopathy lesions based on adaptive multi-label recognition is disclosed, including the following steps:

[0056] S1. Data Collection: Multi-temporal fluorescein angiography (FFA) images of patients with diabetic retinopathy are collected using a fundus camera, and ophthalmologists label the lesion types in the images. In a specific embodiment, image data acquisition is performed by two ophthalmologists using a Heidelberg retinal angiography device under a 55° field of view. This procedure excludes subjects with ischemic optic neuropathy, age-related macular degeneration, optic disc edema, or other fundus diseases. Lesion type labeling is performed by three ophthalmologists independently labeling all images without prior communication. If the labeling results are consistent, the lesion category is determined; otherwise, images with inconsistent labeling are discussed until a consensus is reached.

[0057] S2. Data preprocessing: For each group of 10 time-series FFA images in step S1, the "16-grid input" method is used to combine and stitch the images one by one. The first 10 positions of the 4×4 grid are filled in chronological order. Six images are randomly selected from the remaining images to fill the remaining positions of the 4×4 grid. The images are then cropped and normalized.

[0058] S3, model construction: build an adaptive multi-label recognition model, namely D-LAM model, which includes a class adaptive mapping module, a label-specific decoding module and a smooth focus loss function, wherein the class adaptive mapping module improves the recognition ability of small lesions through a temperature parameter controlled adaptive response mechanism; a transformer decoder is used to build a label-specific decoding module to process label association; a smooth focus loss function is designed by combining label smoothing and focus loss to enhance the learning effect of difficult classification samples;

[0059] The step of controlling the adaptive response mechanism of the class adaptive mapping module through the temperature parameter is:

[0060] S311, calculate the normalized class response map score through the feature mapping generator, as shown in formula (1):

[0061] ,

[0062] In formula (1), head(F) represents the output generated by 1x1 convolution operation, is the L2 norm of the convolution layer weight, used for normalizing the output; the normalization process ensures the scale consistency of the class response map, which helps to improve the training effect of the model;

[0063] S312, calculate the base response base_logit, which is the mean value of the class response map score in the spatial dimension, as shown in formula (2):

[0064] ,

[0065] In formula (2), H and W represent the height and width of the feature map respectively, score ij is the value of the i-th row and j-th column in the class response map score; this mean operation provides a global response reference for the model, which helps to generate adaptive response later;

[0066] S313, the adaptive response att_logit is controlled by the temperature parameter T, according to different values of the temperature parameter T, it is divided into two cases: maximum pooling response and weighted summation response:

[0067] ,

[0068] S314, the base response base_logit and the adaptive response att_logit are combined by weighting to generate the final output response map M':

[0069] ,

[0070] In formula (4), lambda is a hyperparameter for controlling the adaptive response weight, and heads represent the number of attention heads.

[0071] The processing steps of the label-specific decoding module are:

[0072] S321, extract high-dimensional features src and corresponding spatial position encodings pos from the input image F through the backbone network:

[0073] ,

[0074] S322, generate label embeddings and query vectors accordingly to guide label-specific feature extraction during the decoding process; then input the generated high-dimensional features src, spatial position encodings pos, initial query vectors tgt, and label query vectors query_input into the Transformer decoder to extract label-specific responses hs related to specific labels from image features:

[0075] ,

[0076] S323, group the label-specific responses hs output by the Transformer decoder, first extract the responses of the last num_labels classes from the channel dimension, then take the mean in the spatial dimension to obtain the final response M for each label:

[0077] ;

[0078] The calculation steps of the smooth focal loss function are:

[0079] S331, perform smoothing processing on the target label:

[0080] ,

[0081] In formula (8), y represents the original target label, and smoothing is the smoothing coefficient, which is set to 0.1 by default;

[0082] S332, the comprehensive loss is composed of the label smoothing weighted binary cross-entropy loss and the focal loss, and the calculation formula is:

[0083] ,

[0084] In formula (9), alpha is a balance coefficient for adjusting the weight between the smoothing loss and the focal loss;

[0085] In the final label prediction stage, the output results of the class adaptive mapping module and the label-specific decoding module are fused with a weight ratio of 3:2;

[0086] S4. Model Application: The FFA image set processed in step S2 is used to train and validate the D-LAM model, and finally, automatic identification of different types of lesions is achieved on the test set.

[0087] Table 1 shows the performance comparison between the D-LAM model constructed by this invention and nine existing models on the collected dataset. The nine existing models include four classic convolutional neural networks and five innovative models proposed in recent years. The results fully demonstrate the superiority of the method of this invention in the task of identifying diabetic retinopathy lesions.

[0088] Table 1 Performance comparison of the method of the present invention with existing models

[0089] Reference method mAP CP CR CF1 OR OF1 VGG16 95.6 93.9 86.7 90.2 93.4 94.3 ResNet101 92.8 91.8 83.7 87.6 92.2 93.7 DenseNet 93.3 92.2 84.3 88.1 93.1 93.9 EfficientNet 80.1 75 65.2 69.8 86.5 89.8 Vit 75.9 75.6 70.4 72.9 86.2 87.7 Swin 71.4 65.9 63.1 64.5 84.8 86.9 GCN 88.6 90.7 78 83.9 89.3 91.7 CSRA 94.99 94.16 76.98 84.71 90.93 92.96 CTran 93.6 94.2 84.9 89.3 92.5 94.3 D-LAM model 96.27 94.78 87.9 91.21 93.52 94.58

[0090] As can be seen from the above, this invention includes three key steps: data acquisition, data processing, and model implementation. In data acquisition, a standardized 55° field-of-view Heidelberg retinal angiography device is used, operated professionally by experienced ophthalmologists who also annotate lesions, while strictly excluding interference from other fundus diseases. In data processing, an innovative "16-grid input" method is proposed, where 10 temporal FFA images are sequentially filled into the first 10 positions of a 4×4 matrix, and the remaining positions are randomly filled, effectively integrating multi-temporal image information. In model implementation, a D-LAM model with a category adaptive mapping module and a label-specific decoding module is built. Through an adaptive response mechanism controlled by temperature parameters and a transformer decoder structure, and by innovatively employing a smooth focus loss function, accurate identification of different types of lesions is achieved, providing clinicians with a reliable tool for identifying diabetic retinopathy lesions and promoting the deep integration of medicine and artificial intelligence.

[0091] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the present invention. Although detailed descriptions have been provided with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments, and they should all be covered within the protection scope of the claims.

Claims

1. A method for diabetic retinopathy lesion detection based on adaptive multi-label recognition, characterized in that, The method comprises the following steps: S1, data collection: collecting multi-time phase image data of a diabetic retinopathy patient by using a fundus camera for a fundus fluorescein angiography (FFA) image, and labeling the lesion types in the image by an ophthalmologist; S2, data preprocessing: combining and splicing each group of 10 time sequence FFA images in step S1 by using a "16-tile input" method, wherein the first 10 positions of a 4x4 tile are sequentially filled in time order, 6 images are randomly selected from the remaining images to fill the remaining positions of the 4x4 tile, and image cropping and normalization processing are performed; S3, model construction: building a self-adaptive multi-label recognition model (D-LAM model), wherein the D-LAM model comprises a class adaptive mapping module, a label-specific decoding module, and a smooth focal loss function, the class adaptive mapping module improves the recognition ability of small lesions through an adaptive response mechanism controlled by a temperature parameter; the label-specific decoding module is constructed by using a transformer decoder to process label association; and the smooth focal loss function is designed by combining label smoothing and focal loss to enhance the learning effect on difficult classification samples; The class adaptive mapping module controls the adaptive response mechanism through a temperature parameter in the following steps: S311, calculating a normalized class response map score by a feature mapping generator, as shown in formula (1): ， In Equation (1), head(F) represents an output generated by a 1x1 convolution operation, is an L2 norm of the convolution layer weight, used for normalizing the output; S312, calculating a base response base_logit, which is the mean value of the class response map score in the spatial dimension, as shown in formula (2): ， In Equation (2), H and W represent the height and width of the feature map, respectively, and score ij is the value of the i-th row and j-th column in the class response map score. S313, the adaptive response att_logit is controlled by a temperature parameter T, and according to different values of the temperature parameter T, there are two cases of maximum pooling response and weighted summation response: ， S314, the base response base_logit and the adaptive response att_logit are combined by weighting to generate the final output response map M': ， In formula (4), λ is a hyperparameter for controlling the weight of the adaptive response, and heads represent the number of attention heads; The processing steps of the label-specific decoding module are as follows: S321, extracting high-dimensional features src and corresponding spatial position encodings pos from the input image F by a backbone network: ， S322, generating label embeddings and query vectors to guide label-specific feature extraction during the decoding process; then inputting the generated high-dimensional features src, spatial position encodings pos, initial query vectors tgt, and label query vectors query_input into a Transformer decoder to extract label-specific responses hs related to specific labels from image features: ， S323, grouping the label-specific responses hs output by the Transformer decoder, first extracting the responses of the last num_labels classes from the channel dimension, and then taking the mean value in the spatial dimension to obtain the final response M of each label: ； The calculation steps of the smooth focal loss function are as follows: S331, performing smoothing processing on the target label: ， In formula (8), y represents the original target label, and smoothing is a smoothing coefficient, which is set to 0.1 by default. S332, the comprehensive loss is composed of the label smoothing weighted binary cross-entropy loss and the focal loss, and the calculation formula is: ， In formula (9), alpha is a balance coefficient, which is used to adjust the weight between the smoothing loss and the focal loss. S4, model application: using the FFA image set processed in step S2 to train and verify the D-LAM model, and finally realizing automatic identification of different types of lesions on the test set.

2. The method of claim 1, wherein the method is based on adaptive multi-label identification of diabetic retinopathy lesions. In step S1, image data acquisition is performed by two ophthalmologists using a Heidelberg retinal angiography device under a 55° field of view.

3. The method of claim 2, wherein the method is based on adaptive multi-label identification of diabetic retinopathy lesions. In step S1, the lesion type labeling is marked by three ophthalmologists without communication, and if the marking results of the images are consistent, the lesion category is determined. Otherwise, images with inconsistent marking need to be discussed until a consensus is reached.

4. The method of diabetic retinopathy lesion detection based on adaptive multi-label identification according to any one of claims 1 to 3, characterized in that, In step S3, in the final label prediction stage, the output results of the class adaptive mapping module and the label specific decoding module are fused with a weight ratio of 3:2.

Images