Diabetic retinopathy lesion detection method based on adaptive multi-label identification

Abstract

The invention belongs to the technical field of diabetic retinopathy focus detection, and particularly relates to a diabetic retinopathy focus detection method based on adaptive multi-label identification, which comprises the following steps: acquiring multi-temporal image data of an FFA image of a patient; carrying out single-piece combination splicing on each group comprising 10 time sequence FFA images by adopting a sixteen-grid input method; and building a D-LAM model composed of a category adaptive mapping module, a label specificity decoding module and a smooth focus loss function, carrying out model training and verification by using a processed FFA image data set, and finally realizing automatic identification of different types of lesions on a test set. According to the method, the recognition capability of micro lesions such as microhemangioma is remarkably improved, meanwhile, complex semantic association among multiple tags is effectively processed, a doctor is effectively assisted in distinguishing diabetic retinopathy, and the recognition efficiency and accuracy of the diabetic retinopathy are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of diabetic retinopathy lesion detection, and in particular to a diabetic retinopathy lesion detection method based on adaptive multi-label recognition. Background Art

[0002] Diabetic retinopathy (DR) is a common microvascular complication of diabetes. Its occurrence rate increases significantly with the extension of diabetes. The development of DR is accompanied by retinal microvascular lesions, including microaneurysms, hemorrhages, hard exudates, soft exudates and other lesions. The appearance and changes of these features mark the progression of the disease. Therefore, early detection and accurate grading are crucial to prevent visual impairment. Fundus Fluorescein Angiography (FFA) is widely used in the diagnosis of DR because it can clearly present the subtle lesions of retinal vessels and is regarded as the gold standard for diagnosing DR.

[0003] However, the interpretation of FFA images is highly dependent on the doctor's experience and subjective judgment, and due to the complex characteristics and uneven distribution of lesions, misdiagnosis and missed diagnosis are prone to occur. In addition, the DR diagnosis process is time-consuming and labor-intensive, and traditional manual grading methods are difficult to meet clinical needs. Therefore, automated and intelligent DR grading methods have gradually become a research hotspot. With the development of deep learning technology, image-based automated lesion recognition has made significant progress. Existing methods mostly focus on the classification and detection of color fundus photos, but there are relatively few studies on FFA sequence images, and they face two major challenges: first, it is difficult to identify tiny lesions (such as microaneurysms) in FFA sequence images; second, the complex associations between multi-label lesions are difficult to be effectively handled by existing models. These problems restrict the application effect of automated grading models, and it is urgent to improve the model architecture to improve the accuracy and robustness of lesion recognition. Summary of the invention

[0004] In order to overcome the technical defects of the existing lesion processing model that it is difficult to identify tiny lesions in FFA sequence images and difficult to process the complex associations between multi-label lesions, the present invention provides a diabetic retinopathy lesion detection method based on adaptive multi-label recognition.

[0005] The present invention provides a diabetic retinopathy lesion detection method based on adaptive multi-label recognition, comprising the following steps:

[0006] S1. Data collection: Fundus cameras are used to collect multi-phase image data of fluorescent fundus angiography (FFA) images of patients with diabetic retinopathy, and ophthalmologists mark the types of lesions in the images;

[0007] S2, data preprocessing: for each group of 10 time-series FFA images in step S1, a "sixteen-grid input" method is used to perform single-image combination stitching, wherein the first 10 positions of the 4×4 grid are filled in chronological order, and 6 images are randomly selected from the remaining images to fill in the remaining positions of the 4×4 grid, and the images are cropped and normalized;

[0008] S3. Model construction: Build an adaptive multi-label recognition model, namely the D-LAM model. The D-LAM model includes a category adaptive mapping module, a label-specific decoding module, and a smooth focal loss function. The category adaptive mapping module improves the recognition ability of small lesions through an adaptive response mechanism controlled by temperature parameters; a label-specific decoding module is constructed using a transformer decoder to process label associations; a smooth focal loss function is designed by combining label smoothing and focal loss to enhance the learning effect of difficult-to-classify samples;

[0009] The steps of controlling the adaptive response mechanism through the temperature parameter in the category adaptive mapping module are as follows:

[0010] S311. Calculate the normalized category response map score through the feature map generator, as shown in formula (1):

[0011]

[0012] In formula (1), head(F) represents the output generated by the 1×1 convolution operation, which is the L2 norm of the convolution layer weight and is used to normalize the output. The normalization process ensures the scale consistency of the category response map, which helps to improve the training effect of the model.

[0013] S312. Calculate the basic response base_logit, which is the mean of the category response map score in the spatial dimension, as shown in formula (2):

[0014]

[0015] In formula (2), H and W represent the height and width of the feature map respectively, and score ij is the value of the i-th row and j-th column in the category response graph score; this mean operation provides a global response reference for the model, which helps in the subsequent adaptive response generation;

[0016] S313, the adaptive response att_logit is controlled by the temperature parameter T. According to the different values ​​of the temperature parameter T, it is divided into two cases: maximum pooling response and weighted sum response:

[0017]

[0018] S314, perform a weighted combination of the base response base_logit and the adaptive response att_logit to generate a final output response graph M':

[0019]

[0020] In formula (4), λ is a hyperparameter used to control the adaptive response weight, and heads represents the number of attention heads;

[0021] Among them, the processing steps of the tag-specific decoding module are:

[0022] S321, extract high-dimensional features src and corresponding spatial position codes pos from the input image F through the backbone network:

[0023] src, pos=backbone(F) (5),

[0024] S322, based on this, generate a label embedding and a query vector to guide the label-specific feature extraction in the decoding process; then input the generated high-dimensional feature src, the spatial position code pos, the initial query vector tgt and the label query vector query_input into the Transformer decoder to extract the label-specific response hs related to the specific label from the image features:

[0025] hs=Decoder(tgt, src, pos_embed, query_input) (6),

[0026] S323. Group the label-specific responses hs output by the Transformer decoder. First, intercept the responses of the last num_labels categories from the channel dimension, and then take the average in the spatial dimension to obtain the final response M for each label:

[0027] M=mean(hs[:,-num_labels:,:],dim=-1) (7);

[0028] Among them, the calculation steps of the smooth focus loss function are:

[0029] S331, smoothing the target label:

[0030]

[0031] In formula (8), y represents the original target label, smoothing is the smoothing coefficient, and the default setting is 0.1;

[0032] S332, the comprehensive loss is composed of the label smoothing weighted binary cross entropy loss and the focal loss, and the calculation formula is:

[0033] Loss = α·smooth_loss + (1-α)·focal_loss (9), where α is a balance coefficient used to adjust the weight between smooth loss and focal loss;

[0034] S4. Model application: The FFA image set processed in step S2 is used to train and verify the D-LAM model, and finally the automatic recognition of different types of lesions is achieved on the test set.

[0035] Preferably, in step S1, image data acquisition is performed by two ophthalmologists using a Heidelberg retinal angiography device under a 55° field of view. Such an operation can exclude subjects with other fundus diseases such as ischemic optic neuropathy, age-related macular degeneration, and optic disc edema.

[0036] Preferably, in step S1, the lesion type labeling is performed by three ophthalmologists who label all images separately without communication. If the labeling results of the images are consistent, the lesion category is determined; otherwise, the images with inconsistent labels need to be discussed until a consistent labeling opinion is reached.

[0037] Preferably, in step S3, in the final label prediction stage, the output results of the category adaptive mapping module and the label specific decoding module are fused at a weight ratio of 3:2.

[0038] Compared with the prior art, the technical solution provided by the present invention has the following technical effects:

[0039] (I) The present invention proposes a "sixteen-square grid input" method, which orderly combines multiple images of each group of fundus fluorescein angiography (FFA) examinations; this method not only retains the temporal dynamic characteristics of fundus angiography, but also enhances the feature expression of key time points by random supplementation, effectively improving the comprehensiveness and accuracy of lesion identification; compared with traditional single image input, this method can make fuller use of the information obtained during the examination process, significantly improving the efficiency of identifying diabetic retinopathy lesions.

[0040] (ii) The class adaptive mapping module designed by the present invention can accurately capture the characteristics of lesions of different scales through the adaptive response mechanism controlled by temperature parameters and the normalized class response map. In particular, for microscopic lesions such as microvascular tumors, the module can adaptively adjust the sensitivity of feature extraction, significantly improving the ability to identify subtle lesions.

[0041] (III) The label-specific decoding module proposed in the present invention uses the Transformer architecture to achieve accurate modeling of complex label associations. Through the label embedding and query vector generation mechanism, the module can effectively handle complex scenarios where multiple lesions co-occur and accurately identify the associations between different types of lesions. This mechanism significantly improves the model's recognition accuracy for multiple lesions, giving the model a stronger ability to identify complex cases.

[0042] (IV) The present invention innovatively designs a smoothed focal loss function, which effectively solves the sample imbalance problem by combining the advantages of label smoothing and focal loss. This loss function not only improves the learning effect of the model on difficult-to-classify samples, but also enhances the generalization ability of the model.

[0043] (V) The end-to-end adaptive multi-label recognition model of the present invention constructs a complete lesion recognition system by organically combining various innovative modules. The system can simultaneously process temporal information, spatial features and label associations, greatly improving the overall performance of the model. In actual clinical applications, the present invention can effectively assist doctors in distinguishing diabetic retinopathy, improve the recognition efficiency and accuracy of diabetic retinopathy, and has important clinical application value.

[0044] (VI) The model of the present invention has good scalability and adaptability, and is not only suitable for the detection of diabetic retinopathy, but can also be applied to other types of medical image multi-label recognition tasks through appropriate adjustments. This versatility makes the present invention have broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 It is an overall flow chart of a diabetic retinopathy lesion detection method based on adaptive multi-label recognition described in an embodiment of the present invention;

[0048] Figure 2 It is a framework diagram of an adaptive multi-label recognition model in a diabetic retinopathy lesion detection method based on adaptive multi-label recognition described in an embodiment of the present invention;

[0049] Figure 3It is a structural diagram of a category adaptive mapping module in a diabetic retinopathy lesion detection method based on adaptive multi-label recognition described in an embodiment of the present invention;

[0050] Figure 4 It is a structural diagram of a label-specific decoding module in a diabetic retinopathy lesion detection method based on adaptive multi-label recognition described in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to more clearly understand the above-mentioned objectives, features and advantages of the present invention, the scheme of the present invention will be further described below. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0052] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present invention, rather than all of the embodiments.

[0053] The following is combined with Figures 1 to 4 The specific embodiments of the present invention are described in detail.

[0054] In one embodiment, Figure 1 As shown, a diabetic retinopathy lesion detection method based on adaptive multi-label recognition is disclosed, comprising the following steps:

[0055] S1. Data collection: Use a fundus camera to collect multi-phase image data of fluorescent fundus angiography images, i.e. FFA images, from patients with diabetic retinopathy, and ophthalmologists will annotate the lesion types in the images; in a specific embodiment, image data collection is performed by two ophthalmologists using a Heidelberg retinal angiography device under a 55° field of view. Such an operation can exclude subjects with other fundus diseases such as ischemic optic neuropathy, age-related macular degeneration, and optic disc edema; lesion type annotation is performed by three ophthalmologists who separately label all images without communication. If the labeling results of the images are consistent, the lesion category is determined; otherwise, images with inconsistent labels need to be discussed until a consistent labeling opinion is reached;

[0056] S2, data preprocessing: for each group of 10 time-series FFA images in step S1, a "sixteen-grid input" method is used to perform single-image combination stitching, wherein the first 10 positions of the 4×4 grid are filled in chronological order, and 6 images are randomly selected from the remaining images to fill in the remaining positions of the 4×4 grid, and the images are cropped and normalized;

[0057] S3. Model construction: Build an adaptive multi-label recognition model, namely the D-LAM model. The D-LAM model includes a category adaptive mapping module, a label-specific decoding module, and a smooth focal loss function. The category adaptive mapping module improves the recognition ability of small lesions through an adaptive response mechanism controlled by temperature parameters; a label-specific decoding module is constructed using a transformer decoder to process label associations; a smooth focal loss function is designed by combining label smoothing and focal loss to enhance the learning effect of difficult-to-classify samples;

[0058] The steps of controlling the adaptive response mechanism through the temperature parameter in the category adaptive mapping module are as follows:

[0059] S311. Calculate the normalized category response map score through the feature map generator, as shown in formula (1):

[0060]

[0061] In formula (1), head(F) represents the output generated by the 1×1 convolution operation, which is the L2 norm of the convolution layer weight and is used to normalize the output. The normalization process ensures the scale consistency of the category response map, which helps to improve the training effect of the model.

[0062] S312. Calculate the basic response base_logit, which is the mean of the category response map score in the spatial dimension, as shown in formula (2):

[0063]

[0064] In formula (2), H and W represent the height and width of the feature map respectively, and score ij is the value of the i-th row and j-th column in the category response graph score; this mean operation provides a global response reference for the model, which helps in the subsequent adaptive response generation;

[0065] S313, the adaptive response att_logit is controlled by the temperature parameter T. According to the different values ​​of the temperature parameter T, it is divided into two cases: maximum pooling response and weighted sum response:

[0066]

[0067] S314, perform a weighted combination of the base response base_logit and the adaptive response att_logit to generate a final output response graph M':

[0068]

[0069] In formula (4), λ is a hyperparameter used to control the adaptive response weight, and heads represents the number of attention heads;

[0070] Among them, the processing steps of the tag-specific decoding module are:

[0071] S321, extract high-dimensional features src and corresponding spatial position codes pos from the input image F through the backbone network:

[0072] src, pos=backbone(F) (5),

[0073] S322, based on this, generate a label embedding and a query vector to guide the label-specific feature extraction in the decoding process; then input the generated high-dimensional feature src, the spatial position code pos, the initial query vector tgt and the label query vector query_input into the Transformer decoder to extract the label-specific response hs related to the specific label from the image features:

[0074] hs=Decoder(tgt, src, pos_embed, query_input) (6),

[0075] S323. Group the label-specific responses hs output by the Transformer decoder. First, intercept the responses of the last num_labels categories from the channel dimension, and then take the average in the spatial dimension to obtain the final response M for each label:

[0076] M=mean(hs[:,-num_labels:,:],dim=-1) (7);

[0077] Among them, the calculation steps of the smooth focus loss function are:

[0078] S331, smoothing the target label:

[0079]

[0080] In formula (8), y represents the original target label, smoothing is the smoothing coefficient, and the default setting is 0.1;

[0081] S332, the comprehensive loss is composed of the label smoothing weighted binary cross entropy loss and the focal loss, and the calculation formula is:

[0082] Loss = α·smooth_loss + (1-α)·focal_loss (9), where α is a balance coefficient used to adjust the weight between smooth loss and focal loss;

[0083] In the final label prediction stage, the outputs of the category adaptive mapping module and the label specific decoding module are fused in a weight ratio of 3:2;

[0084] S4. Model application: The FFA image set processed in step S2 is used to train and verify the D-LAM model, and finally the automatic recognition of different types of lesions is achieved on the test set.

[0085] Table 1 is a performance comparison of the D-LAM model constructed by the present invention and 9 existing models on the collected data sets. The 9 existing models include 4 classic convolutional neural networks and 5 innovative models proposed in recent years. The results fully demonstrate the superiority of the method of the present invention in the task of identifying diabetic retinal lesions.

[0086] Table 1 Performance comparison between the method of the present invention and existing models

[0087]

[0088] As can be seen from the above, the present invention includes three key steps: data acquisition, data processing and model implementation. In data acquisition, a standardized 55° field of view Heidelberg retinal angiography device is used, and experienced ophthalmologists perform professional operations and mark lesions, while strictly excluding interference from other fundus diseases; in data processing, an innovative "sixteen-square grid input" method is proposed, and 10 time-series FFA images are filled into the first 10 positions of the 4×4 matrix in time sequence and the remaining positions are randomly filled, effectively integrating multi-phase image information; in model implementation, a D-LAM model with a category adaptive mapping module and a label-specific decoding module is built, and an adaptive response mechanism controlled by temperature parameters and a transformer decoder structure are used, and a smooth focus loss function is innovatively used to achieve accurate identification of different types of lesions, providing clinicians with a reliable tool for identifying diabetic retinopathy lesions and promoting the deep integration of medicine and artificial intelligence.

[0089] The above is only a specific implementation of the present invention, which enables those skilled in the art to understand or implement the present invention. Although detailed descriptions are given with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments, and they should all be covered by the protection scope of the claims.

Claims

1. A diabetic retinopathy lesion detection method based on adaptive multi-label recognition, characterized in that: The following steps are involved: S1. Data collection: Fundus cameras are used to collect multi-phase image data of fluorescent fundus angiography (FFA) images of patients with diabetic retinopathy, and ophthalmologists mark the types of lesions in the images; S2, data preprocessing: for each group of 10 time-series FFA images in step S1, a "sixteen-grid input" method is used to perform single-image combination stitching, wherein the first 10 positions of the 4×4 grid are filled in chronological order, and 6 images are randomly selected from the remaining images to fill in the remaining positions of the 4×4 grid, and the images are cropped and normalized; S3. Model construction: Build an adaptive multi-label recognition model, namely the D-LAM model. The D-LAM model includes a category adaptive mapping module, a label-specific decoding module, and a smooth focal loss function. The category adaptive mapping module improves the recognition ability of small lesions through an adaptive response mechanism controlled by temperature parameters; a label-specific decoding module is constructed using a transformer decoder to process label associations; a smooth focal loss function is designed by combining label smoothing and focal loss to enhance the learning effect of difficult-to-classify samples; The steps of controlling the adaptive response mechanism through temperature parameters in the category adaptive mapping module are as follows: S311. Calculate the normalized category response map score through the feature map generator, as shown in formula (1): In formula (1), head(F) represents the output generated by the 1×1 convolution operation, which is the L2 norm of the convolution layer weight and is used to normalize the output; S312. Calculate the basic response base_logit, which is the mean of the category response map score in the spatial dimension, as shown in formula (2): In formula (2), H and W represent the height and width of the feature map respectively, and score ij is the value of the i-th row and j-th column in the category response graph score; S313, the adaptive response att_logit is controlled by the temperature parameter T. According to the different values ​​of the temperature parameter T, it is divided into two cases: maximum pooling response and weighted sum response: S314, perform a weighted combination of the base response base_logit and the adaptive response att_logit to generate a final output response graph M': In formula (4), λ is a hyperparameter used to control the adaptive response weight, and heads represents the number of attention heads; Among them, the processing steps of the tag-specific decoding module are: S321, extract high-dimensional features src and corresponding spatial position codes pos from the input image F through the backbone network: src, pos=backbone(F) (5), S322, based on this, generate a label embedding and a query vector to guide the label-specific feature extraction in the decoding process; then input the generated high-dimensional feature src, the spatial position code pos, the initial query vector tgt and the label query vector query_input into the Transformer decoder to extract the label-specific response hs related to the specific label from the image features: hs=Decoder(tgt, src, pos_embed, query_input) (6), S323. Group the label-specific responses hs output by the Transformer decoder. First, intercept the responses of the last num_labels categories from the channel dimension, and then take the average in the spatial dimension to obtain the final response M for each label: M=mean(hs[:,-num_labels:,:],dim=-1) (7); Among them, the calculation steps of the smooth focus loss function are: S331, smoothing the target label: In formula (8), y represents the original target label, smoothing is the smoothing coefficient, and the default setting is 0.1; S332, the comprehensive loss is composed of the label smoothing weighted binary cross entropy loss and the focal loss, and the calculation formula is: Loss = α·smooth_loss + (1-α)·focal_loss (9), where α is a balance coefficient used to adjust the weight between smooth loss and focal loss; S4. Model application: The FFA image set processed in step S2 is used to train and verify the D-LAM model, and finally the automatic recognition of different types of lesions is achieved on the test set.

2. The method for detecting diabetic retinopathy lesions based on adaptive multi-label recognition according to claim 1, characterized in that: In step S1, image data acquisition was performed by two ophthalmologists using a Heidelberg retinal angiography device under a 55° field of view.

3. The method for detecting diabetic retinopathy lesions based on adaptive multi-label recognition according to claim 2, characterized in that: In step S1, the lesion type labeling is performed by three ophthalmologists who label all images separately without communication. If the labeling results of the images are consistent, the lesion category is determined; Otherwise, images with inconsistent labels need to be discussed until a consensus on the labeling is reached.

4. The method for detecting diabetic retinopathy lesions based on adaptive multi-label recognition according to any one of claims 1 to 3, characterized in that: In step S3, in the final label prediction stage, the output results of the category adaptive mapping module and the label specific decoding module are fused at a weight ratio of 3:2.

Images