A method and system for diabetic retinopathy assessment based on lesion-aware cross-task interaction

Abstract

The application belongs to the technical fields of computer vision and medical image processing, and specifically relates to a diabetic retinopathy evaluation method and system based on lesion perception cross-task interaction. The method comprises the following steps: acquiring a fundus fluorescein angiography image sequence of a patient; inputting the image sequence into a feature extraction backbone network to generate shared features; extracting hierarchical features and lesion features from the shared features; aggregating multi-frame information through an eye-level prediction module to output DR severity grading results and lesion multi-label identification results; performing multi-task joint training on the data set obtained at S1, simultaneously optimizing the DR severity grading task and the lesion label identification task by using a weighted joint loss function to obtain an optimal evaluation model; and inputting an image sequence to be evaluated into the evaluation model to output corresponding DR severity grading results and lesion multi-label identification results. The application can simultaneously realize DR severity grading and lesion multi-label identification.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision and medical image processing technology, specifically a method and system for assessing diabetic retinopathy based on cross-task interaction of lesion perception. Background Technology

[0002] Diabetic retinopathy (DR) is a series of complications caused by damage to the microvessels of the retina due to diabetes, and it is a significant cause of preventable vision impairment and blindness in the working-age population. Early identification and accurate severity grading are crucial for timely intervention and reducing the risk of vision loss. In clinical imaging assessment, fundus fluorescein angiography (FFA), through continuous acquisition of fundus images, can dynamically display retinal vascular perfusion and fluorescence leakage. It has unique advantages in identifying abnormalities such as microaneurysms, non-perfusion areas, and neovascularization, and is an important basis for assessing the severity of DR. However, traditional FFA image interpretation requires physicians to comprehensively analyze multi-phase images and various lesion manifestations. The interpretation process is complex, time-consuming, and highly dependent on subjective experience, making it difficult to meet the needs of large-scale application and standardized assessment.

[0003] In recent years, deep learning technology has been introduced into the assisted diagnosis of diabetic retinopathy (DR). Existing automated analysis methods based on freeform imaging (FFA) images often focus on a single diagnostic objective, typically separating DR severity grading from local lesion identification. This fails to fully explore and model the intrinsic relationship between local lesion evidence and overall severity assessment. Furthermore, FFA images exhibit significant multi-temporal characteristics, with complementary pathological information presented at different stages; relying solely on a single frame can easily lead to information omissions. Therefore, how to fully utilize multi-frame FFA image information and establish a link between lesion identification information and DR severity grading is a crucial problem that urgently needs to be solved to improve the accuracy and stability of automated assessment of diabetic retinopathy. Summary of the Invention

[0004] To address the aforementioned problems, this invention provides a method and system for assessing diabetic retinopathy based on lesion perception and cross-task interaction.

[0005] This invention adopts the following technical solution: a method for assessing diabetic retinopathy based on lesion perception and cross-task interaction, comprising: S1: Obtain the patient's fundus fluorescein angiography image sequence, and perform DR severity grading and lesion labeling to obtain the dataset; S2: Input the image sequence into the feature extraction backbone network to extract features from different stages, and generate shared features through a multi-level feature fusion module; S3: The task decoupling interaction module is used to extract hierarchical features and lesion features from the shared features. The cross-task attention mechanism is used to realize bidirectional information interaction between the two types of features, and the updated hierarchical features and lesion features are obtained. S4: The grading features and lesion features are aggregated through the eye-level prediction module to aggregate information from multiple frames and calibrate the eye-level grading representation using the lesion prediction information, so as to output the DR severity grading results and lesion multi-label recognition results. S5: Perform multi-task joint training on the dataset obtained in S1, and use a weighted joint loss function to simultaneously optimize the DR severity grading task and the lesion label recognition task to obtain the best evaluation model. S6: Input the image sequence to be evaluated into the trained best evaluation model, and output the corresponding DR severity grading results and lesion multi-label recognition results.

[0006] In some embodiments, in S1, the image sequence consists of multiple frames of images acquired by the same eye at different angiography phases; The image sequences were subjected to DR severity classification and lesion multi-label annotation. The DR severity classification annotation was used to supervise the classification task and was divided into 4 levels according to severity. The lesion multi-label annotation was used to supervise the lesion identification task and recorded the existence of DR-related lesions.

[0007] In some embodiments, the feature extraction backbone network uses ConvNeXt to extract multi-stage visual features for each frame of image sequence.

[0008] In some embodiments, the multi-level feature fusion module generates shared features including: Channel alignment is performed on shallow, mid-level, and high-level features using 1×1 convolution; Spatial scale alignment and fusion of high-level and mid-level features are performed to obtain mid-to-high-level semantic features. Detail gating maps are generated based on channel-aligned shallow features and upsampled mid-to-high-level semantic features to filter local lesion details in shallow features. The filtered features are then fused with mid-to-high-level semantic features to obtain shared features.

[0009] In some embodiments, the task decoupling interaction module in step S3 includes: Task-specific dual branches, which include: The hierarchical specific branch normalizes the shared features to obtain hierarchical input features, which are then input into the semantic gating branch and the multi-level context branch respectively. The semantic gating branch generates a semantic gating response through a 1×1 convolutional layer and a SiLU activation function; The multi-level context branch first performs deep convolution on the hierarchical input features to obtain basic context features. Then, the basic context features are divided into three groups along the channel dimension. The first group of features is retained, and the second and third groups of features are processed by deep convolution. The three groups of features are concatenated and fused to obtain the context response. The semantic gated response and the context response are multiplied element-wise, fused by 1×1 convolution, and then back-injected into the shared features according to the learnable scaling factor to obtain hierarchical enhancement features. The hierarchical enhancement features are subjected to global average pooling to obtain channel description vectors. The local dependencies between adjacent channels are modeled by one-dimensional convolution and channel weights are generated by the Sigmoid activation function. The channel weights are weighted element-wise with the hierarchical enhancement features according to the channel to obtain hierarchical branch features. For lesion-specific branches, shared features are extracted using deep convolution to obtain local texture and edge features within each channel. These features are then processed sequentially through batch normalization and ReLU activation to obtain local enhancement features. These local enhancement features are then processed using deep convolution with different receptive fields, followed by 1×1 convolution for channel mapping to obtain lesion features at two scales. These features are then concatenated along the channels to obtain aggregated lesion features. The aggregated lesion features are then subjected to average pooling and max pooling along the channel dimensions. The concatenated pooling results are then processed through 3×3 convolution and a sigmoid activation function to generate spatial selection weights. These spatial selection weights are used to weight and fuse lesion features at different scales. Finally, lesion enhancement weights are generated through 1×1 convolution and a sigmoid activation function. These enhancement weights are then multiplied element-wise with the local enhancement features to obtain the lesion branch features. A cross-task attention mechanism is used, in which shared features are used as the feature alignment benchmark between the DR severity grading task and the lesion label recognition task. The shared features are encoded with the grading branch features and the lesion branch features respectively to obtain the grading task features and the lesion task features. The interaction information between the two types of tasks is calculated based on the grading task features and the lesion task features, and the interaction information is fused into the corresponding task branches to obtain the updated grading features and lesion features.

[0010] In some embodiments, the interaction information includes lesion-related interaction information and hierarchical semantic interaction information, and the calculation process is as follows: In the lesion-to-grading direction, the grading task features that receive interactive information are used as target branch features, and the lesion task features are used as information source features to generate lesion-related interactive information to supplement the grading judgment. In the grading-to-lesion direction, the lesion task features that receive interactive information are used as target branch features, and the grading task features are used as information source features to generate interactive information for supplementing lesion identification with grading semantics. Three independent Convolution performs channel mapping on target branch features and information source features, where query features are generated from target branch features, and key and value features are generated from information source features. The spatial positions of the query features, key features, and value features are then arranged sequentially into a matrix to obtain the query matrix. Key matrix Sum matrix : in, Represents the target branch features. Indicates the characteristics of the information source; , , These represent query mapping, key mapping, and value mapping, respectively. The interaction information is as follows: in, Indicates by Convolution achieves output projection. Indicates spatial resolution. Indicates interactive information.

[0011] In some embodiments, the interaction information is fused into the corresponding task branches to obtain updated hierarchical features and lesion features, including: The lesion-related interactive information and the hierarchical semantic interactive information are respectively divided into preset proportions. Add the corresponding task branch to obtain the updated hierarchical features. and lesion characteristics ; in, This represents the interactive information from the lesion location to the grading direction. To facilitate the exchange of information from grading to the direction of lesions, This is the cross-task interaction strength coefficient, used to control the fusion ratio of cross-task information.

[0012] In some embodiments, S4 includes: Global average pooling is performed on the graded features and lesion features obtained from S3. The T-frame feature vectors of the same eye are aggregated to obtain the corresponding ocular-level grade representation and ocular-level lesion representation; The ocular lesion representation is input into the lesion prediction head, which is composed of a Dropout layer and a fully connected layer connected in sequence, and is used to output the predicted values ​​of various types of lesions. The predicted values ​​of various types of lesions are processed by Sigmoid to obtain the corresponding lesion occurrence probability, and the lesion multi-label recognition result is obtained according to the preset threshold. The probability of lesion occurrence is input into the lesion-guided grading calibration unit, which includes a fully connected layer and a Tanh activation function. This unit maps the probability of lesion occurrence to lesion prompt information and adds the lesion prompt information to the eye-level grading representation according to a preset calibration coefficient to obtain the calibrated grading representation. The calibrated classification representation is input into the classification prediction head, which is composed of a Dropout layer and a fully connected layer connected in sequence, and is used to output the predicted value of each DR severity level. The predicted value is processed by Softmax to obtain the predicted probability of each level, and the level with the highest predicted probability is determined as the DR severity classification result.

[0013] In some embodiments, in S5: From the dataset obtained from S1, multiple frames of images, DR severity level and lesion label corresponding to the same eye are read in batches, and the multiple frames of images are input into the diabetic retinopathy assessment model to obtain the DR severity level prediction value and the lesion multi-label prediction value. The total training loss is calculated using a weighted joint loss function. The loss function for multi-task joint training is a weighted sum of the losses from the DR severity grading task and the lesion label identification task. in, The loss for the DR severity classification task is represented by the cross-entropy loss function with category weights, calculated based on the classification prediction value and the DR severity level label. The loss for lesion label recognition task is represented by a binary cross-entropy loss function with positive sample weights, calculated based on the predicted lesion value and multiple lesion labels. and These represent the loss weights for the DR severity grading task and the lesion identification task, respectively, and are used to adjust the contributions of the two tasks in joint training. In each training batch, the gradients of the model parameters are first cleared to zero, then backpropagation is performed based on the joint loss, and the learnable parameters of the model are updated using the AdamW optimizer to obtain the best evaluation model.

[0014] An assessment system for implementing the aforementioned lesion-sensing cross-task interaction-based diabetic retinopathy assessment method includes: The image sequence construction module is used to acquire and process multiple frames of images from different angiography phases of the same eye to form an image sequence; The lesion perception cross-task interaction network module is used for S2-S4 processing of image sequences; The model training module performs multi-task joint training using training data and corresponding labels to obtain the evaluation model. The results output module is used to output the severity grading results of DR and the multi-label identification results of lesions.

[0015] Compared with the prior art, the present invention has the following beneficial effects: This invention uses multiple frames of FFA images as input, leveraging information on vascular perfusion, fluorescence leakage, and lesion visualization at different angiographic phases to avoid the problem of insufficient utilization of dynamic pathological information in single-frame images. In model construction, a multi-level feature fusion module integrates visual features from different levels to obtain shared features that consider both hierarchical semantics and lesion details. A task decoupling interaction module enhances semantic information related to DR hierarchies and local responses related to lesion identification, respectively, and a cross-task attention mechanism promotes information interaction between the two types of task features, enabling lesion cues to assist in severity hierarchical judgment. Simultaneously, a lesion-guided calibration strategy is introduced in the eye-level multi-task prediction stage, further integrating lesion prediction information into the hierarchical representation, allowing the final hierarchical result to more fully reference the lesion identification result. This invention can simultaneously achieve DR severity hierarchies and multi-label lesion identification, improving the accuracy and stability of diabetic retinopathy assessment in multi-frame FFA images and providing reliable imaging support for clinical decision-making by physicians. Attached Figure Description

[0016] Figure 1 This is the overall flowchart in the embodiments of the present invention; Figure 2 This is a schematic diagram of the dataset construction process in an embodiment of the present invention; Figure 3 This is a model structure diagram in an embodiment of the present invention; Figure 4 This is a structural diagram of an embodiment of the present invention; Figure 5 Figure I shows the ROC curves for DR severity grading in this embodiment of the invention. Figure 6 This is ROC curve diagram II for multi-label lesion identification in this embodiment of the invention; Figure 7 This is a detailed block diagram of an embodiment of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] like Figure 1 As shown, a method for assessing diabetic retinopathy based on lesion perception cross-task interaction includes: S1: Obtain the patient's fundus fluorescein angiography image sequence, and perform DR severity grading and lesion labeling to obtain the dataset; S2: Input the image sequence into the feature extraction backbone network to extract features from different stages, and generate shared features through a multi-level feature fusion module; S3: The task decoupling interaction module is used to extract hierarchical features and lesion features from shared features and perform cross-task interaction; S4: The grading features and lesion features are aggregated through the eye-level prediction module to aggregate information from multiple frames and calibrate the eye-level grading representation using the lesion prediction information, so as to output the DR severity grading results and lesion multi-label recognition results. S5: Perform multi-task joint training on the dataset obtained in S1, and use a weighted joint loss function to simultaneously optimize the DR severity grading task and the lesion label recognition task to obtain the best evaluation model. S6: Input the image sequence to be evaluated into the trained best evaluation model, and output the corresponding DR severity grading results and lesion multi-label recognition results.

[0019] In S5, the "DR severity grading task" and "lesion label recognition task" represent two learning objectives during the model training phase. These objectives are used to update the learnable parameters of the evaluation model through corresponding task losses, and do not refer to the model's output results. In S6, the "DR severity grading result" and "lesion multi-label recognition result" represent two output results obtained by the best evaluation model after training by inferring the image sequence to be evaluated.

[0020] In a specific embodiment, in step S1, the patient's fundus fluorescein angiography (FFA) image sequence is acquired, and lesion grading and lesion labeling are performed. The sequence is then divided into training, validation, and test sets proportionally. Specifically, the FFA image sequence consists of multiple frames of images acquired from the same eye at different angiography phases, used to preserve temporal information such as changes in vascular perfusion, fluorescence leakage, and lesion enhancement during angiography. The image sequence is then graded for DR severity and labeled with multiple lesions. The DR severity grading is used to supervise the grading task, divided into four levels based on severity. The lesion multi-label labeling is used to supervise the lesion identification task, recording the presence of six types of DR-related lesions. The dataset is divided by patient.

[0021] Fundus fluorescein angiography (FFA) image sequences were acquired from patients, and DR severity was graded and lesions were labeled with multiple tags. Training, validation, and test sets were proportionally divided. Ten FFA images from each eye were selected as one eye-level sequence sample, covering different stages of the angiography examination to preserve multi-temporal information during the imaging process. All images were acquired using a Heidelberg Retina Angiograph (Heidelberg Engineering, Germany) device with a 55° field of view. During data acquisition, low-quality images with severe overexposure, refractive media opacity, or unclear retinal structures and fluorescence features were excluded. All included images were anonymized, removing information that could identify the patient. In a specific embodiment, data from diabetic patients who underwent FFA examinations at Shanxi Provincial Eye Hospital between June 2022 and June 2025 were collected, including 2658 eyes from 1503 patients, totaling 26580 images. Each eye was labeled with a DR severity level, and the presence of six types of DR-related lesions was recorded. All annotations were completed by three ophthalmologists. Initially, two doctors independently interpreted the annotations; in case of discrepancies, a senior expert arbitrated the interpretations. The final annotation results were then reviewed and approved by the senior expert. To prevent data leakage, the dataset was divided into training, validation, and test sets based on patient units, ensuring that the same patient's two eyes were not assigned to different subsets. The partitioning ratio was 6:2:2. The entire process is as follows: Figure 2 As shown.

[0022] In a specific embodiment, step S2 involves inputting the image sequence into the feature extraction backbone network to extract features from different stages, and generating shared features through a multi-level feature fusion module. Specifically, this includes: The feature extraction backbone network uses ConvNeXt to extract multi-stage visual features for each frame of image. A multi-level feature fusion module extracts the multi-stage features from the backbone network. shallow features Retains a lot of local details, mid-to-high-level features and It contains stronger semantic expression. First, 1×1 convolution is used to align the channels of features at different stages. Then, high-level features and mid-level features are spatially scale aligned and fused to obtain mid-to-high-level semantic features. This step is shown in the following formula: in, This represents a 1×1 convolution operation. Indicates batch normalization, Represents the ReLU activation function. Indicates an upsampling operation. This indicates concatenation along the channel dimension. This represents the convolution fusion function.

[0023] Then, based on the shallow features after channel alignment and upsampled mid-to-high-level semantic features Generate detail gate mapping To filter out local lesion details in superficial features, and to compare the filtered features with... Integration yields shared features The process is shown in the following formula: in, Represents the gated prediction function. It is the Sigmoid activation function. This indicates element-wise multiplication. Through the above processing, the multi-level feature fusion module can provide a unified shared feature foundation for subsequent task branches.

[0024] In a specific embodiment, the task decoupling interaction module in step S3 includes: Task-specific dual branches, which include: Hierarchical specific branches are used to enhance the contextual semantic information in shared features related to DR severity judgment, such as... Figure 4 As shown. Shared features First, the hierarchical input features are obtained through normalization. Then Input semantic gating branches and multi-level context branches respectively. The semantic gating branches include... Convolutional layers and the SiLU activation function are used to generate semantically gated responses. As shown in the following formula: Furthermore, the multi-level context branch is used to extract hierarchical relevant contextual information under different receptive fields. Unlike the method of extracting features using only a single convolutional kernel, this branch first processes the hierarchical input features. Perform depthwise convolution to obtain basic context features. and will Divided into three groups of features along the channel dimension , , The first set of features is retained, the second set of features is processed using a 5×5 depthwise convolution with a dilation rate of 2, and the third set of features is processed using a 7×7 depthwise convolution. The three sets of processed features are then concatenated and fused using convolution to obtain the contextual response. The expression is: in, This represents a depthwise convolution with a kernel size of 5×5. Indicates the expansion rate. This indicates that the data is stitched along the channel dimension. In some embodiments, the channel number ratio of the three channel groups is 1:2:1. Through the above processing, hierarchical contextual information of local, medium-range, and large-range scopes can be fused.

[0025] Subsequently, the semantically gated response With context response Element-wise multiplication is performed to selectively modulate multi-level contextual information using semantically gated responses, followed by... Convolution is used to fuse features and scale them according to a learnable scaling factor. Inject to shared features Hierarchical enhancement features are obtained. : in, This indicates element-wise multiplication. This is a learnable scaling factor used to control the fusion strength of hierarchical semantic enhancement information.

[0026] Furthermore, regarding hierarchical enhancement features Channel reweighting is performed to highlight characteristic channels that contribute more to the severity classification of DR. Specifically, first, for... Global average pooling is performed to obtain channel description vectors; then, one-dimensional convolutions acting on the channel dimension are used to model the local dependencies between adjacent channels, and channel weights are generated using the sigmoid activation function. The expression is: in, This indicates a global average pooling operation. This represents a one-dimensional convolution operation. This represents the Sigmoid activation function. Finally, the channel weights... With hierarchical enhancement features Weighting by channel yields hierarchical branching features. For example, in the formula: Therefore, the hierarchical specific branch introduces multi-receptive field context modeling and channel adaptive reweighting on the basis of shared features, so that the resulting hierarchical branch features Greater attention is paid to semantic regions and key channels related to DR severity judgment, which are used as hierarchical side input features for subsequent cross-task interaction modules.

[0027] Lesion-specific branches are used to enhance local texture and spatial location information related to the lesion region in shared features, including local feature enhancement and multi-scale lesion selection processing, such as... Figure 4 As shown. Given shared features Firstly, adopt Depth convolution extracts local texture and edge features within each channel, which are then processed sequentially by batch normalization and ReLU activation to obtain local enhancement features of the lesion. The expression is as follows: Furthermore, regarding the local enhancement characteristics of the lesions Processing is performed using depthwise convolutions with different receptive fields, and through... The convolution is used for channel mapping to obtain lesion features at different scales, as shown in the formula: in, Used to represent local lesion information within a small area. This is used to represent lesion context information over a large area. Subsequently, the lesion features from the two scales are concatenated along the channel dimension to obtain the aggregated lesion features. This is used to collect lesion responses from different receptive fields. (The sentence is incomplete and requires further context.) Perform average pooling and max pooling along the channel dimension; then concatenate the two pooling results. Convolution and the Sigmoid activation function generate spatial selection weights for the corresponding scale. and : in, and These represent average pooling and max pooling along the channel dimension, respectively. Finally, the spatial selection weights are used to weight and fuse lesion features at different scales. Convolution and the sigmoid activation function generate lesion enhancement weights and local enhancement features of the lesions. Element-by-element multiplication yields the branching characteristics of the lesions. The expression is as follows: Through the above processing, the lesion-specific branch can fuse lesion information from different receptive fields and adaptively adjust the contribution of features at different scales based on the response at each spatial location, thus obtaining lesion branch features. Greater attention is paid to the local morphology and spatial location of the lesion area, which is then used as the lesion-side input feature for subsequent cross-task interaction modules.

[0028] The process of cross-task attention mechanism is as follows: using shared features As a reference for task alignment, shared features are compared with hierarchical branch features respectively. and lesion branching characteristics Encoding is performed to obtain hierarchical task features and lesion task features. Subsequently, interaction information between the two types of tasks is calculated based on the hierarchical and lesion task features, and this interaction information is fused into the corresponding task branches to obtain updated hierarchical and lesion features. This process avoids the feature mismatch problem caused by directly using the interaction of features from two task branches and promotes information complementarity between local lesion clues and overall hierarchical semantics.

[0029] Specifically, first, regarding shared features conduct Convolution, batch normalization, and SiLU activation function processing are used to obtain shared mapping features. Then share the mapping features With hierarchical branching features and lesion branching characteristics By concatenating and encoding along the channel dimension, hierarchical task features are obtained. and lesion task characteristics : in, and These represent hierarchical task coding operations and lesion task coding operations, respectively, both including... Convolution, batch normalization and SiLU activation function Convolution, batch normalization, and SiLU activation functions, such as Figure 4 As shown.

[0030] The interaction information between the two types of tasks is calculated based on the hierarchical task features and the lesion task features, and the interaction information is then fused into the corresponding task branches to obtain the updated hierarchical features and lesion features.

[0031] Interaction information includes lesion-related interaction information and hierarchical semantic interaction information. For any interaction direction, the task features receiving the interaction information are used as target branch features, and the task features providing the interaction information are used as information source features. Three independent [methods] are employed respectively. Convolution performs channel mapping on target branch features and information source features, where query features are generated from target branch features, and key and value features are generated from information source features. Subsequently, the spatial positions of the query features, key features, and value features are arranged sequentially into a matrix to obtain the query matrix. Key matrix Sum matrix : in, Represents the target branch features. Indicates the characteristics of the information source; , , These represent query mapping, key mapping, and value mapping, respectively, and each uses an independent method. Convolution performs channel mapping and arranges the spatial positions in the convolution result into a matrix. (Query matrix) Generated from target branch features, used to represent the information requirements of the target task; key matrix Generated from information source features, used to represent matching information for the information source task; value matrix Generated from information source features, it provides the feature content to be transmitted to the target task. Subsequently, cross-task interaction information is generated. The calculation process is as follows: in, The spatial resolution of the input features is represented by the total number of spatial locations contained in the feature map. Indicates by The output projection implemented by convolution is used to restore the interactive features to the same number of channels as the target branch features.

[0032] Specifically, the interactive information is fused into the corresponding task branches to obtain updated hierarchical features and lesion features, including: The lesion-related interactive information and the hierarchical semantic interactive information are respectively divided into preset proportions. Add the corresponding task branch to obtain the updated hierarchical features. and lesion characteristics ; in, This represents the interactive information from the lesion location to the grading direction. To facilitate the exchange of information from grading to the direction of lesions, This is the cross-task interaction strength coefficient, used to control the fusion ratio of cross-task information.

[0033] In a specific embodiment, S4 includes: S41: Perform global average pooling on the hierarchical features and lesion features obtained from S3; S42: Aggregate the T-frame feature vectors of the same eye to obtain the corresponding ocular-level grade representation and ocular-level lesion representation; S43: Input the ocular lesion representation into the lesion prediction head. The lesion prediction head is composed of a Dropout layer and a fully connected layer connected in sequence, used to output the predicted values ​​of various types of lesions; the predicted values ​​of various types of lesions are processed by Sigmoid to obtain the corresponding lesion occurrence probability, and the lesion multi-label recognition result is obtained according to a preset threshold.

[0034] S44: Input the lesion occurrence probability into the lesion-guided grading calibration unit. The lesion-guided grading calibration unit includes a fully connected layer and a Tanh activation function, which is used to map the lesion occurrence probability into lesion prompt information, and add the lesion prompt information into the eye-level grading representation according to a preset calibration coefficient to obtain the calibrated grading representation.

[0035] S45: Input the calibrated classification representation into the classification prediction head. The classification prediction head is composed of a Dropout layer and a fully connected layer connected in sequence, used to output the predicted value of each DR severity level; perform Softmax processing on the predicted value to obtain the predicted probability of each level, and determine the level with the highest predicted probability as the DR severity classification result.

[0036] Eye-level prediction module, such as Figure 3 The latter half of the diagram shows that the process mainly involves first performing GAP (Global Average Pooling), then aggregating the T-frame feature vectors, and finally creating the lesion prediction head, the lesion-guided hierarchical calibration unit, and the hierarchical prediction head.

[0037] In a specific embodiment, in step S5: Multiple frames of images, DR severity level labels, and lesion multi-labels corresponding to the same eye are read in batches from the training set obtained from S1. The multiple frames of images are then input into the diabetic retinopathy assessment model to obtain the DR severity level prediction value and the lesion multi-label prediction value.

[0038] The loss function for multi-task joint training is a weighted sum of the losses from the DR severity grading task and the lesion label recognition task: in, The loss for the DR severity classification task is represented by the cross-entropy loss function with category weights, calculated based on the classification prediction value and the DR severity level label. The loss for lesion label recognition task is represented by a binary cross-entropy loss function with positive sample weights, calculated based on the predicted lesion value and multiple lesion labels. and These represent the loss weights for the DR severity grading task and the lesion identification task, respectively, and are used to adjust the contributions of the two tasks in joint training.

[0039] In each training batch, the gradients of the model parameters are first cleared to zero, then backpropagation is performed based on the joint loss, and the learnable parameters of the model are updated using the AdamW optimizer to obtain the best evaluation model.

[0040] In a specific embodiment, in step S6: the FFA image sequence to be evaluated is input into the optimal model, and the corresponding DR severity grading result and lesion multi-label recognition result are output. The model first extracts features from the input sequence and obtains prediction results through a multi-level feature fusion module, a task decoupling interaction module, and an eye-level prediction module. Among them, the DR grading branch outputs the predicted probability of each severity level, and the category with the highest probability is taken as the final DR severity grading result; the lesion recognition branch outputs the occurrence probability of various types of lesions, and determines whether the corresponding lesion exists according to a preset threshold, thereby obtaining the lesion multi-label recognition result.

[0041] The best evaluation model after training was validated using a test set, and receiver operating characteristic curves were plotted for the DR severity grading task and the lesion label identification task, respectively. Figure 5 The results showed that the model had good ability to distinguish different DR grades and also had good ability to identify multiple types of DR-related lesions.

[0042] The method described in this invention can simultaneously complete DR severity grading and lesion multi-label identification in multi-frame FFA image sequences, providing effective imaging auxiliary information for the automatic assessment of diabetic retinopathy.

[0043] An evaluation system for implementing a lesion-perception cross-task interaction-based assessment method for diabetic retinopathy, comprising: an image sequence construction module for acquiring and processing multiple frames of images from different angiography phases of the same eye to form an image sequence; a lesion-perception cross-task interaction network module for performing S2-S4 processing on the image sequence; a model training module for performing multi-task joint training using training data and corresponding labels to obtain an evaluation model; and a result output module for outputting DR severity grading results and lesion multi-label recognition results.

[0044] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for assessing diabetic retinopathy based on lesion perception cross-task interaction, characterized in that, include: S1: Obtain the patient's fundus fluorescein angiography image sequence, and perform DR severity grading and lesion labeling to obtain the dataset; S2: Input the image sequence into the feature extraction backbone network to extract features from different stages, and generate shared features through a multi-level feature fusion module; S3: The task decoupling interaction module is used to extract hierarchical features and lesion features from the shared features. The cross-task attention mechanism is used to realize bidirectional information interaction between the two types of features, and the updated hierarchical features and lesion features are obtained. S4: The grading features and lesion features are aggregated through the eye-level prediction module to aggregate information from multiple frames and calibrate the eye-level grading representation using the lesion prediction information, so as to output the DR severity grading results and lesion multi-label recognition results. S5: Perform multi-task joint training on the dataset obtained in S1, and use a weighted joint loss function to simultaneously optimize the DR severity grading task and the lesion label recognition task to obtain the best evaluation model. S6: Input the image sequence to be evaluated into the trained best evaluation model, and output the corresponding DR severity grading results and lesion multi-label recognition results.

2. The method for assessing diabetic retinopathy based on lesion perception and cross-task interaction according to claim 1, characterized in that, In S1, the image sequence consists of multiple frames of images acquired by the same eye at different angiography phases; The image sequences were subjected to DR severity classification and lesion multi-label annotation. The DR severity classification annotation was used to supervise the classification task and was divided into 4 levels according to severity. The lesion multi-label annotation was used to supervise the lesion identification task and recorded the existence of DR-related lesions.

3. The method for assessing diabetic retinopathy based on lesion perception and cross-task interaction according to claim 1, characterized in that, The feature extraction backbone network uses ConvNeXt to extract multi-stage visual features for each frame of image.

4. The method for assessing diabetic retinopathy based on lesion perception cross-task interaction according to claim 3, characterized in that, The multi-level feature fusion module generates shared features including: Channel alignment is performed on shallow, mid-level, and high-level features using 1×1 convolution; Spatial scale alignment and fusion of high-level and mid-level features are performed to obtain mid-to-high-level semantic features. Detail gating maps are generated based on channel-aligned shallow features and upsampled mid-to-high-level semantic features to filter local lesion details in shallow features. The filtered features are then fused with mid-to-high-level semantic features to obtain shared features.

5. The method for assessing diabetic retinopathy based on lesion perception cross-task interaction according to claim 1, characterized in that, In S3, the task decoupling interaction module includes: Task-specific dual branches, which include: The hierarchical specific branch normalizes the shared features to obtain hierarchical input features, which are then input into the semantic gating branch and the multi-level context branch respectively. The semantic gating branch generates a semantic gating response through a 1×1 convolutional layer and a SiLU activation function; The multi-level context branch first performs deep convolution on the hierarchical input features to obtain basic context features. Then, the basic context features are divided into three groups along the channel dimension. The first group of features is retained, and the second and third groups of features are processed by deep convolution. The three groups of features are concatenated and fused to obtain the context response. The semantic gated response and the context response are multiplied element-wise, fused by 1×1 convolution, and then back-injected into the shared features according to the learnable scaling factor to obtain hierarchical enhancement features. The hierarchical enhancement features are subjected to global average pooling to obtain channel description vectors. The local dependencies between adjacent channels are modeled by one-dimensional convolution and channel weights are generated by the Sigmoid activation function. The channel weights are weighted element-wise with the hierarchical enhancement features according to the channel to obtain hierarchical branch features. For lesion-specific branches, shared features are extracted using deep convolution to obtain local texture and edge features within each channel. These features are then processed sequentially through batch normalization and ReLU activation to obtain local enhancement features. These local enhancement features are then processed using deep convolution with different receptive fields, followed by 1×1 convolution for channel mapping to obtain lesion features at two scales. These features are then concatenated along the channels to obtain aggregated lesion features. The aggregated lesion features are then subjected to average pooling and max pooling along the channel dimensions. The concatenated pooling results are then processed through 3×3 convolution and a sigmoid activation function to generate spatial selection weights. These spatial selection weights are used to weight and fuse lesion features at different scales. Finally, lesion enhancement weights are generated through 1×1 convolution and a sigmoid activation function. These enhancement weights are then multiplied element-wise with the local enhancement features to obtain the lesion branch features. A cross-task attention mechanism is used, in which shared features are used as feature alignment references for the DR severity grading task and the lesion label recognition task. The shared features are encoded with the grading branch features and the lesion branch features respectively to obtain grading task features and lesion task features. The interaction information between the two types of tasks is calculated based on the grading task features and lesion task features, and the interaction information is fused into the corresponding task branches to obtain updated grading features and lesion features.

6. The method for assessing diabetic retinopathy based on lesion perception cross-task interaction according to claim 5, characterized in that, Interactive information includes lesion-related interactive information and hierarchical semantic interactive information. The calculation process is as follows: In the lesion-to-grading direction, the grading task features that receive interactive information are used as target branch features, and the lesion task features are used as information source features to generate lesion-related interactive information to supplement the grading judgment. In the grading-to-lesion direction, the lesion task features that receive interactive information are used as target branch features, and the grading task features are used as information source features to generate interactive information for supplementing lesion identification with grading semantics. Three independent Convolution performs channel mapping on target branch features and information source features, where query features are generated from target branch features, and key and value features are generated from information source features. The spatial positions of the query features, key features, and value features are then arranged sequentially into a matrix to obtain the query matrix. Key matrix Sum matrix : in, Represents the target branch features. Indicates the characteristics of the information source; , , These represent query mapping, key mapping, and value mapping, respectively. The interaction information is as follows: in, Indicates by Convolution achieves output projection. Indicates spatial resolution. Indicates interactive information.

7. The method for assessing diabetic retinopathy based on lesion perception cross-task interaction according to claim 6, characterized in that, The interactive information is fused into the corresponding task branches to obtain the updated hierarchical features and lesion features, including: The lesion-related interactive information and the hierarchical semantic interactive information are respectively divided into preset proportions. Add the corresponding task branch to obtain the updated hierarchical features. and lesion characteristics ; in, This represents the interactive information from the lesion location to the grading direction. To facilitate the exchange of information from grading to the direction of lesions, This is the cross-task interaction strength coefficient, used to control the fusion ratio of cross-task information.

8. The method for assessing diabetic retinopathy based on lesion perception cross-task interaction according to claim 1, characterized in that, S4 includes: Global average pooling is performed on the graded features and lesion features obtained from S3. The T-frame feature vectors of the same eye are aggregated to obtain the corresponding ocular-level grade representation and ocular-level lesion representation; The ocular lesion representation is input into the lesion prediction head, which is composed of a Dropout layer and a fully connected layer connected in sequence, and is used to output the predicted values ​​of various types of lesions. The predicted values ​​of various types of lesions are processed by Sigmoid to obtain the corresponding lesion occurrence probability, and the lesion multi-label recognition result is obtained according to the preset threshold. The probability of lesion occurrence is input into the lesion-guided grading calibration unit, which includes a fully connected layer and a Tanh activation function. This unit maps the probability of lesion occurrence to lesion prompt information and adds the lesion prompt information to the eye-level grading representation according to a preset calibration coefficient to obtain the calibrated grading representation. The calibrated classification representation is input into the classification prediction head, which is composed of a Dropout layer and a fully connected layer connected in sequence, and is used to output the predicted value of each DR severity level. The predicted value is processed by Softmax to obtain the predicted probability of each level, and the level with the highest predicted probability is determined as the DR severity classification result.

9. The method for assessing diabetic retinopathy based on lesion perception cross-task interaction according to claim 1, characterized in that, In S5: From the dataset obtained from S1, multiple frames of images, DR severity level and lesion label corresponding to the same eye are read in batches, and the multiple frames of images are input into the diabetic retinopathy assessment model to obtain the DR severity level prediction value and the lesion multi-label prediction value. The total training loss is calculated using a weighted joint loss function. The loss function for multi-task joint training is a weighted sum of the losses from the DR severity grading task and the lesion label identification task. in, The loss for the DR severity classification task is represented by the cross-entropy loss function with category weights, calculated based on the classification prediction value and the DR severity level label. The loss for lesion label recognition task is represented by a binary cross-entropy loss function with positive sample weights, calculated based on the predicted lesion value and multiple lesion labels. and These represent the loss weights for the DR severity grading task and the lesion identification task, respectively, and are used to adjust the contributions of the two tasks in joint training. In each training batch, the gradients of the model parameters are first cleared to zero, then backpropagation is performed based on the joint loss, and the learnable parameters of the model are updated using the AdamW optimizer to obtain the best evaluation model.

10. An assessment system for implementing the diabetic retinopathy assessment method based on lesion perception cross-task interaction as described in any one of claims 1-9, characterized in that, include: The image sequence construction module is used to acquire and process multiple frames of images from different angiography phases of the same eye to form an image sequence; The lesion perception cross-task interaction network module is used for S2-S4 processing of image sequences; The model training module performs multi-task joint training using training data and corresponding labels to obtain the evaluation model. The results output module is used to output the severity grading results of DR and the multi-label identification results of lesions.

Images