Image recognition-based fundus color photograph retinopathy screening system
By employing modules for multimodal feature decoupling, specific lesion feature enhancement, dynamic decision boundary generation, and hierarchical cascade classification and confidence assessment, the feature confusion problem in the automated screening system for diabetic retinopathy was resolved, achieving highly specific and robust screening for diabetic retinopathy and improving the system's accuracy and reliability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- TONGJI HOSPITAL ATTACHED TO TONGJI MEDICAL COLLEGE HUAZHONG SCI TECH
- Filing Date
- 2026-04-13
- Publication Date
- 2026-07-07
AI Technical Summary
In existing technologies, automated screening systems for diabetic retinopathy suffer from insufficient screening specificity and generalization ability due to the visual similarity of lesion characteristics to those of other fundus lesions, resulting in a high risk of misjudgment.
The system employs a multimodal feature decoupling extraction module, a specific lesion feature enhancement module, a dynamic decision boundary generation module, and a hierarchical cascade classification and confidence assessment module to achieve highly specific and robust screening for diabetic retinopathy.
It significantly improves the feature identification accuracy and system robustness of diabetic retinopathy, enhances the ability to process complex clinical images, realizes an optimized workflow of human-machine collaboration, and improves the practicality and reliability of the screening system.
Smart Images
Figure CN122023950B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of medical image processing and artificial intelligence technology, specifically relating to an automatic screening system for diabetic retinopathy based on fundus color imaging using image recognition. Background Technology
[0002] Artificial intelligence (AI) technology is playing an increasingly important role in the field of medical image analysis. Through methods such as deep learning, it assists doctors in disease diagnosis, significantly improving diagnostic efficiency and accuracy. Automated screening for diabetic retinopathy based on fundus color photography is a key application of AI-assisted diagnosis, aiming to achieve early and rapid identification of diabetic retinopathy by analyzing fundus images.
[0003] Existing technologies typically employ models such as convolutional neural networks to extract and classify features from single fundus images. However, certain pathological features of diabetic retinopathy, such as microaneurysms and hemorrhages, exhibit visual similarities to fundus changes caused by other systemic diseases like hypertensive retinopathy. This leads to confusion during feature learning, making it difficult for models to extract highly specific discriminative features. This feature confusion directly impacts the performance of classification models, resulting in a high risk of misclassification for existing automated screening systems when faced with complex and diverse clinical images. The reliability and generalization ability of their screening results urgently need improvement. Summary of the Invention
[0004] The purpose of this invention is to provide an automatic screening system for diabetic retinopathy based on image recognition of fundus color photography, so as to solve the problems of model feature confusion, insufficient screening specificity and generalization ability caused by the visual similarity of diabetic retinopathy features with other fundus lesion features in the prior art.
[0005] This invention provides an automatic screening system for diabetic retinopathy based on image recognition fundus photography, comprising:
[0006] The multimodal feature decoupling extraction module is used to receive the input fundus color image and simultaneously execute at least two parallel feature extraction paths to generate feature vectors with different pathological semantic orientations.
[0007] The specific lesion feature enhancement module is used to perform cross-path comparison analysis and adaptive weighted fusion on the feature vectors generated by the multimodal feature decoupling extraction module to generate a comprehensive feature representation with high Glycoidic network specificity.
[0008] The dynamic decision boundary generation module is used to calculate and generate a classification decision boundary that matches the image feature distribution in real time based on the comprehensive feature representation of the current input fundus color image.
[0009] The hierarchical classification and confidence assessment module is used to perform multi-level classification judgment on the comprehensive feature representation based on the decision boundary provided by the dynamic decision boundary generation module, and simultaneously output the lesion level classification results and the corresponding confidence scores.
[0010] Preferably, the multimodal feature decoupling extraction module includes a global structural feature extraction path and a local lesion feature extraction path;
[0011] The global structural feature extraction path is configured with a first deep convolutional neural network. This first deep convolutional neural network takes the entire fundus color image as input. Its network architecture has been pre-trained and optimized to focus on capturing the overall morphology, relative positional relationship and macroscopic texture distribution information of the optic disc, macula, and major vascular arches.
[0012] The local lesion feature extraction path is configured with a second deep convolutional neural network and a lesion candidate region proposal network. The local lesion feature extraction path first automatically generates multiple candidate lesion regions in the input image through the lesion candidate region proposal network. Then, the second deep convolutional neural network performs high-resolution cropping and feature encoding on these candidate regions to specifically extract the micromorphological and texture details of local lesions such as microaneurysms, bleeding points, and hard exudates.
[0013] Preferably, the workflow of the specific lesion feature enhancement module is as follows:
[0014] It receives a set of global feature vectors from the global structural feature extraction path and a set of local feature vectors from the local lesion feature extraction path;
[0015] By using the built-in cross-path feature comparison unit, each local feature vector is mapped to the same dimension as the global feature vector through a linear transformation layer. Then, the cosine similarity between the global feature vector and the local feature vectors after each dimension is aligned is calculated. Based on the preset similarity threshold, the local feature vectors are divided into feature subsets with high correlation to the global structure and feature subsets with low correlation.
[0016] The built-in adaptive feature fusion unit applies different fusion weights to the two feature subsets. For local feature subsets with low correlation to the global structure, a higher fusion weight is assigned, and for local feature subsets with high correlation to the global structure, a lower fusion weight is assigned.
[0017] The weighted local feature subsets are aggregated and concatenated with the global feature vector to generate a comprehensive feature representation.
[0018] Preferably, the core of the dynamic decision boundary generation module is a boundary computation network;
[0019] The boundary computation network takes the comprehensive feature representation output by the specific lesion feature enhancement module as input;
[0020] The boundary computation network performs nonlinear transformations through three fully connected layers and finally outputs a decision boundary parameter vector, which defines the normal vector and bias term of the hyperplane in the high-dimensional feature space.
[0021] The hyperplane serves as the real-time classification decision boundary for the current input sample, and its position and orientation are dynamically determined by the comprehensive features of the input sample.
[0022] Preferably, the hierarchical cascaded classification and confidence evaluation module includes a cascaded classifier stack and a confidence evaluator;
[0023] The cascaded classifier stack consists of multiple shallow classifiers connected sequentially. The first-level classifier receives the comprehensive feature representation and performs a preliminary binary classification judgment based on the dynamic decision boundary, i.e., whether the diabetic retinopathy is positive or negative.
[0024] If the Level 1 classifier determines a positive result, the comprehensive feature representation and the Level 1 determination result are passed to the Level 2 classifier. The Level 2 classifier is responsible for classifying the severity of the lesion in the positive sample and outputting the classification result according to the international clinical grading standard.
[0025] The confidence evaluator works in parallel with each level of classifier. It receives the last hidden layer activation value of the current level classifier before outputting the final category. It calculates the minimum Euclidean distance between the hidden layer activation value vector and the prototype vector of each category, and maps the minimum Euclidean distance to a value between 0 and 1 through a preset monotonically decreasing function, which is used as the confidence score of the current classification result.
[0026] Preferably, the system further includes a model continuous optimization interface;
[0027] The model's continuous optimization interface is used to receive screening results that have been reviewed and confirmed by professional physicians after the system is deployed.
[0028] The model's continuous optimization interface will feed back data, including the original fundus image, the system-generated comprehensive feature representation, the physician-corrected lesion label, and the confidence score output by the system itself, and package them into training sample pairs.
[0029] The training sample pairs are stored in an incremental learning buffer pool. When the number of samples in the buffer pool accumulates to a preset threshold, incremental fine-tuning training of the multimodal feature decoupling extraction module, the specific lesion feature enhancement module, and the hierarchical cascade classification and confidence assessment module will be triggered.
[0030] Preferably, the working mechanism of the lesion candidate region suggestion network is as follows:
[0031] A dense grid of anchor points is generated on the input image, and a lesion presence probability score and bounding box position adjustment are predicted for each anchor point.
[0032] A non-maximum suppression algorithm is used to select several candidate regions with high rankings based on the probability scores of lesion presence.
[0033] The coordinate information of the candidate region is used to accurately crop the corresponding image patch from the original image through a spatial transformation layer, which is then processed by the subsequent second deep convolutional neural network.
[0034] Preferably, the system follows a preset confidence level-workflow linkage protocol during operation;
[0035] The confidence-workflow linkage protocol stipulates that when the confidence score of the highest-level lesion classification result output by the hierarchical cascade classification and confidence assessment module is less than the first preset threshold, the system automatically marks the case as requiring manual review and pushes its image and all intermediate feature visualization results to the manual review queue.
[0036] When the confidence score is less than a lower second preset threshold, in addition to marking that manual review is required, the system will automatically activate an internal feature re-extraction process. In this process, the specific lesion feature enhancement module will use a set of alternative, more conservative feature fusion weight coefficients to regenerate the comprehensive feature representation and perform classification and evaluation again.
[0037] Preferably, the lesion candidate region suggestion network in the local lesion feature extraction path adopts a multi-scale feature pyramid mechanism;
[0038] The multi-scale feature pyramid mechanism utilizes the output of the intermediate convolutional layers of the first deep convolutional neural network to construct a top-down feature pyramid.
[0039] The lesion candidate region suggestion network independently deploys anchor point grids on each layer of the feature pyramid, and the anchor point size of each layer is scaled according to the receptive field of each layer's feature map.
[0040] The region proposals for each layer are subjected to independent nonmaximum suppression, and then merged and screened at a uniform scale to generate candidate regions.
[0041] Preferably, the boundary computation network embeds a channel attention module between the second fully connected layer and the third fully connected layer;
[0042] The channel attention module performs global average pooling on the output of the second fully connected layer to generate a channel statistics vector.
[0043] The channel attention module performs a non-linear transformation on the channel statistical vector through two 1×1 convolutional layers to generate channel weights.
[0044] The channel attention module multiplies the channel weights with the original features channel by channel to obtain a reweighted feature representation, which is then processed by the third fully connected layer.
[0045] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0046] 1. This invention constructs a multimodal feature decoupling extraction module, which physically isolates and processes easily confused global structural information and local lesion information in parallel during the feature extraction stage from the technical source. This effectively curbs the mutual interference of different pathological semantic features in the early encoding stage and lays a solid foundation for the subsequent generation of highly specific features.
[0047] 2. This invention, through cross-path comparison analysis and adaptive weighted fusion mechanism in the specific lesion feature enhancement module, can intelligently identify and enhance local lesion features that are highly correlated with diabetic retinopathy but have low correlation with common confusing lesions, while weakening common structural features, thereby actively constructing a comprehensive feature representation with stronger discriminative power, significantly improving the system's feature identification accuracy for diabetic retinopathy.
[0048] 3. The dynamic decision boundary generation module introduced in this invention abandons the fixed and rigid decision boundaries of traditional classification models, and can dynamically adjust the classification criteria according to the feature distribution of each input sample. This adaptive mechanism enables the system to better handle difficult samples located in fuzzy areas or sparsely distributed regions of the feature space, greatly enhancing the robustness and generalization ability of the model when facing the diversity of clinical images.
[0049] 4. The hierarchical cascaded classification and confidence assessment module employed in this invention decomposes complex multi-classification tasks into sequential decision-making processes, reducing the difficulty of individual decisions. Simultaneously, parallel confidence assessment provides a reliability metric for each decision step. Combined with a confidence-workflow linkage protocol, the system can intelligently triage low-confidence cases to manual review. While ensuring the efficiency of automated screening, it provides necessary human intervention entry points for critical and complex cases, achieving an optimized workflow through human-machine collaboration and comprehensively improving the practicality and reliability of the screening system. Attached Figure Description
[0050] Figure 1 This is a schematic diagram of the overall technical solution architecture of the present invention;
[0051] Figure 2 This is a schematic diagram of the core principle framework of the specific lesion feature enhancement module in this invention;
[0052] Figure 3This is a flowchart of the main stages of the multimodal feature decoupling and extraction module in this invention.
[0053] Figure 4 This is a logical flowchart of the hierarchical cascade classification and confidence assessment module in this invention;
[0054] Figure 5 This is a schematic diagram of the multi-level interaction relationship and data flow of the system runtime confidence-workflow linkage protocol in this invention. Detailed Implementation
[0055] Example 1: Please refer to the appendix Figure 1 To be continued Figure 5 This invention proposes an automated screening system for diabetic retinopathy based on image recognition of fundus color images. The system uses a standard-format fundus color image as input and, through a series of highly collaborative and structured functional modules, automates the entire process from the original image to the final lesion classification result and confidence score. The entire system strictly adheres to the technical logic of feature decoupling, specificity enhancement, dynamic decision-making, and hierarchical evaluation during operation, ensuring high specificity, robustness, and interpretability even in complex and ever-changing clinical image scenarios.
[0056] The system first sends the input fundus color image to the multimodal feature decoupling extraction module. This module is the basic feature generation unit of the entire system. Its core objective is to physically isolate information with different pathological semantics in the initial stage of feature extraction, so as to avoid feature confusion between global structural information and local lesion details during the early encoding process.
[0057] Please refer to the attached document. Figure 3 The multimodal feature decoupling extraction module contains two fully parallel and independently optimized feature extraction paths: a global structural feature extraction path and a local lesion feature extraction path. The global structural feature extraction path uses a first deep convolutional neural network pre-trained on a large-scale fundus image dataset as its backbone architecture. This first deep convolutional neural network receives the entire fundus color image as input, and its convolutional layer stacking structure is specifically optimized to prioritize capturing the contour integrity of the optic disc, the reflectivity of the fovea, the course of the major vascular arches, and the macroscopic distribution pattern of the retinal background texture.
[0058] This global structural information is of great reference value for determining the presence of nonspecific fundus degenerative changes (such as age-related macular degeneration or high myopia retinopathy). However, if it is mixed with local lesion features for encoding, it can easily lead to misdiagnosis of diabetic retinopathy by the model. Therefore, the output of this path is a global feature vector with a fixed dimension, which is usually set to 512 dimensions, with each dimension corresponding to a quantitative representation of a certain high-level semantic structural attribute.
[0059] Simultaneously, the local lesion feature extraction path is initiated. This path consists of a cascaded lesion candidate region proposal network and a second deep convolutional neural network. The lesion candidate region proposal network first constructs a dense grid of anchor points on the input image. The grid density is dynamically adjusted according to the image resolution, typically set to place anchor points at 16-pixel intervals.
[0060] For each anchor point, the lesion candidate region proposal network predicts two key outputs:
[0061] The lesion has a probability score, which reflects the likelihood of diabetic reticulum-related lesions (such as microaneurysms, punctate hemorrhages, and hard exudates) in the area surrounding the anchor point.
[0062] The bounding box position adjustment, including center offset and width and height scaling factors, is used to accurately locate potential lesion areas.
[0063] Subsequently, the system employs a non-maximum suppression algorithm to filter all anchor point prediction results, retaining the top 128 candidate regions with the highest lesion presence probability scores and removing redundant boxes with an overlap greater than 0.7. The coordinate information of these filtered candidate regions is then fed into a spatial transformation layer for the lesion candidate regions. This spatial transformation layer, based on the principle of affine transformation, precisely crops corresponding image patches from the original high-resolution fundus image, with the cropping size uniformly normalized to 224×224 pixels. These cropped image patches are then fed into a second deep convolutional neural network for high-resolution feature encoding. This second deep convolutional neural network has also undergone targeted pre-training, with its receptive field and filter parameters optimized to capture microscopic lesion details such as the circular high-brightness features of microaneurysms, the irregular dark morphology of hemorrhage points, and the star-shaped boundaries of hard exudates.
[0064] Finally, the path outputs a set of local feature vectors, which contains 128 local feature vectors, each with a dimension of 256, and each corresponding to a deep semantic encoding of a candidate lesion region.
[0065] After the multimodal feature decoupling extraction module completes its work, its output global feature vector and local feature vector set are synchronously transmitted to the specific lesion feature enhancement module. Please refer to the appendix. Figure 2 The core task of this specific lesion feature enhancement module is to intelligently identify and enhance local lesion features that are highly specific to diabetic retinopathy and highly distinguishable from other common fundus lesions through cross-path comparative analysis. The module first maps each local feature vector to 512 dimensions through a linear transformation layer, aligning it with the global feature vector dimension. Then, it calculates the global feature vector. Local feature vectors aligned with each dimension Cosine similarity between them. The calculation formula is as follows:
[0066] ;
[0067] Represents the global feature vector. Indicates the first A dimension-aligned local feature vector The value range is [-1, 1]. The system presets a similarity threshold. The typical value is 0.35. If Then determine the local feature vector. A low correlation with the overall structure likely indicates isolated, small lesions characteristic of the diabetic reticulum; conversely, if... If the correlation between local features and global structure is high, it is considered that the local features may originate from non-specific inflammation or degenerative changes. Based on this judgment result, the module divides the 128 local feature vectors into two mutually exclusive subsets: a low-correlation subset. Highly correlated subsets .
[0068] Subsequently, the adaptive feature fusion unit applies differentiated fusion weights to the two subsets. For the low-association subset... Assign fusion weights For highly correlated subsets Assign fusion weights The weighting coefficients were set based on statistical analysis of numerous validation experiments, aiming to maximize the characteristic response intensity of diabetic retinopathy lesions while suppressing common structural noise. The feature vector after applying the weights is... Finally, the With 512-dimensional global feature vectors Channel splicing is performed to form a 768-dimensional comprehensive feature representation. This comprehensive feature representation is designed to preserve the discriminative information of diabetic retinopathy to the greatest extent possible, while minimizing feature overlap with other fundus diseases.
[0069] Comprehensive feature representation The sample is then fed into the dynamic decision boundary generation module. This module abandons the static, globally shared decision boundary setting of traditional classifiers, and instead dynamically generates a unique classification hyperplane for each input sample. The module contains a boundary computation network, which consists of three fully connected layers with 512, 256, and 768 neurons in each layer, respectively.
[0070] The input 768-dimensional comprehensive feature representation first undergoes dimensionality reduction and nonlinear activation (using the ReLU function) through the first fully connected layer, then further abstracts it through the second fully connected layer, and finally outputs a 768-dimensional decision boundary parameter vector by the third fully connected layer. . It is analyzed into two parts: the first 767 dimensions constitute the normal vector of the hyperplane. The last dimension is used as a bias term. The equation of the hyperplane thus defined is: , These are the feature points to be classified. The hyperplane serves as the real-time classification decision boundary for the current sample. Its direction and position are entirely determined by the feature distribution of the input sample itself, thus enabling it to adaptively handle difficult samples with sparse distribution or ambiguous boundaries in the feature space.
[0071] After obtaining the dynamic decision boundary, the system enters the hierarchical cascade classification and confidence assessment module. Please refer to the appendix. Figure 4 The hierarchical classification and confidence assessment module adopts a cascaded decision architecture, which decomposes the complex multi-class classification task into two sequentially executed sub-tasks.
[0072] The first level is a binary classification task, executed by the first-level classifier in the cascaded classifier stack. This first-level classifier receives the comprehensive feature representation. The system calculates the signed distance to the hyperplane based on the dynamic decision boundary. If the distance is greater than 0, the result is considered positive for diabetic retinopathy; otherwise, it is considered negative. The result, along with the original comprehensive feature representation, is passed to the next level. If the result is negative, the process terminates, and the conclusion "no diabetic retinopathy" and the corresponding confidence level are output.
[0073] If a positive result is obtained, the second-level classifier is triggered. This second-level classifier is responsible for classifying the severity of the lesion in positive samples, and outputs one of four results based on international clinical grading standards (such as the ICDR grading system): mild non-proliferative phase, moderate non-proliferative phase, severe non-proliferative phase, or proliferative phase. The second-level classifier uses a shallow fully connected network with a Softmax output layer. Its input is a comprehensive feature representation, and its output is the probability distribution of the four categories.
[0074] Working in parallel with the two-stage classifiers is a confidence estimator. This estimator truncates the activation vector of the last hidden layer before each classifier outputs its final class. The system pre-maintains a prototype vector for each class in the feature space, obtained from the average feature values of all positive samples during training, and updates it periodically. The confidence estimator calculates the Euclidean distance between the current activation vector and the prototype vector of its class. and will The confidence score is converted to a range of 0 to 1 using a monotonically decreasing Sigmoid mapping function. The mapping relationship is as follows:
[0075] ;
[0076] and The preset scale and offset parameters are 2.5 and 0.8, respectively. This confidence score intuitively reflects the reliability of the current classification result in the feature space: the smaller the distance, the higher the confidence.
[0077] The system also features a continuous model optimization interface for post-deployment performance self-evolution. This interface continuously monitors screening results confirmed by professional physicians. Upon receiving a feedback result, the system packages and generates training sample pairs, including the original fundus image, the system-generated comprehensive feature representation, the physician-corrected lesion label (including positive / negative and specific grading), and the system's own output confidence score. These sample pairs are stored in an incremental learning buffer.
[0078] When the number of samples in the buffer pool reaches 1000, the system automatically triggers an incremental fine-tuning training. The fine-tuning process only updates the parameters of the multimodal feature decoupling extraction module, the specific lesion feature enhancement module, and the hierarchical cascade classification and confidence assessment module with small steps (learning rate set to 10). -5 (The system) freezes other parts to ensure model stability, enabling the system to continuously absorb new clinical knowledge and gradually improve its ability to identify rare or novel disease patterns.
[0079] Furthermore, the system strictly adheres to the confidence level-workflow linkage protocol during operation. Please refer to the appendix. Figure 5 This confidence-workflow linkage protocol defines the mapping relationship between confidence scores and subsequent processing flows. The system sets two key thresholds: a first preset threshold... The second preset threshold When the confidence score of the highest-level lesion classification result... When a case is identified as "requiring manual review", the system will automatically mark it as "requiring manual review" and push the original image, comprehensive feature representation, classification results at all levels, and intermediate feature visualizations (such as heat maps and candidate region boxes) to the manual review queue for ophthalmologists to make a final decision.
[0080] If the confidence score is further less than ,Right now While marking items as requiring manual review, the system will automatically activate an internal feature re-extraction process. During this process, the specific lesion feature enhancement module switches to a set of alternative, more conservative feature fusion weighting coefficients (e.g., The comprehensive feature representation is regenerated, and the complete classification and confidence assessment process is performed again. If the confidence score after reassessment improves to [a certain level], [then the process continues]. If the above results are obtained, the new results will be adopted; otherwise, the original low-confidence label will be maintained and the case will be manually reviewed. This balances the efficiency of automation with the safety of diagnosis, ensuring that key and difficult cases will not be misdiagnosed or missed by the system.
[0081] In summary, this embodiment constructs a highly specialized, adaptive, and evolvable automated screening system for diabetic retinopathy through the close collaboration of four core technical components: multimodal feature decoupling, cross-path contrast enhancement, dynamic boundary generation, and hierarchical confidence assessment. This system not only solves the industry challenges of feature confusion and insufficient generalization at the technical level, but also achieves an intelligent closed-loop human-machine collaboration at the clinical workflow level, providing reliable technical support for large-scale diabetic retinopathy screening.
[0082] Example 2: Building upon Example 1, this example enhances the region proposal network in the local lesion feature extraction path to further improve the detection sensitivity for small lesions. Specifically, the original region proposal network relies solely on a single-scale anchor grid for initial lesion screening, which may pose a risk of missed detection when dealing with early microaneurysms with a diameter of less than 50 micrometers.
[0083] To address this, this embodiment introduces a multi-scale feature pyramid mechanism. This mechanism first utilizes the outputs of the intermediate convolutional layers of the first deep convolutional neural network (specifically, the output feature maps of the 3rd, 5th, and 7th residual blocks) to construct a top-down feature pyramid. Each pyramid feature map has a different spatial resolution and semantic level: higher-level feature maps are semantically rich but have low resolution, suitable for detecting larger lesions; lower-level feature maps have clear details but weak semantics, suitable for capturing small structures. The region proposal network no longer generates anchor points only at the original image scale, but instead independently deploys an anchor point grid on each layer of the feature pyramid, with the anchor point size scaled according to the receptive field of each layer's feature map.
[0084] For example, at the highest layer (corresponding to a 1 / 32 scale of the original image), the anchor point base size is set to 128×128 pixels; at the middle layer (1 / 16 scale), it is set to 64×64 pixels; and at the lowest layer (1 / 8 scale), it is set to 32×32 pixels. The region proposal results for each layer undergo independent non-maximum suppression, followed by upsampling and feature fusion operations, merging and secondary filtering at a uniform scale to ultimately generate no more than 256 high-quality candidate regions.
[0085] Furthermore, this embodiment optimizes the structure of the boundary computation network for the dynamic decision boundary generation module. The original network uses a fixed stack of fully connected layers, and its expressive power is limited by the preset number of neurons. This embodiment introduces a lightweight attention mechanism, embedding a channel attention module between the second and third fully connected layers. This channel attention module first performs global average pooling on the 256-dimensional output of the second fully connected layer to generate a 256-dimensional channel statistical vector; then, it performs a nonlinear transformation on the channel statistical vector through two 1×1 convolutional layers (reducing the dimensionality to 64 dimensions in the middle) to generate channel weights; finally, it multiplies the channel weights with the original 256-dimensional features channel by channel to obtain the reweighted feature representation.
[0086] This operation enables the network to dynamically adjust the importance of each channel based on the content of the input features, thereby generating a more discriminative decision boundary parameter vector. This is demonstrated on a test set containing a large number of samples with ambiguous boundaries.
[0087] Regarding confidence assessment, this embodiment improves the prototype vector update strategy. The original scheme uses a simple arithmetic mean, which is susceptible to interference from outlier samples. This embodiment employs a sliding window weighted average strategy, using only the 500 most recent samples of the same type confirmed as correct by physicians for prototype updates, and assigning higher weights to recent samples (weights decay exponentially over time with a decay coefficient of 0.98). This strategy allows the prototype vector to adapt more quickly to slow shifts in data distribution, such as style differences introduced by different imaging devices, thereby maintaining the long-term stability of the confidence score.
[0088] Finally, this embodiment expands the data processing capabilities of the model's continuous optimization interface. The original interface only processed feedback for a single image, while this embodiment supports batch feedback and adversarial example mining. Once a sufficient number of samples have accumulated in the buffer pool, the system automatically runs the adversarial example generator. Based on the gradient information of the current model, it generates adversarial examples by making minor perturbations to high-confidence misjudged samples. These adversarial examples are then added to the training set for fine-tuning, improving the model's robustness and making it more stable in the face of interference factors such as image noise and changes in illumination. Clinically validated, this embodiment achieved a comprehensive F1 score of 0.937 in real-world screening scenarios, an improvement of 0.028 compared to Embodiment 1, fully demonstrating the effectiveness and necessity of the various enhancement measures.
[0089] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.
[0090] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. An automatic screening system for diabetic retinopathy based on image recognition fundus photography, characterized in that, include: The multimodal feature decoupling extraction module is used to receive the input fundus color image and simultaneously execute at least two parallel feature extraction paths to generate feature vectors with different pathological semantic orientations. The specific lesion feature enhancement module is used to perform cross-path comparison analysis and adaptive weighted fusion on the feature vectors generated by the multimodal feature decoupling extraction module to generate a comprehensive feature representation with high Glycoidic network specificity. The workflow of the specific lesion feature enhancement module is as follows: It receives a set of global feature vectors from the global structural feature extraction path and a set of local feature vectors from the local lesion feature extraction path; The built-in cross-path feature comparison unit calculates the cosine similarity between the global feature vector and each local feature vector, and distinguishes the local feature vectors into feature subsets with high correlation to the global structure and feature subsets with low correlation based on the preset similarity threshold. The built-in adaptive feature fusion unit applies different fusion weights to the two feature subsets. For local feature subsets with low correlation to the global structure, a higher fusion weight is assigned, and for local feature subsets with high correlation to the global structure, a lower fusion weight is assigned. The weighted local feature subsets are aggregated and concatenated with the global feature vector to generate a comprehensive feature representation. The dynamic decision boundary generation module is used to calculate and generate a classification decision boundary that matches the image feature distribution in real time based on the comprehensive feature representation of the current input fundus color image. The hierarchical classification and confidence assessment module is used to perform multi-level classification judgment on the comprehensive feature representation based on the decision boundary provided by the dynamic decision boundary generation module, and simultaneously output the lesion level classification results and the corresponding confidence scores.
2. The automatic screening system for diabetic retinopathy based on image recognition fundus photography according to claim 1, characterized in that, The multimodal feature decoupling extraction module includes a global structural feature extraction path and a local lesion feature extraction path; The global structural feature extraction path is configured with a first deep convolutional neural network. This first deep convolutional neural network takes the entire fundus color image as input. Its network architecture has been pre-trained and optimized to focus on capturing the overall morphology, relative positional relationship and macroscopic texture distribution information of the optic disc, macula, and major vascular arches. The local lesion feature extraction path is configured with a second deep convolutional neural network and a differentiable region proposal network. The local lesion feature extraction path first automatically generates multiple candidate lesion regions in the input image through the differentiable region proposal network. Then, the second deep convolutional neural network performs high-resolution cropping and feature encoding on these candidate regions to specifically extract the micromorphological and texture details of local lesions such as microaneurysms, bleeding points, and hard exudates.
3. The automatic screening system for diabetic retinopathy based on image recognition in fundus photography according to claim 2, characterized in that, The core of the dynamic decision boundary generation module is the boundary computation network; The boundary computation network takes the comprehensive feature representation output by the specific lesion feature enhancement module as input; The boundary computation network performs nonlinear transformations through three fully connected layers and finally outputs a decision boundary parameter vector, which defines the normal vector and bias term of the hyperplane in the high-dimensional feature space. The hyperplane serves as the real-time classification decision boundary for the current input sample, and its position and orientation are dynamically determined by the comprehensive features of the input sample.
4. The automatic screening system for diabetic retinopathy based on image recognition fundus photography according to claim 3, characterized in that, The hierarchical cascaded classification and confidence evaluation module includes a cascaded classifier stack and a confidence evaluator; The cascaded classifier stack consists of multiple shallow classifiers connected sequentially. The first-level classifier receives the comprehensive feature representation and performs a preliminary binary classification judgment based on the dynamic decision boundary, i.e., whether the diabetic retinopathy is positive or negative. If the Level 1 classifier determines a positive result, the comprehensive feature representation and the Level 1 determination result are passed to the Level 2 classifier. The Level 2 classifier is responsible for classifying the severity of the lesion in the positive sample and outputting the classification result according to the international clinical grading standard. The confidence evaluator works in parallel with each level of classifier. It receives the last hidden layer activation value of the current level classifier before outputting the final category. It calculates the minimum Euclidean distance between the hidden layer activation value vector and the prototype vector of each category, and maps the minimum Euclidean distance to a value between 0 and 1 through a preset monotonically decreasing function, which is used as the confidence score of the current classification result. The prototype vector for each category refers to the prototype vector that the system maintains in advance in the feature space for each category. This prototype vector is obtained by the average feature value of all positive samples during the training phase and is updated periodically.
5. The automatic screening system for diabetic retinopathy based on image recognition fundus photography according to claim 4, characterized in that, The system also includes a model continuous optimization interface; The model's continuous optimization interface is used to receive screening results that have been reviewed and confirmed by professional physicians after the system is deployed. The model's continuous optimization interface will feed back data, including the original fundus image, the system-generated comprehensive feature representation, the physician-corrected lesion label, and the confidence score output by the system itself, and package them into training sample pairs. The training sample pairs are stored in an incremental learning buffer pool. When the number of samples in the buffer pool accumulates to a preset threshold, incremental fine-tuning training of the multimodal feature decoupling extraction module, the specific lesion feature enhancement module, and the hierarchical cascade classification and confidence assessment module will be triggered.
6. The automatic screening system for diabetic retinopathy based on image recognition fundus photography according to claim 5, characterized in that, The working mechanism of the differentiable region proposal network is as follows: A dense grid of anchor points is generated on the input image, and a lesion presence probability score and bounding box position adjustment are predicted for each anchor point. A non-maximum suppression algorithm is used to select several candidate regions with high rankings based on the probability scores of lesion presence. The coordinate information of the candidate region is used to accurately crop the corresponding image patch from the original image through a differentiable spatial transformation layer, which is then processed by the subsequent second deep convolutional neural network.
7. The automatic screening system for diabetic retinopathy based on image recognition in fundus photography according to claim 6, characterized in that, The system follows a preset confidence level-workflow linkage protocol during operation; The confidence-workflow linkage protocol stipulates that when the confidence score of the highest-level lesion classification result output by the hierarchical cascade classification and confidence assessment module is less than the first preset threshold, the system automatically marks the case as requiring manual review and pushes its image and all intermediate feature visualization results to the manual review queue. When the confidence score is less than a lower second preset threshold, in addition to marking that manual review is required, the system will automatically activate an internal feature re-extraction process. In this process, the specific lesion feature enhancement module will use a set of alternative, more conservative feature fusion weight coefficients to regenerate the comprehensive feature representation and perform classification and evaluation again.
8. The automatic screening system for diabetic retinopathy based on image recognition fundus photography according to claim 7, characterized in that, The differentiable region proposal network in the local lesion feature extraction path adopts a multi-scale feature pyramid mechanism. The multi-scale feature pyramid mechanism utilizes the output of the intermediate convolutional layers of the first deep convolutional neural network to construct a top-down feature pyramid. The differentiable region proposal network independently deploys an anchor grid on each layer of the feature pyramid, and the anchor size of each layer is scaled according to the receptive field of each layer's feature map. The region proposals for each layer are subjected to independent nonmaximum suppression, and then merged and screened at a uniform scale to generate candidate regions.
9. The automatic screening system for diabetic retinopathy based on image recognition fundus photography according to claim 8, characterized in that, The boundary computation network embeds a channel attention module between the second fully connected layer and the third fully connected layer; The channel attention module performs global average pooling on the output of the second fully connected layer to generate a channel statistics vector. The channel attention module performs a non-linear transformation on the channel statistical vector through two 1×1 convolutional layers to generate channel weights. The channel attention module multiplies the channel weights with the original features channel by channel to obtain a reweighted feature representation, which is then processed by the third fully connected layer.