A multi-modal fusion-based diabetic retinopathy auxiliary diagnosis method
By using multilayer perceptron mapping and restricted tensor fusion models, the problem of information isolation between image and text modalities is solved, enabling high accuracy and interpretability of early DR diagnosis, reducing the false negative rate, and improving screening efficiency.
Patent Information
- Application Number
- CN202610387917.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-03-27
- Publication Date
- 2026-06-23
Smart Images
Figure CN122266723A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of medical imaging, and more particularly to an auxiliary diagnostic method for diabetic retinopathy based on multimodal fusion. Background Technology
[0002] Diabetic retinopathy (DR) is a common microvascular complication of diabetes and a leading cause of blindness in working-age patients. Currently, intelligent assisted diagnosis of DR primarily relies on single-modal deep learning models based on fundus images. While these methods can automatically extract image features, their diagnostic capabilities are limited in complex samples with early, small lesions, multiple lesions, and underlying diseases due to a lack of clinical information such as patient history and biochemical indicators. In recent years, solutions combining fundus images with large language models have emerged, but these mostly generate general diagnostic text based on images. Although some advanced general multimodal methods (such as vision-language models) have begun to employ cross-modal attention or tensor outer product for deep feature fusion, they have fatal flaws in the specific medical scenario of DR assisted diagnosis: early DR lesions (such as microaneurysms and punctate hemorrhages) account for a very small percentage (usually less than 1%) in full-frame fundus images, while the patient's clinical characteristics (such as glycated hemoglobin and diabetes duration) are macroscopic, global semantic indicators. Directly applying general attention mechanisms or full tensor outer product fusion can trigger a severe "semantic noise avalanche"—that is, performing ineffective cross-calculations between a large amount of irrelevant image background (such as normal retinal areas) and key clinical indicators. This not only leads to dimensionality explosion but also dilutes the weak feature signals of small lesions, resulting in an extremely high rate of missed diagnoses of early-stage DR. Furthermore, existing multimodal methods lack a "text-image joint causal attribution" mechanism for medical diagnosis. A single heatmap cannot explain "how specific biochemical abnormalities affect the model's discrimination weights for specific small lesions in the fundus," limiting its credibility and widespread application in clinical practice.
[0003] Therefore, there is an urgent need for a multimodal diagnostic method for DR that can establish explicit semantic interaction between image modalities and clinical text modalities, and has high robustness and strong interpretability, in order to solve the core problems of existing technologies such as information isolation, insufficient multimodal collaboration, and uninterpretable decision-making. Summary of the Invention
[0004] The purpose of this invention is to address the above-mentioned problems by providing a multimodal fusion-based auxiliary diagnostic method for diabetic retinopathy that fully explores the correlation between fundus image features and patient clinical text information, achieves deep interactive fusion, and improves diagnostic accuracy and interpretability.
[0005] To achieve the above objectives, the technical solution of the present invention is as follows: A multimodal fusion-based auxiliary diagnostic method for diabetic retinopathy includes the following steps: S1. Perform preprocessing operations on the patient's fundus images; S2. Perform preprocessing operations on the patient's text information; S3. Perform feature extraction on the preprocessed fundus image to obtain the image feature vector; S4. Perform feature extraction on the preprocessed text information to obtain the text semantic vector; S5. A spatially guided gating matrix is generated through multilayer perceptron mapping, and a restricted tensor fusion model is established. The spatially guided gating matrix is then used to perform high-order sparse tensor fusion of image feature vectors and text semantic vectors. Simultaneously, based on the application of Tucker low-rank decomposition, a... Sparse regularization penalty is applied to obtain the fused feature vector and the diagnostic category prediction probability. S6. Combine the two-way reverse reasoning mechanism to perform image and text joint tracing of the diagnostic results, and output the diagnostic heat map and its corresponding dynamic linkage interpretation.
[0006] Furthermore, step S1 specifically includes the following steps: S11. Perform spherical correction and retinal region cropping on the fundus image to remove black background and artifacts; S12. Perform illumination normalization processing, the calculation formula is as follows: ; in, These are the pixel values of the original fundus image. , These represent the mean and standard deviation of the fundus images, respectively. These are the standardized pixel values of the fundus image; S13. Perform data augmentation on the fundus images after illumination normalization to expand the training samples.
[0007] Furthermore, the data augmentation operations in step S13 include random rotation, horizontal flipping, brightness adjustment, and contrast adjustment.
[0008] Furthermore, step S2 specifically includes the following steps: S21. Perform word segmentation and named entity recognition on the text information to extract key structured entities; S22. Encode key structured entities and normalize numerical indicators into standardized features; S23. Use TF-IDF or word vectors to further extract features to obtain the patient's clinical text feature vector.
[0009] Furthermore, step S3 specifically includes the following steps: S31. Using a pre-trained convolutional neural network as the backbone, a channel attention module is introduced into the convolutional neural network to adaptively adjust the weights of each channel. The calculation formula is as follows: ; in, This is the channel statistics vector obtained from global average pooling. , For a trainable weight matrix, , These are the ReLU and Sigmoid activation functions, respectively. S32. The formula for calculating the residual of a convolutional neural network is: ; in, Represents a nonlinear mapping of a set of convolutional layers; S32. The preprocessed fundus image is processed by a convolutional neural network with an added attention mechanism to extract features and obtain the image feature vector.
[0010] Furthermore, step S4 specifically includes the following steps: S41. Use the Chinese BERT model to obtain context-sensitive word vectors of clinical text feature vectors; S42. Combine context-sensitive word vectors and model them using bidirectional LSTM or multi-layer Transformer to output text semantic vectors.
[0011] Furthermore, step S5 specifically includes the following steps: S51. Let the image feature vector be... The text semantic vector is The medical entity features are mapped using a multilayer perceptron and spatially broadcast to the same spatial resolution as the image feature vector. After Sigmoid activation, a spatially guided gating matrix is generated. ; S52, Map the image feature vector and the text semantic vector to... , , Then, a spatial guidance gating matrix is added. The formula for calculating restricted attention is as follows: ; in, This represents element-wise multiplication; S53. Establish a constrained tensor fusion model, based on the application of Tucker low-rank decomposition, and in the core tensor... Apply on The sparse regularization penalty compresses the weights of the interaction terms between non-lesion image features and ordinary text features to zero during the training process of the restricted tensor fusion model. S54. The image feature vector and text semantic vector are fused using a restricted tensor fusion model to obtain fused features. These fused features are then input into a Softmax classifier, which outputs the predicted class probabilities. The formula is as follows: ; in, For category The original score.
[0012] Furthermore, in step S6, the joint partial derivatives of the fusion features with respect to the fundus image and text information are calculated, and a structured diagnostic heatmap and its corresponding dynamic linkage interpretation are output based on the joint partial derivatives.
[0013] Compared with the prior art, the advantages and positive effects of this invention are: This invention effectively utilizes the complementary information of the two modalities through deep fusion of fundus images and patient clinical information. It can capture minute lesions in the images and improve early DR detection capabilities by combining background information such as blood glucose and disease course. Furthermore, the invention's two-stage sparse interaction fusion strategy based on pathological feature anchors (Med-CMA and sparse low-rank tensor fusion) deeply explores the correlation between "image features and clinical indicators," which helps to handle complex cases. Its classification accuracy is higher (which can be verified through large-scale clinical data), and its false negative rate is lower, especially sensitive to minor lesions. On the other hand, this invention generates joint attribution tuples containing heatmap and text entity linkages by combining image-text causal tracing mechanism (based on bidirectional back-inference of high-order fusion tensors), which greatly enhances the clinical transparency and credibility of diagnostic results.
[0014] In clinical applications, this invention can provide doctors with rapid diagnostic suggestions, reduce the burden of manual image reading, lower the false negative rate, and accelerate the efficiency of retinal screening for diabetic patients. It can also realize "intelligent" DR screening in grassroots areas or areas with limited resources, which helps to detect and treat diseases early and reduce the social medical burden. Attached Figure Description
[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0016] Figure 1This is a system structure diagram of the present invention; Figure 2 Diagram of the feature extraction network structure for fundus images; Figure 3 Flowchart of feature extraction logic for fundus images; Figure 4 A diagram of the network structure for feature extraction of text information; Figure 5 Flowchart of feature extraction logic for text information; Figure 6 This is a flowchart of the method of the present invention. Detailed Implementation
[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, any modifications, equivalent substitutions, improvements, etc., made by those skilled in the art to all other embodiments obtained without creative effort should be included within the protection scope of the present invention.
[0018] This embodiment discloses a multimodal fusion-based auxiliary diagnostic method for diabetic retinopathy proposed by the present invention. Its core innovation lies in fully exploring the correlation between fundus image features and patient clinical text information, achieving deep interactive fusion, and improving diagnostic accuracy and interpretability. For example... Figure 1 As shown, the operating system on which this diagnostic method is based generally includes four modules: a data preprocessing module, a feature extraction module, a multimodal fusion and diagnostic module, and an interpretability output module.
[0019] The specific steps of the multimodal fusion-based auxiliary diagnostic method for diabetic retinopathy are as follows: 1. Multi-source data preprocessing Fundus image preprocessing: Standard retinal color images of the patient are collected. First, spherical correction and retinal region cropping are performed to remove black backgrounds and artifacts. Then, illumination normalization is performed, which can be achieved using mean-variance standardization to balance pixel value distribution. The formula is as follows: (1) in, These are the original image pixel values. , These are the mean and standard deviation of the entire graph, respectively. These are the standardized pixel values. This operation eliminates biases introduced by different devices or shooting conditions. Data augmentation methods (such as random rotation, horizontal flipping, brightness / contrast adjustment, etc.) are then used to augment the training samples and improve the model's robustness.
[0020] Text information preprocessing: For unstructured texts such as patient medical records and examination reports, word segmentation and named entity recognition (NER) are first performed to extract key entities such as diabetes course, medication history, and complications. For example, a pre-trained Chinese BioBERT model combined with conditional random fields (CRF) can be used to identify medical entities. The extracted structured entities (such as "blood glucose level" and "glycated hemoglobin value") are encoded, and numerical indicators (such as blood glucose and HbA1c) are normalized into standardized features. For text descriptions, TF-IDF or word vectors (Word2Vec / GloVe / BERT) can be used to further extract features. Finally, the feature vector of the patient's clinical text is obtained. ,in For text feature dimensions.
[0021] 2. Single-modal feature extraction Image Feature Extraction Network: This invention designs a dedicated retinal image feature extraction network, such as... Figure 2 , Figure 3 As shown, it can use a pre-trained convolutional neural network (such as ResNet-50 or EfficientNet) as the backbone to extract features from the standardized fundus image. An attention mechanism (such as an SE block or CBAM module) is then added to enhance the response of key channel / spatial features; specifically, a channel attention module (SE) is introduced to adaptively adjust the weights of each channel. (2) in, This is the channel statistics vector obtained from global average pooling. , For a trainable weight matrix, , The ReLU and Sigmoid activation functions are used respectively to enhance lesion-related channels. On the other hand, spatial attention (such as CBAM) learns a spatial attention map to suppress the influence of non-lesion regions. This network can also be designed as a multi-task structure, outputting DR classification results on one hand, and performing lesion segmentation tasks (such as exudate, hemorrhage, etc.) in parallel on the other, thereby enriching the feature representation. The residual learning structure is as follows: (3) ( This represents a set of nonlinear mappings (representing a set of convolutional layers), which can alleviate gradient problems and extract deep features. The network ultimately outputs an image feature vector. .
[0022] Text feature extraction network: For clinical text information, a Transformer-based or recurrent neural network-based model is used. After obtaining context-sensitive word vectors using a Chinese BERT model, a bidirectional LSTM or multi-layer Transformer is used to model the text and capture long-range dependencies, such as... Figure 4 , Figure 5 As shown. The model ultimately outputs a fixed-dimensional text semantic vector. For example, the entire medical description can be encoded into a 768-dimensional vector using BERT, and then projected through several fully connected layers to... Dimensional features.
[0023] 3. Multimodal fusion and decision-making with medical prior constraints The core of this invention lies in solving the problem of accurate alignment between small DR lesions and macroscopic clinical text. To avoid the "semantic noise" brought about by full feature fusion, this invention abandons the conventional full-mapping attention mechanism and innovatively proposes a two-stage sparse interaction fusion strategy based on pathological feature anchors (mid-term prior-guided interaction + high-order sparse tensor fusion), such as... Figure 6 As shown.
[0024] Medical Prior Guided Cross-Modal Masked Attention Interaction (Med-CMA): Before calculating the cross-attention between the image and text, this module first introduces a "pathological relevance mask," using key medical entities extracted from text features (such as "abnormal blood sugar" and "history of hypertension") to generate a spatially guided gating matrix. This is used to suppress smooth background regions without blood vessels in image features; specifically, let the image features be... Text features are Specific medical entity features are mapped using a multilayer perceptron (MLP) and spatially broadcast to the same spatial resolution as the image features. After sigmoid activation, a spatially guided gating matrix is generated. That is, mapping image features and text features to , , Then, add the mask item. Performing restricted attention calculations: (4) in, This represents element-wise multiplication. This mechanism forces the model to activate and focus on extremely faint potential microaneurysms in fundus images only when the text prompts a high-risk biochemical abnormality, achieving precise interaction from medical history to symptom identification.
[0025] Sparse Low-Rank Tensor Fusion with Medical Constraints: To extract higher-order combinatorial information after interactive features, conventional bilinear (tensor) outer products are used instead of traditional methods. This would generate a large number of redundant cross terms; this invention designs a constrained tensor fusion method, which applies Tucker low-rank decomposition (…). Based on this, we... Applying The sparse regularization penalty forces the model to automatically compress the weights of the interaction terms between "non-lesion image features and ordinary text features" to zero during training, which ensures the quality of the fused feature vector. By retaining only feature pairs with strong clinicopathological coupling (such as the strong correlation between "high-frequency fundus hemorrhage texture" and "HbA1c>8%)", the model's robustness in classifying complex complication samples is greatly improved. The fused features are then fed into the fully connected layer to output the classification results.
[0026] Diagnostic Output and Loss: Feature Fusion Output the DR classification probability using a Softmax classifier. In a classification task with 5 categories (from no DR to proliferating DR), the categories are... The predicted probability can be expressed as: (5) in, For category The original score. Cross-entropy loss is used during training. ( (True label) or focus loss can be used to address class imbalance. Diagnostic outputs can be expanded as needed to include probability vectors, confidence scores, or stage labels, along with lesion localization results or clinical recommendation text.
[0027] Joint Causal Tracing: To improve system interpretability, a visualization method is introduced. Unlike traditional approaches that separately generate heatmaps from images using Grad-CAM and visualize attention weights for text, this system introduces a bidirectional back-inference mechanism based on a high-order fusion tensor, calculating fusion decision features. Compared to the joint partial derivatives of the image and text branches in the early stage, the system can output a structured "joint attribution tuple". The final feedback to doctors is not only a heat map, but also a dynamic and linked interpretation, such as: <Microvascular dilation at coordinates (x,y) in the fundus region is thermally highlighted, associated with the clinical entity: 10-year course of diabetes, joint contribution: 85%>. This joint interpretation mechanism directly reflects the doctor's clinical diagnostic thinking (combining medical history with image interpretation), which greatly enhances the transparency and medical credibility of AI-assisted decision-making.
[0028] This invention effectively utilizes the complementary information of the two modalities through deep fusion of fundus images and patient clinical information. It can capture minute lesions in the images and improve early DR detection capabilities by combining background information such as blood glucose and disease course. Furthermore, the invention's two-stage sparse interaction fusion strategy based on pathological feature anchors (Med-CMA and sparse low-rank tensor fusion) deeply explores the correlation between "image features and clinical indicators," which helps to handle complex cases. Its classification accuracy is higher (which can be verified through large-scale clinical data), and its false negative rate is lower, especially sensitive to minor lesions. On the other hand, this invention generates joint attribution tuples containing heatmap and text entity linkages by combining image-text causal tracing mechanism (based on bidirectional back-inference of high-order fusion tensors), which greatly enhances the clinical transparency and credibility of diagnostic results.
[0029] In clinical applications, this invention can provide doctors with rapid diagnostic suggestions, reduce the burden of manual image reading, lower the false negative rate, and accelerate the efficiency of retinal screening for diabetic patients. It can also realize "intelligent" DR screening in grassroots areas or areas with limited resources, which helps to detect and treat diseases early and reduce the social medical burden.
Claims
1. A method for auxiliary diagnosis of diabetic retinopathy based on multimodal fusion, characterized in that: Includes the following steps: S1. Perform preprocessing operations on the patient's fundus images; S2. Perform preprocessing operations on the patient's text information; S3. Perform feature extraction on the preprocessed fundus image to obtain the image feature vector; S4. Perform feature extraction on the preprocessed text information to obtain the text semantic vector; S5. A spatially guided gating matrix is generated through multilayer perceptron mapping, and a restricted tensor fusion model is established. The spatially guided gating matrix is then used to perform high-order sparse tensor fusion of image feature vectors and text semantic vectors. Simultaneously, based on the application of Tucker low-rank decomposition, a... Sparse regularization penalty is applied to obtain the fused feature vector and the diagnostic category prediction probability. S6. Combine the two-way reverse reasoning mechanism to perform image and text joint tracing of the diagnostic results, and output the diagnostic heat map and its corresponding dynamic linkage interpretation.
2. The method for auxiliary diagnosis of diabetic retinopathy based on multimodal fusion as described in claim 1, characterized in that: Step S1 specifically includes the following steps: S11. Perform spherical correction and retinal region cropping on the fundus image to remove black background and artifacts; S12. Perform illumination normalization processing, the calculation formula is as follows: ； in, These are the pixel values of the original fundus image. , These represent the mean and standard deviation of the fundus images, respectively. These are the standardized pixel values of the fundus image; S13. Perform data augmentation on the fundus images after illumination normalization to expand the training samples.
3. The method for auxiliary diagnosis of diabetic retinopathy based on multimodal fusion as described in claim 2, characterized in that: The data augmentation operations in step S13 include random rotation, horizontal flipping, brightness adjustment, and contrast adjustment.
4. The method for auxiliary diagnosis of diabetic retinopathy based on multimodal fusion as described in claim 1, characterized in that: Step S2 specifically includes the following steps: S21. Perform word segmentation and named entity recognition on the text information to extract key structured entities; S22. Encode key structured entities and normalize numerical indicators into standardized features; S23. Use TF-IDF or word vectors to further extract features to obtain the patient's clinical text feature vector.
5. The method for auxiliary diagnosis of diabetic retinopathy based on multimodal fusion as described in claim 1, characterized in that: Step S3 specifically includes the following steps: S31. Using a pre-trained convolutional neural network as the backbone, a channel attention module is introduced into the convolutional neural network to adaptively adjust the weights of each channel. The calculation formula is as follows: ； in, This is the channel statistics vector obtained from global average pooling. , For a trainable weight matrix, , These are the ReLU and Sigmoid activation functions, respectively. S32. The formula for calculating the residual of a convolutional neural network is: ； in, Represents a nonlinear mapping of a set of convolutional layers; S32. The preprocessed fundus image is processed by a convolutional neural network with an added attention mechanism to extract features and obtain the image feature vector.
6. The method for auxiliary diagnosis of diabetic retinopathy based on multimodal fusion as described in claim 1, characterized in that: Step S4 specifically includes the following steps: S41. Use the Chinese BERT model to obtain context-sensitive word vectors of clinical text feature vectors; S42. Combine context-sensitive word vectors and model them using bidirectional LSTM or multi-layer Transformer to output text semantic vectors.
7. The method for auxiliary diagnosis of diabetic retinopathy based on multimodal fusion as described in claim 1, characterized in that: Step S5 specifically includes the following steps: S51. Let the image feature vector be... The text semantic vector is The medical entity features are mapped using a multilayer perceptron and spatially broadcast to the same spatial resolution as the image feature vector. After Sigmoid activation, a spatially guided gating matrix is generated. ; S52, Map the image feature vector and the text semantic vector to... , , Then, a spatial guidance gating matrix is added. The formula for calculating restricted attention is as follows: ； in, This represents element-wise multiplication; S53. Establish a constrained tensor fusion model, based on the application of Tucker low-rank decomposition, and in the core tensor... Apply on The sparse regularization penalty compresses the weights of the interaction terms between non-lesion image features and ordinary text features to zero during the training process of the restricted tensor fusion model. S54. The image feature vector and text semantic vector are fused using a restricted tensor fusion model to obtain fused features. These fused features are then input into a Softmax classifier, which outputs the predicted class probabilities. The formula is as follows: ； in, For category The original score.
8. The method for auxiliary diagnosis of diabetic retinopathy based on multimodal fusion as described in claim 1, characterized in that: In step S6, the joint partial derivatives of the fusion features with respect to the fundus image and text information are calculated, and a structured diagnostic heatmap and its corresponding dynamic linkage interpretation are output based on the joint partial derivatives.