Ophthalmic Disease Image Recognition Method Based on Dual-Stream EfficientNet and Decision Tree

Abstract

An ophthalmic disease image recognition method based on dual-stream EfficientNet and decision tree, which relates to the field of medical detection technology, includes the following steps: collecting multi-modal data of the patient's binocular samples for vectorization preprocessing, inputting the preprocessed binocular images into the dual-stream EfficientNet-b3 network to extract and fuse to generate a binocular comprehensive image feature vector, performing interactive fusion on the binocular comprehensive image feature vector and the text feature vector, and then continuing to perform weighted fusion with the clinical data vector to obtain the final fusion feature vector, performing dynamic gating network fusion on the final fusion feature vector, inputting the fused decision feature vector into the decision tree to judge the final result; the present invention discloses an ophthalmic disease image recognition method and model based on cross-modal features and decision tree, training the decision tree to judge the fused features, effectively improving the reliability and accuracy of the recognition results for complex ophthalmic disease images.

Description

Technical Field

[0001] The present invention relates to the field of medical detection technologies, and particularly to an ophthalmic disease image recognition method based on dual-stream EfficientNet and decision tree. Background Art

[0002] Early diagnosis of ophthalmic diseases is crucial for preventing vision loss and reducing the risk of blindness. However, traditional diagnostic methods mainly rely on the experience of professional doctors and manual film reading. Although this method can ensure the accuracy of diagnosis to a certain extent, it has problems such as low efficiency and strong subjectivity, and it is difficult to meet the needs of large-scale screening and rapid diagnosis. In recent years, with the rapid development of artificial intelligence and deep learning technologies, image recognition systems based on deep learning have gradually been applied to the field of ophthalmology, providing new possibilities for auxiliary diagnosis. However, existing image recognition technologies still have some limitations, restricting their wide application in clinical practice.

[0003] Firstly, existing systems have deficiencies in the utilization of multi-modal data. Most systems only rely on fundus images for recognition, ignoring the important influence of clinical data such as the age, gender, and medical history of patients on disease diagnosis. The combination of these clinical data and fundus images can provide more comprehensive information for disease diagnosis, thereby improving the accuracy of diagnosis.

[0004] Secondly, the interpretability of existing deep learning models is poor. Deep learning models are usually regarded as "black boxes", and their decision-making processes are difficult for doctors to understand and trust. This restricts their wide application in clinical practice to a certain extent, especially in cases where doctors need to further analyze and judge the diagnosis results.

[0005] In addition, the performance of existing models in multi-label classification tasks also has limitations. Fundus lesions are usually accompanied by multiple complications, but the recognition accuracy of existing models for concurrent diseases is insufficient, and it is difficult to meet the clinical need for auxiliary diagnosis of multiple diseases simultaneously.

[0006] In the prior art, single-image models (such as ResNet, traditional EfficientNet) only process image data, cannot fuse clinical information, and lack decision interpretability. Some systems add fixed threshold rules after deep learning output, but this method has poor flexibility and cannot adapt to individual differences of patients. At the same time, the data imbalance problem is also an important challenge faced by the prior art. The sample size of rare disease labels (such as hypertensive retinopathy) is insufficient, resulting in a high misrecognition rate of these diseases by the model. Summary of the Invention

[0007] In view of this, the present invention proposes an ophthalmic disease image recognition method based on dual-stream EfficientNet and decision tree, as well as a multi-modal data-driven decision tree (DEDT) model based on this method. By embedding a trainable decision tree layer in the network structure of the model, fusing image features and clinical data, combining multi-modal fusion technology, jointly optimizing image and non-image data, and introducing a splitting criterion for clinical logic embedding, the accuracy, interpretability, and robustness of the image recognition results are effectively improved. At the same time, the resource efficiency of the system is optimized, enabling it to better assist in the diagnosis of ophthalmic diseases.

[0008] To solve the above-mentioned at least one technical problem, the technical solution provided by the present invention is an ophthalmic disease image recognition method based on dual-stream EfficientNet and decision tree, including the following steps:

[0009] Step S1: Collect multi-modal data of the binocular samples of patients, preprocess the multi-modal data to obtain vectorized multi-modal data, where the multi-modal data includes fundus images, text descriptions, and clinical data;

[0010] Step S2: Input the preprocessed binocular fundus images into two groups of dual-stream EfficientNet-b3 networks with shared weights to extract feature vectors, and fuse the binocular image features to generate a binocular comprehensive image feature vector;

[0011] Step S3: Interact and fuse the binocular comprehensive image feature vector and the text feature vector;

[0012] Step S4: Perform weighted fusion on the image-text fusion feature vector obtained after interaction and fusion and the clinical data vector to obtain a final fusion feature vector;

[0013] Step S5: According to the clinical logic embedding node splitting rule, perform dynamic gating network fusion on the clinical logic embedding node splitting rule and the final fusion feature vector to obtain a decision feature vector;

[0014] Step S6: Input the decision feature vector into the decision tree, judge the flow direction of the decision feature vector in the branches of each intermediate node of the decision tree, so as to match the binocular samples of the patient corresponding to the decision feature vector to the recognition results at the end of the decision tree, and output the final result;

[0015] Among them, the decision tree is composed of multiple connected nodes. The beginning of the decision tree is the root node, and the end of the decision tree is multiple groups of leaf nodes. A group of subtrees are formed by connecting the root node and any leaf node through multiple groups of intermediate nodes. Each group of intermediate nodes contains left and right branches, and each branch is connected to the subsequent adjacent intermediate nodes.

[0016] The technical effects achieved by the present invention are:

[0017] 1. The present invention discloses an ophthalmic disease image recognition method and model based on cross-modal features and decision trees. By fusing multi-modal data, correlating multi-factor features affecting diseases, and training a decision tree to judge the fused features, the synergistic influence of multi-factor features on disease judgment is fully considered, thereby effectively improving the reliability and accuracy of the image recognition results for complex ophthalmic diseases.

[0018] 2. The present invention emphasizes the interaction of cross-modal features, can fully simulate the comprehensive diagnostic thinking in the actual medical diagnosis process, ensure that the fused features not only contain rich diagnostic information but also conform to the pathophysiological laws of ophthalmic diseases; at the same time, a deep connection is established between image features and clinical indicators, which can further explore complex medical laws that are difficult to intuitively grasp in routine diagnosis and treatment, ensuring the comprehensiveness of the recognition results and enabling it to better assist the doctor's diagnosis process.

[0019] 3. The present invention realizes the adaptive allocation of modal weights through a gated attention mechanism. The system can dynamically adjust the feature extraction strategy according to the quality of the input data, automatically enhance the weight of structured clinical features when the image quality is poor, and focus on image and text feature analysis when clinical data is missing, ensuring that the most discriminative feature combination can be extracted under any data conditions, thereby significantly improving the robustness of the model in a non-ideal data environment and expanding the applicable range of the model.

[0020] 4. The present invention introduces a clinical logic embedding node splitting rule to guide the distribution direction of nodes in the decision tree, introduces actual medical diagnosis experience in the image recognition process and converts it into a learnable node splitting rule, not limited to numerical judgment, greatly improving the coincidence degree between the decision path and the real clinical diagnosis process, and further improving the accuracy of image recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention, and therefore should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0022] Figure 1 It is a schematic diagram of the overall process of the image recognition method in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0023] The present invention will be further described in detail below in conjunction with the embodiments and the drawings.

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention. Therefore, the detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed present invention, but merely represents selected embodiments of the present invention.

[0025] See Figure 1 , an ophthalmic disease image recognition method based on dual-stream EfficientNet and decision tree, comprising the following steps:

[0026] Step S1: Collect multimodal data of the binocular samples of patients, preprocess the multimodal data to obtain vectorized multimodal data, wherein the multimodal data includes fundus images, text descriptions, and clinical data.

[0027] The method steps for preprocessing the fundus image are as follows:

[0028] Step S11-1: Use a circular detection algorithm with improved Hough transform for sub-pixel level pupil localization to determine the pupil center position and radius.

[0029] The system first preprocesses the input multimodal data, including fundus images, text descriptions, and clinical structured data. For the fundus image, the system uses an improved circular Hough transform to achieve sub-pixel level pupil localization and determine the pupil center position and radius, as shown in Equation (1):

[0030] (1);

[0031] In Equation (1), x * represents the sub-pixel level abscissa of the pupil center; y * represents the sub-pixel level ordinate of the pupil center; r * represents the radius of the pupil; I ( x + r cosθ , y + r sinθ ) represents the pixel intensity value of the fundus image at the coordinate ( x + r cosθ , y + r sinθ ); θ represents the polar coordinate angle; λ represents the regularization coefficient; r represents the candidate value of the pupil radius;

[0032] The central position and radius of the pupil are accurately determined thereby.

[0033] Step S11-2: Based on the through-hole positioning result, perform dual-channel registration and alignment on the left and right eye images, and extract the region of interest.

[0034] Based on the pupil positioning result, the system also performs dual-channel registration on the left and right eye images to ensure image alignment, and extracts the region of interest (ROI), providing an accurate image basis for subsequent analysis.

[0035] Step S11-3: Use a blood vessel segmentation method based on Gabor filtering and morphological reconstruction to accurately segment the retinal blood vessels.

[0036] To better analyze the characteristics of retinal blood vessels, the system uses a blood vessel segmentation method based on Gabor filtering and morphological reconstruction, specifically as shown in Equation (2):

[0037] (2);

[0038] In Equation (2), G σ,θ represents the Gabor filter; B disk represents the morphological structure element; V seg represents the segmented retinal blood vessel image; I represents the original fundus image; Reconstruct represents the reconstruction operation; represents the element-wise dot product operation.

[0039] The method for preprocessing the text description is as follows:

[0040] Step S12-1: Perform standardization mapping of professional terms on the text description, and map the non-standard text description to the standard medical term library by minimizing the Levenshtein distance.

[0041] For the text description, the system first needs to perform standardization mapping of professional terms on these received left and right eye descriptions, and map the non-standard text description to the standard medical term library by minimizing the Levenshtein distance to ensure the accuracy and consistency of the text description, specifically as shown in Equation (16):

[0042] (16);

[0043] In Equation (16), desc std represents the standardized text description; desc raw represents the original text description; DRepresents a standard medical terminology library; d Is a set D An element in.

[0044] Step S12-2: Input the text description after standardized mapping into the BioBERT model to generate text feature vectors.

[0045] After that, input the standardized text description into the domain-adaptive BioBERT model to generate text feature vectors, as specifically shown in Equation (3):

[0046] (3);

[0047] In Equation (3), h text Represents the text feature vector, [CLS] represents the start token of the text, [SEP] represents the separator token of the text, Represents the semantic information containing the text description of the left eye, Represents the semantic information containing the text description of the right eye; BioBERT represents a pre-trained deep learning model based on the BERT architecture; [0] represents taking the first feature vector of the output.

[0048] Thus, embed the text descriptions of the left and right eyes into a high-dimensional space, where "[CLS]" and "[SEP]" are special tokens used to indicate the start and separation of the text. The generated text feature vectors will contain the semantic information of the text descriptions of the left and right eyes, providing the input of the text modality for subsequent multimodal fusion.

[0049] The method for preprocessing clinical data is as follows:

[0050] Step S13-1: Compensate for missing values in clinical data.

[0051] For clinical data, first process the missing values based on the missing value compensation method of population distribution, as specifically shown in Equation (4):

[0052] (4);

[0053] In Equation (4), w k Represents the population weight; x obs Represents the observed features; y k Represents the population label; p Represents under the observed features x obs Of the condition, the population label y k Of the conditional probability; ximp Represents the expected value of the missing value; K Represents the total number of group labels; x Represents the missing features to be estimated; Represents that given the group label y k the expected value of the missing feature x under this condition; Represents that given the observed features x obs the conditional probability of the group label y k under this condition.

[0054] Weighted summation is performed through the conditional probability distribution of the group label to compensate for the missing clinical data.

[0055] Step S13-2: Cross-combine the clinical data after missing value compensation to generate a new clinical data vector.

[0056] Next, in order to better capture the complex relationships between clinical data, the system generates high-order feature crosses, and these high-order features will provide richer information for subsequent identification, as shown in Equation (5) specifically:

[0057] (5);

[0058] In Equation (5), X cross is the clinical data feature vector after crossing; X Represents the clinical data feature vector; Represents the set of features to be crossed; x k Represents the k th clinical data variable, k ∈ [i, j]; Represents the set of continuous features that need to be non-linearly transformed; f k ( x k ) represents the specific transformation function for the feature x k ; x i and x j respectively represent the i th and j th feature variables in the set of features to be crossed in the clinical data.

[0059] Among them, the features that need to be crossed include age, gender, and medical history; the features that need non-linear transformation include BMI, blood pressure, blood glucose level, and intraocular pressure; logarithmic transformation rules are used for BMI and blood glucose level transformation, and Sigmoid transformation rules are used for blood pressure and intraocular pressure transformation.

[0060] Step S2: Input the preprocessed binocular fundus images into two groups of two-stream EfficientNet-b3 networks with shared weights to extract feature vectors, and fuse the binocular image features to generate a binocular comprehensive image feature vector;

[0061] The preprocessed left and right fundus images are respectively input into two EfficientNet-b3 networks with shared weights. Through its efficient architecture and deep feature extraction ability, the EfficientNet-b3 network extracts the feature vectors of the left and right eye images respectively. During the extraction process, the SE (Squeeze-and-Excitation) attention module in the network will focus on the pathological areas in the image and enhance the feature representation related to the disease. The feature vectors of the left and right eye images extracted from the EfficientNet-b3 network are preliminarily fused through a fusion module to generate a comprehensive image feature vector, as shown in Equation (17) specifically:

[0062] (17);

[0063] In Equation (17), represents the binocular comprehensive image feature vector; represents the left eye image feature vector; represents the right eye image feature vector; Fusion represents the fusion operation. The fused image feature vector not only contains the independent information of the left and right eyes but also captures the correlation information between the two eyes.

[0064] Step S3: Interactively fuse the binocular comprehensive image feature vector and the text feature vector;

[0065] The image features and text features are interactively fused, as shown in Equation (6) specifically:

[0066] (6);

[0067] In Equation (6), represents the image-text fusion feature vector, represents the binocular comprehensive image feature vector, represents the normalized text feature vector; LayerNorm represents the layer normalization operation; ATT represents the attention mechanism.

[0068] Among them, the method for normalizing the text feature vector is shown in Equation (7):

[0069] (7);

[0070] In Equation (7), represents the text feature vector, is the L2 norm of

[0071] In this way, the interaction enhancement between image features and text features is realized, where ATT represents the attention mechanism, and LayerNorm is the layer normalization operation used to stabilize the training process. The feature vector will fuse the information of the image and text modalities.

[0072] When calculating ATT( f img , f text ), the similarity S img,text is introduced to adjust the attention weights, as shown in Equation (18) specifically:

[0073] (18);

[0074] In Equation (18), represents the similarity between the image feature and the text feature. And the calculation method of the similarity is shown in Equation (19):

[0075] (19);

[0076] In Equation (19), S ij represents the similarity between the i-modal feature and the j-modal feature, where and respectively represent the feature vectors of different modalities, is the attention weight matrix, is the feature dimension, so as to quantify the similarity between the image feature modality and the text feature modality. When the similarity S img,text is higher, the attention interaction between the image feature and the text feature will be more significant.

[0077] Step S4: Weightedly fuse the image-text fusion feature vector obtained after the interaction fusion with the clinical data vector to obtain the final fusion feature vector;

[0078] The weighted fusion of the image-text feature after the interaction and the clinical data is specifically shown in Equation (8):

[0079] (8);

[0080] In formula (8), f fused represents the final fused feature vector, f img-text represents the image-text fused feature vector, f clin represents the normalized cross-clinical data vector, w clin represents the weight vector of clinical data, represents the dimension of d； MLP represents generating the final fused feature vector through a multi-layer perceptron.

[0081] The method for normalizing the cross-clinical data feature vector is shown in formula (9):

[0082] (9);

[0083] In formula (9), X cross is the cross-clinical data feature vector, is X the L2 norm of cross;

[0084] Thereby generating the final fused feature vector. In formula (8), MLP is a multi-layer perceptron, ⊙ represents element-wise multiplication, and w clin is the weight vector of clinical data. Weighted fusion ensures that clinical data plays an important role in the final decision.

[0085] For the clinical data weight vector w clin , it is mainly determined through model training, including the following steps:

[0086] Randomly initialize the weight vector w clin from a normal distribution. During the training process, with the goal of minimizing the loss function, dynamically adjust the weight vector w clin backward. Among them, the total loss function of the model can be defined as formula (20):

[0087] (20);

[0088] In formula (20), represents the classification loss function, represents the regularization term, which is used to prevent overfitting, λ 0 represents the regularization coefficient, represents the total loss function of the model.

[0089] In each iteration of the model, the gradient of the loss function with respect to w needs to be calculated according to Equation (21): clin ;

[0090] (21);

[0091] In Equation (21), represents the gradient of the classification loss with respect to w clin ; represents the gradient with respect to w clin ;

[0092] On this basis, w clin is updated by the gradient descent method, as specifically shown in Equation (22):

[0093] (22);

[0094] In Equation (22), η represents the learning rate, which is determined according to actual needs; w clin2 , w clin1 are the clinical data weight vectors before and after update, respectively.

[0095] When the total loss function of the model no longer significantly decreases after multiple consecutive iterations, it can be considered that the model has converged. At this time, the update is stopped, and the latest weight vector at this time is used as the clinical data base weight vector .

[0096] After that, by combining the clinical data base weight vector with the modality weight of the clinical data, the clinical data weight vector w clin can be calculated, as specifically shown in Equation (23):

[0097] (23);

[0098] In Equation (23), is the modality weight of the clinical data.

[0099] For the modality weight of the clinical data, its calculation method is shown in Equation (24):

[0100] (24);

[0101] In Equation (24), represents the weight of each modality, W m and b m represent the modality-specific weight matrix and bias term, respectively, , m represents the data of different modalities, represents a fundus image, represents a text description, represents clinical data; f m represents feature vectors of different modalities; GeLU represents a non - linear activation function; σ represents the sigmoid function; represents the global weight vector;

[0102] When m is clin the object calculated is the modality weight of clinical data .

[0103] Among them, the global weight vector can be obtained by training the model using the same gradient descent method as the clinical data weight vector w clin . Similarly, the weight vectors of other modalities, such as the weight vector of the fundus image w img , can also be determined using the same method as the clinical data weight vector w clin .

[0104] Step S5: According to the clinical logic embedding node splitting rule, perform dynamic gated network fusion on the clinical logic embedding node splitting rule and the final fused feature vector to obtain a decision feature vector;

[0105] Introducing the clinical logic embedding node splitting rule can transform the medical diagnosis path into a learnable node splitting rule. Among them, the clinical logic embedding node splitting rule is specifically shown in Equation (10):

[0106] , (10);

[0107] In Equation (10), Split( f ) represents the clinical logic embedding node splitting rule, represents the indicator function, T age represents the age threshold, w T img represents the transposed matrix of the image feature weight vector, w T clin represents the transposed matrix of the clinical data weight vector, f img represents the binocular comprehensive image feature vector, f clin represents the normalized cross - clinical data vector, denotes a dimension of k ; Age represents age.

[0108] Introducing the clinical logic embedding node splitting rule enables the model to dynamically adjust the weights of features according to the patient's age, realizing the clinical thinking modeling of "emphasizing imaging features for young patients and clinical indicators for elderly patients", thus better simulating the clinical diagnosis logic.

[0109] Among them, the optimal age threshold can be obtained by training the model using a data-driven method. First, analyze the training dataset to understand the age distribution of patients and the disease distribution in different age groups. Based on these analysis results, select a series of candidate age thresholds, with each candidate value being every 5 years. Then, use the cross-validation method to divide the dataset into a training set and a validation set, train the model for each candidate threshold, and evaluate its performance on the validation set, mainly examining indicators such as accuracy and recall. Finally, select the candidate threshold that optimizes the model performance as the age threshold.

[0110] For the method of gated network fusion, it mainly includes the following steps:

[0111] Step S51: Map the clinical logic embedding node splitting rule Split( f ) to the same dimension as the final fused feature vector f fused , and the specific mapping process is shown in Equation (11):

[0112] (11);

[0113] In Equation (11), h split represents the high-dimensional projection of the clinical logic embedding node splitting rule, b s represents the bias term that compensates for the offset of the feature distribution after projection, W s is the weight matrix that maps the clinical logic embedding node splitting rule Split( f ) to the high-dimensional space, with a dimension of d × k ;

[0114] Step S52: Calculate the weight balance coefficient between the data-driven feature and the clinical data to generate the gating information, and the specific process is shown in Equation (12);

[0115] (12);

[0116] In Equation (12), represents the weight balance coefficient between the data-driven feature and the clinical data, σrepresents the sigmoid function, represents the weights of the fully connected layer for calculating the gating signal, represents that the input dimension is 2 d , and the output dimension is d , represents the bias term of the gating mechanism, represents the dimension of d ;

[0117] Step S53: Weightedly merge the high-dimensional projection of the clinical logic embedding node splitting rule and the final fusion feature vector, and output the decision feature vector. The specific process is shown in Equation (13):

[0118] (13);

[0119] In Equation (13), f final represents the decision feature vector.

[0120] Through the fusion of the gating network, the final fusion feature vector f fused and the clinical logic embedding node splitting rule Split( f ) are combined. The clinical logic embedding node splitting rule Split( f ) is the result of dynamically adjusting the feature weights according to the patient's age, which directly reflects the clinical logic. The combination of the two enables the model to utilize multi-modal information and clinical logic simultaneously during the decision-making process, thereby effectively improving the accuracy and interpretability of the recognition results.

[0121] Step S6: Input the decision feature vector into the decision tree, determine the flow direction of the decision feature vector at each intermediate node branch of the decision tree, so as to match the binocular samples of the patient corresponding to the decision feature vector to the recognition results at the end of the decision tree, and output the final result;

[0122] Among them, the decision tree is composed of multiple connected nodes. The beginning of the decision tree is the root node, and the end of the decision tree is multiple groups of leaf nodes. The root node and any leaf node are connected by multiple groups of intermediate nodes to form a subtree. Each group of intermediate nodes contains left and right branches, and each branch is connected to the subsequent adjacent intermediate nodes.

[0123] For every group of intermediate nodes, there is a left and right branch. Therefore, starting from the root node at the beginning of the decision tree, different branches can be selected at each intermediate node to reach different leaf nodes at the end.

[0124] The decision feature vector flows sequentially in the decision tree in the order of the root node, intermediate node, and leaf node. The root node is the entry port of the decision feature vector, and different disease diagnosis results are randomly assigned at the leaf nodes.

[0125] Among them, the method for determining the flow direction of the decision-making feature vector in the branches of each intermediate node of the decision tree is as follows: calculate the flow probabilities of the decision-making feature vector in the left and right groups of branches of the intermediate node, and take the side with the larger probability as the flowing branch. The specific calculation of the flow probability is shown in Equation (14):

[0126] (14);

[0127] In Equation (14), P left represents the probability that the decision-making feature vector flows to the left branch of the intermediate node, P right represents the probability that the decision-making feature vector flows to the right branch of the intermediate node, represents the transposed matrix of the splitting weight vector; σ () represents the Sigmoid function, which is used to map the input value to the interval (0, 1) to calculate the probability;

[0128] The probability that the decision-making feature vector flows from the root node along the subtree to any leaf node is shown in Equation (15):

[0129] (15);

[0130] In Equation (15), P path represents the probability that the decision-making feature vector flows from the root node along the subtree to any leaf node, represents the th splitting node in the decision tree, represents the total number of nodes on the path from the root node to the current leaf node; represents at the th splitting node, the probability that the decision-making feature vector flows to the left subtree or the right subtree; dir represents the flow direction of the decision-making feature vector at each splitting node of the decision tree; left represents the left subtree direction; right represents the right subtree direction.

[0131] Any leaf node at the end of the decision tree corresponds to a randomly set disease diagnosis result. When the decision-making feature vector passes through multiple groups of intermediate node branches in the decision tree and finally reaches a certain leaf node, the binocular samples of the patient corresponding to the decision-making feature vector will be matched to the diagnosis result at the end of the decision tree and output as the final result of ophthalmic disease image recognition.

[0132] To further improve the interpretability of the system, the uncertainty of the decision path can also be quantified through the decision path entropy, which is beneficial to intuitively reflect the quality of the decision path. The specific calculation of the decision path entropy is shown in Equation (25):

[0133] (25);

[0134] Among them, H path represents the decision path entropy, P ( path t ) represents the probability of the sample on the th path in the decision tree. The decision path entropy H path The lower it is, the more certain the decision path is, and the better the interpretability of the model. T represents the total number of decision paths, t represents the index of the decision path.

[0135] It can be seen that the probability of the decision feature vector moving along any intermediate node branch is mainly determined by the split weight vector w s . The split weight vector w s is mainly determined during model training, and the specific method is as follows:

[0136] 1) Randomly generate the split weight vectors w s of all intermediate nodes. During the training process, when the decision feature vector passes through any intermediate node, perform the modality consistency detection of multimodal data, as shown in Equation (26):

[0137] (26);

[0138] In Equation (26), κ represents the decision consistency between different modalities, is the Jensen-Shannon divergence, which is used to measure the difference between the decision probability distributions of different modalities, P i represents the decision probability distribution of the i th modality, P j represents the decision probability distribution of the j th modality.

[0139] Among them, the calculation method of the decision probability distribution of any modality is to map the feature vector of this modality to the decision probability distribution through a classifier (softmax layer), as shown in Equation (27):

[0140] (27);

[0141] In Equation (27), P mis the decision probability distribution of any modality; MLP is a multi-layer perceptron; f m is the feature vector of any modality.

[0142] The consistency detection result reflects the coordination degree among data of each modality. When κ < 0.5, it indicates that the coordination among modality data is poor and the splitting parameter needs to be adjusted; when κ ≥ 0.5, it indicates that the coordination among modality data is good and the current parameter can be maintained.

[0143] The correction of the splitting parameter in the decision tree is mainly carried out by Equation (28):

[0144] (28);

[0145] In Equation (28), Δ w s represents the correction value of the splitting weight vector, represents the learning rate, represents the consistency loss function, represents the gradient of the consistency loss function.

[0146] 2) When the modality consistency detection result of the multi-modal data meets the above requirements, the splitting weight vector w s of this intermediate node is determined; while when the modality consistency detection result of the multi-modal data does not meet the above requirements, the splitting weight vector w s is adjusted by Equation (28) to obtain the adjusted splitting weight vector w s of this intermediate node, and the decision consistency κ of this node is recalculated in the new round of training, and the steps are repeated until the modality consistency detection result meets the requirements. And so on, train to obtain the results where all intermediate nodes meet the consistency detection, and use them as the decision tree for image recognition.

[0147] Embodiment: To verify the effectiveness of the above multi-modal data-driven decision tree (DEDT) model, 2885 groups of fundus image data were used to conduct image recognition experiments based on the method of the present invention. These data contain the age and gender labels of patients, and through the data augmentation technology, the data set was expanded to 15460 images.

[0148] At the same time, ResNet50 and the traditional EfficientNet-b3 were also selected as comparison models to conduct image recognition experiments. The specific results are shown in Table 1:

[0149] Table 1 Comparison experiment of fundus image data recognition

[0150]

[0151] As can be seen from Table 1, the DEDT model achieved an accuracy of 92.03% in multi-label classification, while the accuracies of ResNet50 and the traditional EfficientNet-b3 were 88.40% and 89.63% respectively. For the recall rate of glaucoma, the DEDT model reached 89%, while ResNet50 and the traditional EfficientNet-b3 were 72% and 75% respectively. In terms of the interpretability score, the score of the DEDT model was 4.5 / 5, much higher than 2.1 / 5 of ResNet50 and 2.3 / 5 of the traditional EfficientNet-b3. These results indicate that the DEDT model is superior to the prior art in terms of recognition accuracy, rare disease recognition ability and clinical interpretability. At the same time, the clinical interpretability of the model in the present invention has been significantly enhanced. The decision tree path provides a visual recognition basis for doctors, enabling doctors to better understand and trust the results, and thus better assisting doctors in making diagnoses. In addition, the model in the present invention also performs well in terms of resource efficiency. The single-case recognition time is less than 3 seconds, supporting deployment in primary hospitals, and the GPU video memory occupancy is less than 2GB. This enables the system to be applicable not only to large medical institutions but also to be widely used in primary hospitals with limited resources, improving the accessibility of medical services.

[0152] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of the present invention should be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. An ophthalmic disease image recognition method based on dual-stream EfficientNet and decision tree, characterized in that, The following steps are involved: Step S1: collecting multimodal data of samples of both eyes of a patient, and preprocessing the multimodal data to obtain vectorized multimodal data, wherein the multimodal data includes fundus images, text descriptions, and clinical data; Step S2: Input the preprocessed binocular fundus images into two sets of weight-sharing dual-stream EfficientNet-b3 networks to extract feature vectors, and fuse binocular image features to generate binocular comprehensive image feature vectors; Step S3: interactively fuse the binocular integrated image feature vector and the text feature vector; Step S4: weighted fusion of the image-text fusion feature vector obtained after interactive fusion and the clinical data vector to obtain a final fusion feature vector; Step S5: According to the clinical logic embedding node splitting rule, a dynamic gating network fusion is performed on the clinical logic embedding node splitting rule and the final fusion feature vector to obtain a decision feature vector; Step S6: input the decision feature vector into the decision tree, determine the flow direction of the decision feature vector in each intermediate node branch of the decision tree, so as to match the patient's binocular samples corresponding to the decision feature vector to the recognition result at the end of the decision tree, and output the final result; Among them, the decision tree is composed of multiple nodes connected together. The beginning of the decision tree is the root node, and the end of the decision tree is multiple groups of leaf nodes. The root node and any leaf node are connected to a group of subtrees through multiple groups of intermediate nodes. Each group of intermediate nodes contains two groups of left and right branches, and each group of branches is connected to the subsequent adjacent intermediate nodes.

2. The ophthalmic disease image recognition method based on dual-stream EfficientNet and decision tree according to claim 1, wherein: The clinical data included age, gender, BMI, blood pressure, blood glucose level, intraocular pressure and medical history.

3. The ophthalmic disease image recognition method based on dual-stream EfficientNet and decision tree according to claim 1, wherein: The method for preprocessing the fundus image in step S1 comprises the following steps: Step S11-1: Use the improved Hough transform circle detection algorithm to perform sub-pixel pupil positioning to determine the pupil center position and radius, as shown in formula (1): （1）； In formula (1), x * represents the sub-pixel abscissa of the pupil center; y * represents the sub-pixel ordinate of the pupil center; r * represents the radius of the pupil; I ( x+rcosθ , y+rsinθ ) represents the pixel intensity value of the fundus image at the coordinate ( x+rcosθ , y+rsinθ ); θ represents the polar coordinate angle; λ represents the regularization coefficient; r represents the candidate value of the pupil radius; Step S11-2: Based on the through-hole positioning result, dual-channel registration and alignment are performed on the left and right eye images to extract the region of interest; Step S11-3: Use a blood vessel segmentation method based on Gabor filtering and morphological reconstruction to accurately segment the retinal blood vessels, as shown in formula (2): （2）； In formula (2), G σ,θ represents a Gabor filter; B disk represents a morphological structuring element; V seg represents the segmented retinal blood vessel image; I represents the original fundus image; Reconstruct represents the reconstruction operation; represents the element-wise dot product operation.

4. The ophthalmic disease image recognition method based on dual-stream EfficientNet and decision tree according to claim 1, characterized in that: The method for preprocessing the text description in step S1 comprises the following steps: Step S12-1: Performing a standardized mapping of professional terms on the text description, mapping the non-standard text description to a standard medical terminology library by minimizing the Levenshtein distance; Step S12-2: Input the standardized mapped text description into the BioBERT model to generate a text feature vector, as shown in formula (3): （3）； In formula (3), h text represents the text feature vector; [CLS] represents the start token of the text; [SEP] represents the separator token of the text; left_desc represents the semantic information containing the text description of the left eye; right_desc represents the semantic information containing the text description of the right eye; BioBERT represents a pre-trained deep learning model based on the BERT architecture; [0] represents taking the first feature vector of the output.

5. The ophthalmic disease image recognition method based on dual-stream EfficientNet and decision tree according to claim 2, characterized in that: The method for preprocessing clinical data in step S1 comprises the following steps: Step S13-1: Compensate for missing values in clinical data, as shown in formula (4): （4）； In formula (4), w k represents the group weight; x obs represents the observed features; y k represents the group label; p represents under the condition of the observed features x obs the conditional probability of the group label y k ; x imp represents the expected value of the missing value; K represents the total number of group labels; x represents the missing features to be estimated; represents under the condition of the given group label y k the expected value of the missing features x ; represents under the condition of the observed features x obs the conditional probability of the group label y k ; Step S13-2: Cross-combine the clinical data after missing value compensation to generate a new clinical data vector, as shown in formula (5): （5）； In formula (5), X cross is the clinical data feature vector after crossing; X represents the clinical data feature vector; represents the feature set to be crossed; x k represents the k th clinical data variable, k ∈ i , j ; represents the set of continuous features that need non - linear transformation; f k ( x k ) represents the transformation function for feature x k ; x i and x j respectively represent the i th and j th feature variables in the feature set to be crossed in the clinical data; Among them, the features that need to be crossed include age, gender, and medical history; the features that need non-linear transformation include BMI, blood pressure, blood glucose level, and intraocular pressure.

6. The ophthalmic disease image recognition method based on dual-stream EfficientNet and decision tree according to claim 1, wherein: The method of interacting and fusing the binocular comprehensive image feature vector and the text feature vector described in step S3 is shown in formula (6): （6）； In formula (6), f img-text represents the image-text fusion feature vector, f img represents the binocular comprehensive image feature vector, f text represents the normalized text feature vector; LayerNorm represents the layer normalization operation; ATT represents the attention mechanism; Among them, the method of normalizing the text feature vector is shown in formula (7): （7）； In formula (7), represents the text feature vector, is the L2 norm of 7. The ophthalmic disease image recognition method based on dual-stream EfficientNet and decision tree according to claim 1, wherein: The method of generating the final fused feature vector described in step S4 is shown in formula (8): （8）； In formula (8), f fused represents the final fusion feature vector, f img-text represents the image-text fusion feature vector, f clin represents the normalized cross-clinical data vector, w clin represents the weight vector of the clinical data, represents the dimension of d ; MLP represents generating the final fusion feature vector through a multi-layer perceptron; The method of normalizing the crossed clinical data feature vector is shown in formula (9): （9）； In formula (9), X cross is the clinical data feature vector after crossing, is X the L2 norm of cross.

8. The ophthalmic disease image recognition method based on dual-stream EfficientNet and decision tree according to claim 7, wherein: The clinical logic embedding node splitting rule described in step S5 is specifically shown in formula (10) as follows: , (10); In formula (10), Split( f ) represents the clinical logic embedding node splitting rule, represents the indicator function, T age represents the age threshold, w T img represents the transposed matrix of the image feature weight vector, w T clin represents the transposed matrix of the clinical data weight vector, f img represents the binocular comprehensive image feature vector, f clin represents the normalized cross-clinical data vector, represents the dimension of k ; Age represents age.

9. The ophthalmic disease image recognition method based on dual-stream EfficientNet and decision tree according to claim 8, characterized in that: The method of performing dynamic gating network fusion described in step S5 includes the following steps: Step S51: Embed the clinical logic into the node splitting rule Split( f ) and map it to the same dimension as the final fusion feature vector f fused . The specific mapping process is shown in Equation (11): （11）； In formula (11), h split represents the high-dimensional projection of the clinical logic embedding node splitting rule, b s represents the bias term for compensating the offset of the feature distribution after projection, W s is the weight matrix that maps the clinical logic embedding node splitting rule Split( f ) to the high-dimensional space, with dimensions of d × k ; Step S52: Calculate the weight balance coefficient between the data-driven feature and the clinical data to generate gating information. The specific process is shown in formula (12): （12）； In formula (12), represents the weight balance coefficient between the data-driven feature and the clinical data, σ represents the sigmoid function, represents the weight of the fully connected layer for calculating the gating signal, represents that the input dimension is 2 d , and the output dimension is d , represents the bias term of the gating mechanism, represents the dimension of d ; Step S53: Weightedly combine the high-dimensional projection of the clinical logic embedding node splitting rule and the final fused feature vector, and output the decision feature vector. The specific process is shown in formula (13): （13）； In formula (13), f final represents the decision feature vector.

10. The ophthalmic disease image recognition method based on dual-stream EfficientNet and decision tree according to claim 9, wherein: The decision feature vector flows sequentially in the decision tree in the order of the root node, intermediate nodes, and leaf nodes. Among them, the method of judging the flow direction of the decision feature vector in the branches of each intermediate node of the decision tree is: calculate the flow probabilities of the decision feature vector in the left and right groups of branches of the intermediate node, and take the side with the larger probability as the flowing branch. The specific calculation of the flow probability is shown in formula (14): （14）； In formula (14), P left represents the probability that the decision feature vector flows to the left branch of the intermediate node, P right represents the probability that the decision feature vector flows to the right branch of the intermediate node, represents the transpose matrix of the splitting weight vector; σ () represents the Sigmoid function, which is used to map the input value to the interval (0,1) to calculate the probability; The probability that the decision feature vector flows from the root node along the subtree to any leaf node is shown in formula (15): （15）； In formula (15), Ppath represents the probability that the decision feature vector flows from the root node along the subtree to any leaf node. represents the th splitting node in the decision tree, and represents the total number of nodes on the path from the root node to the current leaf node; represents the probability that the decision feature vector flows to the left subtree or the right subtree at the th splitting node; dir represents the flow direction of the decision feature vector at each splitting node of the decision tree; left represents the left subtree direction; right represents the right subtree direction.

Images