Ear disease prediction method based on multi-modal data fusion and confidence evaluation

Abstract

The present application relates to the field of otology disease prediction, and particularly relates to an otology disease prediction method based on multi-modal data fusion and confidence evaluation. The technical scheme comprises: collecting an ear endoscope digital image, wideband tympanometry measurement data, and structured or unstructured clinical information; performing integrity check and quality check on the collected data, checking whether the ear endoscope digital image resolution meets the minimum pixel requirement and whether the information is complete; then performing otology data preprocessing and multi-modal feature extraction, extracting an ear endoscope digital image feature vector, a feature vector representing wideband tympanometry data, and a clinical information feature vector; after feature extraction, fusion is performed, and after fusion, confidence evaluation and recurrence risk prediction are performed. The present application significantly improves the accuracy of otology disease prediction by deeply fusing image, physiological signal and clinical text three modal data through an attention mechanism. The present application is suitable for otology disease prediction.

Description

Technical Field

[0001] This invention relates to the field of ear disease prediction, and more specifically to an ear disease prediction method based on multimodal data fusion and confidence assessment. Background Technology

[0002] Diagnosing ear diseases, especially otitis media spectrum disorders, is a crucial aspect of clinical practice. Traditional diagnostic methods rely on visual examination by physicians using an otoscope, and their accuracy is highly dependent on the physician's clinical experience. Studies have shown that even otolaryngologists experience fluctuations in diagnostic accuracy.

[0003] To improve the objectivity and accuracy of diagnosis, computer-aided diagnostic techniques, particularly deep learning-based methods, have been introduced into this field. Current research primarily employs convolutional neural networks for automatic classification of endoscopic ear images. For example, using an ensemble convolutional neural network model to perform six-class classification on a database containing multiple images has achieved a high average accuracy. Alternatively, deep metric learning techniques have been applied to differentiate between acute otitis media and effusion-related otitis media, which share similar clinical characteristics, achieving high accuracy on the defined tasks.

[0004] However, existing technologies still have the following defects and shortcomings:

[0005] The information fusion mechanism is rudimentary and fails to simulate the comprehensive diagnostic logic of experts: existing models are mostly limited to image analysis or simply integrate two data sources. They fail to deeply and organically integrate key information such as patients' textual clinical symptoms and signs (e.g., fever, ear pain) with imaging and physiological measurement data, resulting in a significant gap from the thought process of clinical experts making comprehensive judgments.

[0006] Diagnostic outputs lack quantifiable uncertainty, leading to low clinical trust: Existing AI systems typically output a discrete classification result, a "black box" conclusion, without providing quantifiable confidence levels. This makes it difficult for clinicians to assess the reliability of AI diagnoses, making them less likely to accept them when faced with high-risk or ambiguous decisions, thus limiting the safe integration of AI into actual clinical workflows.

[0007] Insufficient optimization for key differential diagnosis scenarios: For diseases such as acute otitis media and effusion otitis media, which have drastically different treatment plans but similar characteristics, although there have been dedicated studies on metric learning, existing general computer-aided diagnostic systems lack an internalized, efficient mechanism to specifically handle these key differential diagnosis challenges, resulting in the risk of confusion in practical applications.

[0008] The diagnostic paradigm is limited to immediacy and lacks forward-looking risk assessment: Existing technologies are all focused on diagnosing the current state and fail to use patients' longitudinal data (such as past medical history and treatment response) to build predictive models. They cannot provide forward-looking assessment of the risk of disease recurrence, which prevents the auxiliary value of AI from extending from "treatment" to the "prevention" and "management" stages. Summary of the Invention

[0009] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method for predicting ear diseases based on multimodal data fusion and confidence assessment, thereby achieving accurate prediction of ear diseases.

[0010] This invention achieves the above objectives by adopting the following technical solution: This invention provides a method for predicting ear diseases based on multimodal data fusion and confidence assessment, comprising:

[0011] S1. Ear data collection and verification;

[0012] S2, Otology data preprocessing;

[0013] S3. Multimodal feature extraction: Extract feature vectors from digital images of otoscopes, feature vectors representing broadband tympanogram data, and clinical information feature vectors.

[0014] S4. Perform feature fusion on the feature vectors of the otoscope digital image, the feature vectors representing broadband tympanogram data, and the feature vectors of clinical information.

[0015] S5. Confidence assessment;

[0016] S6. Recurrence risk prediction.

[0017] Furthermore, step S1 specifically includes:

[0018] Acquire digital images from ear endoscopy, broadband tympanogram measurement data, and structured or unstructured clinical information;

[0019] The collected data were subjected to integrity and quality checks, including whether the resolution of the otoscope digital images met the minimum pixel requirements, and whether the broadband tympanogram measurement matrix and structured or unstructured clinical information were complete.

[0020] Furthermore, step S2 specifically includes:

[0021] Digital image preprocessing for otoscopes:

[0022] Region cropping: Automatically detects and removes invalid black areas at the edges of the image to maximize the proportion of effective information;

[0023] Size normalization: Bicubic interpolation algorithm is used to standardize the size of all images to match the input requirements of subsequent models;

[0024] Pixel value normalization: scaling the pixel values ​​of the RGB channels of an image;

[0025] Broadband tympanogram measurement matrix preprocessing:

[0026] Grid resampling: Bilinear interpolation is used to resample all broadband tympanogram measurement matrix data onto a common grid.

[0027] Clinical information preprocessing:

[0028] The collected text data undergoes deep natural language processing, including Chinese word segmentation using a word segmentation library enhanced with a professional medical dictionary, removal of stop words, conversion of word sequences into unique numerical indexes through a word embedding model, and standardization of continuous numerical features using Z-score.

[0029] Furthermore, step S3 specifically includes:

[0030] Feature extraction from digital images of ear endoscopes:

[0031] The Inception-V3 network is used as the backbone network of the image encoder. After the Inception-V3 network outputs the initial feature map, a convolutional block attention module is cascaded to perform secondary processing on the output initial feature map. The convolutional block attention module includes a channel attention module and a spatial attention module.

[0032] First, the global spatial information of the feature map is aggregated through global average pooling and global max pooling operations of the channel attention module. A channel attention weight vector is then generated by a shared multilayer perceptron. This weight vector is multiplied channel by channel with the original feature map to enhance the feature channels most relevant to ear diseases while suppressing noise or redundant information. Then, the feature map processed by the channel attention module is fed into the spatial attention module. Average pooling and max pooling are applied to the channel axes of the spatial attention module to compress the received feature map into two two-dimensional spatial maps. A convolutional layer is then used to generate the final spatial attention map. Finally, after adaptive feature optimization by the convolutional block attention module, the final endoscopic digital image feature vector is output.

[0033] Furthermore, step S3 specifically includes:

[0034] Broadband tympanogram feature extraction:

[0035] Feature extraction is achieved through a broadband tympanogram encoder, which employs a hybrid architecture combining hierarchical convolutional neural networks and bidirectional gated recurrent units. Broadband tympanogram data first undergoes hierarchical feature extraction through two cascaded convolutional neural modules, followed by dimensionality reduction using max pooling to form a primary feature sequence. This primary feature sequence is then fed into a second-level convolutional neural module to learn complex combinations of the primary feature sequence, thereby forming high-level features that characterize the relevant otological disease state. The high-level feature sequence is then input into the bidirectional gated recurrent unit, which simultaneously models the feature sequence from both forward and reverse directions, ultimately generating a feature vector representing all spatiotemporal information of the broadband tympanogram data. The forward direction refers to low to high frequencies, and the reverse direction refers to high to low frequencies.

[0036] Furthermore, step S3 specifically includes:

[0037] Clinical information feature extraction:

[0038] Clinical information features are extracted through a dual-path cross-attention encoder. The first path of the dual-path cross-attention encoder is used to process structured data, encoding continuous numerical features through an embedding layer and a multilayer perceptron to generate structured feature vectors. The second path of the dual-path cross-attention encoder focuses on deep semantic understanding of unstructured text, using a pre-trained ClinicalBERT model as its core backbone. The ClinicalBERT model captures the complex contextual semantics and implicit relationships between medical terms in the description of otological medical history, generating a contextual embedding sequence for each word.

[0039] The dual-path cross-attention encoder introduces a cross-attention mechanism, using the structured feature vector generated by the first path as the query in the attention mechanism, and the context embedding sequence generated by the second path as both key and value. Through the cross-attention mechanism, the text information is dynamically assigned to match the individual patient situation, generating a text feature vector that is context-adjusted and contains clinical reasoning. Finally, this text feature vector is concatenated with the original structured feature vector and integrated through the final multilayer perceptron layer to output a clinical information feature vector.

[0040] Furthermore, step S4 specifically includes:

[0041] The feature vectors of the endoscopic digital image, the feature vectors representing broadband tympanogram data, and the clinical information feature vectors are concatenated by a multi-head self-attention mechanism. Then, the multi-head self-attention mechanism is applied to the concatenated vectors for weighted fusion. The concatenated vectors are used as query, key and value inputs to finally generate a fused feature vector.

[0042] The fused feature vector is input into a classification decoder to generate the final diagnostic probability distribution. This classification decoder is jointly optimized using a composite loss function, which consists of two parts:

[0043] The first part is the cross-entropy loss function, a cornerstone of multi-class classification tasks. , ,in The total number of categories, It is the one-hot encoding of the real label. The cross-entropy loss is used to drive the model to learn the general discriminative features of each otological disease category, which are the corresponding category probabilities predicted by the model.

[0044] The second part is the triplet loss function. The triplet loss function constructs a triplet consisting of an anchor sample, a positive sample, and a negative sample. It explicitly constrains the sample distribution in the feature space, aiming to minimize the distance between the anchor sample and its class of positive samples, while maximizing the distance between it and its class of negative samples, ensuring a pre-defined boundary value exists between them. The method is as follows: ,in Represents from input to The entire feature extraction and fusion network, Indicates anchor sample, Indicates a positive sample. Indicates a negative sample;

[0045] Add the triplet loss function to the summation function. , , Indicates adjustable hyperparameters. Represents the total function. Represents the cross-entropy loss function. This represents the triplet loss function.

[0046] Furthermore, step S5 specifically includes:

[0047] Based on the output of the classification decoder, the comprehensive confidence score is calculated as follows:

[0048] ;

[0049] In the formula, This represents the confidence score. Represents information entropy. , This represents the Euclidean distance between the fused feature vector of the current sample and the cluster center of its predicted class. This represents the distance between the fused feature vector of the current sample and the farthest unpredicted category cluster center. , This represents the adjustable weighting coefficient.

[0050] Furthermore, step S6 specifically includes:

[0051] A relapse risk prediction module is constructed, which employs a deep network architecture combining bidirectional gated recurrent units and temporal attention mechanisms. The input to the relapse risk prediction module is a sequence of feature vectors of patients sorted by timestamps. ,in This is the patient's first The fusion feature vector generated during the second medical visit , To represent the total number of visits, this feature vector sequence is fed into a bidirectional gated recurrent unit (GRU) network. This GRU network simultaneously learns long-term dependencies in the sequence from both forward and backward dimensions, thereby comprehensively capturing the dynamic patterns of otological disease development and the cumulative effect of treatment response. This generates a series of hidden state outputs containing contextual information, as follows: ;

[0052] A temporal attention mechanism is used to dynamically assign weights to the hidden states at different times. This mechanism learns a weight vector to identify the time that has the greatest influence on predicting future recurrence events. The calculation process of the attention weights and the final sequence context vector is as follows:

[0053] ;

[0054] ;

[0055] ;

[0056] In the formula, Indicates attention weights. Represents the learnable parameters. To represent an intermediate variable, Let represent the hidden state at the i-th time. Represents the final sequence context vector;

[0057] The final context vector It is a highly condensed and intelligently weighted representation of the patient's entire medical history;

[0058] The recurrence risk prediction module constructs the prediction task as a discrete-time time-event problem, dividing the future time axis into... Non-overlapping intervals The prediction head of the recurrence risk prediction module is a multilayer perceptron, which receives the context vector. As input, it outputs the conditional probability, i.e., the risk rate, of the recurrence event occurring within each time interval k. The output layer of the risk assessment module uses the Sigmoid activation function, so the patient at any time point... The probability of no recurrence afterwards for:

[0059] ;

[0060] Finally, based on clinical needs, the cumulative recurrence probability within a specific future time window is calculated and output:

[0061] ;

[0062] In the formula, This indicates the cumulative recurrence probability within a specific future time window. In a specific future time window, It represents the probability of no recurrence within a specific time window in the future.

[0063] The beneficial effects of this invention are as follows:

[0064] This invention utilizes an attention mechanism to deeply fuse trimodal data from images, physiological signals (broadband tympanograms), and clinical text, thus highly simulating the comprehensive diagnostic process of an expert. Compared to unimodal or bimodal methods, it significantly improves the accuracy of predicting otological diseases.

[0065] This invention enhances the safety of clinical applications by introducing, for the first time, a quantitative confidence assessment model based on information entropy and feature space distance. It provides a transparent reliability metric for AI's "black box" decision-making, achieving a safe closed loop for human-machine collaboration and resolving the core trust issue in the clinical application of AI disease prediction.

[0066] By introducing triplet loss into the loss function, this invention specifically optimizes the model's ability to distinguish between key ambiguous cases such as acute otitis media and effusion otitis media, ensuring high separability within the feature space. Attached Figure Description

[0067] Figure 1 This is a flowchart of a method for predicting ear diseases based on multimodal data fusion and confidence assessment provided by the present invention;

[0068] Figure 2 This is a block diagram of the image encoder structure provided by the present invention;

[0069] Figure 3 This is a structural block diagram of the broadband tympanogram encoder provided by the present invention;

[0070] Figure 4This is a block diagram of the clinical information encoder structure provided by the present invention;

[0071] Figure 5 This is a structural block diagram of the risk assessment module provided by the present invention. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0073] This invention provides a method for predicting ear diseases based on multimodal data fusion and confidence assessment, such as... Figure 1 As shown, it specifically includes:

[0074] S1. Ear data collection and verification;

[0075] Securely acquire and standardize multi-source heterogeneous input data from heterogeneous data sources such as hospital information systems, image archiving and communication systems through standard medical interfaces (such as HL7, DICOM), including: digital images of ear endoscopy. Broadband tympanogram measurement data and structured or unstructured clinical information .

[0076] Data reception and verification: The received data is verified for integrity and quality. For example, the resolution of the otoscope digital image meets the minimum requirements (e.g., 640×480 pixels), and the broadband tympanogram data matrix and structured or unstructured clinical information are checked for completeness.

[0077] S2, Otology data preprocessing;

[0078] Image preprocessing:

[0079] Region cropping: Automatically detects and removes invalid black areas at the edges of the image to maximize the proportion of effective information.

[0080] Size normalization: The bicubic interpolation algorithm is used to uniformly adjust the size of all images to 299×299 pixels to match the input requirements of the subsequent Inception-V3 model.

[0081] Pixel value normalization: linearly scales the RGB channel pixel values ​​of an image from [0,255] to the range [0,1].

[0082] Broadband tympanogram measurement data preprocessing: namely, grid resampling. Since the original broadband tympanogram measurements were non-uniformly sampled on the pressure axis, bilinear interpolation was used to resample all broadband tympanogram data onto a common 84×84 grid. The pressure axis range of this grid is [180, 280] daPa (linear interval), and the frequency axis range is [226, 4000] Hz (logarithmic interval), thus providing regularized input for subsequent convolution operations.

[0083] Clinical information preprocessing: Deep natural language processing is performed on text data (such as chief complaint and present medical history) obtained from the hospital information system, including but not limited to: Chinese word segmentation using a word segmentation library enhanced with a professional medical dictionary, removal of stop words, and conversion of word sequences into unique numerical indexes (Token IDs) through a word embedding model. For continuous numerical features such as age, Z-score standardization is used to eliminate the influence of dimensions.

[0084] S3. Multimodal feature extraction: Extract feature vectors from digital images of otoscopes, feature vectors representing broadband tympanogram data, and clinical information feature vectors.

[0085] Multimodal feature extraction aims to map raw input data (endoscopic digital images, broadband tympanogram measurement data matrices, and clinical information) from different sources and with varying structures to a high-dimensional, semantically aligned feature space. To achieve this, this invention designs three parallel encoder subnetworks, each deeply customized for a specific modality. Each subnetwork is dedicated to extracting the most informative and discriminative representations from its corresponding data, thereby providing high-quality input for downstream attention fusion and diagnostic decision-making.

[0086] Image encoder (digital image from an otoscope):

[0087] For digital images of ear endoscopy For visual feature extraction, this invention employs the Inception-V3 architecture, which has been widely validated in the field of medical image analysis, as the image encoder. The backbone network of Inception-V3. The core advantage of Inception-V3 lies in its multi-scale convolutional kernel design, which can capture rich visual cues ranging from fine textures to macroscopic structures in parallel. This is crucial for recognizing subtle pathological changes in the tympanic membrane (such as congestion, effusion, and calcifications). After the original Inception-V3 network outputs preliminary feature maps, this invention cascades a CBAM (Convolutional Block Attention Module) to further refine the features.

[0088] like Figure 2As shown, CBAM comprises two sequential sub-modules: First, the global spatial information of the feature map is aggregated through global average pooling and global max pooling operations in the channel attention module, and a channel attention weight vector is generated by learning via a shared multilayer perceptron. This vector is then multiplied channel-by-channel with the original feature map to enhance the feature channels most relevant to disease diagnosis (e.g., channels associated with "red" should be given higher weights when diagnosing tympanic membrane congestion), while suppressing noise or redundant information. Next, the channel-refined feature map is fed into the spatial attention module. The spatial attention module applies average pooling and max pooling along the channel axis to the channel-refined feature map, compressing it into two two-dimensional spatial maps, which are then passed through a convolutional layer to generate the final spatial attention map. This spatial attention map enables precise focusing on the lesion region. After adaptive feature optimization by CBAM, a highly condensed and semantically rich one-dimensional image feature vector is finally output. , This represents the parameters (weights and biases) in the internal neural network of the image encoder. Represent a A 3D real vector space, Represents image feature vectors Number of dimensions.

[0089] Wideband tympanogram encoder (wideband tympanogram data):

[0090] Wideband tympanogram encoder designed for physiological signals It employs a hybrid architecture combining hierarchical convolutional neural networks and bidirectional gated recurrent units. This design aims to comprehensively capture the local morphological patterns contained in the broadband tympanogram matrix and their dynamic dependencies on the frequency-pressure spectrum.

[0091] like Figure 3As shown, the data first undergoes hierarchical feature extraction through two cascaded convolutional neural modules. The first-level convolutional neural module operates with a relatively large convolutional kernel (5*5) to capture basic local morphological features from the original 84×84 grid data, such as the initial slope of absorption peaks and peak positions. This first-level module consists of one-dimensional convolutional layers, batch normalization, and ReLU activation functions, and undergoes dimensionality reduction using max pooling (MaxPooling1D) to form a primary feature sequence. This sequence is then fed into the second-level convolutional neural module, which uses a smaller convolutional kernel (3*3) to learn complex combinations of these primary patterns, thereby forming higher-level features that characterize specific disease states (e.g., the "flat" curve corresponding to effusion-induced otitis media). After extracting the hierarchical spatial features through the convolutional neural network, the resulting high-level feature sequence is input into a bidirectional gated recurrent unit. As an advanced recurrent neural network, the bidirectional gated recurrent unit (GRU) can effectively capture long-term dependencies in sequences. Its bidirectional design allows it to model feature sequences simultaneously from both the forward (low-frequency to high-frequency) and reverse (high-frequency to low-frequency) directions, thus providing a complete understanding of the dynamic evolution of broadband tympanogram data across the entire frequency-pressure spectrum. Ultimately, it generates feature vectors representing all spatiotemporal information of the broadband tympanogram data. , This represents the parameters (weights and biases) in the internal neural network of the broadband tympanogram encoder. Represent a A 3D real vector space, Represents the eigenvectors of a broadband tympanogram. The number of dimensions.

[0092] Clinical Information Encoder:

[0093] Clinical Information The processing of this information is crucial for the comprehensive diagnostic logic of the simulation expert. Therefore, this invention constructs an innovative dual-path cross-attention encoder. Its structure is as follows Figure 4 As shown, the cross-attention encoder can organically fuse structured (e.g., age, body temperature) and unstructured (e.g., chief complaint, present medical history) clinical data. The first pathway is dedicated to processing structured data, encoding continuous numerical features such as age through an embedding layer and a multilayer perceptron to generate structured feature vectors. The second approach focuses on deep semantic understanding of unstructured text, employing the ClinicalBERT model pre-trained on massive clinical text corpora as its core backbone. ClinicalBERT can capture the complex contextual semantics and implicit relationships between medical terms in medical history descriptions, generating a contextual embedding sequence for each token. .

[0094] The core innovation of the cross-attention encoder lies in introducing a cross-attention mechanism to achieve intelligent interaction between information from two pathways. Specifically, this invention uses the structured vector generated by the first pathway... As a query in the attention mechanism, the second path generates a sequence of text terms. Simultaneously serving as both a key and a value, this design ingeniously simulates the diagnostic thinking of a clinician: basic patient information (such as "infant") intelligently guides the model to give higher weight to specific descriptions such as "crying" and "ear-scratching" when interpreting medical history texts. In this way, the importance of dynamically matching textual information allocation with individual patient circumstances is reflected, and then a context-adjusted textual feature vector containing clinical reasoning is generated. Finally, this vector is compared with the original structured vector. The data is then concatenated and integrated through a final multilayer perceptron layer, outputting a feature vector that deeply fuses all clinical information. , This represents the parameters (weights and biases) in the internal neural network of the clinical information encoder. Represent a A 3D real vector space, Represents clinical information feature vector The number of dimensions or features.

[0095] S4. Perform feature fusion on the feature vectors of the otoscope digital image, the feature vectors representing broadband tympanogram data, and the feature vectors of clinical information.

[0096] This invention introduces an advanced multi-head self-attention mechanism, aiming to deeply simulate the nonlinear diagnostic thinking process of clinical experts when facing complex cases, involving the weighing, correlation, and integration of information from different sources. This process first integrates three independent feature vectors into a unified multimodal sequence through vector concatenation operations: Subsequently, a multi-head self-attention mechanism is applied to the concatenated vector. Weighted fusion is performed, and the concatenated vector is then used as the input for query, key, and value simultaneously. The core advantage of this mechanism lies in its ability to compute attention weights in parallel across multiple independent "representation subspaces," enabling the model to capture complex dependencies between features from different perspectives. For example, one "head" might focus on learning how to increase attention to tympanic membrane congestion features in endoscopic images when "ear pain" and "fever" appear in clinical text descriptions; while another "head" might focus on discovering the association between specific peak patterns in broadband tympanograms and "recurrent episodes" mentioned in the medical history. In this way, the system no longer processes all information statically and with equal weight, but dynamically assigns importance weights to features of different modalities and dimensions based on the specific "context" of the current case, ultimately generating a highly condensed and expressive fused feature vector. :

[0097] ;

[0098] get Then, it is fed into a classification decoder to generate the final diagnostic probability distribution. The training process of this decoder is not driven by a single objective, but rather by joint optimization through a carefully designed composite loss function. This ensures overall classification accuracy while addressing the key identification and diagnostic challenges mentioned in the background section. This composite loss function... It consists of two parts:

[0099] The first part is the cross-entropy loss function, a cornerstone of multi-class classification tasks. , ,in The total number of categories, It is the one-hot encoding of the real label. The cross-entropy loss is used to drive the model to learn the general discriminative features of each otological disease category, which are the corresponding category probabilities predicted by the model.

[0100] More importantly, in order to directly address the diagnostic challenge of the highly similar clinical manifestations of acute otitis media and effusion otitis media, which are easily confused, we have innovatively introduced differential diagnostic loss, namely the ternary loss.

[0101] The second part is the triplet loss function, which is constructed by using anchor samples (anchor, ), positive samples ) and negative samples The triplet, consisting of , explicitly constrains the sample distribution within the feature space. The goal is to minimize the distance between the anchor sample and its class of positive samples, while maximizing its distance from its class of negative samples, ensuring a pre-defined margin exists between them. The method is as follows: ,in Represents from input to The entire feature extraction and fusion network, Indicates anchor sample, Indicates a positive sample. Indicates a negative sample;

[0102] Add the triplet loss function to the summation function. , , Indicates adjustable hyperparameters. Represents the total function. Represents the cross-entropy loss function. This represents the triplet loss function.

[0103] This invention forces the model to push "difficult pairs" such as acute otitis media and effusion otitis media to far apart positions in the feature space while learning classification, forming clear decision boundaries. This design not only greatly improves the ability to distinguish key fuzzy cases, but also provides a deep mathematical interpretation of the expert differential diagnosis logic, thereby ensuring the clinical applicability and accuracy of the diagnostic results.

[0104] S5. Confidence assessment;

[0105] Based on the output of the classification decoder, the comprehensive confidence score is calculated as follows:

[0106] ;

[0107] In the formula, This represents the confidence score. Represents information entropy. , This represents the Euclidean distance between the fused feature vector of the current sample and the cluster center of its predicted class. This represents the distance between the fused feature vector of the current sample and the farthest unpredicted category cluster center. , This represents the adjustable weighting coefficient.

[0108] S6. Recurrence risk prediction;

[0109] To achieve a paradigm shift from immediate diagnosis to longitudinal disease management, this invention designs a recurrence risk prediction module based on sequence modeling and survival analysis. This module aims to dynamically assess the likelihood of a patient's recurrence within a specific future time window. Instead of simply classifying data from a single medical visit, the recurrence risk prediction module deeply mines the temporal dependencies of a patient's past medical information to construct a forward-looking, individualized risk assessment model.

[0110] like Figure 5 As shown, the core input to the recurrence risk prediction module is a sequence of feature vectors for a specific patient, sorted by timestamp. ,in This is the patient's first During the second visit, the multimodal fusion feature vector generated upstream... , This represents the total number of visits. This sequence not only encodes a diagnostic snapshot at each visit, but more importantly, it captures the evolutionary trajectory of the disease state, such as the transition from effusion-related otitis media to acute otitis media, dynamic changes in tympanic membrane lesions (such as increases or decreases in effusion volume and changes in congestion), and responses to specific treatments (such as antibiotic use).

[0111] The relapse risk prediction module employs a deep network architecture combining bidirectional gated recurrent units and temporal attention mechanisms to accurately capture the disease evolution trajectory and dynamically focus on key nodes. First, the patient's feature sequence is fed into a bidirectional gated recurrent network. This network can simultaneously learn long-term dependencies in the sequence from both forward (history to present) and backward (current to backtracking) dimensions, thereby comprehensively capturing the dynamic patterns of disease development and the cumulative effects of treatment response, generating a series of hidden state outputs containing contextual information. .

[0112] Considering that not all historical medical records are equally important in indicative of future risk, this invention further introduces a temporal attention mechanism to dynamically assign weights to the hidden states at different times. This mechanism learns a weight vector. It intelligently identifies the "critical moments" most influential in predicting future relapses; for example, a medical record of a sudden deterioration of the condition should be given far more attention than a routine follow-up visit. Attention Weighting and the final sequence context vector The calculation process is as follows:

[0113] ;

[0114] ;

[0115] ;

[0116] In the formula, This represents the attention weight for the i-th time. Represents the learnable parameters. and To represent intermediate variables, the calculation method is the same. Let represent the hidden state at the i-th time. Represents the final sequence context vector;

[0117] The final context vector It is a highly condensed and intelligently weighted representation of the patient's entire medical history.

[0118] Unlike traditional models that output a single relapse probability, the relapse risk prediction module draws inspiration from clinical survival analysis, constructing the prediction task as a discrete-time time-to-event problem. The future timeline is divided into... Non-overlapping intervals The prediction head of the relapse risk prediction module is a multilayer perceptron (MLP), which receives a context vector. As input, it outputs the conditional probability of a recurrence event occurring within each time interval k, i.e., the hazard rate. The model's output layer uses the Sigmoid activation function to ensure that the probability values ​​are within the range [0,1]. Based on this, we can calculate the probability values ​​of the patient at any time point. The probability of survival afterwards (i.e., the probability of no recurrence). :

[0119] ;

[0120] In this way, the model can not only predict whether a relapse will occur, but also provide a dynamic probability distribution of when the risk of relapse is highest. Ultimately, the system can calculate and output the cumulative relapse probability within a specific future time window based on clinical needs.

[0121] ;

[0122] In the formula, Indicates the cumulative recurrence probability within a specific future time window. Represents any point in time. In a specific future time window, It represents the probability of no recurrence within a specific time window in the future.

[0123] This provides strong quantitative evidence for developing precise and prospective follow-up plans (such as shortening the interval between follow-up examinations for high-risk patients) and intervention decisions (such as considering tympanostomy tube insertion when appropriate). The model is trained using a negative log-likelihood loss function to maximize the consistency between the predicted survival function and real-world clinical event observations.

[0124] S7. Comprehensive report generation and storage;

[0125] Finally, the diagnostic category, quantified confidence score, and prospective recurrence risk prediction probability are integrated into a standardized, clinician-friendly electronic report, which is then securely written into a database for review and traceability.

[0126] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A method for predicting otological diseases based on multimodal data fusion and confidence assessment, characterized in that, include: S1. Ear data collection and verification; S2, Otology data preprocessing; S3. Multimodal feature extraction: Extract feature vectors from digital images of otoscopes, feature vectors representing broadband tympanogram data, and clinical information feature vectors. S4. Perform feature fusion on the feature vectors of the otoscope digital image, the feature vectors representing broadband tympanogram data, and the feature vectors of clinical information. The feature vectors of the endoscopic digital image, the feature vectors representing broadband tympanogram data, and the clinical information feature vectors are concatenated by a multi-head self-attention mechanism. Then, the multi-head self-attention mechanism is applied to the concatenated vectors for weighted fusion. The concatenated vectors are used as query, key and value inputs to finally generate a fused feature vector. The fused feature vector is input into a classification decoder to generate the final diagnostic probability distribution. This classification decoder is jointly optimized using a composite loss function, which consists of two parts: The first part is the cross-entropy loss function, a cornerstone of multi-class classification tasks. , ,in The total number of categories, It is the one-hot encoding of the real label. The cross-entropy loss is used to drive the model to learn the general discriminative features of each otological disease category, which are the corresponding category probabilities predicted by the model. The second part is the triplet loss function. The triplet loss function constructs a triplet consisting of an anchor sample, a positive sample, and a negative sample. It explicitly constrains the sample distribution in the feature space, aiming to minimize the distance between the anchor sample and its class of positive samples, while maximizing the distance between it and its class of negative samples, ensuring a pre-defined boundary value exists between them. The method is as follows: ,in Represents from input to The entire feature extraction and fusion network, Indicates anchor sample, Indicates a positive sample. Indicates a negative sample; Add the triplet loss function to the summation function. , , Indicates adjustable hyperparameters. Represents the total function. Represents the cross-entropy loss function. Represents the triplet loss function; S5. Confidence assessment; Based on the output of the classification decoder, the comprehensive confidence score is calculated as follows: ； In the formula, This represents the confidence score. Represents information entropy. , This represents the Euclidean distance between the fused feature vector of the current sample and the cluster center of its predicted class. This represents the distance between the fused feature vector of the current sample and the farthest unpredicted category cluster center. , Indicates the adjustable weighting coefficient; S6. Recurrence risk prediction.

2. The method for predicting otological diseases based on multimodal data fusion and confidence assessment according to claim 1, characterized in that, Step S1 specifically includes: Acquire digital images from ear endoscopy, broadband tympanogram measurement data, and structured or unstructured clinical information; The collected data were subjected to integrity and quality checks, including whether the resolution of the otoscope digital images met the minimum pixel requirements, and whether the broadband tympanogram measurement matrix and structured or unstructured clinical information were complete.

3. The method for predicting otological diseases based on multimodal data fusion and confidence assessment according to claim 1, characterized in that, Step S2 specifically includes: Digital image preprocessing for otoscopes: Region cropping: Automatically detects and removes invalid black areas at the edges of the image to maximize the proportion of effective information; Size normalization: Bicubic interpolation algorithm is used to standardize the size of all images to match the input requirements of subsequent models; Pixel value normalization: scaling the pixel values ​​of the RGB channels of an image; Broadband tympanogram measurement matrix preprocessing: Grid resampling: Bilinear interpolation is used to resample all broadband tympanogram measurement matrix data onto a common grid. Clinical information preprocessing: The collected text data undergoes deep natural language processing, including Chinese word segmentation using a word segmentation library enhanced with a professional medical dictionary, removal of stop words, conversion of word sequences into unique numerical indexes through a word embedding model, and standardization of continuous numerical features using Z-score.

4. The method for predicting otological diseases based on multimodal data fusion and confidence assessment according to claim 1, characterized in that, Step S3 specifically includes: Feature extraction from digital images of ear endoscopes: The Inception-V3 network is used as the backbone network of the image encoder. After the Inception-V3 network outputs the initial feature map, a convolutional block attention module is cascaded to perform secondary processing on the output initial feature map. The convolutional block attention module includes a channel attention module and a spatial attention module. First, the global spatial information of the feature map is aggregated through global average pooling and global max pooling operations of the channel attention module. A channel attention weight vector is then generated by a shared multilayer perceptron. This weight vector is multiplied channel by channel with the original feature map to enhance the feature channels most relevant to ear diseases while suppressing noise or redundant information. Then, the feature map processed by the channel attention module is fed into the spatial attention module. Average pooling and max pooling are applied to the channel axes of the spatial attention module to compress the received feature map into two two-dimensional spatial maps. A convolutional layer is then used to generate the final spatial attention map. Finally, after adaptive feature optimization by the convolutional block attention module, the final endoscopic digital image feature vector is output.

5. The method for predicting otological diseases based on multimodal data fusion and confidence assessment according to claim 1, characterized in that, Step S3 specifically also includes: Broadband tympanogram feature extraction: Feature extraction is achieved through a broadband tympanogram encoder, which employs a hybrid architecture combining hierarchical convolutional neural networks and bidirectional gated recurrent units. Broadband tympanogram data first undergoes hierarchical feature extraction through two cascaded convolutional neural modules, followed by dimensionality reduction using max pooling to form a primary feature sequence. This primary feature sequence is then fed into a second-level convolutional neural module to learn complex combinations of the primary feature sequence, thereby forming high-level features that characterize the relevant otological disease state. The high-level feature sequence is then input into the bidirectional gated recurrent unit, which simultaneously models the feature sequence from both forward and reverse directions, ultimately generating a feature vector representing all spatiotemporal information of the broadband tympanogram data. The forward direction refers to low to high frequencies, and the reverse direction refers to high to low frequencies.

6. The method for predicting otological diseases based on multimodal data fusion and confidence assessment according to claim 1, characterized in that, Step S3 specifically also includes: Clinical information feature extraction: Clinical information features are extracted through a dual-path cross-attention encoder. The first path of the dual-path cross-attention encoder is used to process structured data, encoding continuous numerical features through an embedding layer and a multilayer perceptron to generate structured feature vectors. The second path of the dual-path cross-attention encoder focuses on deep semantic understanding of unstructured text, using a pre-trained ClinicalBERT model as its core backbone. The ClinicalBERT model captures the complex contextual semantics and implicit relationships between medical terms in the description of otological medical history, generating a contextual embedding sequence for each word. The dual-path cross-attention encoder introduces a cross-attention mechanism, using the structured feature vector generated by the first path as the query in the attention mechanism, and the context embedding sequence generated by the second path as both key and value. Through the cross-attention mechanism, the text information is dynamically assigned to match the individual patient situation, generating a text feature vector that is context-adjusted and contains clinical reasoning. Finally, this text feature vector is concatenated with the original structured feature vector and integrated through the final multilayer perceptron layer to output a clinical information feature vector.

7. The method for predicting otological diseases based on multimodal data fusion and confidence assessment according to claim 1, characterized in that, Step S6 specifically includes: A relapse risk prediction module is constructed, which employs a deep network architecture combining bidirectional gated recurrent units and temporal attention mechanisms. The input to the relapse risk prediction module is a sequence of feature vectors of patients sorted by timestamps. ,in This is the patient's first The fusion feature vector generated during the second medical visit , To represent the total number of visits, this feature vector sequence is fed into a bidirectional gated recurrent unit (GRU) network. This GRU network simultaneously learns long-term dependencies in the sequence from both forward and backward dimensions, thereby comprehensively capturing the dynamic patterns of otological disease development and the cumulative effect of treatment response. This generates a series of hidden state outputs containing contextual information, as follows: ; A temporal attention mechanism is used to dynamically assign weights to the hidden states at different times. This mechanism learns a weight vector to identify the time that has the greatest influence on predicting future recurrence events. The calculation process of the attention weights and the final sequence context vector is as follows: ； ； ； In the formula, This represents the attention weight for the i-th time. , Represents the learnable parameters. and Both represent intermediate variables and are calculated in the same way. Let represent the hidden state at the i-th time. Represents the final sequence context vector; The final context vector It is a highly condensed and intelligently weighted representation of the patient's entire medical history; The recurrence risk prediction module constructs the prediction task as a discrete-time time-event problem, dividing the future time axis into... Non-overlapping intervals The prediction head of the recurrence risk prediction module is a multilayer perceptron, which receives the context vector. As input, and output as the conditional probability, i.e., the risk rate, of the recurrence event occurring within each time interval k. The output layer of the risk assessment module uses the Sigmoid activation function, so the patient at any time point... The probability of no recurrence afterwards for: ； Finally, based on clinical needs, the cumulative recurrence probability within future time windows is calculated and output: ； In the formula, This indicates the cumulative recurrence probability within a future time window. Future time window, This indicates the probability of no recurrence within a future time window.

Images