Pulmonary tuberculosis recognition system based on multi-modal fusion and expert-assisted optimization

Abstract

The invention discloses a pulmonary tuberculosis recognition system based on multi-modal fusion and expert-assisted optimization, which belongs to the technical field of artificial intelligence and comprises a data acquisition module, an imaging examination image, a laboratory examination result and an electronic medical record. The data preprocessing module is used for extracting a first feature by utilizing an iconography examination image, extracting a second feature by utilizing a laboratory examination result and extracting a third feature by utilizing an electronic medical record; the third feature comprises 32 sub-features and contribution degrees corresponding to the sub-features; the optimization module is used for optimizing the contribution degree based on expert diagnosis experience; and the fusion module is used for performing modal fusion on the first feature, the second feature and the optimized third feature to generate a pulmonary tuberculosis recognition result. According to the method, imaging examination images, laboratory examination results and electronic medical records are fused, the contribution degree of expert experience to the features is optimized, intelligent detection of pulmonary tuberculosis is finally achieved, and the method has good generalization ability and pulmonary tuberculosis detection precision.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a tuberculosis identification system based on multimodal fusion and expert-assisted optimization. Background Technology

[0002] Tuberculosis is a chronic respiratory disease. Studies show that most tuberculosis patients first seek medical attention at general hospitals rather than specialized prevention and treatment institutions. The median time from the first visit to the final diagnosis is more than half a month. Therefore, delayed diagnosis often occurs, which has several serious consequences: first, it accelerates the spread of tuberculosis in the community; second, it delays the treatment window, causing the patient's condition to deteriorate further and making treatment more difficult.

[0003] Research has found that the reasons for delayed diagnosis of pulmonary tuberculosis are complex and diverse, mainly including the following three categories: First, limitations in detection technology. Currently used etiological detection methods, such as smear microscopy and culture techniques, have limited sensitivity and are difficult to identify patients with low bacterial loads in the early stages. Second, the clinical manifestations of the disease are complex. Pulmonary tuberculosis often presents with "different symptoms in different diseases and different symptoms in the same disease," especially in terms of imaging and clinical manifestations, making it easy to confuse with pneumonia, lung cancer, etc. Traditional diagnostic methods based on X-ray images, electronic medical records, and laboratory data have limited accuracy and rely heavily on the subjective judgment of doctors. Third, uneven distribution of medical resources. Doctors in general hospitals have fewer opportunities to see pulmonary tuberculosis cases, and their cognitive level and diagnostic experience are relatively lacking, urgently requiring the introduction of intelligent early identification tools to assist in diagnosis. Fourth, although artificial intelligence has made some progress in medical image recognition, existing AI models still have problems such as insufficient interpretability, difficulty in gaining clinical trust due to black-box decision-making, reliance on a large amount of manual parameter tuning for model iteration and optimization, high training costs, and difficulty in practical application, which limit their widespread promotion and application in the early diagnosis of pulmonary tuberculosis. Summary of the Invention

[0004] The purpose of this invention is to provide a tuberculosis identification system based on multimodal fusion and expert-assisted optimization. This system integrates imaging examination images, laboratory test results, and electronic medical records, and optimizes the contribution of features by combining expert experience, ultimately achieving intelligent detection of tuberculosis with good generalization ability and tuberculosis detection accuracy.

[0005] To achieve the above-mentioned objectives, the present invention provides a tuberculosis identification system based on multimodal fusion and expert-assisted optimization, comprising: The data acquisition module collects imaging examination images, laboratory test results, and electronic medical records; The data preprocessing module extracts the first feature from imaging examination images, the second feature from laboratory test results, and the third feature from electronic medical records. The third feature includes 32 sub-features and their corresponding contribution values. The 32 sub-features include: miliary nodules, lung destruction, tree-in-bud sign, diabetes, bronchial stenosis, hemoptysis, history of contact with pulmonary tuberculosis, history of tuberculosis, history of AIDS, calcification, cavitation, cough, malnutrition, multiple nodules, patchy shadows, pleural thickening, pleural effusion, sputum production, fever, pericardial effusion, poor appetite, weight loss, enlarged lymph nodes, night sweats, linear shadows, consolidation, dullness on percussion, dyspnea, chest pain, moist rales, history of alcohol consumption, and history of smoking. The optimization module optimizes contribution based on expert diagnostic experience; The fusion module performs modal fusion on the first feature, the second feature, and the optimized third feature to generate tuberculosis identification results.

[0006] Preferably, the optimized contribution values ​​of the 32 sub-features include: 0.79 for miliary nodules, 0.79 for lung destruction, 0.79 for tree-in-bud sign, 0.77 for diabetes, 0.74 for bronchial stenosis, 0.71 for hemoptysis, 0.71 for a history of exposure to pulmonary tuberculosis, 0.7 for a history of tuberculosis, 0.7 for a history of AIDS, 0.7 for calcification, 0.69 for cavitation, 0.67 for cough, 0.66 for malnutrition, 0.66 for multiple nodules, and 0.66 for patchy shadows. The contribution of pleural thickening was 0.66, pleural effusion was 0.66, coughing up sputum was 0.64, fever was 0.63, pericardial effusion was 0.63, poor appetite was 0.61, weight loss was 0.61, enlarged lymph nodes were 0.6, night sweats were 0.56, linear opacities were 0.54, consolidation was 0.53, dullness on percussion was 0.51, dyspnea was 0.5, chest pain was 0.49, moist rales were 0.47, history of alcohol consumption was 0.46, and history of smoking was 0.41.

[0007] Preferably, the extraction of the first feature from the imaging examination image includes: First, data augmentation is performed on the imaging examination images; Then, a 3D ResNet18 model is used to extract 3D image features.

[0008] Preferably, the extraction of the second feature using laboratory test results includes: The first structured data feature in laboratory test results is extracted using a custom module; the first structured data feature includes the test item name and the test value. A custom module is used to extract second structured data features from laboratory test results; these features include the patient's age and gender.

[0009] Preferably, the extraction of the third feature using electronic medical records includes: The text is refined by using the BERT-EMR model to structure the natural language text of the electronic medical record; the refined text of the electronic medical record is reconstructed to contain only the tuberculosis features selected by the tuberculosis specialist. Vocabulary standardization involves converting non-standard vocabulary in the refined text of electronic medical records into standard vocabulary. Feature extraction: Using NLP models to extract textual features from the refined text of electronic medical records; The risk contribution is integrated by incorporating the expert-assessed risk contribution of tuberculosis into the text features to obtain the third feature.

[0010] Preferably, the refined text includes: First, the BERT-EMR model is used to generate electronic medical record feature vectors; Then, it is transformed into 512-dimensional features required by the multimodal model through the text_proj text projection layer; Finally, dynamic text_pos_embed position encoding is superimposed, and invalid text features are filtered out in conjunction with text_mask.

[0011] Preferably, the modal fusion of the first feature, the second feature, and the optimized third feature includes: First, the second and third features are classified as text features, and the first feature is classified as an image feature; Then, a hybrid attention module is used to process textual and image features, including intramodal self-attention and cross-modal bidirectional cross-attention mechanisms, subsequently generating enhanced features. In the cross-modal bidirectional cross-attention mechanism, one path uses textual features (processed by intramodal self-attention) and population features as the query, image features (processed by intramodal self-attention) and population features as the key, and image features (processed by intramodal self-attention) as the value, focusing textual descriptions on relevant image regions. The other path uses image features (processed by intramodal self-attention) and population features as the query, textual features (processed by intramodal self-attention) and population features as the key, and textual features (processed by intramodal self-attention) as the value, allowing image regions to query their corresponding textual descriptions. Each attention sub-layer is followed by a feedforward network, residual connections, and layer normalization. The features enhanced by the hybrid attention module are pooled along the sequence dimension to obtain aggregated embeddings of images and text. Finally, a bidirectional gating fusion module is used to perform attention correction of demographic information (age, gender), generation of bidirectional cross-features between images and text, generation of gating weights, and weighted feature fusion. The in-modal gating integrates the original features, the enhanced features processed by the hybrid attention module, and the bidirectional cross-features between text and images. Then, based on the gating weights, the contributions of each type of feature are dynamically allocated to generate the final fused features required by the classification head module.

[0012] Compared with the prior art, the present invention has the following technical effects: This invention utilizes multimodal fusion technology to integrate features from imaging examination images, laboratory test results, and electronic medical records. By dynamically optimizing the feature contribution based on expert experience, it achieves intelligent detection and diagnosis of pulmonary tuberculosis. This system not only improves the model's generalization ability, enabling it to adapt to data differences across different medical institutions, but also significantly enhances the accuracy and reliability of pulmonary tuberculosis detection, providing strong support for clinical decision-making.

[0013] During the doctor's diagnosis and treatment process, this invention can process multi-source information in real time as soon as the data is generated, quickly calculate the risk probability of contracting tuberculosis, and issue timely visual alerts to doctors through an early warning module integrated into the medical information system, assisting doctors in early intervention and treatment planning, thereby optimizing the diagnosis and treatment process and improving efficiency.

[0014] This invention employs a gated Transformer architecture as its core framework to effectively capture long-range dependencies in multimodal data. It also integrates expert diagnostic experience as prior knowledge and combines it with an MLP (Multilayer Perceptron) for nonlinear feature transformation and risk probability inference. This design ensures that the model maintains high-reliability output while efficiently processing large amounts of case data, reducing the possibility of misdiagnosis and missed diagnosis.

[0015] Furthermore, this invention innovatively embeds the experience of medical experts (clinicians) into the multimodal attention mechanism in the form of feature contribution percentages. By allocating attention weights, it prioritizes key features, thereby significantly reducing the model's demand for computational resources and training data. This not only reduces computing power and data costs but also enables the model to possess high accuracy and strong interpretability. Doctors can intuitively understand the model's decision-making basis, enhancing clinical trust and adoption. Attached Figure Description

[0016] Figure 1 A schematic diagram provided for a preferred embodiment of the present invention; Figure 2 A flowchart of a preferred embodiment of the present invention; Figure 3 This is an optimized operation diagram based on expert feedback in a preferred embodiment of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are only a part of the embodiments of the present invention, and not all of them. Generally, the embodiments of the present invention described and shown in the accompanying drawings are characteristic technologies and solutions. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0018] Please see Figures 1 to 3 A tuberculosis identification system based on multimodal fusion and expert-assisted optimization, comprising: The data acquisition module collects basic data: imaging examination images, laboratory test results, and electronic medical records; In the process of identifying pulmonary tuberculosis, the main sources of information include imaging examination images (pulmonary computed tomography (CT) images), laboratory test results (white blood cell count, neutrophil percentage, C-reactive protein, procalcitonin), and electronic medical record texts (containing natural language descriptions, such as chief complaint, present illness, past medical history, and physical examination records). However, any single data source has its limitations: imaging examinations may be affected by equipment accuracy and interpretation experience; laboratory results have a certain delay and limited sensitivity; and electronic medical record texts may lead to judgment bias due to subjective descriptions or missing information. Therefore, to improve the accuracy and efficiency of pulmonary tuberculosis identification, it is necessary to rely on the fusion and complementarity of multimodal data to build a comprehensive and complete basic data system, thereby supporting more reliable early screening, differential diagnosis, and disease assessment.

[0019] The data preprocessing module extracts the first feature from imaging examination images, the second feature from laboratory test results, and the third feature from electronic medical records. The third feature includes 32 sub-features and their corresponding contribution values. The 32 sub-features include: miliary nodules, lung destruction, tree-in-bud sign, diabetes, bronchial stenosis, hemoptysis, history of contact with pulmonary tuberculosis, history of tuberculosis, history of AIDS, calcification, cavitation, cough, malnutrition, multiple nodules, patchy shadows, pleural thickening, pleural effusion, sputum production, fever, pericardial effusion, poor appetite, weight loss, enlarged lymph nodes, night sweats, linear shadows, consolidation, dullness on percussion, dyspnea, chest pain, moist rales, history of alcohol consumption, and history of smoking. To achieve automated computer identification of pulmonary tuberculosis, a comprehensive and systematic preprocessing of the collected basic data is first required to ensure its accuracy, consistency, and usability. The specific preprocessing process includes: extracting first features from imaging examination images (CT scans), which primarily capture morphological changes and abnormal areas in lung structure; extracting second features from laboratory test results (white blood cell count, neutrophil percentage, C-reactive protein, procalcitonin), which reflect biomarkers, inflammation levels, or pathogen presence indicators; and extracting third features from electronic medical records (including medical history, symptom descriptions, and diagnostic conclusions), using natural language processing techniques to parse key terms, semantic relationships, and clinical patterns in the text. The third feature is further decomposed into 32 sub-features, each corresponding to a specific clinical factor or contextual element and assigned a corresponding contribution weight to quantify its relative importance in the pulmonary tuberculosis identification task. The preprocessing stage may also involve data cleaning, normalization, and feature alignment to eliminate noise and improve the efficiency of subsequent model training.

[0020] The optimization module optimizes contribution based on expert diagnostic experience; The content of electronic medical records is highly diverse and random, primarily due to the recording habits of different healthcare professionals, the variations in patient complaints, and the complexity of clinical scenarios. Specifically, the number of sub-features is numerous, and their descriptions often involve uncertainty. For example, the frequency, duration, or intensity of symptoms such as "occasional fever" may be vague, posing a significant challenge to automated recognition models. This uncertainty can lead to inaccurate feature extraction, thus affecting the model's recognition results, such as reducing classification accuracy or increasing the risk of misclassification. Therefore, to effectively improve the accuracy of model recognition, it is essential to rely on the experience of domain experts and fine-tune the contribution of specific features, for example, through weight allocation, feature selection, or parameter optimization, to ensure that the model better reflects the needs of actual medical scenarios.

[0021] In this embodiment, an expert committee of 25 experts with over 10 years of clinical experience was formed. These 25 experts were divided into five expert groups: 5 respiratory specialists, 5 infectious disease specialists, 10 tuberculosis specialists, and 5 radiology specialists. Using the Delphi method, the initially constructed first, second, and third features were comprehensively reviewed and revised. Then, based on clinical experience and evidence-based medicine, the experts eliminated features with weak relevance to the diagnosis of pulmonary tuberculosis, added key features of high clinical concern, and optimized the contribution of each feature—that is, the contribution was reassessed and redefined.

[0022] The fusion module performs modal fusion on the first feature, the second feature, and the optimized third feature to generate a tuberculosis identification result. The tuberculosis identification result includes the risk level (high risk, medium risk, low risk) and the probability value of tuberculosis.

[0023] To better understand the technical content of this invention, the following non-limiting description is provided: Choosing the appropriate number, specific category, and contribution level of sub-features can effectively enhance the stability, accuracy, and interpretability of the model for identifying tuberculosis features. This process not only helps to select the most representative sub-features but also reduces the interference of redundant information during modeling, thereby improving the model's generalization ability in real-world medical scenarios. By reasonably setting the contribution level (feature weights) and optimizing the combination method, the model can more accurately capture subtle differences in tuberculosis lesions while enhancing the interpretability of the results, providing more reliable and transparent auxiliary evidence for clinical diagnosis.

[0024] The contribution value of each sub-feature ranges from 0 to 1, as shown in Table 1 below. The optimized contributions of the 32 sub-features include: miliary nodules (0.79), lung destruction (0.79), tree-in-bud sign (0.79), diabetes (0.77), bronchial stenosis (0.74), hemoptysis (0.71), history of tuberculosis exposure (0.71), history of tuberculosis (0.7), history of AIDS (0.7), calcification (0.7), cavity (0.69), cough (0.67), malnutrition (0.66), multiple nodules (0.66), and so on. The contribution of patchy opacities was 0.66, pleural thickening was 0.66, pleural effusion was 0.66, coughing up sputum was 0.64, fever was 0.63, pericardial effusion was 0.63, poor appetite was 0.61, emaciation was 0.61, enlarged lymph nodes were 0.6, night sweats were 0.56, linear opacities were 0.54, consolidation was 0.53, dullness on percussion was 0.51, dyspnea was 0.5, chest pain was 0.49, moist rales were 0.47, history of alcohol consumption was 0.46, and history of smoking was 0.41.

[0025] Table 1 shows the contribution of the 32 sub-features after optimization.

[0026] In this embodiment, the recognition model is trained using Natural Language Processing (NLP) technology, aiming to improve the parsing accuracy of medical clinical texts and enhance the recognition, extraction, and generalization performance of medical entities. Specifically, Chinese RoBERTa-wwm-ext is selected as the basic architecture. This model is a bidirectional pre-trained large language model based on a Transformer encoder-decoder structure, which has strong advantages in Chinese language understanding tasks. The training corpus comes from more than 200,000 real-world hospital electronic medical record texts. These data have undergone a rigorous manual annotation process, with a professional team ensuring the consistency and accuracy of the annotations. The annotation categories cover a variety of entity and attribute labels, including diseases, diagnoses, symptoms, drugs, surgeries, body parts, tests, examinations, onset dates, trends, frequencies, measurements, and physical examinations. Through supervised learning, the model is finely tuned using the cross-entropy loss function and the Adam optimizer. Overfitting is avoided during training, ultimately resulting in a BERT-EMR model specifically designed for medical scenarios. With a parameter size of 0.7B, this lightweight model can run efficiently on a CPU after optimization, without the need for high-end GPU support. Its Named Entity Recognition (NER) accuracy reaches 95%, significantly surpassing the 75% accuracy of the base model Chinese RoBERTa-wwm-ext. This is due to the thorough learning of domain-specific data and the effective application of fine-tuning strategies, thus demonstrating excellent practicality and reliability in medical NLP tasks.

[0027] Electronic Medical Record Text Refinement: To reduce the interference of textual noise on feature representation, this embodiment uses the Named Entity Recognition (NER) task of the BERT-EMR model to perform structured processing on the natural language text of the electronic medical record, thereby reconstructing a refined electronic medical record text that only contains tuberculosis features selected by the tuberculosis specialist. This process helps to focus on key medical information and improve the accuracy of subsequent analysis. For example, the original electronic medical record data may contain a lot of redundant details, such as "Chief complaint: lung shadow and pleural effusion discovered 1 month ago; Present illness: none; Physical examination: T: 36.8℃, P: 95 bpm, R: 22 bpm, BP: 131 / 88 mmHg, H: 175 cm." W: 60kg; General condition: Conscious, mentally sound, normal development, good nutrition, moderate build, walks into the ward independently, normal facial appearance, natural expression; Auxiliary examinations: None; CT scan report: Lung window: Spots, patches, consolidation, and masses are visible in the lower lobe of the right lung and the left lung. Bronchovascular bundles are thickened bilaterally, and linear shadows are visible in both lungs. The openings of the trachea and main bronchi are patent. Mediastinal window: The chest is symmetrical. The heart and great vessels are clearly visible; the heart is not enlarged. A dense, arc-shaped shadow is visible in the aortic wall and coronary artery course. The mediastinum is midline; the mediastinal lymph nodes are not enlarged. The left interlobar fissure and left pleural cavity are visible. Watery low-density shadows are observed. Bilateral pleural thickening and adhesions are present. Nodules and linear dense shadows are visible in the stomach. 1. Infectious lesions of the left lung are suspected, tuberculosis should be ruled out, and histological examination is recommended to rule out neoplastic lesions in the upper lobe of the left lung. 2. Linear shadows in both lungs suggest chronic inflammation or atelectasis. 3. Aortic and coronary artery sclerosis. 4. Left interlobar fissure effusion, loculated pleural effusion in the left side. 5. Bilateral pleural thickening and adhesions. 6. Nodules and linear dense shadows in the stomach, please correlate with clinical findings. After NER task processing, the output is simplified to the core features of "pleural effusion, patchy shadows, consolidation, bronchial stenosis, linear shadows, and pleural thickening".

[0028] To eliminate overfitting and underfitting of the model caused by imbalanced training data, this embodiment incorporates the expert-assessed contribution of features to tuberculosis risk. Numerical representation (values ​​range from 0 to 1, with larger values ​​indicating greater risk contribution). Risk factors for pulmonary tuberculosis. This includes multiple aspects, such as miliary nodules, lung destruction, tree-in-bud sign, diabetes, bronchial stenosis, hemoptysis, history of contact with pulmonary tuberculosis, history of tuberculosis, history of AIDS, calcification, cavitation, cough, malnutrition, multiple nodules, patchy shadows, pleural thickening, pleural effusion, sputum production, fever, pericardial effusion, poor appetite, weight loss, enlarged lymph nodes, night sweats, linear shadows, consolidation, dullness on percussion, dyspnea, chest pain, moist rales, history of alcohol consumption, and smoking history. This fusion process adjusts the importance of features through weighting, ensuring that the model focuses more on high-contribution features during training, thereby improving robustness.

[0029] Furthermore, to improve the model's generalization ability, this embodiment also establishes a thesaurus for disease-related vocabulary, uniformly converting non-standard words into standard words. For example, "poor appetite" is mapped to "loss of appetite." This standardization process reduces feature variation caused by differences in electronic medical record writing habits, enabling the model to extract feature vectors more consistently. By adjusting the text feature vectors, the model can maintain the stability of feature extraction under various electronic medical record writing styles, thereby significantly improving generalization ability.

[0030] After multiple rounds of training, the system automatically saves the optimal model parameters and uses this model to perform embedding tasks to extract features from the text modalities. This entire process ensures the effectiveness and reliability of electronic medical record text refinement, laying a solid foundation for subsequent tuberculosis risk assessment and clinical decision support.

[0031] The formula is expressed as follows:

[0032] in: The vectorization of the electronic medical record text has a dimension of 768; (String) indicates that the input string returns an m-dimensional vector, where m=768, and calls the Embedding algorithm of the BERT-EMR model; This indicates that the input string will be a string, and the output string will be a string in which synonyms are replaced with standard vocabulary. This represents the original text of the electronic medical record, which is a string. Element-wise multiplication (HadamardProduct) multiplies corresponding elements of two tensors / vectors with identical dimensions, while keeping the output dimension unchanged. Static weighting: 1. Calculate the contribution of physician assessments to risk factors. This indicates the doctor's assessment of a certain risk factor. The average contribution of the assessments, n>=7, indicates that at least 7 doctors participated in the assessment. The dimension vector represents the number of risk factors, with a value of 32. Values ​​range from 0 to 1; 2. Establish a mapping between risk factors and text tokens: map(i) = t; 3. Generate a static weight matrix. ; Dynamic weight adjustment: Dynamic weights are generated through a two-layer linear network and a gating mechanism (Sigmoid). The hidden_states are adjusted by combining residual connections to achieve dynamic optimization of weights in combination with context, thus making up for the fixed defects of static weights. Global feature aggregation: Take the weighted features of the [CLS] token as the 768-dimensional aggregated vector of the entire text.

[0033] Laboratory test result characterization method: Laboratory test results are structured data, such as the test item name and the form of test value pairs (the test value of C-reactive protein is 12.6 mg / L, the normal value of the reference range is less than 10 mg / L, and the mild elevation is 10 mg / L to 40 mg / L), and are processed as text.

[0034] The formula is expressed as follows:

[0035] in: This represents the laboratory test result embedding, with an input dimension of 4 (4 represents the number of test items) and an output of model dimension 512. This represents a multivariate set of values ​​for the test item; This represents a clinical reference range normalization function (which converts raw values ​​into "standardized values ​​relative to the reference range," preserving the semantics of abnormalities). This represents a linear projection layer (which maps normalized low-dimensional numerical vectors to 512 dimensions). Demographic characteristics (age, gender) representation method: Demographic characteristics are structured data, such as age: 60, gender: male, and are processed using a separate structured module.

[0036] The formula is expressed as follows:

[0037] in: This represents the population feature embedding, with an input dimension of 2 and an output dimension of 512 (the model dimension). Value represents a 2-dimensional vector containing age and gender information; ; B represents the batch size, and W represents the original structured feature dimension 2 (corresponding to age + gender). Stack(.) means concatenating the last dimension, merging two 1-dimensional features into a 2-dimensional vector; Struct: The structured processing module includes linear mapping, nonlinear activation GELU, layer normalization LayerNorm, and random deactivation Dropout, ultimately yielding a structured embedding.

[0038] The extraction of the first feature from the imaging examination image includes: First, data augmentation is performed on the imaging examination images; Then, a 3D ResNet18 model is used to extract 3D image features.

[0039] CT Image Feature Embedding: This invention first preprocesses a large number of CT images, i.e., data augmentation, such as adjusting the image size to 224×224, random rotation (±45°), converting color images to single-channel grayscale images, and standardizing single-channel images; then, it significantly improves the model's performance through transfer learning, such as modifying the number of fully connected layers to 2, the number of channels in the model's convolutional layers to 1, and defining the model training process. After multiple rounds of training on the images, the optimal model parameters are saved, and this model is used to extract multimodal image features.

[0040] The 3D ResNet18 model was used to extract features from 3D chest CT slice images. The advantage of using the ResNet1 model is that it utilizes a deep residual learning framework to solve the degradation problem that as the number of network layers increases, more features can be extracted, but at the same time, the training error will gradually increase, thus significantly improving the image classification ability.

[0041]

[0042] in: This represents the volume of a 3D image composed of D consecutive CT slices stacked together, with dimensions (C, D, H, W); C represents the number of channels (grayscale). Figure 1 D represents the depth (number of slices 96), H represents the height (256), and W represents the width (256). The 3DResNet model processing includes: initial convolutional layer (ConvInit); residual block groups (Block1-Block4): 3DResNet-18 contains 4 groups of residual blocks (2 residual blocks in each group), which gradually increases the feature dimension and compresses the spatial size; global pooling (GlobalAvgPool3D) represents the compression of the spatial dimension of the 3D feature map; flatten: converts it into a 1D vector, i.e., the final embedding.

[0043] The extraction of the second feature using laboratory test results includes: The first structured data feature in laboratory test results is extracted using a custom module; the first structured data feature includes the test item name and the test value. A custom module is used to extract second structured data features from laboratory test results; these features include the patient's age and gender.

[0044] The extraction of the third feature using electronic medical records includes: The text is refined by using the BERT-EMR model to structure the natural language text of the electronic medical record; the refined text of the electronic medical record is reconstructed to contain only the tuberculosis features selected by the tuberculosis specialist. Vocabulary standardization involves converting non-standard vocabulary in the refined text of electronic medical records into standard vocabulary. Feature extraction: Using NLP models to extract textual features from the refined text of electronic medical records; The risk contribution is integrated by incorporating the expert-assessed risk contribution of tuberculosis into the text features to obtain the third feature.

[0045] The refined text includes: First, the BERT-EMR model is used to generate electronic medical record feature vectors; Then, it is transformed into 512-dimensional features required by the multimodal model through the text_proj text projection layer; Finally, dynamic text_pos_embed position encoding is superimposed, and invalid text features are filtered out in conjunction with text_mask.

[0046] The modality fusion of the first feature, the second feature, and the optimized third feature includes: First, the second and third features are classified as text features, and the first feature is classified as an image feature; Then, a hybrid attention module is used to process textual and image features, including intramodal self-attention and cross-modal bidirectional cross-attention mechanisms, ultimately generating enhanced features. In the cross-modal bidirectional cross-attention mechanism, one path uses textual features (processed by intramodal self-attention) and population features as the query, image features (processed by intramodal self-attention) and population features as the key, and image features (processed by intramodal self-attention) as the value, focusing the text description on the relevant image region. The other path uses image features (processed by intramodal self-attention) and population features as the query, text features (processed by intramodal self-attention) and population features as the key, and text features (processed by intramodal self-attention) as the value, allowing the image region to query its corresponding text description. Each attention sub-layer is followed by a feedforward network, residual connections, and layer normalization. The features enhanced by the hybrid attention module are pooled along the sequence dimension to obtain aggregated embeddings of images and text. Finally, a bidirectional gating fusion module is used to perform attention correction of demographic information (age, gender), generation of bidirectional cross-features between images and text, generation of gating weights, and weighted feature fusion. The in-modal gating integrates the original features, the enhanced features processed by the hybrid attention module, and the bidirectional cross-features between text and images. Then, based on the gating weights, the contributions of each type of feature are dynamically allocated to generate the final fused features required by the classification head module.

[0047] As mentioned earlier, the prediction results of a single model are often limited by its own structure and characteristics, making it difficult to fully capture the complex information in images and electronic medical records.

[0048] First, the saved optimal BERT-EMR model is used to generate electronic medical record feature vectors. Then, the vectors are transformed into 512-dimensional features through the text_proj text projection layer and dynamic text_pos_embed position encoding is superimposed. At the same time, text_mask is used to filter invalid text features (such as blank fields) to adapt to the variable length of clinical text. Generate feature vectors of laboratory test results using custom modules; The 3DResNet18 model is used to generate CT image feature vectors, which are then mapped to sequence features with a uniform dimension of 512 by the img_pro projection layer. A custom module is used to process demographic information (age, gender), encoding the input 2D vector into 512-dimensional features. This avoids the structured data being "overwhelmed" by the high-dimensional features of text and image modalities, ensuring the effective participation of individual baseline information.

[0049] Secondly, a hybrid attention module consisting of three layers of hybrid attention (including intramodal self-attention mechanism and bidirectional cross-attention mechanism) is used to generate enhanced features (attended_img / attended_text).

[0050] Expressed as a formula:

[0051]

[0052] in: This represents the enhanced image features after multi-layered hybrid attention processing, i.e., attended_img; This represents the enhanced text features after multi-layered hybrid attention processing, i.e., attended_text; This represents the set of cross-attention weights across all attention layers, where each element is the attention weight after single-layer processing. HybridAttention refers to a hybrid attention module, which consists of 3 HybridAttentionLayer layers.

[0053] This represents the image features after the original CT image features have been processed by the projection layer. This represents the text features after the original text features have been processed by the projection layer.

[0054] This indicates the bias of population features on the image side. This indicates a bias in the demographic features of the text side.

[0055] HybridAttentionLayer represents a single-layer hybrid attention module, consisting of modal self-attention and cross-attention. This represents modal self-attention (including demographic feature bias), including image modal self-attention (self_attn_img) and text modal self-attention (self_attn_text). For example, image modal self-attention:

[0056] This represents the output features of self-attention within an image modality. Represents the self-attention weights within the image modality. The model has 512 dimensions. After processing with Dropout and Norm, the final output image features are obtained after self-attention enhancement, residual connections, and layer normalization. ).

[0057] Modal cross-attention includes image-to-text cross-attention (cross_attn_img) and text-to-image cross-attention (cross_attn_text). For example, image-to-text cross-attention:

[0058] This represents the final output image features after self-attention enhancement, residual connection, and layer normalization. Represents image-to-text cross-attention features; This represents the image-to-text cross-attention weights. After Dropout and Norm processing, the final output image features are obtained after cross-modal cross-attention enhancement, residual connections, and layer normalization. ).

[0059] Similarly, the text underwent the same processing described above, resulting in... That is, the final output text features after cross-modal cross-attention enhancement + residual connection + layer normalization, and then both ( The features are then passed through a feedforward network to output the final features.

[0060] Next, the BidirectionalGateFusion module is used to perform attention correction, directional cross-feature generation, gating weight generation, and feature weighted fusion of demographic features (age, gender). Structured attention correction: struct_attn_img and struct_attn_text are used to adjust the enhanced features. Directional cross-feature generation: cross_attn_img2text and cross_attn_text2img are used to calculate the attention between text and image to supplement the correlation between the original and enhanced features. Gating weight generation: The gate_gen module contains two layers of Linear, GELU, and Dropout to generate 6-dimensional weights (3 types of image features + 3 types of text features, summed to 1), which are then dynamically allocated the contribution ratio of each feature after softmax normalization. Feature weighted fusion: Based on the adjusted features and weights, multiple features are dynamically weighted and fused. The three types of image features (original, enhanced, and cross) are weighted and summed according to the image gating weights to obtain fused_img; the three types of text features are weighted and summed according to the text gating weights to obtain fused_text. After fusion, the core information of the original features is preserved through residual connections and layer normalization.

[0061] Expressed as a formula:

[0062] in: This represents the enhanced image features, i.e., the output after processing by the HybridAttention module. ; This represents the enhanced text features, i.e., the output after processing by the HybridAttention module. .

[0063] S represents the feature vector of raw population information (age, gender). .

[0064] The fused image features represent the final image features after integrating the original image features through a gating mechanism, the enhanced image features after processing by the hybrid attention module, and the image-to-text cross-attention features. It is the optimized result of image modality fusion of multi-source information. The fused text features are the final image features resulting from the integration of original text features through a gating mechanism, enhanced text features after processing by a hybrid attention module, and text-image cross-attention features. They represent the optimized result of text modality fusion with multi-source information.

[0065] explain represents model interpretability information, which is a dictionary that records key associations and weights in the model's decision-making process. It is used to explain questions such as "why the model focuses on certain features" and "how demographic information affects text and images".

[0066] The BidirectionalGateFusion module includes the following steps: Bidirectional cross-attention includes image-to-text cross-attention (cross_attn_img2text) based on original and enhanced features, and text-to-image cross-attention (cross_attn_text2img) based on original and enhanced features. For example, image-to-text cross-attention... , This represents the image-to-text secondary cross-attention feature.

[0067] Population feature information calibration (including attention weights) includes population feature to image attention struct_attn_img (quantifying the impact of population features on the image) and population feature to text attention struct_attn_text (quantifying the impact of population features on the text).

[0068] For example, population features to image attention:

[0069] Represents extended structured features, Representing population features to image attention features, This represents the population features to image attention weights.

[0070] Gating weight generation and feature fusion include generating image gating weights and text gating weights, and fusing their respective features. Image and text gating weights are generated; the global gating output is:

[0071] A representative vector representing the original image features, i.e., a vector obtained by globally aggregating the original CT features; A representative vector representing the original text features, i.e., a vector obtained by globally aggregating the original text features; This represents a representative vector of the attention-enhanced image features, which is the vector obtained by globally aggregating the enhanced image features output by the HybridAttention module. This represents a representative vector of attention-enhanced text features, which is the vector obtained by globally aggregating the enhanced text features output by the HybridAttention module.

[0072] Image fusion gating weights:

[0073] Text incorporates gating weights:

[0074] Feature fusion, such as image feature fusion (original + enhanced + cross), results in the following image features: , .

[0075] The Kth feature component of image fusion .

[0076] Finally, the fused image features (fused_img), text features (fused_text), and population features (structured information structured_feat) are concatenated into a final feature of 1536 dimensions (512×3), which is then input into a classifier (including GELU activation, Dropout, and LayerNorm, etc.) for prediction, generating binary logits. After Softmax activation, these logits are converted into "negative confidence" and "positive confidence". The classification criteria are then set with reference to the model's risk level threshold table, and the risk level and confidence are output.

[0077] Expressed as a formula:

[0078]

[0079] in: Logits represent the raw scores output by the classifier; MLP stands for Multilayer Perceptron, a neural network module consisting of linear layers, activation functions, and regularization, responsible for mapping high-dimensional features to a low-dimensional classification space; Concat_feat indicates the fusion of multimodal features after concatenation [fusedtext;fusedimage;] ]; prob represents the normalized prediction probability, with the same dimension as logits ([B,2]). Softmax() represents the normalization exponential function, which converts any real number input into a probability value in the interval (0,1), and the sum of the probabilities of all classes is 1.

[0080] During model training, setting appropriate training parameters is crucial for improving model performance. This invention meticulously tunes and determines key parameters such as learning rate, number of iterations, and batch size for the BERT text model, the 3DResNet18 image model, and multimodal models, ensuring that each model achieves optimal fitting and generalization ability on its corresponding modality data. Through cross-validation and early stopping strategies, the risk of overfitting is effectively avoided, while fully leveraging the advantages of various models in feature extraction and pattern recognition.

[0081] In terms of model application scenarios and automatic model optimization, the system design closely aligns with actual clinical workflows. In hospital outpatient or inpatient settings, after a doctor writes electronic medical records or a patient completes laboratory tests and imaging examinations such as X-rays and CT scans, the system automatically retrieves the patient's electronic medical record data from LIS (Laboratory Information System), HIS (Hospital Information System), and PACS (Picture Archiving and Communication System), and performs multi-source information aggregation and alignment. The integrated data is then pushed to the tuberculosis risk assessment model, which generates a tuberculosis risk probability through multimodal fusion inference and classifies it into high, medium, and low risk levels based on preset thresholds. The results are communicated to attending physicians and public health physicians in real time via interface pop-ups and push notifications, assisting their decision-making and prompting them to conduct further targeted tests to clarify the diagnosis.

[0082] The model possesses a continuous optimization mechanism. After running for a period of time, the system collects expert evaluations of the model's predictions and actual judgment probabilities, forming newly labeled training samples. These samples are merged with the original data to retrain the model, iteratively fine-tuning the model structure and hyperparameters. The training process generates updated model parameter changes Δm. During actual inference, the system simultaneously loads the base model M and the incremental model Δm, introducing feature contribution weights based on expert experience to achieve dynamic fusion of multiple model outputs. This mechanism not only improves the model's adaptability and accuracy but also enables the model's self-evolution and continuous learning in a real clinical environment.

[0083] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented, in whole or in part, as a computer program product, the computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0084] The above description is only a preferred embodiment of the present invention. It should be noted that any improvements, modifications, substitutions or variations made by those skilled in the art without departing from the principle of the present invention should be considered as being included within the protection scope of the present invention.

Claims

1. A tuberculosis identification system based on multimodal fusion and expert-assisted optimization, characterized in that, include: The data acquisition module collects imaging examination images, laboratory test results, and electronic medical records; The data preprocessing module extracts the first feature from imaging examination images, the second feature from laboratory test results, and the third feature from electronic medical records. The third feature includes 32 sub-features and the contribution of each sub-feature. The 32 sub-features include: miliary nodules, lung destruction, tree-in-bud sign, diabetes, bronchial stenosis, hemoptysis, history of contact with pulmonary tuberculosis, history of tuberculosis, history of AIDS, calcification, cavitation, cough, malnutrition, multiple nodules, patchy shadows, pleural thickening, pleural effusion, sputum production, fever, pericardial effusion, poor appetite, weight loss, enlarged lymph nodes, night sweats, linear shadows, consolidation, dullness on percussion, dyspnea, chest pain, moist rales, history of alcohol consumption, and history of smoking. The optimization module optimizes contribution based on expert diagnostic experience; The fusion module performs modal fusion on the first feature, the second feature, and the optimized third feature to generate tuberculosis identification results.

2. The tuberculosis identification system based on multimodal fusion and expert-assisted optimization according to claim 1, characterized in that, The optimized contributions of the 32 sub-features include: miliary nodules (0.79), lung destruction (0.79), tree-in-bud sign (0.79), diabetes (0.77), bronchial stenosis (0.74), hemoptysis (0.71), history of tuberculosis exposure (0.71), history of tuberculosis (0.7), history of HIV / AIDS (0.7), calcification (0.7), cavitation (0.69), cough (0.67), malnutrition (0.66), multiple nodules (0.66), patchy shadows (0.66), and pleura. The contribution of thickening was 0.66, pleural effusion was 0.66, coughing up sputum was 0.64, fever was 0.63, pericardial effusion was 0.63, poor appetite was 0.61, emaciation was 0.61, enlarged lymph nodes were 0.6, night sweats were 0.56, linear opacities were 0.54, consolidation was 0.53, dullness on percussion was 0.51, dyspnea was 0.5, chest pain was 0.49, moist rales were 0.47, history of alcohol consumption was 0.46, and history of smoking was 0.

41.

3. The tuberculosis identification system based on multimodal fusion and expert-assisted optimization according to claim 1 or 2, characterized in that, The extraction of the first feature from the imaging examination image includes: First, data augmentation is performed on the imaging examination images; Then, a 3D ResNet18 model is used to extract 3D image features.

4. The tuberculosis identification system based on multimodal fusion and expert-assisted optimization according to claim 1 or 2, characterized in that, The extraction of the second feature using laboratory test results includes: The first structured data feature in laboratory test results is extracted using a custom module; the first structured data feature includes the test item name and the test value. A custom module is used to extract second structured data features from laboratory test results; these features include the patient's age and gender.

5. The tuberculosis identification system based on multimodal fusion and expert-assisted optimization according to claim 1 or 2, characterized in that, The extraction of the third feature using electronic medical records includes: The text is refined by using the BERT-EMR model to structure the natural language text of the electronic medical record; the refined text of the electronic medical record is reconstructed to contain only the tuberculosis features selected by the tuberculosis specialist. Vocabulary standardization involves converting non-standard vocabulary in the refined text of electronic medical records into standard vocabulary. Feature extraction: Using NLP models to extract textual features from the refined text of electronic medical records; The risk contribution is integrated by incorporating the expert-assessed risk contribution of tuberculosis into the text features to obtain the third feature.

6. The tuberculosis identification system based on multimodal fusion and expert-assisted optimization according to claim 5, characterized in that, The refined text includes: First, the BERT-EMR model is used to generate electronic medical record feature vectors; Then, it is transformed into 512-dimensional features required by the multimodal model through the text_proj text projection layer; Finally, dynamic text_pos_embed position encoding is superimposed, and invalid text features are filtered out in conjunction with text_mask.

7. The tuberculosis identification system based on multimodal fusion and expert-assisted optimization according to claim 1, characterized in that, The modality fusion of the first feature, the second feature, and the optimized third feature includes: First, the second and third features are classified as text features, and the first feature is classified as an image feature; Then, a hybrid attention module is used to process textual and image features, including intramodal self-attention and cross-modal bidirectional cross-attention mechanisms, subsequently generating enhanced features. In the cross-modal bidirectional cross-attention mechanism, one path uses textual and demographic features processed by the intramodal self-attention mechanism as the query, image and demographic features processed by the intramodal self-attention mechanism as the key, and image features processed by the intramodal self-attention mechanism as the value, focusing textual descriptions on relevant image regions. The other path uses image and demographic features processed by the intramodal self-attention mechanism as the query, textual and demographic features processed by the intramodal self-attention mechanism as the key, and textual features processed by the intramodal self-attention mechanism as the value, allowing image regions to query their corresponding textual descriptions. Each attention sub-layer is followed by a feedforward network, residual connections, and layer normalization. The features enhanced by the hybrid attention module are pooled along the sequence dimension to obtain aggregated embeddings of images and text. Finally, the bidirectional gating fusion module is used to perform attention correction of population feature information, generation of bidirectional cross features of images and text, generation of gating weights, and weighted fusion of features. The gating integration of the original features, the enhanced features after processing by the hybrid attention module, and the bidirectional cross features of text and images within the modality is then dynamically allocated according to the gating weights to generate the final fused features required by the classification head module.

Images