A disease high-risk population screening method based on multi-modal fusion

Abstract

The application discloses a disease high-risk population screening method based on multi-modal fusion, first collects multi-modal medical data related to lung cancer high-risk screening of a to-be-screened object, and carries out pretreatment and data quality evaluation on the collected multi-modal medical data, then constructs a disease high-risk population screening model based on multi-modal fusion, carries out multi-modal feature extraction through a modal feature extraction network, introduces a context adaptive fusion (CAF) module for multi-modal fusion, obtains a fusion feature vector, and carries out risk grading and report generation through a risk output layer. The method of the application significantly improves the accuracy and reliability of lung cancer high-risk screening of the population, effectively reduces the false positive rate and the missed diagnosis rate, enhances the robustness in a complex clinical environment, the model decision is more in line with medical common sense, can effectively handle the information conflict between modes, and greatly improves the credibility and explainability of the decision.

Description

Technical Field

[0001] This invention belongs to the field of medical artificial intelligence-assisted screening technology, specifically relating to a screening method for high-risk populations of diseases based on multimodal fusion. Background Technology

[0002] Early screening of high-risk groups is crucial for improving patient survival rates. Taking lung cancer as an example, lung cancer is a malignant tumor with the highest incidence and mortality rates worldwide. Current lung cancer screening methods mainly rely on single-modality medical data analysis, such as low-dose spiral CT (LDCT) imaging screening, blood tumor marker testing, or clinical history assessment based on electronic health records (EHRs).

[0003] In existing technologies, medical imaging, especially LDCT, is considered one of the main methods for early lung cancer screening. By analyzing lung CT images, doctors can detect abnormalities such as lung nodules and lesions and assess their malignancy risk. However, image interpretation is highly dependent on the radiologist's experience and suffers from problems such as high false positive rates and overdiagnosis. Blood tumor markers (such as CEA, CYFRA21-1, NSE, etc.) also play a role in the auxiliary diagnosis and treatment monitoring of lung cancer. These indicators provide numerical evidence for lung cancer risk assessment by detecting the levels of specific biomolecules in the blood. However, the sensitivity and specificity of a single tumor marker are often insufficient for early independent diagnosis and are easily affected by non-specific factors such as inflammation and other diseases. Electronic health records (EHRs) contain rich textual and structured numerical information, including the patient's medical history, family history, lifestyle habits (such as smoking history and occupational exposure history), previous medical records, and symptom descriptions. This information is also of great reference value for assessing a patient's lung cancer risk. For example, a long history of smoking and a family history of lung cancer are recognized high-risk factors for lung cancer. However, EHR data is usually in unstructured or semi-structured form and is very large in volume, making it difficult to perform efficient and comprehensive automated analysis.

[0004] In addition, some existing studies have begun to explore simple combinations of data from different modalities, such as combining imaging results with clinical symptoms or tumor markers, in order to improve the accuracy of screening. Common fusion methods include feature concatenation, weighted summation, or decision-level fusion. These methods, to some extent, compensate for the shortcomings of single-modality data and provide multi-dimensional information support for lung cancer screening.

[0005] Despite some progress in existing lung cancer screening methods, significant limitations remain, restricting their clinical efficacy and reliability:

[0006] (1) Limitations of single-modality information: Imaging (such as LDCT) has problems such as high false positive rate and strong subjective interpretation; blood tumor markers have low specificity and are easily affected by non-tumor factors; electronic health record (EHR) data is unstructured, information is redundant and of inconsistent quality, making it difficult to use efficiently.

[0007] (2) Original multimodal fusion strategies: Most existing fusion methods adopt fixed fusion strategies, which have four major problems: First, they lack context adaptation capabilities and cannot dynamically adjust information weights according to individual patient characteristics and data quality; second, they have poor robustness and are difficult to suppress noise interference caused by low-quality or missing data; third, the fusion mechanism is simple and fails to fully explore the deep complementary relationship between various modalities; and fourth, they have poor interpretability and their "black box" decision-making process hinders clinical trust and application.

[0008] (3) Insufficient use of medical prior knowledge: Most existing methods are purely data-driven and ignore medical prior knowledge (such as combinations of high-risk factors, typical clinical manifestations, etc.), which leads to the model lacking the ability to make judgments and corrections based on medical common sense when faced with information conflicts or rare cases, and is prone to making wrong diagnoses.

[0009] In summary, existing lung cancer screening methods have significant shortcomings in terms of data processing, fusion strategies, contextual adaptability, robustness, and utilization of medical knowledge, and there is an urgent need for a new, more intelligent and reliable screening paradigm. Summary of the Invention

[0010] To address the aforementioned technical problems, this invention provides a method for screening high-risk populations for diseases based on multimodal fusion, which overcomes the challenges of existing multimodal fusion methods, such as lack of context adaptation capabilities, difficulty in effectively handling data quality issues, and insufficient utilization of prior medical knowledge.

[0011] The technical solution adopted in this invention is: a method for screening high-risk populations for diseases based on multimodal fusion, the specific steps of which are as follows:

[0012] S1. Collect multimodal medical data related to high-risk lung cancer screening of the subjects to be screened, and preprocess the collected multimodal medical data;

[0013] The multimodal medical data includes: electronic health record (EHR) text, lung CT images, and structured numerical index data.

[0014] The structured numerical index data includes: demographic information, laboratory test results, and vital signs.

[0015] Then, the collected multimodal medical data is preprocessed to ensure that the data formats of each modality are consistent, the quality meets the standards, and the data is aligned in the time dimension.

[0016] The specific preprocessing of the disease data is as follows:

[0017] 1) Preprocessing of lung CT images;

[0018] The lung CT images were subjected to format conversion, resampling, noise reduction and enhancement, and quality verification to form a two-dimensional slice sequence.

[0019] 2) EHR text preprocessing;

[0020] First, the EHR text data was anonymized to ensure patient privacy. Then, word segmentation and terminology standardization were performed, mapping synonymous medical terms to a unified code. Irrelevant and redundant information and obvious errors were removed. After word segmentation and lexical normalization, the text data was converted into a sequence format acceptable to the model, while retaining necessary contextual information.

[0021] 3) Preprocessing of structured numerical index data;

[0022] The structured numerical index data of patients is normalized and missing values ​​are handled, including filling missing values ​​with statistical or interpolation methods, normalizing indices of different dimensions, and detecting outliers for correction or labeling as needed; at the same time, it is ensured that the structured numerical index data from different sources are aligned with the corresponding images and text in time.

[0023] S2. Based on step S1, conduct a data quality assessment on the preprocessed multimodal medical data;

[0024] S21. Image modal quality assessment;

[0025] First, the signal-to-noise ratio is calculated using the Region of Interest (ROI) analysis method. In homogeneous regions of lung CT images, the mean pixel value is calculated as the signal intensity within the selected region of interest (ROI). The pixel standard deviation in the image background region is calculated as the noise intensity. ,Right now .

[0026] Then, the sharpness index is calculated using the Laplacian operator. This involves performing convolution operations on the two-dimensional slice sequence using a Laplacian convolution kernel to obtain the corresponding edge maps, and determining the sharpness index based on the variance of pixel values ​​in each edge map. .

[0027] The two-dimensional slices are sequences of two-dimensional slices formed from preprocessed lung CT images.

[0028] Then, the index values ​​are mapped to the [0,1] interval using the min-max normalization method, and the final image quality score is calculated by normalized weighted sum. The expression is as follows:

[0029] (1);

[0030] in, and This represents the preset weighting coefficient. This represents the normalization function, which maps the index to the [0,1] interval. and represents the normalized signal-to-noise ratio and Laplace variance, respectively.

[0031] S22, Text Modal Quality Assessment;

[0032] Text quality score Information integrity score And recording clarity score The weighted combination yields the following:

[0033] 1) Information completeness score ;

[0034] Referring to the predefined set of key information fields (KIFs) for disease-related information defined by clinical experts and the preset weight of each field, key information fields are extracted from the patient's EHR, and the coverage of the key information fields (KIFs) is calculated as follows:

[0035] (2);

[0036] in, This indicates the total number of predefined key information fields; Indicates the first The pre-defined weights of each KIF represent its clinical importance; The score indicates the detection status. If the information is successfully extracted from the text, it is 1; otherwise, it is 0.

[0037] 2) Recording clarity score ;

[0038] By quantifying the density of non-standard abbreviations in text Clinical signal-to-noise ratio calculate The expressions for the density of non-standard abbreviations and the clinical signal-to-noise ratio are as follows:

[0039] (3);

[0040] (4);

[0041] in, This refers to the number of non-standard abbreviations, custom terms, or misspelled words identified in the text after comparison with authoritative medical dictionaries and standard abbreviation databases; This indicates the total number of words in the EHR text. This indicates the number of words in the text that match a predefined "signal" vocabulary. This indicates the number of words in the text that match a predefined "noise" vocabulary; This represents a very small smooth term to prevent the denominator from being zero.

[0042] Then the clarity score The expression is as follows:

[0043] (5);

[0044] in, This represents the balancing weight coefficient.

[0045] 3) Text quality score ;

[0046] (6);

[0047] in, and This indicates the preset weight.

[0048] S23. Modal quality assessment of structured numerical index data;

[0049] Structured numerical index data quality score Score based on raw data completeness and data reliability score The weighted combination yields the following calculation expression:

[0050] (7);

[0051] (8);

[0052] (9);

[0053] in, and Indicates the preset weight. This indicates the number of non-missing feature values ​​before imputation; This indicates the total number of features that should be included in the "standard structured feature panel" as predefined in the screening protocol based on the actual situation, and which should be present in all patients; This indicates the number of outliers identified among the non-missing features.

[0054] S3. Construct a screening model for high-risk populations of diseases based on multimodal fusion;

[0055] The screening model includes: a modality feature extraction network, a context-adaptive fusion CAF module, and a risk output layer.

[0056] The modal feature extraction network includes: an image modal feature extraction network, a text modal feature extraction network, and a structured numerical index data feature extraction network.

[0057] S4. Based on step S3, multimodal feature extraction is performed through a modal feature extraction network, that is, feature extraction is performed on each modal data after quality assessment.

[0058] S41. Image modal feature extraction;

[0059] Visual feature extraction was performed on the standardized multi-slice CT image tensors using a 3D deep convolutional neural network (3D-CNN). The 3D-CNN network directly processes the three-dimensional volumetric data through 3D convolutional kernels, capturing key semantic information in the images, and finally obtaining a feature vector representing the entire lung image through a global average pooling layer. .

[0060] The key semantic information includes: the size, density, and shape of the suspicious nodules, as well as the spatial distribution of the lesions.

[0061] S42. Text modality feature extraction;

[0062] A pre-trained medical language model is used to encode EHR text sequences. First, the segmented text is input into the model's embedding layer to generate a sequence of word vectors. Then, deep semantic features are extracted through stacked Transformer encoding layers. Finally, global text semantic vectors are extracted through pooling or self-attention mechanisms. .

[0063] S43. Extraction of modal features from structured numerical index data;

[0064] For structured numerical index data, a multilayer perceptron (MLP) is used for encoding. All numerical features are arranged into vectors in a predetermined order and input into the network. Nonlinear combination features are extracted through several fully connected layers to obtain the structured data feature vector. .

[0065] S5. Based on the modal features extracted in step S4, multimodal fusion is performed using the context adaptive fusion CAF module to obtain the fused feature vector;

[0066] The CAF module includes: a context encoding layer, a hierarchical gating mechanism, a cross-modal attention fusion module, and a knowledge-guided dynamic adjustment module.

[0067] The context coding layer is used to perform context-aware coding based on the patient's electronic health record (EHR) text and structured numerical indicator data to generate a patient context vector. The hierarchical gating mechanism applies two levels of control to the fusion of multimodal features: global gating at the overall modality level and local gating at a fine-grained level within each modality. The cross-modal attention fusion module employs a multi-head cross-modal attention mechanism to achieve deep information interaction and alignment between modalities. The knowledge-guided dynamic adjustment module adjusts based on prior medical knowledge and the patient's individual context vector. Dynamic adjustment factors are generated based on reward and punishment rules. .

[0068] S6. Based on the fused feature vector output in step S5, risk classification and report generation are performed through the risk output layer, that is, high-risk identification and interpretation report generation are performed.

[0069] Furthermore, step S5 is specifically as follows:

[0070] S51, Context Coding Layer;

[0071] The context coding layer is used to perform context-aware coding based on the patient's electronic health record (EHR) text and structured numerical indicator data to generate a patient context vector. .

[0072] 1) Define a global context feature set, that is, determine a globally shared feature set based on clinical guidelines. A set consisting of the most critical background features;

[0073] First, clinical entities and structured candidate variables were extracted from the electronic health record (EHR) text and structured numerical index data of all samples to form an initial feature set. Then, candidate variables related to lung cancer risk were screened, and the Boruta-SHAP and robust univariate Cox regression methods were used to rank the candidate variables by importance, resulting in a refined feature candidate list. Finally, a forward selection strategy was used to determine the optimal number of features. And select the corresponding feature from the feature candidate list. These features constitute the global context feature set.

[0074] The text entities extracted from the EHR text include: smoking history, family history, symptom description, past medical history, and occupational exposure history. The structured variables extracted from the structured numerical index data include: age, gender, laboratory test results, and vital signs.

[0075] 2) Generate individual context vectors;

[0076] When the model performs individual screening, it extracts corresponding data from the patient's EHR text and structured numerical index data based on the global context feature set. Each feature value constitutes its individual feature vector. ,Will Input a small feedforward neural network to generate a patient-specific context vector. The expression is as follows:

[0077] (10);

[0078] in, This indicates that the patient corresponds to An individual context feature vector composed of context feature values; This represents the learnable weight matrix of the context encoding layer; This represents the bias vector of the context coding layer; Represents a nonlinear activation function; This represents the generated patient context vector, and is used to construct the patient context vector. The features are limited to EHR text data and structured numerical index data only.

[0079] S52, layered gating mechanism;

[0080] 1) Global gating layer;

[0081] The global gating layer assigns weight coefficients to each mode at the modal level. Measure modality The credibility and importance in the current decision are calculated using the following formula:

[0082] (11);

[0083] in, This represents the Sigmoid activation function. , , Indicates learnable parameters; Represents the set of real numbers; Representing the patient context vector The dimension; This indicates the transpose operation. Representing modes Data quality score, , , , These represent the data quality scores for the image modality, text modality, and structured numerical index data modality, respectively.

[0084] 2) Departmental control layer;

[0085] For mode eigenvectors To regard it as containing Each local feature representation is processed through an attention mechanism. Calculate a scalar score The local weights are then obtained by Softmax normalization. The expression is as follows:

[0086] (12);

[0087] (13);

[0088] in, , Representing modes The Local feature representation. , and Indicates learnable parameters, Represents the weight matrix. This represents the bias vector. This represents the weight vector used to map the hidden layer representation to the final scalar score.

[0089] 3) Unification and integration;

[0090] For modes The local features are summed according to their weights to obtain the mode. Feature vectors fused with internal attention; then multiplied by global gating coefficients. The final gating coefficient is obtained. The expression is as follows:

[0091] (14);

[0092] in, . These represent the final gating coefficients corresponding to the image modality, text modality, and structured numerical index data modality, respectively.

[0093] S53, Cross-modal attention fusion;

[0094] 1) Single-head cross-modal attention computation;

[0095] For any pair of modes and Setting the mode For query, modal To be retrieved, the attention output expression is defined as follows:

[0096] (15);

[0097] in, , , , This represents the learnable projection matrix. This represents the scaling factor.

[0098] 2) Multi-head fusion output;

[0099] use A parallel attention head independently learns the alignment relationships between three pairs of modalities in different projection subspaces. Attention head to modality and The calculation expression is as follows:

[0100] (16);

[0101] in, , , , , Indicates the first Each head has its own projection matrix. Each pair of modal directions is obtained... The output of each head is then concatenated along the vector dimension and linearly projected to obtain... The expression is as follows:

[0102] (17);

[0103] in, This represents a learnable projection matrix that maps the concatenated outputs of each attention head along the vector dimension to the target feature space. This represents a vector concatenation operation. Indicates the first Attention in modality Pointing mode Cross-modal attention output.

[0104] Finally, the fused representations of the three directions are concatenated again to obtain the feature representations of the three modalities, i.e., the preliminary fused features. The expression is as follows:

[0105] (18);

[0106] S54, Knowledge-guided dynamic adjustment module;

[0107] 1) Dynamic adjustment factor Formal definition;

[0108] (19);

[0109] in, This represents a dynamic adjustment factor that guides knowledge acquisition. ∈{0,1} represents the activation flag of the reward rule; when a high-risk rule in the knowledge graph is matched... =1, otherwise 0; ∈{0,1} represents the activation flag of the penalty rule, which is triggered when inter-modal conflicts or extremely low data quality are detected. =1, otherwise 0; and This represents two small neural networks that receive context vectors. As input, output the reward intensity and punishment intensity in the context of this patient.

[0110] 2) Triggering mechanism for reward and punishment rules:

[0111] Award logo and punishment signs The value is determined by the three-stage algorithm process, as follows:

[0112] Phase 1: Extraction of clinical atomic entities;

[0113] From the original features or intermediate vectors of each modality, discrete, clinically significant, and indivisible "clinical atomic entities" are extracted to form a patient entity set. .

[0114] Structured data: For each structured numerical indicator, a threshold is set according to clinical guidelines for binarization judgment; if an indicator meets the corresponding threshold condition, the threshold judgment result is transformed into a clinical atomic entity with clear clinical significance and added to the patient entity set. .

[0115] Text data: Based on a predefined medical dictionary, high-risk symptom entities that match in the EHR text are extracted using a keyword matching algorithm and added to the patient entity set as clinical atomic entities. ;

[0116] Image Data: A 3D-CNN was constructed as a multi-task learning model. The primary task was to predict the high risk probability of lung cancer in patients, which outputs the model's risk score. Multiple image feature classification tasks were set up in parallel as auxiliary tasks to learn interpretable image feature representations. Each auxiliary task head was specifically responsible for identifying a specific image feature. When screening new patients, the trained auxiliary task heads output the probability of the presence of key image features. If this probability met a preset threshold, the corresponding clinical atomic entity in the image was added to the patient entity set. .

[0117] The auxiliary task shares the 3D-CNN image feature extraction backbone network with the final main task, and is jointly optimized with the main task through the total loss function.

[0118] Phase Two: Knowledge Graph Rule Base Matching;

[0119] A medical knowledge graph is constructed by integrating publicly available medical ontology databases and authoritative clinical guidelines on high-risk lung cancer. The database stores... Each rule It is itself a condition set composed of multiple clinical atomic entities, representing a logical "AND" combination.

[0120] The publicly available medical ontology includes: SNOMED CT, ICD-11; the authoritative clinical guidelines include: NCCN, CSCO.

[0121] The matching algorithm uses set inclusion relationship judgment, that is, it traverses every rule in the rule base. If the condition set of the rule is the patient entity set If a subset is selected, then the decision rule is... A match was successfully made.

[0122] Phase 3, Assigning Reward / Penalty Flags:

[0123] Award logo If at least one rule is successfully matched in Phase Two, the reward rule is activated. Set to 1; if no rule is matched, then Set to 0.

[0124] Punishment sign If intermodal information conflict is detected, or if the data quality score of any modality is lower than a preset extreme quality threshold, the penalty rule activation flag will be set. Set to 1; if there is no intermodal information conflict and the data quality of each modality is not lower than the preset extreme quality threshold, then Set to 0.

[0125] The intermodal information conflict refers to a clear contradiction in the evidence provided by different modalities; the extremely low data quality refers to a data quality score for any modality being lower than a preset extreme quality threshold, indicating that the data of that modality is basically unreliable.

[0126] Finally, after dynamic adjustment factors After adjustment, the CAF module outputs the final, knowledge-corrected fused feature vector. .

[0127] Furthermore, step S6 is specifically as follows:

[0128] S61. Risk score calculation;

[0129] Fusion feature vectors After processing by a fully connected discriminator, the patient's continuous risk score is obtained. The specific calculation expression is as follows:

[0130] (20);

[0131] in, This represents the final fused feature vector after knowledge correction; and This represents the output layer weight matrix and bias terms. Represents the Sigmoid activation function, ensuring This refers to the lung cancer risk probability predicted by the model.

[0132] Then, the continuous risk score output by the model is applied using a pre-defined lung cancer risk grading standard. Discrete grading is used, with each grade corresponding to specific clinical recommendations.

[0133] The risk classification criteria are as follows:

[0134] First, the continuous risk score output by the model. Mapping to discrete risk levels corresponding to preset risk grading standards; the mapping process involves setting a set of thresholds. accomplish, This indicates the total number of thresholds set.

[0135] S62. Generation of explanatory reports;

[0136] 1) Analyze global modal contributions;

[0137] The system first calculates the global gating weights based on the multimodal fusion process. The report outputs the contribution percentage of each data modality to the final risk assessment, providing doctors with a high-level attribution overview.

[0138] 2) Extract key evidence for each modality;

[0139] For each modal data, the most influential specific features are identified, as follows:

[0140] Text Modality: Utilizing the attention weights learned by the model in the local control layer, automatically highlight the keywords and phrases in the EHR text that contribute the most to the decision.

[0141] Image modality: A saliency heatmap is generated using a gradient-weighted class activation mapping method, and the suspicious regions of most interest to the model are delineated on the original CT images.

[0142] Structured numerical indicator data modality: The impact is quantified by calculating the gradient of the final risk score relative to each structured input indicator. Indicators with larger absolute gradient values ​​are identified as "key structured indicators" and included in the report.

[0143] 3) Backtracking knowledge graph reasoning;

[0144] If the dynamic adjustment factor guided by the knowledge graph Triggered during the calculation, the report will automatically reference the matched medical rules or detected information conflicts, explaining in text form why the risk score was adjusted, making the model's dynamic intervention process transparent.

[0145] Furthermore, in step S3, the screening model undergoes model training, specifically as follows:

[0146] The training dataset comprises authoritatively annotated lung CT images, EHR text, and structured test metrics, and is divided into training, validation, and test sets using stratified sampling based on risk level. The model is jointly optimized end-to-end, with the total loss function... It is a weighted sum of the main task loss, auxiliary task loss, and regularization term, expressed as follows:

[0147] (twenty one);

[0148] in, The loss for the main task is represented by the binary cross-entropy loss, expressed as follows:

[0149] (twenty two);

[0150] in, Indicates the training batch size. Indicates the first The true label for each sample is 1 if the patient has cancer, and 0 otherwise. This represents the predicted risk probability of lung cancer by the model, i.e., the [missing value]. Continuous risk score for each patient. Indicates the first in the image module The loss function of each auxiliary feature classification head includes There are 1 parallel auxiliary task, and each task is a binary classification problem, which also uses binary cross-entropy loss. This represents all learnable parameters of the model. Represents the L2 norm; This represents the weighting coefficient for auxiliary tasks; This represents the regularization decay coefficient.

[0151] Furthermore, in the screening model, each modality feature extraction network is pre-trained to obtain initial parameters, and during the overall training phase, the parameters of each modality feature extraction network are not fixed. They are jointly fine-tuned end-to-end with the CAF module and the risk output layer through the total loss function, while an early stopping mechanism is introduced.

[0152] The beneficial effects of this invention are as follows: The method of this invention first collects multimodal medical data related to high-risk lung cancer screening of the subjects to be screened, and preprocesses and assesses the data quality of the collected multimodal medical data. Then, it constructs a screening model for high-risk populations based on multimodal fusion, extracts multimodal features through a modal feature extraction network, introduces a context-adaptive fusion (CAF) module for multimodal fusion, and obtains a fused feature vector. Finally, it performs risk classification and report generation through a risk output layer. The method of this invention introduces a context-adaptive fusion (CAF) module, which, through hierarchical gating and cross-modal attention, achieves dynamic and adaptive deep fusion of multi-source medical data (images, text, and structured data) based on individual patient characteristics and data quality. This effectively suppresses noise and highlights key information. It utilizes prior medical knowledge (including knowledge graph rules) to perform domain reinforcement and conflict correction on the fusion results, ensuring that the decision conforms to medical common sense and is interpretable. This overcomes the shortcomings of existing methods in terms of context adaptability, data quality robustness, and insufficient utilization of medical knowledge. The method of this invention significantly improves the accuracy and reliability of high-risk lung cancer screening in the population, effectively reduces the false positive rate and the missed diagnosis rate, enhances robustness in complex clinical environments, and the model decision is more in line with medical common sense. It can also effectively handle information conflicts between modalities, greatly improve the credibility and interpretability of the decision, and generate detailed explanatory reports to provide clinicians with quantitative risk assessment and intuitive decision-making basis. Attached Figure Description

[0153] Figure 1 This is a flowchart of a disease high-risk population screening method based on multimodal fusion according to the present invention.

[0154] Figure 2 This is a diagram showing the internal structure of the Context Adaptive Fusion (CAF) module in an embodiment of the present invention. Detailed Implementation

[0155] The method of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0156] like Figure 1 The flowchart of a disease high-risk population screening method based on multimodal fusion of the present invention is shown below. The specific steps are as follows:

[0157] S1. Collect multimodal medical data related to high-risk lung cancer screening of the subjects to be screened, and preprocess the collected multimodal medical data;

[0158] This embodiment is set in a high-risk lung cancer screening scenario. The multimodal medical data includes: electronic health record (EHR) text (medical history, treatment records), lung CT images (medical imaging data), and structured numerical index data.

[0159] The structured numerical index data includes: demographic information, laboratory test results, and vital signs.

[0160] Then, the collected multimodal medical data is preprocessed to ensure that the data formats of each modality are consistent, the quality meets the standards, and the data is aligned in the time dimension.

[0161] The specific preprocessing of the disease data is as follows:

[0162] 1) Preprocessing of lung CT images;

[0163] The lung CT images (medical imaging data) are subjected to format conversion, resampling, denoising enhancement and quality verification to form a two-dimensional slice sequence, so as to obtain standardized image data that meets the requirements of subsequent data quality assessment and feature extraction.

[0164] This embodiment reads low-dose spiral CT (LDCT) images in DICOM format into a standard matrix tensor and adjusts the voxel spacing or resolution as needed. Basic denoising and artifact removal processing (Gaussian filtering or thresholding to remove noise) is then performed to improve the image signal-to-noise ratio. The original CXR / CT images are then converted from DICOM to PNG, resampled to a 1mm pixel pitch, and enhanced with CLAHE and Gaussian filtering for denoising to obtain a uniform and clear input tensor.

[0165] 2) EHR text preprocessing;

[0166] First, the EHR text data was anonymized to ensure patient privacy. Then, word segmentation and terminology standardization were performed, mapping synonymous medical terms to unified codes (e.g., unifying "lung cancer" as a standard term). Irrelevant and redundant information and obvious errors were removed. After word segmentation and lexical normalization, the text data was converted into a sequence format acceptable to the model, while retaining necessary contextual information.

[0167] 3) Preprocessing of structured numerical index data;

[0168] The structured numerical index data of patients is normalized and missing values ​​are handled, including filling missing values ​​with statistical or interpolation methods, normalizing indices of different dimensions, and detecting outliers for correction or labeling as needed. At the same time, it is ensured that the structured numerical index data from different sources are aligned with the corresponding images and text in time (matching the acquisition date of CT images with the laboratory test results of the same day) to provide synchronous multimodal input.

[0169] S2. Based on step S1, conduct a data quality assessment on the preprocessed multimodal medical data;

[0170] S21. Image modal quality assessment;

[0171] The image quality score is obtained by using signal-to-noise ratio (SNR) and sharpness as core indicators for comprehensive evaluation. .

[0172] First, the signal-to-noise ratio is calculated using the Region of Interest (ROI) analysis method. In homogeneous regions of lung CT images (aortic arch or tracheal lumen), the mean pixel value is calculated as the signal intensity. The pixel standard deviation is calculated as the noise intensity in the image background region (external air region). ,Right now .

[0173] The sharpness index is then calculated using the Laplacian operator. This involves performing convolution operations on the two-dimensional slice sequence using a Laplacian convolution kernel to obtain the corresponding edge maps, and determining the sharpness index based on the variance of pixel values ​​in each edge map. The larger the variance, the sharper the image edges and the higher the clarity.

[0174] The two-dimensional slices are sequences of two-dimensional slices formed from preprocessed lung CT images.

[0175] Then, the index values ​​are mapped to the [0,1] interval using the min-max normalization method, and the final image quality score is calculated by normalized weighted sum. The expression is as follows:

[0176] (1);

[0177] in, and This indicates the preset weighting coefficients (0.6 and 0.4 in this embodiment). This represents the normalization function, which maps the index to the [0,1] interval. and represents the normalized signal-to-noise ratio and Laplace variance, respectively.

[0178] S22, Text Modal Quality Assessment;

[0179] Text quality score Information integrity score And recording clarity score The weighted combination yields the following:

[0180] 1) Information completeness score ;

[0181] Referring to a predefined set of disease-related key information fields (KIFs) and the preset weight of each field, key information fields are extracted from the patient's EHR, and the coverage of key information fields (KIFs) is calculated as follows:

[0182] (2);

[0183] in, This indicates the total number of predefined key information fields; Indicates the first The pre-defined weights of each KIF represent its clinical importance; The score indicates the detection status. If the information is successfully extracted from the text, it is 1; otherwise, it is 0.

[0184] 2) Recording clarity score ;

[0185] By quantifying the density of non-standard abbreviations in text Clinical signal-to-noise ratio calculate The expressions for the density of non-standard abbreviations and the clinical signal-to-noise ratio are as follows:

[0186] (3);

[0187] (4);

[0188] in, represents the number of non-standard abbreviations, custom terms or misspelled words identified in the text after comparison with the authoritative medical dictionary (UMLS) and the standard abbreviation lexicon; represents the total number of words in the EHR text. represents the number of words that hit the predefined "signal" lexicon (including high-value medical terms) in the text; represents the number of words that hit the predefined "noise" vocabulary (such as stop words like "the", "of") in the text; represents a very small smoothing term to prevent the denominator from being zero.

[0189] is the clarity score the expression is as follows:

[0190] (5);

[0191] wherein, represents the balance weight coefficient.

[0192] 3) Text quality score ;

[0193] (6);

[0194] wherein, and represent preset weights.

[0195] S23, quality assessment of structured numerical index data modality;

[0196] Quality score of structured numerical index data is obtained by weighted combination of the original data completeness score and the data reliability score , and the calculation expression is as follows:

[0197] (7);

[0198] (8);

[0199] (9);

[0200] wherein, and represent preset weights, represents the number of non-missing feature values before interpolation; represents the total number of expected features of the "standard structured feature panel" that all patients should include, which is predefined according to the actual screening scheme; represents the number of outliers determined among the non-missing feature values.

[0201] S3. Construct a screening model for high-risk populations of diseases based on multimodal fusion;

[0202] The screening model includes: a modality feature extraction network, a context-adaptive fusion CAF module, and a risk output layer.

[0203] The modal feature extraction network includes: an image modal feature extraction network, a text modal feature extraction network, and a structured numerical index data feature extraction network.

[0204] S4. Based on step S3, multimodal feature extraction is performed through a modal feature extraction network, that is, feature extraction is performed on each modal data after quality assessment.

[0205] S41. Image modal feature extraction;

[0206] CT scans used clinically for lung cancer screening are all multi-slice sequences, generating three-dimensional volume data. Therefore, in this embodiment, a 3D deep convolutional neural network (3D-CNN) is used to extract visual features from the standardized multi-slice CT image tensor.

[0207] 3D-CNN networks directly process three-dimensional volumetric data through 3D convolutional kernels, capturing key semantic information in images, and finally obtaining feature vectors representing the entire lung image through global average pooling layers. .

[0208] The key semantic information includes: the size, density, and shape of the suspicious nodules, as well as the spatial distribution of the lesions.

[0209] S42. Text modality feature extraction;

[0210] A pre-trained medical language model (a BERT-based medical text encoder) is used to encode the EHR text sequence. First, the segmented text is input into the model's embedding layer to generate a sequence of word vectors. Then, deep semantic features are extracted through stacked Transformer encoding layers. Finally, global text semantic vectors are extracted through pooling or self-attention mechanisms. .

[0211] S43. Extraction of modal features from structured numerical index data;

[0212] For structured numerical index data, a multilayer perceptron (MLP) is used for encoding. All numerical features are arranged into vectors in a predetermined order and input into the network. Nonlinear combination features are extracted through several fully connected layers to obtain the structured data feature vector. .

[0213] S5. Based on the modal features extracted in step S4, multimodal fusion is performed using the context adaptive fusion CAF module to obtain the fused feature vector;

[0214] This embodiment proposes a context-adaptive fusion method that dynamically adjusts the contributions of each modality to suppress noise and highlight key information. For example... Figure 2 As shown, the CAF (Context-Adaptive Fusion) module includes: a context encoding layer, a hierarchical gating mechanism, a cross-modal attention fusion module, and a knowledge-guided dynamic adjustment module.

[0215] The context coding layer is used to perform context-aware coding based on the patient's electronic health record (EHR) text and structured numerical indicator data to generate a patient context vector. The hierarchical gating mechanism applies two levels of control to the fusion of multimodal features: global gating at the overall modality level and local gating at a fine-grained level within each modality. The cross-modal attention fusion module employs a multi-head cross-modal attention mechanism to achieve deep information interaction and alignment between modalities. The knowledge-guided dynamic adjustment module adjusts based on prior medical knowledge and the individual patient context. Dynamic adjustment factors are generated based on reward and punishment rules. .

[0216] By using hierarchical gating, low-quality or irrelevant modalities can be effectively suppressed, while highlighting the most diagnostically valuable parts of each modality.

[0217] S6. Based on the fused feature vector output in step S5, risk classification and report generation are performed through the risk output layer, that is, high-risk identification and interpretation report generation are performed.

[0218] In this embodiment, step S5 is specifically as follows:

[0219] S51, Context Coding Layer;

[0220] The context coding layer is used to perform context-aware coding based on the patient's electronic health record (EHR) text and structured numerical indicator data to generate a patient context vector. .

[0221] 1) Define a global context feature set, that is, determine a globally shared feature set based on clinical guidelines (NCCN-2024). A set consisting of the most critical background features;

[0222] First, clinical entities and structured candidate variables were extracted from the electronic health record (EHR) text and structured numerical index data of all samples to form an initial feature set. Then, candidate variables related to lung cancer risk were screened, and the Boruta-SHAP and robust univariate Cox regression methods were used to rank the candidate variables by importance, resulting in a refined feature candidate list. Finally, a forward selection strategy was used to determine the optimal number of features. And select the corresponding feature from the feature candidate list. These features constitute the global context feature set.

[0223] The text entities extracted from the EHR text include: smoking history, family history, symptom description, past medical history, and occupational exposure history. The structured variables extracted from the structured numerical index data include: age, gender, laboratory test results, and vital signs.

[0224] 2) Generate individual context vectors;

[0225] When the model performs individual screening, it extracts the corresponding EHR text and structured numerical index data of the patient according to the global context feature set. Each feature value constitutes its individual feature vector. ,Will Input a small feedforward neural network to generate a patient-specific context vector. The expression is as follows:

[0226] (10);

[0227] in, This indicates that the patient corresponds to An individual context feature vector composed of context feature values; This represents the learnable weight matrix of the context encoding layer; This represents the bias vector of the context coding layer; Represents a nonlinear activation function; This represents the generated patient context vector, and is used to construct the patient context vector. The features are limited to EHR text data and structured numerical index data; EHR text data and structured numerical index data are used to characterize the patient's stable, long-term background risk baseline, while imaging features serve as real-time evidence that needs to be dynamically interpreted in subsequent fusion stages. Separating "background" from "evidence" and then using "background" to guide the interpretation of "evidence" is the core design idea of ​​the CAF module in this embodiment.

[0228] S52, Hierarchical Gating Mechanism (HGM);

[0229] 1) Global gating layer;

[0230] The global gating layer assigns weight coefficients to each mode at the modal level. Measure modality The credibility and importance in the current decision are calculated using the following formula:

[0231] (11);

[0232] in, This represents the Sigmoid activation function. , , Indicates learnable parameters; Represents the set of real numbers; Representing the patient context vector The dimension; This indicates the transpose operation. Representing modes Data quality score, , , , These represent the data quality scores for the image modality, text modality, and structured numerical index data modality, respectively.

[0233] During the training process, Used to characterize patient context vectors For modes The impact of global gating weights; Used to characterize modal quality score For modes The impact of global gating weights; through learnable parameters Training the patient context vector Modal quality score Projecting onto a unified logit decision space yields the modalities. Global gating weights .

[0234] 2) Departmental control layer;

[0235] The local control layer further allocates fine-grained weights within each modality to capture the differences in importance of different feature elements within the modality.

[0236] For mode eigenvectors To regard it as containing Each local feature representation is processed through an attention mechanism. Calculate a scalar score The local weights are then obtained by Softmax normalization. The expression is as follows:

[0237] (12);

[0238] (13);

[0239] in, , Representing modes The Local feature representation. , and Indicates learnable parameters, Represents the weight matrix. This represents the bias vector. This represents the weight vector used to map the hidden layer representation to the final scalar score.

[0240] 3) Unification and integration;

[0241] For modes The local features are summed according to their weights to obtain the mode. Feature vectors fused with internal attention; then multiplied by global gating coefficients. The final gating coefficient is obtained. The expression is as follows:

[0242] (14);

[0243] in, . These represent the final gating coefficients for the image modality, text modality, and structured numerical index data modality, respectively. Through a hierarchical gating mechanism, modalities with low reliability can be suppressed. Conversely, if the overall weight of a modality is high, the features within that modality are amplified.

[0244] S53, Cross-modal attention fusion;

[0245] After obtaining the weighted features of each modality, this embodiment adopts a multi-head cross-modal attention mechanism to achieve deep information interaction and alignment between modalities.

[0246] 1) Single-head cross-modal attention computation;

[0247] For any pair of modes and Setting the mode For query, modal To be retrieved, the attention output expression is defined as follows:

[0248] (15);

[0249] in, , , , This represents the learnable projection matrix. This represents the scaling factor. The calculation yields the weight matrix pair. The weighted output realizes modal From modality Extract relevant information from it.

[0250] 2) Multi-head fusion output;

[0251] To enhance representation capabilities, the following approach is adopted. Multi-head attention independently learns the alignment relationships between three pairs of modalities in different projection subspaces. Attention head to modality and The calculation expression is as follows:

[0252] (16);

[0253] in, , , , , Indicates the first Each head has its own projection matrix. Each pair of modal directions is obtained... The output of each head is then concatenated along the vector dimension and linearly projected to obtain... The expression is as follows:

[0254] (17);

[0255] in, This represents a learnable projection matrix that maps the concatenated outputs of each attention head along the vector dimension to the target feature space. This represents a vector concatenation operation. Indicates the first Attention in modality Pointing mode Cross-modal attention output.

[0256] Finally, the fused representations of the three directions are concatenated again to obtain the feature representations of the three modalities, i.e., the preliminary fused features. The expression is as follows:

[0257] (18);

[0258] S54, Knowledge-guided dynamic adjustment module;

[0259] After obtaining the initial fusion features, this embodiment introduces a knowledge-guided dynamic adjustment module to generate a dynamic adjustment factor based on prior medical knowledge and the individual patient context. This factor is used to address the enhancement of key evidence and conflicts in intermodal information; its default value is 1.0.

[0260] 1) Dynamic adjustment factor Formal definition;

[0261] (19);

[0262] in, This represents a dynamic adjustment factor that guides knowledge acquisition. ∈{0,1} represents the activation flag of the reward rule; when a high-risk rule in the knowledge graph is matched... =1, otherwise 0; ∈{0,1} represents the activation flag of the penalty rule, which is triggered when inter-modal conflicts or extremely low data quality are detected. =1, otherwise 0; and This represents two small neural networks that receive context vectors. As input, the reward intensity and punishment intensity are output for the patient's specific context. This allows the adjustment effect to be "personalized," for example, the reward intensity will be higher for patients who are already in a high-risk group.

[0263] 2) Triggering mechanism for reward and punishment rules:

[0264] Award logo and punishment signs The value is determined by the three-stage algorithm process, as follows:

[0265] Phase 1: Extraction of clinical atomic entities;

[0266] From the original features or intermediate vectors of each modality, extract discrete, clinically significant, and indivisible "clinical atomic entities" to form an entity set. .

[0267] Structured data: For each numerical indicator, a threshold is set according to clinical guidelines for binarization. If an indicator meets the corresponding threshold condition, the threshold judgment result is transformed into a clinically meaningful atomic entity and added to the patient entity set. .

[0268] For example, if a patient's CEA level is 12 ng / mL and the preset threshold is 10 ng / mL, a clinical atomic entity "CEA>10 ng / mL" is generated and added to the database. .

[0269] Text data: Based on a predefined medical dictionary (e.g., "hemoptysis", "sudden weight loss"), a keyword matching algorithm is used to extract high-risk symptom entities from the EHR text and add them as clinical atomic entities to the patient entity set. ;

[0270] Image data: A 3D-CNN is constructed as a multi-task learning model, namely an image feature extraction network with an auxiliary feature classification head, whose backbone network is used to extract image features. and the image features The input is then fed into the subsequent CAF module to participate in the final multimodal risk prediction. The primary task is set as predicting the high risk probability of lung cancer in patients, which is used to output the model's risk score. Multiple image feature classification tasks are set in parallel as auxiliary tasks to learn interpretable image feature representations. Each auxiliary task head is specifically responsible for identifying a specific image feature, including: the presence of "ground-glass nodules," the presence of "spiculated features," and whether the maximum diameter of the nodule is greater than 8 mm. When screening new patients, the trained auxiliary task heads output the probability of the presence of key image features. If this probability meets a preset threshold, the corresponding clinical atomic entity in the image (e.g., "spiculated features present") is added to the patient entity set. .

[0271] The auxiliary task shares the 3D-CNN image feature extraction backbone network with the final main task, and is jointly optimized with the main task through the total loss function to help the main task obtain image features with more medical semantics.

[0272] Phase Two: Knowledge Graph Rule Base Matching;

[0273] The medical knowledge graph used in this embodiment is constructed by integrating publicly available medical ontology libraries and a high-risk rule base for lung cancer from authoritative clinical guidelines. The library stores... Each rule It is itself a condition set composed of multiple clinical atomic entities, representing a logical "AND" combination.

[0274] The publicly available medical ontology includes SNOMED CT and ICD-11; the authoritative clinical guidelines include NCCN and CSCO. Phase one extracts the patient's objective "atomic facts," while the rules in this phase represent "composite clinical logic" that requires multiple facts to be satisfied simultaneously to be valid; the two are in a progressive relationship.

[0275] The matching algorithm uses set inclusion relationship judgment, that is, it traverses every rule in the rule base. If the condition set of the rule is the patient entity set If a subset is selected, then the decision rule is... A match was successfully made.

[0276] This approach employs strict binary subset matching, rather than fuzzy similarity matching. This design aims to ensure that rewards are triggered only when a complete chain of clinical evidence is established, thereby avoiding risky misjudgments due to partial evidence and improving the reliability of the model's decisions.

[0277] Phase 3, Assigning Reward / Penalty Flags:

[0278] Award logo If at least one rule is successfully matched in Phase Two, the reward rule is activated. Set to 1; if no rule is matched, then Set to 0.

[0279] Punishment sign If intermodal information conflict is detected, or if the data quality score of any modality is lower than a preset extreme quality threshold, the penalty rule activation flag will be set. Set to 1; if there is no intermodal information conflict and the data quality of each modality is not lower than the preset extreme quality threshold, then Set to 0.

[0280] The intermodal information conflict refers to a clear contradiction in the evidence provided by different modalities. For example, the EHR text records "no obvious symptoms," but the structured test indicators show abnormally elevated tumor markers. The extremely low data quality refers to the data quality score of any modality being lower than a preset extreme quality threshold, indicating that the data of that modality is basically unreliable.

[0281] Finally, after dynamic adjustment factors After adjustment, the CAF module outputs the final, knowledge-corrected fused feature vector. .

[0282] In this embodiment, step S6 is specifically as follows:

[0283] S61. Risk score calculation;

[0284] Fusion feature vectors After processing by a fully connected discriminator (output layer), the patient's continuous risk score is obtained. The specific calculation expression is as follows:

[0285] (20);

[0286] in, This represents the final fused feature vector after knowledge correction; and This represents the output layer weight matrix and bias terms. Represents the Sigmoid activation function, ensuring This refers to the lung cancer risk probability predicted by the model.

[0287] To ensure the clinical applicability and standardization of risk stratification, a pre-defined lung cancer risk stratification standard is used to discretely stratify the continuous risk score output by the model. This embodiment uses the internationally recognized Lung-RADS® (Lung Imaging Reporting and Data System) as the reference standard for risk level classification. Lung-RADS® classifies the malignancy risk of lung nodules into several clearly defined levels (such as Class 1, Class 2, Class 3, Class 4A, Class 4B, and Class 4X), with each level corresponding to specific clinical recommendations.

[0288] The risk classification criteria in this embodiment are as follows:

[0289] First, the continuous risk score output by the model. Mapping to discrete risk levels corresponding to preset risk grading standards; this embodiment uses Lung-RADS® as a reference standard to map the continuous risk score. Mapping to the corresponding Lung-RADS® category. This mapping process involves setting a set of thresholds. To achieve, This represents the total number of thresholds set. These thresholds are not determined based on subjective consensus, but rather through calibration on a validation set: using a validation dataset labeled by radiology experts according to the Lung-RADS® standard, the thresholds are optimized (e.g., maximizing Cohen's Kappa coefficient or classification accuracy) to find the continuous scoring range for the model. The threshold point that best matches the classification results with the expert's Lung-RADS® classification results.

[0290] S62. Generation of explanatory reports;

[0291] To improve the transparency of model decision-making, this invention proposes a method for automatically generating explanatory reports. The details are as follows:

[0292] 1) Analyze global modal contributions;

[0293] The system first calculates the global gating weights based on the multimodal fusion process. The report outputs the contribution percentage of each data modality (image, text, structured data) to the final risk assessment, providing doctors with a high-level attribution overview.

[0294] 2) Extract key evidence for each modality;

[0295] For each modal data, the most influential specific features are identified, as follows:

[0296] Text Modality: Utilizing the attention weights learned by the model in the local control layer, automatically highlight the keywords and phrases in the EHR text that contribute the most to the decision.

[0297] Image modality: Gradient weighted class activation mapping (Grad-CAM) is used to generate saliency heatmaps and to define the suspicious regions of most interest to the model on the original CT images.

[0298] Structured numerical indicator data modality: The influence of an indicator is quantified by calculating the gradient of the final risk score relative to each structured input indicator. Indicators with larger absolute gradient values ​​represent those whose small changes in value have a greater impact on the result, and are therefore identified as "key structured indicators" and included in the report.

[0299] 3) Backtracking knowledge graph reasoning;

[0300] If the knowledge graph-guided dynamic adjustment factor Triggered during computation (i.e.) The report will automatically cite the matched medical rules or detected information conflicts, explaining in text form why the risk score was adjusted, making the model's dynamic intervention process transparent.

[0301] In summary, this method for automatically generating explanatory reports integrates explanatory information from different levels into a single structured report, providing clinicians with decision-making support from macro to micro perspectives.

[0302] In this embodiment, in step S3, the screening model undergoes model training, specifically as follows:

[0303] The training dataset comprises authoritatively annotated lung CT images, EHR text, and structured test metrics, and is divided into training, validation, and test sets using stratified sampling based on risk level. The model is jointly optimized end-to-end, with the total loss function... It is a weighted sum of the main task loss, auxiliary task loss, and regularization term, expressed as follows:

[0304] (twenty one);

[0305] in, This represents the loss for the main task. The main task of this invention is to predict the high risk probability of a patient developing lung cancer (i.e., outputting a continuous value between 0 and 1), therefore, a binary cross-entropy loss is used, expressed as follows:

[0306] (twenty two);

[0307] in, This indicates the training batch size. Indicates the first The true label for each sample (1 for cancer, 0 otherwise). This represents the predicted risk probability of lung cancer by the model, i.e., the [missing value]. Continuous risk score for each patient. Indicating the first in the image module The loss function for each auxiliary feature classification head, this part includes There are two parallel auxiliary tasks, each of which is a binary classification problem, and they also use binary cross-entropy loss. This represents all learnable parameters of the model. Represents the L2 norm; This represents the weighting coefficient for auxiliary tasks; This represents the regularization decay coefficient.

[0308] In this embodiment, in the screening model, each modality feature extraction network is pre-trained to obtain initial parameters. During the overall training phase, the parameters of each modality feature extraction network are not fixed. They are jointly fine-tuned end-to-end with the CAF module and the risk output layer through the total loss function. At the same time, an early stopping mechanism is introduced to prevent overfitting.

[0309] In summary, the method of this invention significantly improves the accuracy and reliability of high-risk lung cancer screening in the population, effectively reduces the false positive rate and false negative rate, enhances robustness in complex clinical environments, and makes the model's decisions more consistent with common medical sense. It also effectively handles intermodal information conflicts, greatly improving the credibility and interpretability of the decisions, and generates detailed explanatory reports, providing clinicians with quantitative risk assessments and intuitive decision-making basis. Furthermore, the method of this invention is not limited to high-risk lung cancer screening scenarios, but is also applicable to early screening of high-risk populations for other diseases.

[0310] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Various modifications and variations can be made to the invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the scope of the claims of the invention.

Claims

1. A method for screening high-risk populations for diseases based on multimodal fusion, the specific steps of which are as follows: S1. Collect multimodal medical data related to high-risk lung cancer screening of the subjects to be screened, and preprocess the collected multimodal medical data; The multimodal medical data includes: Electronic health record (EHR) text, lung CT images, and structured numerical index data; The structured numerical index data includes: demographic information, laboratory test results, and vital signs; Then, the collected multimodal medical data is preprocessed to ensure that the data formats of each modality are consistent, the quality meets the standards, and the data is aligned in the time dimension. The specific preprocessing of the disease data is as follows: 1) Preprocessing of lung CT images; The lung CT images were subjected to format conversion, resampling, noise reduction, enhancement, and quality verification to form a two-dimensional slice sequence; 2) EHR text preprocessing; First, the EHR text data is anonymized to ensure patient privacy. Then, the text is segmented and professional terminology is standardized, mapping synonymous medical terms to a unified code. At the same time, irrelevant redundant information and obvious erroneous characters are removed. After segmentation and word form normalization, the text data is converted into a sequence format acceptable to the model while retaining necessary contextual information. 3) Preprocessing of structured numerical index data; The structured numerical index data of patients is normalized and missing values ​​are handled, including filling missing values ​​with statistical or interpolation methods, normalizing indices of different dimensions, and detecting outliers for correction or labeling as needed; at the same time, it is ensured that the structured numerical index data from different sources are aligned with the corresponding images and text in time. S2. Based on step S1, conduct a data quality assessment on the preprocessed multimodal medical data; S21. Image modal quality assessment; First, the signal-to-noise ratio is calculated using the Region of Interest (ROI) analysis method. In homogeneous regions of lung CT images, the mean pixel value is calculated as the signal intensity within the selected region of interest (ROI). The pixel standard deviation in the image background region is calculated as the noise intensity. ,Right now ; Then, the sharpness index is calculated using the Laplacian operator. This involves performing convolution operations on the two-dimensional slice sequence using a Laplacian convolution kernel to obtain the corresponding edge maps, and determining the sharpness index based on the variance of pixel values ​​in each edge map. ; The two-dimensional slices are two-dimensional slice sequences formed from preprocessed lung CT images; Then, the index values ​​are mapped to the [0,1] interval using the min-max normalization method, and the final image quality score is calculated by normalized weighted sum. The expression is as follows: （1）； in, and This represents the preset weighting coefficient. This represents the normalization function, which maps the index to the [0,1] interval. and These represent the normalized signal-to-noise ratio and the Laplace variance, respectively. S22, Text Modal Quality Assessment; Text quality score Information integrity score And recording clarity score The weighted combination yields the following: 1) Information completeness score ; Referring to the predefined set of key information fields (KIFs) for disease-related information defined by clinical experts and the preset weight of each field, key information fields are extracted from the patient's EHR, and the coverage of the key information fields (KIFs) is calculated as follows: （2）； in, This indicates the total number of predefined key information fields; Indicates the first The pre-defined weights of each KIF represent its clinical importance; The score indicates the detection status. If the information is successfully extracted from the text, it is 1; otherwise, it is 0. 2) Recording clarity score ; By quantifying the density of non-standard abbreviations in text and clinical signal-to-noise ratio calculate The expressions for the density of non-standard abbreviations and the clinical signal-to-noise ratio are as follows: （3）； （4）； in, This refers to the number of non-standard abbreviations, custom terms, or misspelled words identified in the text after comparison with authoritative medical dictionaries and standard abbreviation databases; This indicates the total number of words in the EHR text; This indicates the number of words in the text that match a predefined "signal" vocabulary. This indicates the number of words in the text that match a predefined "noise" vocabulary; This represents a very small smooth term to prevent the denominator from being zero; Then the clarity score The expression is as follows: （5）； in, Indicates the balancing weight coefficient; 3) Text quality score ; （6）； in, and Indicates the preset weight; S23. Modal quality assessment of structured numerical index data; Structured numerical index data quality score Based on the original data completeness score and data reliability score The weighted combination yields the following calculation expression: （7）； （8）； （9）； in, and Indicates the preset weight. This indicates the number of non-missing feature values ​​before imputation; This indicates the total number of features that should be included in the "standard structured feature panel" as predefined in the screening protocol based on the actual situation, and which should be present in all patients; This indicates the number of non-missing features that are identified as outliers. S3. Construct a screening model for high-risk disease populations based on multimodal fusion; The screening model includes: a modality feature extraction network, a context-adaptive fusion CAF module, and a risk output layer; The modal feature extraction network includes: an image modal feature extraction network, a text modal feature extraction network, and a structured numerical index data feature extraction network; S4. Based on step S3, multimodal feature extraction is performed through a modal feature extraction network, that is, feature extraction is performed on each modal data after quality assessment. S41. Image modal feature extraction; Visual feature extraction was performed on the standardized multi-slice CT image tensors using a 3D deep convolutional neural network (3D-CNN). The 3D-CNN network directly processed the three-dimensional volumetric data through 3D convolutional kernels, capturing key semantic information in the images, and finally obtained a feature vector representing the entire lung image through a global average pooling layer. ; The key semantic information includes: the size, density, and shape of the suspicious nodules, as well as the spatial distribution of the lesions; S42. Text modality feature extraction; A pre-trained medical language model is used to encode EHR text sequences. First, the segmented text is input into the model's embedding layer to generate a sequence of word vectors. Then, deep semantic features are extracted through stacked Transformer encoding layers. Finally, global text semantic vectors are extracted through pooling or self-attention mechanisms. ; S43. Extraction of modal features from structured numerical index data; For structured numerical index data, a multilayer perceptron (MLP) is used for encoding. All numerical features are arranged into vectors in a predetermined order and input into the network. Nonlinear combination features are extracted through several fully connected layers to obtain the structured data feature vector. ; S5. Based on the modal features extracted in step S4, multimodal fusion is performed using the context adaptive fusion CAF module to obtain the fused feature vector; The CAF module includes: a context encoding layer, a hierarchical gating mechanism, a cross-modal attention fusion module, and a knowledge-guided dynamic adjustment module; The context coding layer is used to perform context-aware coding based on the patient's electronic health record (EHR) text and structured numerical indicator data to generate a patient context vector. The hierarchical gating mechanism applies two levels of control to the fusion of multimodal features: global gating at the overall modality level and local gating at a fine-grained level within each modality. The cross-modal attention fusion module employs a multi-head cross-modal attention mechanism to achieve deep information interaction and alignment between modalities. The knowledge-guided dynamic adjustment module adjusts based on prior medical knowledge and the patient's individual context vector. Dynamic adjustment factors are generated based on reward and punishment rules. ; S6. Based on the fused feature vector output in step S5, risk classification and report generation are performed through the risk output layer, that is, high-risk identification and interpretation report generation are performed.

2. The method for screening high-risk populations for disease according to claim 1, characterized in that, Step S5 is as follows: S51, Context Coding Layer; The context coding layer is used to perform context-aware coding based on the patient's electronic health record (EHR) text and structured numerical indicator data to generate a patient context vector. ; 1) Define a global context feature set, that is, determine a globally shared feature set based on clinical guidelines. A set consisting of the most critical background features; First, clinical entities and structured candidate variables are extracted from the electronic health record (EHR) text and structured numerical index data of all samples to form an initial feature set. Then, candidate variables associated with lung cancer risk were selected, and the Boruta-SHAP and robust univariate Cox regression methods were used to rank the candidate variables by importance, resulting in a refined list of candidate features. Finally, a forward selection strategy was used to determine the optimal number of features. And select the corresponding feature from the feature candidate list. These features constitute the global context feature set; Among them, text entities were extracted from EHR text, including: smoking history, family history, symptom description, past medical history, and occupational exposure history; and structured variables were extracted from structured numerical index data, including: age, gender, laboratory test results, and vital signs. 2) Generate individual context vectors; When the model performs individual screening, it extracts corresponding data from the patient's EHR text and structured numerical index data based on the global context feature set. Each feature value constitutes its individual feature vector. ,Will Input a small feedforward neural network to generate a patient-specific context vector. The expression is as follows: （10）； in, This indicates that the patient corresponds to An individual context feature vector composed of context feature values; This represents the learnable weight matrix of the context encoding layer; This represents the bias vector of the context coding layer; Represents a nonlinear activation function; This represents the generated patient context vector, and is used to construct the patient context vector. The features are limited to EHR text data and structured numerical index data only; S52, layered gating mechanism; 1) Global gating layer; The global gating layer assigns weight coefficients to each mode at the modal level. Measure modality The credibility and importance in the current decision are calculated using the following formula: （11）； in, This represents the Sigmoid activation function. , , Indicates learnable parameters; Represents the set of real numbers; Representing the patient context vector The dimension; This indicates the transpose operation. Representing modes Data quality score, , , , These represent the data quality scores for image modality, text modality, and structured numerical index data modality, respectively. 2) Departmental control layer; For mode eigenvectors To regard it as containing Each local feature representation is processed through an attention mechanism. Calculate a scalar score The local weights are then obtained by Softmax normalization. The expression is as follows: （12）； （13）； in, , Representing modes The Local feature representation; , and Indicates learnable parameters, Represents the weight matrix. This represents the bias vector. This represents the weight vector used to map the hidden layer representation to the final scalar score; 3) Unification and integration; For modes The local features are summed according to their weights to obtain the mode. Feature vectors fused with internal attention; then multiplied by global gating coefficients. The final gating coefficient is obtained. The expression is as follows: （14）； in, These represent the final gating coefficients corresponding to the image modality, text modality, and structured numerical index data modality, respectively. S53, Cross-modal attention fusion; 1) Single-head cross-modal attention computation; For any pair of modes and Setting the mode For query, modal To be retrieved, the attention output expression is defined as follows: （15）； in, , , , This represents the learnable projection matrix. Indicates the scaling factor; 2) Multi-head fusion output; use A parallel attention head independently learns the alignment relationships between three pairs of modalities in different projection subspaces. Attention head to modality and The calculation expression is as follows: （16）； in, , , , , Indicates the first Each head has its own projection matrix; each pair of modal directions is obtained The output of each head is then concatenated along the vector dimension and linearly projected to obtain... The expression is as follows: （17）； in, This represents a learnable projection matrix that maps the concatenated outputs of each attention head along the vector dimension to the target feature space. This represents a vector concatenation operation. Indicates the first Attention in modality Pointing mode Cross-modal attention output; Finally, the fused representations of the three directions are concatenated again to obtain the feature representation of the fused three modalities, i.e., the preliminary fused features. The expression is as follows: （18）； S54, Knowledge-guided dynamic adjustment module; 1) Dynamic adjustment factor Formal definition; （19）； in, This represents a dynamic adjustment factor that guides knowledge acquisition. ∈{0,1} represents the activation flag of the reward rule; when a high-risk rule in the knowledge graph is matched... =1, otherwise 0; ∈{0,1} represents the activation flag of the penalty rule, which is triggered when inter-modal conflicts or extremely low data quality are detected. =1, otherwise 0; and This represents two small neural networks that receive context vectors. As input, output the reward intensity and punishment intensity in the context of this patient. 2) Triggering mechanism for reward and punishment rules: Award logo and punishment signs The value is determined by the three-phase algorithm process, as follows: Phase 1: Extraction of clinical atomic entities; From the original features or intermediate vectors of each modality, discrete, clinically significant, and indivisible "clinical atomic entities" are extracted to form a patient entity set. ; Structured data: For each structured numerical indicator, a threshold is set according to clinical guidelines for binarization judgment; if an indicator meets the corresponding threshold condition, the threshold judgment result is transformed into a clinical atomic entity with clear clinical significance and added to the patient entity set. ; Text data: Based on a predefined medical dictionary, high-risk symptom entities that match in the EHR text are extracted using a keyword matching algorithm and added to the patient entity set as clinical atomic entities. ; Image Data: A 3D-CNN was constructed as a multi-task learning model. The primary task was to predict the high risk probability of lung cancer in patients, which was used to output the model's risk score. Multiple image feature classification tasks were set up in parallel as auxiliary tasks to learn interpretable image feature representations. Each auxiliary task head was specifically responsible for identifying a specific image feature. When screening new patients, the trained auxiliary task heads output the probability of the presence of key image features. If the probability met a preset threshold, the corresponding clinical atomic entity in the image was added to the patient entity set. ; The auxiliary task shares the 3D-CNN image feature extraction backbone network with the final main task, and is jointly optimized with the main task through the total loss function; Phase Two: Knowledge Graph Rule Base Matching; A medical knowledge graph is constructed by integrating publicly available medical ontology databases and authoritative clinical guidelines on high-risk lung cancer. The database stores... Each rule It is itself a condition set composed of multiple clinical atomic entities, representing a logical "AND" combination; The publicly available medical ontology databases include: SNOMED CT and ICD-11; the authoritative clinical guidelines include: NCCN and CSCO. The matching algorithm uses set inclusion relationship judgment, that is, it traverses every rule in the rule base. If the condition set of the rule is the patient entity set If a subset is selected, then the decision rule is... Successfully matched; Phase 3, Assigning Reward / Penalty Flags: Award logo If at least one rule is successfully matched in Phase Two, the reward rule is activated. Set to 1; if no rule is matched, then Set to 0; Punishment sign If intermodal information conflict is detected, or if the data quality score of any modality is lower than a preset extreme quality threshold, the penalty rule activation flag will be set. Set to 1; if there is no intermodal information conflict and the data quality of each modality is not lower than the preset extreme quality threshold, then Set to 0; Among them, the intermodal information conflict means that the evidence provided by different modalities is clearly contradictory; the extremely low data quality means that the data quality score of any modality is lower than a preset extreme quality threshold, indicating that the data of that modality is basically unreliable. Finally, after dynamic adjustment factors After adjustment, the CAF module outputs the final, knowledge-corrected fused feature vector. .

3. The method for screening high-risk populations for disease according to claim 2, characterized in that, Step S6 is as follows: S61. Risk score calculation; Fusion feature vectors After processing by a fully connected discriminator, the patient's continuous risk score is obtained. The specific calculation expression is as follows: （20）； in, This represents the final fused feature vector after knowledge correction; and This represents the output layer weight matrix and bias terms. Represents the Sigmoid activation function, ensuring That is, the probability of lung cancer risk predicted by the model; Then, the continuous risk score output by the model is applied using a pre-defined lung cancer risk grading standard. The classification is discrete, with each level corresponding to specific clinical recommendations; The risk classification criteria are as follows: First, the continuous risk score output by the model. Mapping to discrete risk levels corresponding to preset risk grading standards; the mapping process involves setting a set of thresholds. accomplish, Indicates the total number of set thresholds; S62. Generation of explanatory reports; 1) Analyze global modal contributions; The system first calculates the global gating weights based on the multimodal fusion process. The report outputs the contribution percentage of each data modality to the final risk assessment, providing doctors with a high-level attribution overview. 2) Extract key evidence for each modality; For each modal data, the most influential specific features are identified, as follows: Text modality: Utilizing the attention weights learned by the model at the local control layer, automatically highlight the keywords and phrases in the EHR text that contribute the most to the decision-making process; Image modality: A saliency heatmap is generated using the gradient-weighted class activation mapping method, and the suspicious regions of most interest to the model are delineated on the original CT images; Structured numerical indicator data modality: The impact is quantified by calculating the gradient of the final risk score relative to each structured input indicator; indicators with larger absolute gradient values ​​are identified as "key structured indicators" and included in the report; 3) Backtracking knowledge graph reasoning; If the knowledge graph-guided dynamic adjustment factor Triggered during the calculation, the report will automatically reference the matched medical rules or detected information conflicts, explaining in text form why the risk score was adjusted, making the model's dynamic intervention process transparent.

4. The method for screening high-risk populations for disease according to claim 1, characterized in that, In step S3, the screening model undergoes model training, specifically as follows: The training dataset includes authoritatively labeled lung CT images, EHR text, and structured test indicators, and is divided into training, validation, and test sets using stratified sampling based on risk levels. The model is jointly optimized end-to-end, with the total loss function... It is a weighted sum of the main task loss, auxiliary task loss, and regularization term, expressed as follows: （21）； in, The loss for the main task is represented by the binary cross-entropy loss, expressed as follows: (22)； in, Indicates the training batch size. Indicates the first The true label for each sample is 1 if the patient has cancer, and 0 otherwise. This represents the predicted risk probability of lung cancer by the model, i.e., the [missing value]. Continuous risk score for each patient; Indicating the first in the image module The loss function of each auxiliary feature classification head includes There are 1 parallel auxiliary task, and each task is a binary classification problem, which also uses binary cross-entropy loss. This represents all learnable parameters of the model. Represents the L2 norm; This represents the weighting coefficient for auxiliary tasks; This represents the regularization decay coefficient.

5. The method for screening high-risk populations for disease according to claim 4, characterized in that, In the screening model, each modality feature extraction network is pre-trained to obtain initial parameters. During the overall training phase, the parameters of each modality feature extraction network are not fixed. They are jointly fine-tuned end-to-end with the CAF module and the risk output layer through the total loss function, and an early stopping mechanism is introduced.

Images