A method for evaluating the feature importance of POGD predictions based on a large model

Abstract

This invention discloses a feature importance assessment method for POGD prediction based on a large model. The method includes: first, acquiring a clinical dataset, converting structured features into cue text, and supervising the fine-tuning of a large language model to construct a baseline prediction model; then, performing text-level ablation on each feature on a validation set, calculating the change in prediction probability and entropy change to characterize the dual impact of the feature on prediction results and model uncertainty; finally, combining the two to obtain a comprehensive importance score and ranking the features. This invention not only identifies key clinical features that simultaneously affect prediction results and uncertainty, but also improves the robustness of feature selection by combining ablation and entropy measurement. It is suitable for high-dimensional heterogeneous medical data scenarios and can provide reliable support for personalized clinical intervention in POGD.

Description

Technical Field

[0001] This invention belongs to the field of medical artificial intelligence technology, specifically relating to a method for evaluating the feature importance of POGD prediction based on a large model. Background Technology

[0002] Postoperative gastrointestinal dysfunction (POGD) is a common adverse complication after general anesthesia surgery, significantly prolonging hospital stays and increasing medical costs. The risk of developing POGD varies significantly among patients, therefore, there is an urgent clinical need for a predictive technology that can accurately identify high-risk patients preoperatively or intraoperatively to develop individualized intervention plans.

[0003] Existing methods typically employ statistical models such as logistic regression, using indicators like p-values ​​and OR values ​​to assess risk factors. However, these methods have inherent limitations: first, they rely on the assumption of linear independence, making it difficult to characterize complex nonlinear relationships between clinical variables; second, they struggle to handle high-dimensional, significantly collinear, multi-source clinical data; and third, statistical significance does not equate to strong predictive power, resulting in limited model performance in clinical applications. While subsequent machine learning models can uncover more complex feature patterns, their feature importance metrics (such as those based on split gain, permutation, and SHAP) only focus on the feature's contribution to the prediction results, failing to characterize the impact of features on the model's predictive uncertainty.

[0004] In high-risk clinical prediction scenarios such as POGD, a truly critical feature should not only significantly influence the model's prediction results but also reduce uncertainty in the model's decision-making process, thereby improving the stability and reliability of the prediction. However, existing feature importance assessment systems generally lack the ability to measure "prediction confidence," making it difficult to identify key risk factors that are truly robust and generalizable in clinical practice.

[0005] As large models such as Transformer demonstrate their advantages in representation learning, they are able to capture deeper nonlinear dependencies between clinical features, making feature importance assessment based on large models possible. However, there is currently a lack of a unified assessment framework that can simultaneously quantify both the "impact of features on prediction results" and the "contribution of features to model uncertainty." Summary of the Invention

[0006] The technical problem to be solved by this invention is to provide a feature importance assessment method for POGD prediction based on a large model that can identify key clinical features that simultaneously affect prediction results and uncertainty, and improve the robustness of feature selection by combining ablation and entropy measurement. This method is applicable to high-dimensional heterogeneous medical data scenarios and can provide reliable support for personalized clinical intervention of POGD.

[0007] The technical solution of this invention is as follows:

[0008] A method for evaluating the feature importance of POGD predictions based on a large model includes the following steps:

[0009] Step 1: Building a baseline prediction model based on prompting engineering: Obtaining clinical datasets The structured clinical features of patients are converted into natural language prompt templates, and the entire dataset is then processed. Dataset converted to Prompt format ,Will Divided into training set and verification set and using the training set Supervised fine-tuning of a large language model yields a baseline prediction model. ;

[0010] Step 2: Two-dimensional impact quantification based on feature ablation: On the validation set, the predicted probability of POGD is calibrated using the Softmax function. An ablation operation is performed on each feature, and the change in predicted probability is calculated. and entropy change ,in, This is the expected value of the difference between the predicted probabilities under the original hints and ablation hints on the validation set. To verify the expected value of the Shannon entropy difference between ablation hints and original hints on the validation set;

[0011] Step 3: Calculation of Overall Importance Index and Feature Ranking: Calculate the overall importance score for each feature. : ,in, This represents a comprehensive importance index, emphasizing dual contributions through a product; and features are then sorted in descending order based on this.

[0012] Furthermore, in step 1, the expression for the clinical dataset is:

[0013] ,

[0014] in, Let represent the feature vector of the j-th sample, which contains M clinical features; The binary label indicating the occurrence of POGD 1 represents occurrence, 0 represents non-occurrence; N represents the sample size;

[0015] In step 1, the natural language prompt template includes task instructions, feature display, and risk assessment questions; the answer is based on... Mapped to high or low.

[0016] Furthermore, in step 1, the fine-tuning of the large language model uses the cross-entropy loss function, with the goal of generating the correct risk words after a given prompt.

[0017] Furthermore, in step 2, the expression for the prediction probability calibration is:

[0018] ,

[0019] in, This indicates the probability that the model output is high risk. This indicates that the model outputs words with high logarithmic scores. This indicates that the model output words have low logarithmic scores.

[0020] Furthermore, in step 2, the feature ablation operation is performed at the prompt text level, generating an ablation prompt by deleting lines containing specific features. .

[0021] Furthermore, in step 2, the predicted probability change is... The expression is:

[0022] ,

[0023] in, This represents the average change in the probability of occurrence after the i-th feature is ablated, used to quantify the strength of the feature's influence on the predicted probability. Indicates in the validation set The expectation on the surface is that prompt represents the original text input after being serialized from the features of a single sample through a standardized template. This means deleting items containing the i-th feature in the prompt. The input following the corresponding text is used to achieve text-level feature ablation.

[0024] Furthermore, in step 2, the entropy change... The expression is:

[0025] ,

[0026] ,

[0027] in, This represents the average increment of uncertainty caused by the ablation of the i-th feature. The larger the value, the more the feature can reduce prediction uncertainty. Indicates in the validation set The expectation on the surface is that prompt represents the original text input after being serialized from the features of a single sample through a standardized template. This means deleting items containing the i-th feature in the prompt. The input following the corresponding text. This represents the Shannon entropy function, used to measure the uncertainty of binary classification output; This represents the probability of a positive class in a binary classification, with a value range of [0,1]. At that time, ablation of this feature increases uncertainty, indicating that this feature helps to stabilize discrimination; At other times, the opposite is true.

[0028] Furthermore, the method was applied to predict the risk of gastrointestinal dysfunction after general anesthesia, with clinical characteristics including age, sex, operation duration, and total remifentanil dosage.

[0029] The beneficial effects of this invention are:

[0030] 1. Achieved a two-dimensional characterization of feature importance:

[0031] Existing technologies, whether traditional statistical methods or mainstream machine learning interpretability methods (such as permutation importance and SHAP values), are primarily one-dimensional in their evaluation, mainly measuring the contribution of features to the numerical value of the model's prediction results. This invention fundamentally breaks through this limitation by innovatively introducing "model uncertainty" as a second orthogonal evaluation dimension. By simultaneously calculating the change in probability and the change in entropy, this invention can not only identify "who can change the prediction results," but also "who can make the model's judgment more reliable and confident." This two-dimensional evaluation system can more comprehensively and profoundly reveal the true role of features in complex decision-making processes, providing a more complete understanding of feature value.

[0032] 2. Significantly improved the robustness and reliability of feature selection:

[0033] This invention's unique comprehensive importance index establishes an extremely rigorous screening criterion through a product form. It effectively distinguishes and filters out "foundational" features that simultaneously possess high "predictive driving force" and high "deterministic contribution." This index automatically suppresses "unstable" features that, while altering prediction results, increase model uncertainty, and "auxiliary" features that only slightly increase model confidence but have little impact on the outcome. Therefore, the feature subset selected by this method exhibits robustness and reliability in practical applications far exceeding traditional methods, making it particularly suitable for medical decision-making fields with extremely high requirements for safety and reliability.

[0034] 3. It fully leverages the advantages of large-scale language models, making it more adaptable to complex, high-dimensional medical data:

[0035] This invention uses LLM (Linguistic Modeling) as its technological foundation and employs Prompt engineering to "linguize" structured clinical tabular data, fully leveraging LLM's powerful capabilities in handling context, capturing nonlinear relationships, and identifying higher-order interactions. Compared to traditional methods, this invention can better uncover deep connections from real-world medical data containing heterogeneous information such as numerical values, text, and categories. Feature ablation is performed at the natural language Prompt level, which is more in line with human cognition and allows the method to be seamlessly applied to future scenarios of multimodal data fusion, demonstrating strong technological foresight and scalability. Attached Figure Description

[0036] Figure 1 This is a flowchart illustrating the feature importance assessment method for POGD prediction based on a large model according to the present invention. Detailed Implementation

[0037] like Figure 1 As shown, the technical solution of this invention uses a large language model as its core and employs a prompting engineering paradigm to construct the prediction model, rather than modifying the internal architecture of the LLM. Specifically, this method converts structured clinical data into natural language prompts (Prompts), and uses supervised fine-tuning to adapt the LLM to the POGD prediction task. Subsequently, feature ablation is performed on the validation set to quantify the impact of each missing feature on the model's output probability and uncertainty entropy. Finally, a comprehensive index is calculated and ranked. The entire process does not rely on the installation of additional packages, using only basic libraries in the Python environment (such as NumPy for expected value calculation).

[0038] The following are the specific implementation steps of this invention, which are logically divided into three main stages: benchmark model construction, two-dimensional influence quantification, and comprehensive index calculation and ranking.

[0039] Phase 1: Building a large-scale baseline prediction model based on the Prompt project:

[0040] This stage aims to build a reliable benchmark model. This is used for subsequent feature evaluation. Model Composed of a fine-tuned LLM, a standardized Prompt template, and a probability mapping function, it can output the POGD risk probability based on the patient's clinical information.

[0041] (1) Dataset preparation and Prompt serialization:

[0042] Obtain a clinical dataset containing N patient samples:

[0043] ,

[0044] in, This represents the feature vector of the j-th sample, which contains M clinical features (e.g., age, gender, operation duration, total remifentanil dosage, etc.). The binary label indicating the occurrence of POGD 1 represents occurrence, 0 represents non-occurrence; N represents the sample size;

[0045] Design a standardized Prompt template, including each Convert to natural language text. The template includes task instructions, structured feature presentation, and risk assessment questions. A typical Prompt example is as follows:

[0046] "Based on the patient's clinical information provided below, please assess the patient's risk of developing postoperative gastrointestinal dysfunction (POGD) after general anesthesia."

[0047] Age: {age value} years

[0048] Gender: {Gender value}

[0049] Surgery duration: {duration value} minutes

[0050] Total remifentanil dosage: {dosage value} micrograms... (List all M characteristics and their values) Based on the above information, is this patient's risk of developing POGD "high" or "low"?

[0051] Dataset Convert to Prompt format For each sample, according to Fill in the answer ( Corresponding to "high", (corresponding to "low"), then, Divided into training set and verification set (The typical ratio is 8:2).

[0052] (2) Fine-tuning of large-scale language models:

[0053] Choose a pre-trained, instruction-fine-tuned LLM as the base model (e.g., a model based on the Transformer architecture).

[0054] Using the training set Supervised fine-tuning aims to ensure the model accurately generates "high" or "low" given a prompt. Fine-tuning employs standard gradient descent optimization with cross-entropy as the loss function. After fine-tuning, a specialized model is obtained. It was used to handle clinical prediction tasks (using the open-source Transformer-based LLM, with fine-tuned parameters: learning rate 1e-5, batch size 8, epoch=3).

[0055] Phase 2: Two-Dimensional Impact Quantification Based on Feature Ablation

[0056] This stage is based on a fixed model. Above, on the verification set Each feature in (e.g., "operation duration") is evaluated, and the impact on the prediction probability and uncertainty entropy is calculated by simulating feature loss through ablation operations.

[0057] (1) Probability calibration:

[0058] For each Prompt in the validation set, input the model Obtain the output vocabulary probabilities, focusing on the logits values ​​for "high" and "low", denoted as... and The probability of POGD occurring (i.e., the probability of y=1) is calibrated using the Softmax function:

[0059] ,

[0060] in, This indicates the probability that the model output is high risk. A high log score indicates that the model outputs a word with a high log score, representing the model's original confidence level for high risk. This indicates that the model output words have low log scores, representing the model's original confidence level for low risk;

[0061] (2) Characteristic ablation:

[0062] For each prompt, generate a variant that ablates the i-th feature. By directly removing the text containing The line (e.g., deleting "- age: {age value} years") is performed at the natural language level to ensure that the real missing scenario is simulated;

[0063] (3) Change in probability Calculation: Calculate the expected value of the probability difference before and after ablation:

[0064] ,

[0065] in, This represents the average change in the probability of occurrence after the i-th feature is ablated, used to quantify the strength of the feature's influence on the predicted probability. Indicates in the validation set The expectation on the surface is that prompt represents the original text input after being serialized from the features of a single sample through a standardized template. This means deleting items containing the i-th feature in the prompt. The input following the corresponding text is used to achieve text-level feature ablation;

[0066] (4) Change in entropy Calculation: First, calculate the Shannon entropy as a measure of uncertainty:

[0067] ,

[0068] Then calculate the expected value of the entropy difference:

[0069] ,

[0070] in, This represents the average increment of uncertainty caused by the ablation of the i-th feature. The larger the value, the more the feature can reduce prediction uncertainty. Indicates in the validation set The expectation on the surface is that prompt represents the original text input after being serialized from the features of a single sample through a standardized template. This means deleting items containing the i-th feature in the prompt. The input following the corresponding text. This represents the Shannon entropy function, used to measure the uncertainty of binary classification output; This represents the probability of a positive class in a binary classification, with a value range of [0,1]. At that time, ablation of this feature increases uncertainty, indicating that this feature helps to stabilize discrimination; At other times, the opposite is true.

[0071] Phase 3: Comprehensive Importance Index Calculation and Feature Ranking: This phase integrates the results from both dimensions to generate the final ranking;

[0072] (1) Calculation of comprehensive index: Calculate the comprehensive importance score of the i-th feature:

[0073] ,

[0074] in, This represents a comprehensive importance index, emphasizing dual contributions through a product form; and features are then ranked in descending order based on this index.

[0075] (2) Feature importance ranking: For all M features... Sort the data in descending order and output a sorted list. Features with high rankings are considered core risk factors and used to build simplified predictive models or clinical guidelines.

[0076] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for evaluating the feature importance of POGD prediction based on a large model, characterized in that, Includes the following steps: Step 1: Building a baseline prediction model based on prompting engineering: Obtaining clinical datasets The structured clinical features of patients are converted into natural language prompt templates, and the entire dataset is then processed. Dataset converted to Prompt format ,Will Divided into training set and verification set and using the training set Supervised fine-tuning of a large language model yields a baseline prediction model. ; Step 2: Two-dimensional impact quantification based on feature ablation: On the validation set, the predicted probability of POGD is calibrated using the Softmax function. An ablation operation is performed on each feature, and the change in predicted probability is calculated. and entropy change ,in, This is the expected value of the difference between the predicted probabilities under the original hints and ablation hints on the validation set. To verify the expected value of the Shannon entropy difference between ablation hints and original hints on the validation set; The expression for the prediction probability calibration is: , in, This indicates the probability that the model output is high risk. This indicates that the model outputs words with high logarithmic scores. This indicates that the model output words have low logarithmic scores; The expression is: , in, This represents the average change in the probability of occurrence after the i-th feature is ablated, used to quantify the strength of the feature's influence on the predicted probability. Indicates in the validation set The expectation on the surface is that prompt represents the original text input after being serialized from the features of a single sample through a standardized template. This means deleting items containing the i-th feature in the prompt. The input following the corresponding text is used to achieve text-level feature ablation; The expression is: ， ， in, This represents the average increment of uncertainty caused by the ablation of the i-th feature. The larger the value, the more the feature can reduce prediction uncertainty. This represents the Shannon entropy function, used to measure the uncertainty of binary classification output; This represents the probability of a positive class in a binary classification, with a value range of [0,1]. When this feature is ablated, the uncertainty increases, indicating that this feature helps to stabilize the discrimination. At other times, the opposite is true; Step 3: Calculation of Overall Importance Index and Feature Ranking: Calculate the overall importance score for each feature. : ,in, This represents a comprehensive importance index, emphasizing dual contributions through a product; and features are then sorted in descending order based on this.

2. The feature importance evaluation method for POGD prediction based on a large model according to claim 1, characterized in that, In step 1, the expression for the clinical dataset is: ， in, Let represent the feature vector of the j-th sample, which contains M clinical features; The binary label indicating the occurrence of POGD 1 represents occurrence, 0 represents non-occurrence; N represents the sample size; In step 1, the natural language prompt template includes task instructions, feature display, and risk assessment questions; the answer is based on... Mapped to high or low.

3. The feature importance evaluation method for POGD prediction based on a large model according to claim 1, characterized in that, In step 1, the fine-tuning of the large language model uses the cross-entropy loss function, with the goal of generating the correct risk words after a given prompt.

4. The feature importance evaluation method for POGD prediction based on a large model according to claim 1, characterized in that, In step 2, the feature ablation operation is performed at the prompt text level, generating an ablation prompt by deleting lines containing specific features. .

5. The feature importance evaluation method for POGD prediction based on a large model according to claim 1, characterized in that, The method was applied to predict the risk of gastrointestinal dysfunction after general anesthesia, with clinical characteristics including age, sex, operation duration, and total remifentanil dosage.

Images