A Multi-Interpreter Partial Order Analysis Method Based on Shadow Calibration

By constructing a shadow reference set using the shadow calibration method and comparing it with the actual interpretable quantity, a partial order graph is generated and a maximal feature set is output. This solves the problem of the inability to directly compare the results of multiple interpreters and provides a robust decision-making reference.

CN122862951BActive Publication Date: 2026-11-17NINGBO UNIV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202611364440.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2026-09-04
Publication Date
2026-11-17
Estimated Expiration
2046-09-04

AI Technical Summary

Technical Problem

In existing methods for predicting the severity of injuries in traffic accidents, the incomparability of results from multiple interpreters and the inconsistency of calculation results lead to overconfidence or misjudgment in the analysis results.

Method used

By using the shadow calibration method, a shadow reference set is constructed and compared with the actual interpretable quantity. A partial order graph is generated and a maximal feature set is output, thereby achieving unified calibration and robust comparison of the results of each interpreter.

Benefits of technology

It enables robust comparison of results from multiple interpreters, avoids specification selection bias caused by computational failures, and provides a traceable and unexaggerated basis for decision-making.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122862951B_ABST
    Figure CN122862951B_ABST
Patent Text Reader

Abstract

This invention discloses a multi-interpreter partial order analysis method based on shadow calibration, relating to the field of electrical data processing technology. It acquires accident data and hierarchically divides it into training / holdout sets for each external fold according to injury category. Based on multiple interpretation sources, it calculates the predictive contribution of each feature relative to each category on the holdout set. A reference set is formed through in-situ replacement of shadow values ​​and retraining, and correlation percentiles are compared. Models effective across all features, interpretation sources, and shadow repetitions are selected to form a common support set. The minimum and maximum percentiles of each feature at each fold are taken as the lower and upper bounds of correlation. Under the same fold and interpretation source, the difference between any two feature percentiles is calculated, and the minimum value is taken across all combinations. If the difference in any direction is greater than a threshold, the former is considered superior to the latter. A partial order graph is generated based on this superiority relationship, and a maximal feature set is output. This invention can provide robust predictive correlation analysis results without forced ranking misleading without assigning weights.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of electrical data processing technology, and specifically to a multi-interpreter partial order analysis method based on shadow calibration. Background Technology

[0002] Existing methods for predicting the severity of traffic accident injuries typically use road environment, vehicle status, and collision characteristics as inputs to train classification models, and rely on interpretability methods to analyze the model's decision-making basis. Currently, methods such as SHAP (SHapley Additive Explanations), permutation feature importance, leave-one-out covariates, and feature importance ranking measures have been applied in this field. SHAP calculates the marginal contribution of features to the prediction results based on cooperative game theory, reflecting the attribution allocation within the model; permutation feature importance measures the model's dependence on features by shuffling feature values ​​and observing the decrease in prediction accuracy; leave-one-out covariates assess the amount of non-substitutable information carried by features by deleting features, retraining the model, and observing performance changes; and feature importance ranking measures focus on the magnitude of changes in prediction probability when feature values ​​change, reflecting the sensitivity of the prediction response to feature values.

[0003] Because the computational principles and statistical definitions of the above methods differ, the resulting importance values ​​differ fundamentally in terms of dimensions and meanings, making them incomparable. Existing techniques, when processing results from multiple interpreters, typically employ normalized weighted summation or averaging to calculate a composite score and then rank it. While this approach unifies the numerical range, it fails to determine whether each interpreter's output is significantly different from its random baseline. When discrepancies exist between different interpreters or model specifications, these discrepancies are compressed into a single score, forcing the ranking of features with insufficient evidence, potentially leading to overconfidence in the analysis results.

[0004] Some existing techniques provide a reference benchmark by constructing shadow features, that is, by adding randomly shuffled shadow feature columns to the original data, allowing the original features and shadow features to compete in a unified model to select effective features. However, adding shadow features changes the original feature dimensions and model splitting rules, leading to a shift in feature importance allocation and making it difficult to form an independent reference consistent with the explanatory statistics processing flow. In addition, in multi-model, multi-explainer combined analysis, some combinations may fail to execute, different features have different effective analysis specifications, and comparison results are affected by computational availability. On the other hand, predictive interpretation and variable structure analysis belong to different levels. The former describes the decision logic of the model using features, while the latter characterizes the conditional independence relationship between variables. Existing methods present the two types of results together, which can easily misjudge predictive correlation as causal effect. Summary of the Invention

[0005] To obtain a scheme that can separately calibrate different interpretation sources, limit the unified comparison range, retain specification discrepancies, and avoid forced overall ranking, this invention proposes a multi-interpreter partial order analysis method based on shadow calibration, including the following steps: S1: Obtain the traffic accident dataset, and divide it into external training sets and external holdout sets under each external fold according to the injury category. The external training set includes internal cross-validation. S2: Train different candidate classification models using internal cross-validation, select the admission model according to the validation results, and then refit on the external training set corresponding to each external fold. S3: Using the refitted admission model, calculate the predicted contribution of each accident feature to each injury category on the external holdout set based on different sources of explanation, and obtain the actual explanatory power in a unified direction. S4: Within the external training set, shadow values ​​are sampled from the empirical marginal distribution of each accident feature and replaced in situ. The admission model is then refitted and the shadow explanatory power is calculated. This process is repeated multiple times to form a shadow reference set and compare it with the actual explanatory power to obtain the correlation percentile. S5: Select the admission models that produce complete results in all accident features, explanatory sources and shadow repetitions to form a common support set; The minimum and maximum values ​​of the correlation percentiles of each accident feature in the common support set under each external fold and each source of explanation are taken as the lower and upper bounds of the correlation of the accident feature under the corresponding injury category, respectively. Under the same admission model, external fold and interpretation source, the difference of correlation percentiles of any two accident features is calculated and the minimum value is taken. If the minimum value in any direction of the difference is greater than the threshold, the feature in that direction is determined to be superior to the other feature. A partial order graph is generated based on the superiority relationship and the maximum feature set is output.

[0006] This invention transforms heterogeneous interpretation results into comparable correlation percentiles by constructing a homologous zero reference through in-situ replacement of shadow values ​​and repeated refitting. After comparing the consistency of specifications by constraining the common support set, it determines the strict superiority relationship by the pairwise lower bound difference under the same analysis specification, generates a partial order graph and outputs a maximal feature set, thus realizing robust feature correlation comparison without weight fusion and without forced sorting.

[0007] Furthermore, in step S3, the different sources of interpretation include at least two of the following: SHAP, permutation feature importance, leave-one-out covariate, and feature importance ranking measure.

[0008] Furthermore, in step S3, the predicted contribution is calculated as follows: for each damage category, the current damage category is taken as the target category, all other damage categories are merged into non-target categories, and the contribution of each feature is calculated in a binary classification manner; wherein the target category and the non-target category are assigned equal total weights in the calculation.

[0009] Furthermore, in step S3, unifying the direction specifically means: setting the statistical direction of each explanation source to a value that indicates a more prominent correlation between the accident characteristics and the prediction results of the corresponding injury category.

[0010] Furthermore, in step S4, the shadow value is obtained by extraction with replacement, and the calculation process of the shadow interpretation quantity is consistent with the calculation process of the actual interpretation quantity.

[0011] Furthermore, in step S4, the number of times the shadow repeats is determined according to the required numerical resolution, where the resolution is... , The number of times the shadow repeats.

[0012] Further, in step S4, the in-situ replacement specifically involves: for the target accident feature, extracting shadow values ​​with replacement from the empirical marginal distribution of the target accident feature within the external training set under the corresponding external fold and directly replacing the original value of the target accident feature, while keeping the feature dimension, field position, and data type unchanged; after each replacement, refitting the admission model and then calculating the shadow explanatory power.

[0013] Furthermore, in step S5, if the common support set is empty, the external fold, admission model, or explanation source information that caused the failure is recorded, and subsequent partial order comparisons under that damage category are terminated. Comparisons of different damage categories are independent of each other.

[0014] Furthermore, the partial order graph is a Hasse graph after transitive reduction, and the maximal feature set is the set of accident features that cannot be directly or indirectly dominated by other accident features through superior relations.

[0015] Compared with the prior art, the present invention has at least the following beneficial effects: (1) The present invention proposes a multi-explainer partial order analysis method based on shadow calibration. By constructing the actual explanatory quantity and the same source shadow reference set under each external fold, the explanatory sources from different principles are all converted into unified bounded correlation percentiles by the same zero reference calibration. Thus, while preserving the statistical meaning of SHAP, permutation feature importance, leave-one covariate and feature importance ranking measure, the difference in dimensions is eliminated, and the fundamental problem that the results of multiple explainers cannot be directly compared is solved. (2) By screening the admission models that produce complete and finite results in all features, sources of explanation and shadow repetition, a common support set is formed to ensure that any two features are always compared under the same analytical specification, thus avoiding specification selection bias caused by calculation failure. (3) Take the minimum value of the correlation percentile of each feature in the common support set under each external fold and each explanatory source as the lower bound of the correlation, and characterize the predictive correlation strength of the feature under different reasonable specifications in the most conservative way. Then, under the same model, external fold and explanatory source, only establish a strict superior relationship when the evidence is sufficient, otherwise retain the incomparable state between features. The resulting partial order graph and maximal feature set can distinguish between the two essentially different situations of "insufficient evidence to determine the order relationship" and "two feature results are the same". (4) Without relying on any model weights or interpreter weights, a list of priority investigation targets with robust predictive relevance under each injury category and not dominated by other features is obtained, providing a traceable, robust and non-exaggerated evidence for decision-making reference in traffic safety analysis. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the steps of a multi-interpreter partial order analysis method based on shadow calibration. Detailed Implementation

[0017] This invention provides a detailed description of its method using accident report data from an autonomous driving system as an example. It should be noted that the data sources, sample size, number of features, and model types used in this embodiment are merely illustrative of the implementation process and do not constitute any limitation on the scope of protection of this invention. Those skilled in the art can adaptively adjust the dataset source, feature type, model selection, and parameter settings according to actual application scenarios and analysis objectives, all of which fall within the scope of protection of this invention.

[0018] The multi-interpreter partial order analysis method proposed in this invention follows a logical thread of "first, hierarchical partitioning and model admission; then, actual interpretation and shadow calibration; and finally, joint support constraints and partial order determination," as follows: Figure 1 As shown, the main steps include the following: S1: Obtain the traffic accident dataset, and divide it into external training sets and external holdout sets under each external fold according to the injury category. The external training set includes internal cross-validation. S2: Train different candidate classification models using internal cross-validation, select the admission model according to the validation results, and then refit on the external training set corresponding to each external fold. S3: Using the refitted admission model, calculate the predicted contribution of each accident feature to each injury category on the external holdout set based on different sources of explanation, and obtain the actual explanatory power in a unified direction. S4: Within the external training set, shadow values ​​are sampled from the empirical marginal distribution of each accident feature and replaced in situ. The admission model is then refitted and the shadow explanatory power is calculated. This process is repeated multiple times to form a shadow reference set and compare it with the actual explanatory power to obtain the correlation percentile. S5: Select the admission models that produce complete results in all accident features, explanatory sources and shadow repetitions to form a common support set; The minimum and maximum values ​​of the correlation percentiles of each accident feature in the common support set under each external fold and each source of explanation are taken as the lower and upper bounds of the correlation of the accident feature under the corresponding injury category, respectively. Under the same admission model, external fold and interpretation source, the difference of correlation percentiles of any two accident features is calculated and the minimum value is taken. If the minimum value in any direction of the difference is greater than the threshold, the feature in that direction is determined to be superior to the other feature. A partial order graph is generated based on the superiority relationship and the maximum feature set is output.

[0019] Specifically, each record in the traffic accident dataset (e.g., using accident reports of autonomous driving systems published by the National Highway Traffic Safety Administration as the data source) contains multiple accident features and corresponding injury category labels. For ease of description, let n denote the total number of accident records in the dataset, p denote the number of accident features in each record, and C denote the total number of injury categories. The entire analysis process is conducted within a framework of multiple external cross-validations. Each external cross-validation generates an external training set and a corresponding external holdout set. The external training set is used for model training and the construction of shadow zero references, while the external holdout set is used for calculating the actual explanatory power. Within each external training set, internal cross-validation is further implemented for performance evaluation and admission selection of candidate models.

[0020] For the dataset partitioning, this embodiment adopts a stratified strategy based on injury category. Specifically, all accident records are first grouped according to their respective injury categories. Then, within each category, random partitioning is performed independently, ensuring that the proportion of samples from each category in each external training set and each external holdout set remains consistent with the proportion in the original dataset. This stratified partitioning strategy aims to avoid partitioning bias caused by class imbalance, ensuring that the data distribution used for subsequent model training and explanatory power calculation is representative. This embodiment employs five-fold external cross-validation, dividing the dataset into five non-overlapping parts. One part is used as the external holdout set in turn, while the remaining four parts are merged as the external training set, resulting in five external folds. For each external fold, a complete model training, explanatory power calculation, and shadow calibration process is performed. Within each external training set, five-fold internal cross-validation is also set up. This internal cross-validation is performed only within the corresponding external training set and does not involve any samples in the external holdout set.

[0021] In the model training and admission phases, several classification models based on different principles are selected as candidate models. In this embodiment, the candidate models include logistic regression, random forest, extreme random trees, histogram gradient boosting, XGBoost, LightGBM, and CatBoost, totaling seven different types of classification models. These models operate on different principles, covering various technical approaches such as linear models, ensemble tree models, and gradient boosting models. The reason for selecting multiple candidate models based on different principles is that different models may have varying degrees of dependence on the same feature; only features exhibiting robust correlation across multiple reasonable models have high credibility. Model admission is based on predictive performance in internal cross-validation, specifically using balanced accuracy as the evaluation metric and a one-standard-error rule as the admission rule. That is, the mean and standard deviation of the balanced accuracy of each candidate model at each fold of internal cross-validation are first calculated. The model corresponding to the highest mean is determined, and then all models whose mean balanced accuracy is not lower than the highest mean minus one standard error are retained as admission models. This admission rule can avoid over-reliance on a single model while ensuring predictive performance, thus providing a basis for subsequent multi-model comparisons. The model admission process is entirely based on the internal cross-validation results, without reading any information from the external hold-out set. This ensures that the actual explained values ​​calculated on the external hold-out set are true out-of-sample evaluation results. For each external fold, after model admission is completed, all admitted models are refitted on the corresponding complete external training set. This refitting process uses all external training set samples and does not retain the validation set, aiming to bring the model parameters to their optimal state before the final calculation of explained values.

[0022] After model refitting is completed, the next step is to calculate the actual explanatory power. The actual explanatory power refers to the statistical measure calculated on the external hold-out set using various explanatory sources, reflecting the contribution of each accident feature to the prediction results of each injury category. This embodiment employs four explanatory sources based on different principles: SHAP, permutation feature importance, leave-one-out covariates, and feature importance ranking measures. The four interpretations are based on different computational principles: SHAP reflects the model attribution of feature to class prediction probability, that is, it fairly distributes the prediction results to each input feature based on a cooperative game theory framework. Its value can be positive or negative; a positive value indicates that the feature has a positive effect on the prediction result, and a negative value indicates that it has an inhibitory effect. The importance of the permuted feature is represented by the difference in class balance Brier loss before and after the target feature permutation, that is, the loss after permutation minus the loss before permutation. A positive value indicates that the permutation leads to an increase in prediction loss, and a negative value indicates that the loss after permutation actually decreases. The larger the value, the greater the adverse effect of the permutation of the feature on the model prediction, that is, the more prominent the model's dependence on the feature. Leave-one covariate is represented by the difference between the loss after deleting the target feature and refitting the model and the loss of the complete model. A positive value indicates that the loss after deletion increases the prediction loss, and a negative value indicates that the loss after deletion actually decreases. The larger the value, the more prominent the model's dependence on the feature. The feature importance ranking metric reflects the degree of change in prediction probability when the feature changes between different levels of the observed data. This metric measures the actual influence of the feature within the observation range. Because these four sources of interpretation have different calculation principles and statistical standards, the raw statistics they produce have different dimensions and numerical ranges, and cannot be directly compared numerically.

[0023] To integrate the four heterogeneous interpretation results into a unified comparison framework, this embodiment standardizes the calculation method and output direction for each interpretation source. For each interpretation source, the explanatory power of each feature is calculated one by one according to the injury category. Specifically, the injury category currently being analyzed is taken as the target category, and all other injury categories are merged into non-target categories, forming a binary classification prediction scenario. In the binary classification scenario, the target category and non-target categories are given equal total weights in the calculation, that is, each accounts for half of the overall evaluation weight. The purpose of this approach is to avoid the majority category dominating the explanatory power calculation, so that the explanatory power corresponding to each injury category can truly reflect the feature contribution structure when the category is predicted. After obtaining the original explanatory power of each interpretation source through the above method, the statistical direction of each interpretation source is uniformly set to "the larger the value, the more prominent the correlation between the accident feature and the prediction result of the corresponding injury category". For SHAP, since its original values ​​are inherently positive or negative, the absolute impact on the prediction results is used as the benchmark for subsequent comparisons when the directions are unified, while retaining its sign to distinguish between promoting and inhibiting effects. For permutation feature importance and leave-one-out covariates, since both use the change in predicted loss as the original statistic, a positive value indicates that the feature makes a positive contribution to the prediction, while a negative value indicates that the feature may have a pervasive effect or no contribution. After direction unification, the signed difference is used directly, and subsequent comparisons are made according to the principle that the larger the value, the more prominent the model's dependence on that feature. The feature importance ranking metric is inherently non-negative, and its original value is used directly after direction unification. The explanatory quantities after direction unification are the actual explanatory quantities, and all subsequent comparisons are based on these actual explanatory quantities.

[0024] After obtaining the actual explanatory values, the construction phase of the shadow zero-reference begins. The purpose of the shadow zero-reference is to provide a homologous random reference benchmark for each actual explanatory value. "Homologous" here means that the shadow explanatory values ​​and actual explanatory values ​​use the exact same statistical definitions and calculation procedures, differing only in the input data—actual explanatory values ​​are calculated based on real accident feature values, while shadow explanatory values ​​are calculated based on feature values ​​that have undergone artificial randomization. The shadow values ​​are constructed as follows: For a target accident feature, samples with replacement are taken from its empirical marginal distribution within the external training set under the corresponding external fold. The empirical marginal distribution refers to the empirical distribution of the observed values ​​of this feature across all samples in the external training set; that is, it does not consider the correlation between this feature and any other features or labels, only retaining the value distribution shape of the feature itself. The sampled shadow values ​​directly replace the original values ​​of the target accident feature in the current record, rather than being added to the data as new features. This in-situ replacement operation preserves the feature's dimensions, field positions, and data types. The only change is the correspondence between the feature and the damage label—after the replacement, the feature's value no longer has a practical relationship with the damage label of its sample. Since shadow values ​​are derived from sampling with replacement, the same feature in the same sample may obtain the same shadow value or different shadow values ​​in multiple samplings.

[0025] After each shadow replacement, the admission model under that external fold needs to be refitted, and then the shadow explanatory power is calculated using the same explanatory algorithm as the actual explanatory power. This retraining process is a crucial step in shadow calibration, aiming to ensure that the shadow explanatory power reflects the complete analytical process of "the explanatory power of a feature in the retrained model after it has been replaced with random noise." If retraining is not performed and mask calculation is only performed on the existing model, the combined impact of feature replacement on the preprocessing and model fitting processes cannot be captured. In this embodiment, for each accident feature, 49 independent shadow replacement and retraining processes are performed, i.e., the number of shadow repetitions. The value is 49. The number of shadow repetitions determines the numerical resolution of the shadow calibration, and its resolution is... In this embodiment, 1 / 50 = 0.02, meaning the accuracy of the correlation percentile can reach 0.02. This resolution is sufficient to meet the decision-making needs of practical engineering applications. It is important to note that the construction of shadow values, model refitting, and calculation of shadow explained quantities are all completed within the external training set, without reading the corresponding external holdout set. This is because the goal of shadow calibration is to provide a reference benchmark established within the training data for the actual explained quantities, rather than evaluating model performance on the holdout data.

[0026] After completing all shadow replacements and shadow interpretation calculations, the actual interpretation is compared with its corresponding shadow interpretation. For the first... The first external fold, the first The first admission model, the first The first explanation source, the first The damage category and the first Let each accident characteristic have an actual explanatory power of 1. The superscript obs indicates the actual value, the first... The shadow interpretation quantity calculated after the shadow replacement is: The superscript 0 indicates the shadow value. First, calculate the values ​​across all... In each shadow replacement, the number of times the shadow explanatory power is greater than or equal to the actual explanatory power is denoted as [the number of times the shadow explanatory power is greater than or equal to the actual explanatory power]. ,Right now ,in This is the indicator function, taking the value 1 if the condition is met, and 0 otherwise. Then, the right-tail proportion of the finite sample is calculated. . The meaning is the relative frequency of reaching or exceeding the actual explanatory quantity in the random reference distribution constructed by the shadow replacement. The larger the value, the less prominent the actual explanatory power is in the random reference distribution; conversely, the smaller the value, the less prominent the actual explanatory power is in the random reference distribution. The smaller the value, the more significantly the actual explanatory power deviates from the level of random noise. Based on this, the correlation percentile is defined. . The value ranges from 0 to 1. A higher correlation percentile indicates a higher position of the actual explanatory power of the accident feature within its corresponding zero-reference distribution, meaning a more prominent association between the feature and the predicted injury category. It is important to emphasize the correlation percentile. It is a relative measure obtained by comparing actual and shadow explanations within the same analytical specification, and between different sources of explanation. Although the values ​​have the same scale, their underlying original statistical meanings are still preserved, that is... The value merely indicates "how prominent the contribution of this feature is relative to the random baseline within the computational framework of this explanatory source," and does not indicate a direct conversion relationship between different explanatory sources.

[0027] After the aforementioned shadow calibration, the original statistics from each explanatory source were converted into uniformly bounded correlation percentiles. However, not all combinations of models and interpretation sources can successfully complete computations under all conditions. Due to structural differences between different models, variations in the numerical stability of different interpretation algorithms, and changes in data distribution caused by shadow substitution, some models may fail to produce finite numerical results under certain external folds or interpretation sources. For example, computational anomalies such as division by zero, matrix invertibility, or constant prediction probabilities may occur. If these anomalies are arbitrarily ignored in subsequent comparisons, it may lead to comparisons between different features based on different sets of models and interpretation sources, thus introducing selection bias. To address this, this embodiment introduces a common support constraint mechanism.

[0028] The core idea of ​​common support constraints is to allow comparisons of different features only under identical analytical specifications. An analytical specification refers to a specific combination of an external fold, a specific admission model, and a specific source of interpretation. For all features to be compared under a given injury category, comparisons are only permitted if an admission model produces complete and finite support for all features to be compared, all sources of interpretation, and all shadow repetitions within that injury category. A model is only included in the common support set when it is found to be valid. If a model fails to compute on any feature, any explanatory source, or any shadow repetition, it is completely excluded from the common support set for that injury category, not just the failed feature or explanatory source. This is to ensure that in subsequent comparisons, the model sets used for comparing any two features are completely identical, avoiding asymmetric situations such as feature A using models 1, 2, and 3, while feature B uses models 1, 2, and 4. If the common support set for a certain injury category is empty, meaning no model can produce complete finite results on all features, all explanatory sources, and all shadow repetitions, then subsequent partial-order comparisons for that injury category are terminated. In this case, the external fold, model, or explanatory source information that caused the failure can be recorded for subsequent troubleshooting analysis. Comparisons between different injury categories are independent; an empty common support set for one injury category does not affect the analysis process for other injury categories.

[0029] After constructing the common support set, this embodiment further calculates the upper and lower bounds of the correlation for each accident feature. The lower and upper bounds of the correlation reflect the fluctuation range of the correlation level of the accident features under different reasonable analysis specifications. For the first... The first external fold, the first The first explanation source, the first The damage category and the first For each accident feature, firstly, among all admission models within the common support set of this external fold, the percentile of the relevance of that feature is taken. The minimum and maximum values ​​are used as the minimum and maximum values ​​under this external fold and this source of explanation, respectively. Then, considering that the sample sizes of the external holdout sets of different external folds may not be exactly the same, a sample size-weighted strategy is adopted when summarizing each external fold. Specifically, for the , Each external fold has a weight equal to the sum of the external holdout sample sizes of that fold and the sum of the external holdout sample sizes of all external folds. For each explanatory source, the sum of the weighted minimum and the sum of the weighted maximum values ​​are calculated under each external fold to obtain the aggregate lower bound and aggregate upper bound for that explanatory source. Finally, among all explanatory sources, the minimum value of the aggregate lower bound is taken as the lower bound of the relevance of the accident feature under the corresponding injury category, and the maximum value of the aggregate upper bound is taken as the upper bound of the relevance of the accident feature under the corresponding injury category. The relevance lower bound and upper bound together constitute the result envelope of the accident feature under all reasonable analytical specifications, where the lower bound represents the relevance level of the feature under the most unfavorable reasonable specification, and the upper bound represents its relevance level under the most favorable reasonable specification. It should be noted that the relevance upper and lower bounds are deterministic envelope values, not frequent confidence intervals, nor do they have the probabilistic interpretation of Bayesian confidence intervals; their sole function is to record the range of discrepancies between different reasonable specifications.

[0030] While calculating the upper and lower bounds of the correlation between various accident features, a partial order determination is also performed simultaneously. The purpose of the partial order determination is to establish a partial sequential relationship among all features to be compared, rather than forcibly excluding the complete overall order. This embodiment does not directly compare whether the endpoints of the upper and lower bounds of the correlation between two features overlap, but adopts a more stringent standard: under the same external fold, the same admission model, and the same interpretation source analysis specifications, comparisons are performed in pairs.

[0031] Specifically, for the same damage category The following two accident characteristics and Under the analytical specifications covered by each common support set, the computational features are... Correlation percentiles and characteristics The difference in correlation percentiles. Since the correlation percentiles themselves have already been shadow-calibrated, different characteristics under the same analytical specifications... The values ​​are comparable. However, the differences under different analytical specifications may have different directions, for example, under certain models, the characteristics... Superior features Features under other models Superior features To establish a robust partial order relation, this embodiment employs a worst-case criterion: among all commonly supported analysis specifications, the feature is selected. Relative to features The minimum value of the difference, i.e., the most unfavorable feature. The difference under the specified conditions. If the worst-case difference is still greater than the preset threshold, then the feature is determined. Strictly superior to features The preset threshold here is only used to exclude rounding errors generated by the computing device during floating-point operations. For example, when two values ​​are theoretically equal but slightly different due to floating-point precision limitations, this threshold can prevent such numerical errors from being misjudged as a true superior-inferior relationship. The value of the preset threshold is related to the numerical precision of the computing device and is not used as a critical value for statistical significance testing.

[0032] If among all commonly supported analysis specifications, when the feature Relative to features The worst-case difference is no greater than a preset threshold, and the feature Relative to features If the worst-case difference is no greater than the threshold, then no feature is determined. Superior features Nor to determine features Superior features Instead, it preserves the incomparability between two features. This incomparability is a key feature of this invention, indicating that based on existing evidence and reasonable analytical specifications, it is impossible to determine the relative order of superiority of the two features. This does not mean that the actual correlation levels of the two features are equal, but rather that the currently available data and model specifications are insufficient to provide a definitive ranking. Preserving the incomparability avoids misleading conclusions caused by forced ranking. For traffic safety decisions, clearly stating "insufficient evidence to distinguish" is more valuable than providing an unreliable ranking.

[0033] Based on the strict dominance relation, a directed graph can be constructed among all features to be compared, where nodes represent features and directed edges point from the strictly dominant feature to the dominated feature. This directed graph is necessarily transitive, meaning that if the strictly dominant feature... Superior features and features Superior features Then the feature It is also superior to features To further simplify the representation of this graph structure, this embodiment performs transitive reduction, that is, deleting all edges that can be derived from transitive relations, retaining only direct covering relations. The directed graph after transitive reduction is a Hasse graph. In a Hasse graph, a maximal feature set can be identified, which is a set of features that cannot be directly or indirectly dominated by other features through superiority relations. The features in the maximal feature set are all incomparable, meaning that existing evidence is insufficient to establish a stable order of superiority between any two. These features represent a set of candidate factors with high and irreplaceable predictive relevance under the corresponding injury category. For traffic safety personnel, the maximal feature set provides a list of accident factors worthy of priority attention and further investigation.

[0034] This embodiment also includes an optional structural interpretation channel to supplement the prediction interpretation results from the perspective of causal structure. This channel is independent of the aforementioned shadow calibration and partial order analysis processes and runs on the external training set of each external fold. In the structural interpretation channel, a fast causal inference algorithm is used to generate a partial ancestor graph. The input of this algorithm is the original accident features and injury result nodes in the external training set, without shadow data or reading the corresponding external holdout set. For discrete accident variables, the G-squared conditional independence test is used as the basis for judging conditional independence. To enhance the robustness of structural discovery, multiple significance levels and maximum condition set depths are set. The parameters of each set are predetermined before the analysis begins and are not adjusted based on the results of the aforementioned shadow calibration or partial order analysis. In this embodiment, three significance levels, three maximum condition set depths, and five external folds are combined to form forty-five structural specifications. After running the fast causal inference algorithm under each structural specification, a partial ancestor graph containing accident feature nodes and overall injury result nodes is obtained. For each accident feature, the shortest distance between it and the injury result node in the undirected skeleton is calculated. If the distance is 1, it means that the feature node and the damage result node are directly adjacent in the undirected skeleton; if the distance is 2, it means that there is exactly one intermediate node between them; if the distance is less than or equal to 2, it includes both direct adjacency and two-step proximity. After summarizing all forty-five structural specifications, the direct adjacency frequency and the two-step proximity frequency between each feature and the damage result node are calculated.

[0035] The results from the structural interpretation channel are output alongside those from the shadow calibration partial order analysis. They do not modify the upper and lower bounds of correlation or the partial order relationship, nor do they determine a unique causal direction, mediation relationship, or causal effect size. The purpose of this channel is to provide a description of the structural relationships between variables from the perspective of conditional independence, helping analysts understand the association patterns between various factors and accident injuries from multiple angles. This structural interpretation information forms cross-validation with the aforementioned maximal feature set. When a feature appears simultaneously in the maximal feature set and is frequently directly adjacent to injury outcome nodes in some ancestor graphs, it can provide more multi-dimensional evidence supporting the importance of that feature.

[0036] In summary, this invention, through a shadow calibration mechanism, unifies the heterogeneous predictive contributions from different explanatory sources—such as SHAP, permutation feature importance, leave-one-out covariate, and feature importance ranking measures—into comparable relevance percentiles based on a zero-reference homologous shadow. This fundamentally solves the technical challenge of directly comparing results from multiple interpreters due to their different dimensions. Furthermore, by selecting admission models that produce complete and finite results across all accident features, explanatory sources, and shadow repetitions to form a common support set, it ensures that any two accident features are compared under identical analytical specifications, avoiding inconsistencies in comparison benchmarks caused by partial computational failures. Meanwhile, this invention does not assign fusion weights to the model or explanatory sources in any way. Instead, it uses the minimum correlation percentile of each accident feature under each external fold and explanatory source as the lower bound of correlation. Based on the worst-case scenario under the same model, external fold, and explanatory source, it determines the superiority of the difference. A strict partial order is established only when the difference exceeds a preset threshold; otherwise, the incomparability between features is preserved. This effectively distinguishes between the two fundamentally different situations: "insufficient evidence to determine the order" and "two feature results are the same," avoiding the misleading conclusions caused by forced ranking in existing technologies. Finally, by transmitting the reduced Hasse diagram and the maximal feature set, it intuitively presents accident factors under each injury category that are not stably dominated by other features in an unweighted form, providing a traceable, robust, and unexaggerated evidence-based decision-making reference for traffic safety analysis.

[0037] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a specific posture. If the specific posture changes, the directional indication will also change accordingly.

[0038] Furthermore, in this invention, descriptions involving terms such as "first," "second," and "a" are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0039] Furthermore, the technical solutions of the various embodiments of the present invention can be combined with each other, but only if they are feasible for those skilled in the art. If the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

Claims

1. A multi-interpreter partial order analysis method based on shadow calibration, characterized in that, Including the following steps: S1: Obtain the traffic accident dataset, and divide it into external training sets and external holdout sets under each external fold according to the injury category. The external training set includes internal cross-validation. S2: Train different candidate classification models using internal cross-validation, select the admission model according to the validation results, and then refit on the external training set corresponding to each external fold. S3: Using the refitted admission model, calculate the predicted contribution of each accident feature to each injury category on the external holdout set based on different sources of explanation, and obtain the actual explanatory power in a unified direction. S4: Within the external training set, shadow values ​​are sampled from the empirical marginal distribution of each accident feature and replaced in situ. The admission model is then refitted and the shadow explanatory power is calculated. This process is repeated multiple times to form a shadow reference set and compare it with the actual explanatory power to obtain the correlation percentile. S5: Select the admission models that produce complete results in all accident features, explanatory sources and shadow repetitions to form a common support set; The minimum and maximum values ​​of the correlation percentiles of each accident feature in the common support set under each external fold and each source of explanation are taken as the lower and upper bounds of the correlation of the accident feature under the corresponding injury category, respectively. Under the same admission model, external fold and interpretation source, the difference of correlation percentiles of any two accident features is calculated and the minimum value is taken. If the minimum value in any direction of the difference is greater than the threshold, the feature in that direction is determined to be superior to the other feature. A partial order graph is generated based on the superiority relationship and the maximum feature set is output.

2. The multi-interpreter partial order analysis method based on shadow calibration as described in claim 1, characterized in that, In step S3, different sources of interpretation include at least two of the following: SHAP, permutation feature importance, leave-one-out covariate, and feature importance ranking measure.

3. The multi-interpreter partial order analysis method based on shadow calibration as described in claim 1, characterized in that, In step S3, the predicted contribution is calculated as follows: for each damage category, the current damage category is taken as the target category, all other damage categories are merged into non-target categories, and the contribution of each feature is calculated in a binary classification manner; wherein the target category and the non-target category are assigned equal total weights in the calculation.

4. The multi-interpreter partial order analysis method based on shadow calibration as described in claim 1, characterized in that, In step S3, the unified direction is specifically defined as follows: the statistical direction of each explanation source is set so that the larger the value, the more prominent the correlation between the accident characteristics and the prediction results of the corresponding injury category.

5. The multi-interpreter partial order analysis method based on shadow calibration as described in claim 1, characterized in that, In step S4, the shadow value is obtained by extraction with replacement, and the calculation process of the shadow interpretation quantity is consistent with the calculation process of the actual interpretation quantity.

6. The multi-interpreter partial order analysis method based on shadow calibration as described in claim 1, characterized in that, In step S4, the number of times the shadow repeats is determined according to the required numerical resolution, where the resolution is... , The number of times the shadow repeats.

7. The multi-interpreter partial order analysis method based on shadow calibration as described in claim 1, characterized in that, In step S4, the in-situ replacement specifically involves: for the target accident feature, extracting shadow values ​​with replacement from the empirical marginal distribution of the target accident feature within the external training set under the corresponding external fold, and directly replacing the original value of the target accident feature, while keeping the feature dimension, field position, and data type unchanged; after each replacement, refitting the admission model, and then calculating the shadow explanatory power.

8. The multi-interpreter partial order analysis method based on shadow calibration as described in claim 1, characterized in that, In step S5, if the common support set is empty, the external fold, admission model, or explanation source information that caused the failure is recorded, and subsequent partial order comparisons under that damage category are terminated. Comparisons of different damage categories are independent of each other.

9. The multi-interpreter partial order analysis method based on shadow calibration as described in claim 1, characterized in that, The partial order graph is a Hasse graph after transitive reduction, and the maximal feature set is the set of accident features that cannot be directly or indirectly dominated by other accident features through superior relations.

Citation Information

Patent Citations

  • Maritime accident prediction method and device based on interpretable integrated machine learning

    CN121457753A

  • Electronic system for dynamic, quasi-realtime measuring and identifying driver maneuvers solely based on mobile phone telemetry, and a corresponding method thereof

    WO2019048070A1