Algorithm for realizing artificial intelligence black box model fine decision logic and human cognition alignment degree evaluation

Abstract

The invention relates to the technical field of machine learning, and discloses a method and a system for realizing artificial intelligence black box model fine decision logic and human cognition alignment degree evaluation. According to the method and the system, the degree of alignment between model decision logic and human cognition can be evaluated by analyzing interactive distribution modeled by the model. Implementations of the method and system include the steps of: providing an input sample; predicting an input sample by using a black box model to obtain a prediction result of the model; based on the output of the black box model, modeling the interaction between the input units of the sample, calculating the interaction strength of the combination formed between the input units, and expressing the output of the black box model as the interaction utility between the input unit combinations; and evaluating the degree of alignment between the model decision logic and human cognition by using an evaluation index based on interaction. The method has the advantages that the quantitative method for evaluating the alignment degree of the fine decision logic and the human cognition of the artificial intelligence black box model is provided, and compared with previous research, the difference between the model decision logic and the human cognition can be better described, and potential characterization defects of the model can be reflected.

Description

Technical Field

[0001] The present invention relates to the field of machine learning technology, and in particular to a method and system for evaluating the degree of alignment between the refined decision-making logic of an artificial intelligence black box model and human cognition. Background Art

[0002] At present, deep learning has been widely used in various fields and has shown powerful performance, but it is difficult for users and researchers to analyze the black box nature of artificial intelligence models. In existing technologies, interactive explanation methods can analyze an artificial intelligence model and obtain sparse and concise explanations. However, the explanations obtained still cannot directly evaluate the difference between the decision logic of the black box model and human cognition, reflecting the potential representation defects of the model. The degree of alignment between the decision logic of the evaluation model and human cognition is precisely a necessary condition for humans to trust black box models and apply black box models in high-risk scenarios.

[0003] Therefore, how to reliably evaluate the degree of alignment between the black box model's sophisticated decision-making logic and human cognition is an urgent problem to be solved in the field of explainability. Summary of the invention

[0004] The purpose of the present invention is to invent a method and system for evaluating the degree of alignment between the refined decision logic of an artificial intelligence black box model and human cognition. The method and system can evaluate the degree of alignment between the model's decision logic and human cognition by analyzing the interaction distribution modeled by the model.

[0005] The present invention discloses a method for evaluating the degree of alignment between the fine decision logic of an artificial intelligence black box model and human cognition, characterized in that the method comprises the following steps:

[0006] (1) Select a black box AI model;

[0007] Selecting a black box artificial intelligence model to be analyzed, wherein the black box artificial intelligence model includes an artificial intelligence model pre-trained based on a certain data set;

[0008] (2) Select input samples and perform recognition;

[0009] Select an input sample for interactive utility calculation, and identify the input sample, so as to decompose the input sample into n input units, and combine the n input units to obtain 2 n The input sample is selected from the following group: table data, picture, text, voice or a combination thereof;

[0010] (3) Calculate and combine “interaction utilities”;

[0011] In step (2), nThe combination of input units is respectively input into the black box artificial intelligence model, and the output of the black box artificial intelligence model is obtained; based on the output of the black box artificial intelligence model, the interaction between the input units is modeled, so as to obtain the "interaction utility" of the black box artificial intelligence model for each combination of the input units; and the output of the black box artificial intelligence model on a certain input unit combination is interpreted as the combination of its "interaction utility" on the input unit combination;

[0012] (4) Divide the “interaction utility” into “reliable component of interaction utility” and “unreliable component of interaction utility”;

[0013] According to whether each input unit is relevant to the output of the black box artificial intelligence model in human cognition, all input units are divided into "relevant input units", "irrelevant input units" and "mutually exclusive units" by algorithm or user customization.

[0014] The "interaction utility" of the combination of all input units in step (3) is divided into "reliable component of interaction utility" and "unreliable component of interaction utility". Among them, the "reliable component of interaction utility" represents the interaction utility consistent with human cognition, and the "unreliable component of interaction utility" represents the interaction utility inconsistent with human cognition. Interaction utility can be divided into "and interaction utility" and "or interaction utility", which respectively represent the "and relationship" between the input units modeled by the model and the "or relationship" between the input units modeled by it. Preferably, the "reliable component of and interaction utility" can be calculated as the interaction utility of the combination containing "related input units" and not containing "mutually exclusive units"; the "unreliable component of and interaction utility" can be calculated as the interaction utility of the combination not containing "related input units" or containing "mutually exclusive units". Preferably, a "reliable component of or interaction utility" can be calculated as the interaction utility of the combination containing "related input units" and not containing "mutually exclusive units"; the "unreliable component of or interaction utility" can be calculated as the interaction utility of the combination not containing "related input units" or containing "mutually exclusive units". Another algorithm is to distribute the "or interaction utility" evenly to each input unit included in this interaction, so that the "reliable component of the or interaction utility" can be calculated as the interaction utility component of the "or interaction utility" evenly distributed to the "related input unit"; the "unreliable component of the or interaction utility" can be calculated as the interaction utility component of the "or interaction utility" evenly distributed to the "irrelevant input unit" and the "mutually exclusive unit".

[0015] (5) Use interaction-based evaluation indicators to evaluate the degree of alignment between the model’s decision logic and human cognition;

[0016] According to the proposed interaction-based evaluation indicators, the ratio of “reliable component of interaction utility” to “unreliable component of interaction utility” is counted to evaluate the degree of alignment between the model’s refined decision-making logic and human cognition.

[0017] The order of step (1) and step (2) can be replaced at will or performed simultaneously.

[0018] In a preferred example, the output of the black box artificial intelligence model on the combination of the input units is defined as follows: if the input units in the set N\S in the input sample x are blocked, and the input units in the set S are not blocked, the sample x is obtained. S . With x S The output obtained by inputting the model is denoted as v(x S ); If the input sample is input into the black box model without occlusion, the output obtained is expressed as v(x N ); If all input units in the input sample are completely blocked and input into the black box model, the output obtained is expressed as Preferably, in natural language processing applications, the masking operation can be achieved by replacing the word embedding vectors of each token covered by each input unit included in S with a specific reference value vector.

[0019] In a preferred example, if the black box artificial intelligence model is a classification model, the output v(x S ) can be expressed as the logit value corresponding to the true label dimension, that is If the black box artificial intelligence model is a language generation model, the model output can be expressed as the logit value of the dimension corresponding to the next token generated with the maximum probability predicted by the model, that is, Where ymax represents the dimension corresponding to the next token generated by the maximum probability predicted by the model; the model output v(x S ) can also be expressed as the sum of the logits values ​​of the input dimensions corresponding to each token with the highest probability (or the target token determined in advance) in generating the subsequent T tokens. Here, the subsequent T target tokens generated are represented as y1, y2, ..., y T , then v(x S ) can be calculated as

[0020]

[0021] Similarly, v(x S ) can also be calculated as

[0022]

[0023] In a preferred embodiment, the step (3) further comprises the following steps: output v(x S ) is interpreted as the combination of the "interaction utility" on the combination of the input units. Preferably, "interaction utility" includes "and interaction utility" and "or interaction utility".

[0024] In a preferred example, the “and interaction utility” means “the additional utility generated by the combination for a black box model output when and only when none of the units in the combination of the input units is blocked”;

[0025] In a preferred embodiment, for the black box artificial intelligence model, the combination of input units Definition I and (S|v and , x) is the “and interaction utility” corresponding to the combination S of input units of the artificial intelligence model, which can be calculated by the following formula:

[0026]

[0027] Herev and (x L ) means from v(x L ) is the output component determined by the “and interaction utility”, corresponding to the formula v(x L )=v and (x L )+v or (x L ), and v or (x L ) means from v(x L ) is the output component determined by the "or interaction utility" decomposed in .

[0028] In a preferred example, the “or interaction utility” means “the additional utility generated by the combination for a set of black box model outputs when at least one of the units in the combination of the input units is not blocked”;

[0029] In a preferred embodiment, for the black box artificial intelligence model, the combination of input units Definition I or (S|v or , x) is the “or interaction utility” corresponding to the combination S of input units of the artificial intelligence model, which can be calculated by the following formula.

[0030]

[0031] Herev or (xL ) means from v(x L ) is the output component determined by the “or interaction utility”, corresponding to the formula v(x L )=v and (x L )+v or (x L ), and v and (x L ) means from v(x L ) is the output component determined by the “and interaction utility” decomposed in .

[0032] The output of the neural network can be expressed as a sum of different interactions. In a preferred embodiment, the output v(x S ) is interpreted as a combination of “and interaction utility” and “or interaction utility”, that is, v(x S )=v and (x S )+v or (x S ). The output v(x S ) is interpreted as a combination of “and interaction utility”, that is, v(x S )=v and (x S ). It should be understood that, without causing ambiguity, the “interaction utility” hereinafter may represent all combinations of “and interaction utility” and “or interaction utility”.

[0033] In a preferred embodiment, the step (4) further includes the following steps: all input units are divided into "related input units", "irrelevant input units" and "mutually exclusive units". The interactive utility of a combination containing "related input units" but not "mutually exclusive units" is defined as a "reliable component of interactive utility", and the interactive utility of a combination containing "irrelevant input units" or "mutually exclusive units" is defined as an "unreliable component of interactive utility".

[0034] In a preferred example, according to whether each input unit is relevant to the output of the black box artificial intelligence model in human cognition, the set N of all input units is divided into a set R of "related input units", a set T of "irrelevant input units" and a set M of "mutually exclusive units" by using an algorithm or user-defined. Among them, "relevant input unit" means an input unit that is strongly correlated with the model output in human cognition, or an input unit with a direct causal relationship; "irrelevant input unit" means an input unit that is weakly correlated with the model output in human cognition, or an input unit without a direct causal relationship; "mutually exclusive unit" means an input unit that should not affect the model output in human cognitive logic, or an input unit that is cognitively exclusive with the model output. Some input samples may only contain one or more of "relevant input unit", "irrelevant input unit" and "mutually exclusive unit". Preferably, for text input, the input unit can be a word embedding vector of a token, or all word embedding vectors corresponding to a word, phrase, sentence or paragraph; for image input, the input unit can be a pixel or an image area.

[0035] For example, given a legal case involving multiple people, "On Sunday, Zhang San attacked Li Si's head during a dispute, causing Li Si to be slightly injured. Seeing this, his enemy Wang Wu had evil thoughts and stabbed Li Si's abdomen with a knife, causing Li Si's death.", and evaluate the fine decision logic of the legal big model for Zhang San's verdict of "intentional injury". In this legal case, the set of all input units N = {"Sunday", "dispute", "attack", "minor injury", "stabbing", "death"}. Among them, the set of "relevant input units" R = {"dispute", "attack", "minor injury"} is the input unit that has a direct causal relationship with the verdict of Zhang San "intentional injury" in human cognition. The set of "irrelevant input units" T = {"Sunday"} is the input unit that has no direct causal relationship with the verdict of Zhang San "intentional injury" in human cognition. The set of "mutually exclusive units" M = {"stabbing", "death"} is Wang Wu's behavior, not Zhang San's behavior. It is a unit that is logically mutually exclusive with the verdict of "intentional injury" against Zhang San in human cognition.

[0036] In a preferred example, in the task of evaluating the black box artificial intelligence model, the user can label the input units in the vocabulary as "relevant input units", "irrelevant input units" and "mutually exclusive units" from the perspective of human cognition; the user can also use an algorithm to automatically determine or label the collection of "relevant input units", "irrelevant input units" and "mutually exclusive units"; the user can use attribution algorithms including Shapley values ​​to measure the importance of all input units to the model output, and automatically divide all input units into "relevant input units", "irrelevant input units" and "mutually exclusive units" according to the importance scores; the user can also use complex artificial intelligence algorithms to measure the degree of correlation between all input units and model outputs at the semantic and logical level, and automatically divide all input units into "relevant input units", "irrelevant input units" and "mutually exclusive units" according to the correlation scores.

[0037] In a preferred embodiment, the "interaction utility" of the combination of all input units in step (4) is further divided into "reliable component of interaction utility" and "unreliable component of interaction utility".

[0038] The "reliable component of interaction utility" is the interaction utility that is consistent with human cognition, and can be defined as the interaction utility of a combination that includes "related input units" and does not include "mutually exclusive units"; the "unreliable component of interaction utility" is the interaction utility that is inconsistent with human cognition, and can be defined as the interaction utility of a combination that includes "irrelevant input units" or "mutually exclusive units".

[0039] In a preferred embodiment, the interaction utility can be divided into "AND interaction utility" and "OR interaction utility", which respectively represent the "AND relationship" between the input units modeled by the model and the "OR relationship" between the input units modeled by it. and (S|v and , x) can be further divided into “reliable components with interactive utility” and "unreliable components of interaction utility" It can be calculated by the following formula:

[0040]

[0041]

[0042] Among them, the set N of all input units is divided into a set R of "related input units", a set T of "unrelated input units" and a set M of "mutually exclusive units".

[0043] In a preferred embodiment, the interaction utility can be divided into "and interaction utility" and "or interaction utility", which respectively represent the "and relationship" between the input units modeled by the model and the "or relationship" between the input units modeled by the model. or (S|v or , x) can be further divided into "or reliable components of interaction utility" and "or unreliable components of interaction utility" Preferably, a partition of "or interaction utility" is similar to the partition of "and interaction utility", which can be calculated by the following formula:

[0044]

[0045]

[0046] Among them, the set N of all input units is divided into a set R of "related input units", a set T of "unrelated input units" and a set M of "mutually exclusive units".

[0047] Another way to divide the "or interaction utility" is to consider the ratio of the number of "related input units" |S∩R| in the combination S to the number of input units in the combination S |S|, which can be calculated by the following formula:

[0048]

[0049]

[0050] In a preferred example, the step (5) further includes the following steps: using the proposed evaluation index based on interaction utility, calculating the ratio of "reliable component of interaction utility" to "unreliable component of interaction utility", and evaluating the degree of alignment between the model's fine decision logic and human cognition. The user can quantify the degree of alignment between the model's fine decision logic and human cognition by the ratio of "reliable component of interaction utility" in interaction utility.

[0051] In a preferred embodiment, significant interactions are defined as follows: Given a preset threshold τ, significant interactions are defined as the set of interactions Ω greater than the threshold τ and and Ω or , can be calculated by the following formula:

[0052]

[0053]

[0054] Among them, the threshold τ can be set as the output of the black box artificial intelligence model Preferably, The threshold τ can also be set to the value of the absolute value of the kth interaction utility after the absolute values ​​of all interaction utilities are sorted from large to small. Preferably, in the input sample containing 10 input units, there are 2 10 "and interaction utilities"; or both 10 "and interaction utility" and 2 10 The threshold τ distinguishes a certain proportion of "significant and interaction" and "significant or interaction". The threshold τ can also be set according to user experience, for example, it can be set to the absolute value of the 100th largest interaction utility among all the absolute values ​​of interaction utilities.

[0055] In a preferred example, the sum of the intensities of all significant interaction utilities is:

[0056]

[0057] In a preferred embodiment, the sum of the intensities of all interaction utilities is:

[0058]

[0059] In a preferred embodiment, the order of interaction is defined as the number of input variables in the set S, that is, order(S) = |S|. For each order of interaction o, the sum of the reliable components of the interaction utility is:

[0060]

[0061] In a preferred embodiment, for each order of interaction o, the strength of the reliable component of the interaction utility is:

[0062]

[0063] In a preferred example, the total strength of the reliable component of the interaction utility is:

[0064]

[0065] In a preferred embodiment, for each order of interaction o, the sum of the unreliable components of the interaction utility is:

[0066]

[0067] In a preferred example, for each order of interaction o, the strength of the unreliable component of the interaction utility is:

[0068]

[0069] In a preferred example, the total strength of the unreliable component of the interaction utility is:

[0070]

[0071] In a preferred embodiment, the following indicators It represents the ratio of the strength of the "reliable component of interaction effect" to the strength of the significant interaction effect.

[0072]

[0073] Among them, the ratio of the strength of the "reliable component of interaction utility" to the strength of the interaction utility of the significant interaction The higher the value, the higher the degree of alignment between the model's refined decision logic and human cognition. It should be higher.

[0074] In a preferred embodiment, the following indicators It represents the ratio of the strength of the "reliable component of interaction utility" to the strength of the interaction utilities of all interactions.

[0075]

[0076] Among them, the strength of the "reliable component of interaction utility" is the proportion of the strength of the interaction utility of all interactions. The higher the value, the higher the degree of alignment between the model's refined decision logic and human cognition. It should be higher.

[0077] In a preferred embodiment, the following indicators and They respectively represent the proportion of the "reliable component of interaction utility" in the sum of the positive utilities and the sum of the negative utilities of significant interactions of each order.

[0078]

[0079]

[0080] Where ∈ represents a small positive real number to prevent the fraction from being divided by 0. For each order interaction o, the sum of the positive interaction utilities of the significant interactions is calculated. The sum of the negative interaction utilities of the significant interactions Calculate the positive interaction utility of the "reliable component of interaction utility" and the negative interaction utility of the “reliable component of interaction utility” index and The higher the value, that is, for each order of interaction o, the higher the proportion of the "reliable component of interaction utility" in the positive and negative utilities of significant interactions, indicating that the model's refined decision logic is more aligned with human cognition. A model with a high degree of alignment with human cognition has a proportion of the sum of the positive and negative utilities of each order of significant interactions. or indicator It should be higher.

[0081] In a preferred embodiment, the following indicators It represents the ratio of positive and negative offsets in each order of significant interaction effects.

[0082]

[0083] Among them, the indicator The higher it is, the higher the proportion of effective interactions remaining after the significant interaction utilities are offset. This indicator helps to understand whether low-order interactions or high-order interactions play a major role in the model output. A well-trained model tends to model low-order interactions.

[0084] In a preferred embodiment, the following index O reliable Represents the weighted average order of "reliable interactions" among the significant interactions.

[0085] or

[0086]

[0087] Among them, the weighted average order of "reliable interaction" is O reliable The lower the value, the more susceptible the model's fine-grained decision logic is to low-order interactions. On the contrary, the lower the value, the more susceptible the model's fine-grained decision logic is to low-order interactions. reliable The higher the value, the more susceptible the model’s refined decision logic is to high-order interactions. A model that is highly aligned with human cognition has a weighted average order of “reliable interactions” in significant interactions. reliable ∈[0,|N|] should be lower, indicating that the model tends to model lower-order "reliable interactions".

[0088] In a preferred example, the statistical diagrams of the significant interaction utilities of each order, the reliable components of the interaction utilities, and the unreliable components of the interaction utilities can be visualized to show the difference between the model decision logic and human cognition and reflect the potential representation defects of the model. Specifically, for each order of interaction o, the sum of the positive interaction utilities of the significant interactions is shown. + (o), the sum of the negative interaction utilities of significant interactions - (o), the sum of the positive utilities of the reliable interaction components + (0), the sum of the negative utilities of the reliable interaction components - (o), the sum of the positive utilities of the unreliable interaction components + (o):=Salient + (o)-Reliable + (o), the sum of negative utilities of unreliable interaction components - (o):=Salient - (o)-Reliable - (o).

[0089] For each order interaction o, when the indicator and The higher it is, that is, the higher the proportion of reliable components of interaction utility in significant interactions, the higher the degree of alignment between the model's sophisticated decision-making logic and human cognition.

[0090] In summary, we can use statistical graphs of significant interaction utilities of each order, reliable components of interaction utilities, unreliable components of interaction utilities, and indicators and It depicts the differences between the model’s decision-making logic and human cognition, and reflects the model’s potential representational defects.

[0091] A second aspect of the present invention discloses a system for evaluating the degree of alignment between the refined decision logic of an artificial intelligence black box model and human cognition, the system comprising:

[0092] (1) An input module, wherein the input module is configured to receive a pre-trained black box artificial intelligence model and a set of data to be analyzed.

[0093] (2) A calculation module, wherein the calculation module is configured to calculate the "interaction utility" between the input units of the data to be analyzed modeled by the model based on the model and data in the input module. The "interaction utility" is divided into a "reliable component of interaction utility" and an "unreliable component of interaction utility". The ratio of the "reliable component of interaction utility" to the "unreliable component of interaction utility" is calculated using the proposed interaction-based evaluation index.

[0094] (3) An output module, wherein the output module is configured to evaluate the evaluation index based on interaction utility, and statistics of significant interaction utility of each order, reliable component of interaction utility, and unreliable component of interaction utility, so as to evaluate the degree of alignment between the fine decision-making logic of the evaluation model and human cognition.

[0095] The advantages of the present invention are:

[0096] 1) The present invention reasonably classifies the input components and introduces the definition and calculation of reliable components, so as to reliably evaluate the degree of alignment between the black box model's fine decision logic and human cognition, thereby reflecting the model's potential representation defects.

[0097] 2) The present invention introduces multiple standards to evaluate the degree of alignment between the black box model's refined decision logic and human cognition, ensuring that the method used in the present invention is applicable to most black box models.

[0098] A large number of technical features are recorded in the specification of this application, which are distributed in various technical solutions. If all possible combinations of technical features of this application (i.e., technical solutions) are to be listed, the specification will be too long. In order to avoid this problem, the various technical features disclosed in the above-mentioned invention content of this application, the various technical features disclosed in the various embodiments and examples below, and the various technical features disclosed in the accompanying drawings can be freely combined with each other to form various new technical solutions (these technical solutions are all deemed to have been recorded in this specification), unless the combination of such technical features is technically infeasible. For example, in one example, feature A+B+C is disclosed, and in another example, feature A+B+D+E is disclosed, and features C and D are equivalent technical means that play the same role. Technically, only one can be used, and it is impossible to use them at the same time. Feature E can be combined with feature C technically. Then, the solution of A+B+C+D should not be deemed to have been recorded because it is technically infeasible, and the solution of A+B+C+E should be deemed to have been recorded. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] Figure 1 It is a flow chart of evaluating the degree of alignment between the refined decision logic of an artificial intelligence black box model and human cognition according to the first embodiment of the present invention;

[0100] Figure 2 a is a schematic diagram of the sample to be analyzed and the input variables according to the present invention;

[0101] Figure 2 b is a schematic diagram of the present invention dividing all input units of the sample to be analyzed into "related input units", "irrelevant input units" and "mutually exclusive units";

[0102] Figure 3 is a schematic diagram showing the absolute value utilities of "AND interaction utility" and "OR interaction utility" obtained according to the present invention arranged in descending order;

[0103] FIG4( a ) is an evaluation index based on interactive utility obtained according to the present invention;

[0104] FIG4( b ) is a statistical diagram of significant interaction utilities of each order, reliable components of interaction utilities, and unreliable components of interaction utilities; DETAILED DESCRIPTION

[0105] After careful and in-depth research, the inventors have developed for the first time a method and system for evaluating the degree of alignment between the refined decision-making logic of an artificial intelligence black box model and human cognition.

[0106] It should be noted that in the invention document of this patent, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one" do not exclude the existence of other identical elements in the process, method, article or device including the element. In the invention document of this patent, if it is mentioned that an action is performed according to an element, it means that the action is performed at least according to the element, which includes two situations: performing the action only according to the element, and performing the action according to the element and other elements. Expressions such as multiple, multiple, and multiple include 2, 2 times, 2 kinds, and more than 2, more than 2 times, and more than 2 kinds.

[0107] The present invention comprises the following steps:

[0108] (1) Selecting a black box AI model

[0109] A black box artificial intelligence model to be analyzed is selected, wherein the black box artificial intelligence model includes an artificial intelligence model pre-trained based on a certain data set.

[0110] (2) Select input samples and perform recognition

[0111] Select an input sample for interactive utility calculation, and identify the input sample, so as to decompose the input sample into n input units, and combine the n input units to obtain 2 n The input sample is selected from the following group: table data, picture, text, voice or a combination thereof;

[0112] (3) Calculate and combine “interaction utilities”;

[0113] In step (2), n The combination of input units is respectively input into the black box artificial intelligence model, and the output of the black box artificial intelligence model is obtained; based on the output of the black box artificial intelligence model, the interaction between the input units is modeled, so as to obtain the "interaction utility" of the black box artificial intelligence model for each combination of the input units; and the output of the black box artificial intelligence model on a certain input unit combination is interpreted as the combination of its "interaction utility" on the input unit combination;

[0114] (4) Divide the “interaction utility” into “reliable component of interaction utility” and “unreliable component of interaction utility”;

[0115] According to whether each input unit is relevant to the output of the black box artificial intelligence model in human cognition, all input units are divided into "relevant input units", "irrelevant input units" and "mutually exclusive units" by algorithm or user customization.

[0116] The "interaction utility" of the combination of all input units in step (3) is divided into "reliable component of interaction utility" and "unreliable component of interaction utility". Among them, the "reliable component of interaction utility" is the interaction utility consistent with human cognition, which can be expressed as the interaction utility of the combination containing "related input units" and not containing "mutually exclusive units"; the "unreliable component of interaction utility" is the interaction utility inconsistent with human cognition, which can be expressed as the interaction utility of the combination containing "irrelevant input units" and "mutually exclusive units". Preferably, the interaction utility can be divided into "and interaction utility" and "or interaction utility", which respectively represent the "and relationship" between the input units modeled by the model and the "or relationship" between the input units modeled by it. Preferably, the "reliable component of interaction utility" can be calculated as the interaction utility of the combination containing "related input units" and not containing "mutually exclusive units"; the "unreliable component of interaction utility" can be calculated as the interaction utility of the combination not containing "related input units" or containing "mutually exclusive units". Preferably, a "reliable component of the or interaction utility" can be calculated as the interaction utility of a combination that includes "related input units" and does not include "mutually exclusive units"; and an "unreliable component of the or interaction utility" can be calculated as the interaction utility of a combination that does not include "related input units" or includes "mutually exclusive units". Another algorithm is to evenly distribute the "or interaction utility" to each input unit included in the interaction, so that the "reliable component of the or interaction utility" can be calculated as the interaction utility component evenly distributed to the "related input units" in this "or interaction utility"; and the "unreliable component of the or interaction utility" can be calculated as the interaction utility component evenly distributed to the "irrelevant input units" and the "mutually exclusive units" in this "or interaction utility".

[0117] (5) Use interaction-based evaluation indicators to evaluate the degree of alignment between the model’s decision logic and human cognition;

[0118] According to the proposed interaction-based evaluation indicators, the ratio of “reliable component of interaction utility” to “unreliable component of interaction utility” is counted to evaluate the degree of alignment between the model’s refined decision-making logic and human cognition.

[0119] Example

[0120] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0121] The first embodiment of the present invention relates to a method for evaluating the degree of alignment between the fine decision logic of an artificial intelligence black box model and human cognition, the process of which is as follows: Figure 1 As shown, the method comprises the following steps:

[0122] In step 101: Based on a certain data set, a black box artificial intelligence model is trained as a model to be analyzed. Optionally, a model can be a deep neural network. In one embodiment, the model is selected as the Magnolia Law Model fine-tuned with Chinese legal documents, which is a deep neural network.

[0123] Then, proceed to step 102, which can be further divided into the following two sub-steps:

[0124] (a) Collecting data to be analyzed. Optionally, if the model in step (1) can be used for natural language generation, the data to be analyzed can be a text.

[0125] (b) Preprocessing the data to be analyzed. Optionally, the data to be analyzed may be processed by word segmentation, encoding into embedding vectors, etc., so as to make it suitable for the above model input.

[0126] In one embodiment, Figure 2 As shown in (a), the data is selected as any sentence of Chinese legal case description, including 10 input variables, representing the 1st to 10th phrases, respectively, and recorded as phrase 1, phrase 2, ..., phrase 10. In one embodiment, a sentence of Chinese legal case description is input into the Magnolia Law Model, and the corresponding crime output of the case can be obtained.

[0127] Then, proceed to step 103, which can be further divided into the following five sub-steps:

[0128] (a) Calculate the output value of the black box model under any occlusion of the above-mentioned data to be analyzed;

[0129] (b) Calculate the “and interaction utility” in the data to be analyzed modeled by the black box model;

[0130] (c) Calculate the “or interaction utility” in the data to be analyzed modeled by the black box model;

[0131] (d) Based on the “AND interaction utility” and “OR interaction utility”, the black box model output is interpreted as a combination of the “AND interaction utility” and “OR interaction utility” between the input unit combinations;

[0132] (e) Based on the combination of “and interaction utility” and “or interaction utility”, the combination of “and interaction utility” and “or interaction utility” of the model is further optimized to make the interaction more sparse and concise.

[0133] In the above sub-step (a), the black box artificial intelligence model is represented as v, and the data sample x to be analyzed contains n input units, which are represented as a set N = {1, 2, ..., n}, where the number of input units is generally 5 to 20. Optionally, each token in the input data can be regarded as an input unit, or the input data can be divided into multiple phrases or short sentences, and the word embedding vectors of all tokens covered by each phrase or short sentence can be regarded as an input unit. For extremely long input samples, all tokens covered by multiple sentences or multiple paragraphs and chapters can be counted as the same input unit, and different input units correspond to different paragraphs and chapters in the input sample.

[0134] Any subset of the input cells is called a combination of input units, v(x S ) represents the output of the black box model when a combination S of input units is given and the input units in N\S are blocked. In particular, if the input sample is input to the model without occlusion, the output obtained is expressed as v(x N ), if all input units in the input sample are completely blocked, the output is expressed as

[0135] Calculate the output v(x) of the black box model on a certain input unit combination S S ), it is necessary to keep the input value of the sample to be analyzed on the input unit contained in S, and replace the input units in all S's complement N\S with the reference value. In particular, for natural language processing tasks, assuming that the reference value is b, and assuming that each input unit contains only one token, then the masked sample to be analyzed x S The individual input units of are defined by:

[0136]

[0137] Here, (x S ) i =x i Indicates that the word embedding vector corresponding to the ith input unit (i.e., the ith token) of the occluded sample remains unchanged, and is still the word embedding vector of the ith input unit (i.e., the ith token) in the original input sample; (x S ) i =b i The word embedding vector corresponding to the i-th input unit (i.e., the i-th token) of the occluded sample is replaced with the word embedding vector baseline value.

[0138] Thus, v(x S ) can be calculated as SThe output value obtained by inputting the model. Optionally, the benchmark value can be set to the mean of the sample set, a random value, a zero value, etc., or can be obtained through learning. For natural language processing tasks, the benchmark value can be set to a special word embedding vector, and the masking of a single token can be achieved by replacing the word embedding vector of this token with a special word embedding vector.

[0139] If the ith input unit includes multiple tokens (for example, the ith input unit is a word, phrase, short sentence, etc.), then the occlusion of the ith input unit needs to simultaneously replace the word embedding vectors corresponding to all the tokens it contains. Furthermore, the black box model output v(x S ) is specifically denoted as v(x S |b), it should be understood that v(x S |b) is abbreviated as v(x S ).

[0140] We can use "and interaction utility" and "or interaction utility" to explain the output of the model. Given a black box model output v and an input sample x, in calculating the output v(x S ), the model outputs v(x S ) is decomposed into two terms, namely v(x S )=v and (x S )+v or (x S ). In this way, we can use the interaction with v and (x S ), and use or interactively interpret v or (x S ).

[0141] In the above sub-step (b), the “and interaction utility” is calculated to evaluate the output v corresponding to the and interaction between any input unit combination in the sample to be analyzed for each model. and The contribution value of (x). "Interaction utility" means the additional utility generated by the output corresponding to the interaction of a black box model when the units in the combination are triggered at the same time (i.e., not blocked). It should be understood that in a black box model, the input units often do not contribute to the output of the model individually, but interact with each other to affect the output of the model.

[0142] Therefore, for Definition I and (S|v and , x) is the “and interaction utility” corresponding to the combination S of input units of an artificial intelligence black box model, so that when all input variables in S are not blocked, Iand (S|v and , x) is triggered and accumulated to the output v corresponding to the interaction of the black box model and (x) in. Specifically, I and (S|v and , x) can be calculated by the following formula.

[0143]

[0144] The above “and interaction utility” satisfies that for any given input unit combination of the sample to be analyzed The output v corresponding to the interaction of an artificial intelligence model under this input unit combination and (x T ) can be split into all triggered and interactive I and (S|v and , x), as shown in the following formula.

[0145]

[0146]

[0147] Among them, e T Indicates whether the input variable is blocked. When i∈T, (e T ) i =1, indicating that the input variable i is not blocked (or called triggered), otherwise, (e T ) i =0 means that the input variable i is blocked; ∧ i∈S (e T ) i Indicates whether all the units in the input unit combination S are not blocked. The result of the "and" operation on the elements after the ∧ symbol is ∧ i∈S (e T ) i =Π i∈s (e T ) i .

[0148] Furthermore, the above calculation formula for interaction can be equivalent to the following form:

[0149] Assumptions and Exhaustively enumerate the interactions of all input unit combinations, then there exists Make I and =Av and .

[0150] In the above sub-step (c), the OR interaction utility is calculated to evaluate the OR relationship between any input unit combination of the sample to be analyzed for the OR interaction output v of the black box model. or The contribution value of (x). "Or interactive utility" means the additional utility generated by the output of the black box model or interaction when at least one of the units in the combination is triggered (i.e. not blocked). It should be understood that in the black box model, the input units often do not contribute to the output of the model individually, but interact with each other to affect the output of the model.

[0151] Therefore, for Definition I or (S|v or , x) is the “or interaction utility” corresponding to the combination S between input units, so that when at least one of all input variables in S is not blocked, I or (S|v or , x) is triggered and added to the output v of the black box model or interaction or (x) in. Specifically, I or (S|v or , x) can be calculated by the following formula.

[0152]

[0153] The above “or interaction utility” satisfies that for any given input unit combination of the sample to be analyzed The black box model outputs v under this input unit combination or interaction or (x T ) can be split into all triggered or interactive I or (S|v or , x), as shown in the following formula.

[0154]

[0155] Among them, e T Indicates whether the input variable is not blocked. When i∈T, (e T ) i =1, which means that the input variable i is not blocked, otherwise, (e T ) i =0, which means that the input variable i is not blocked. i∈S (e T ) i Indicates whether at least one of the cells in the input cell combination S is not blocked. The result of the “or” operation on the elements after the V symbol is V. i∈S (e T ) i =1-Π i∈s [1-(eT ) i ].

[0156] Furthermore, the above or interactive calculation formula can be equivalent to the following form, assuming

[0157] Exhaustively enumerate the interactions on all input unit combinations, then there exists Make I or =Bv or .

[0158] In one embodiment of the present invention, based on the Magnolia Law Model in step 101, the calculation formulas of "and interaction utility" and "or interaction utility" are used to calculate the values ​​of "and interaction utility" and "or interaction utility" of each model. Figure 3 As shown, the absolute values ​​of the "AND interaction utility" and "OR interaction utility" of different input unit combinations obtained by the present invention and the traditional method are arranged in descending order to show the distribution schematic diagram.

[0159] In the above sub-step (d), based on the "AND interaction utility" and "OR interaction utility", the black box model output is interpreted as a combination of the "AND interaction utility" and "OR interaction utility" between the input unit combinations.

[0160] In sub-step (d), based on the “AND interaction utility” and “OR interaction utility” in step (3), the output v(x T ) is interpreted as a combination of “and interaction utility” and “or interaction utility”.

[0161]

[0162] In sub-step (e), based on the combination of "and interaction utility" and "or interaction utility" of each model obtained in sub-step (d), in order to further obtain sparse interactions, the present invention hopes to minimize the sum of the L1-norms of all interactions, that is,

[0163]

[0164] in, ||·||1 represents the L1-norm of the vector.

[0165] Optionally, when optimizing the above expression, you can define Make The above optimization problem can be converted into the following form.

[0166]

[0167] st|q S |<τ S

[0168] Among them, τ S is a preset threshold. In this embodiment,

[0169] Then, proceed to step 104, which can be further divided into the following sub-steps:

[0170] (a) Divide all input units into “related input units”, “irrelevant input units” and “mutually exclusive units”;

[0171] (b) Dividing the “interaction utility” of the combination of all input units in step 103 into a “reliable component of interaction utility” and an “unreliable component of interaction utility”.

[0172] In the above sub-step (a), all input units are divided into "related input units", "irrelevant input units" and "mutually exclusive units" by algorithm or user definition according to whether each input unit is relevant to the output of the black box artificial intelligence model in human cognition.

[0173] Depending on whether each input unit is relevant to the output of the black box artificial intelligence model in human cognition, the user can customize / empirically divide the set N of all input units into a set R of "relevant input units", a set T of "irrelevant input units", and a set M of "mutually exclusive units" according to specific tasks. Among them, "relevant input units" refer to input units that are strongly relevant to the model output in human cognition, or input units that have a direct causal relationship with the model output, "irrelevant input units" refer to input units that are weakly relevant to the model output in human cognition, or input units that have no direct causal relationship with the model output, and "mutually exclusive units" refer to input units that should not affect the model output in human cognitive logic. In the Magnolia Law Model, "relevant input units" can be set as input units that are strongly related to criminal behavior and sentencing, or input units with direct causal relationships, such as specific criminal behavior; "irrelevant input units" can be set as input units that are weakly related to criminal behavior and sentencing, or units without direct causal relationships, such as crime time, crime location, person's name, etc.; "mutually exclusive units" can be set as case descriptions of other people in multi-person cases. For example, if Zhang San's behavior in the case description is unrelated to Wang Wu's behavior, Zhang San's behavior cannot be used as the basis for Wang Wu's crime judgment. Figure 2 (b) shows a schematic diagram of dividing all input units of the sample to be analyzed into "related input units", "irrelevant input units" and "mutually exclusive units".

[0174] In the above sub-step (b), the "interaction utility" of the combination of all input units in step 103 is divided into "reliable component of interaction utility" and "unreliable component of interaction utility". The "reliable component of interaction utility" is the interaction utility that is consistent with human cognition, and can be expressed as the interaction utility of the combination that includes "related input units" and does not include "mutually exclusive units"; the "unreliable component of interaction utility" is the interaction utility that is inconsistent with or conflicts with human cognition, and can be expressed as the interaction utility of the combination that includes "irrelevant input units" and "mutually exclusive units".

[0175] In a preferred embodiment, the "interaction utility" I and (S|v and , x) can be further divided into “reliable components with interactive utility” and "unreliable components of interaction utility" It can be calculated by the following formula:

[0176]

[0177]

[0178] Among them, the set N of all input units is divided into a set R of "related input units", a set T of "unrelated input units" and a set M of "mutually exclusive units".

[0179] In a preferred embodiment, the "or interactive utility" I or (S|v or , x) can be further divided into "or reliable components of interaction utility" and "or unreliable components of interaction utility" Preferably, a partition of "or interaction utility" is similar to the partition of "and interaction utility", which can be calculated by the following formula:

[0180]

[0181]

[0182] Among them, the set N of all input units is divided into the set R of "related input units", the set T of "unrelated input units" and the set M of "mutually exclusive units". Another "or interactive utility" division considers the ratio of the number of "related input units" |S ∩R| in the combination S to the number of input units in the combination S |S|, which can be calculated by the following formula:

[0183]

[0184]

[0185] Then, proceed to step 105, which can be further divided into the following sub-steps:

[0186] (a) Calculate the interaction-based evaluation index and count the ratio of “reliable component of interaction utility” to “unreliable component of interaction utility”;

[0187] (b) The degree of alignment between the model’s refined decision-making logic and human cognition is evaluated through statistics of significant interaction utilities of each order, reliable components of interaction utilities, and unreliable components of interaction utilities.

[0188] In the above sub-step (a), the following evaluation indicators based on interaction utility are calculated, and the ratio of "reliable component of interaction utility" to "unreliable component of interaction utility" is counted. As shown in Figure 4(a), users can use the ratio of "reliable component of interaction utility" in interaction utility to quantify the degree of alignment between the model's refined decision logic and human cognition.

[0189] The significant interaction is defined as follows. Given a preset threshold τ, a significant interaction is defined as the set of interactions Ω greater than the threshold τ and and Ω or , can be calculated by the following formula:

[0190]

[0191]

[0192] The threshold τ is set to the absolute value of the 100th interaction utility after the absolute values ​​of all interaction utilities are sorted from large to small.

[0193] The sum of the intensities of all significant interaction utilities is:

[0194]

[0195] The sum of the intensities of all interaction utilities is:

[0196]

[0197] The order of interaction is defined as the number of input variables in the set S, that is, order(S) = |S|. For each order of interaction o, the sum of the reliable components of the interaction utility is:

[0198]

[0199] For each order interaction o, the strength of the reliable component of the interaction utility is:

[0200]

[0201] The total strength of the reliable component of the interaction utility is:

[0202]

[0203] For each order interaction o, the sum of the unreliable components of the interaction utility is:

[0204]

[0205] For each order interaction o, the strength of the unreliable component of the interaction utility is:

[0206]

[0207] The total strength of the unreliable component of the interaction utility is:

[0208]

[0209] The following indicators It represents the ratio of the strength of the "reliable component of interaction effect" to the strength of the significant interaction effect.

[0210]

[0211] Among them, the ratio of the strength of the "reliable component of interaction utility" to the strength of the interaction utility of the significant interaction The higher it is, the more closely the model’s refined decision-making logic is aligned with human cognition.

[0212] The following indicators It represents the ratio of the strength of the "reliable component of interaction utility" to the strength of the interaction utilities of all interactions.

[0213]

[0214] Among them, the strength of the "reliable component of interaction utility" is the proportion of the strength of the interaction utility of all interactions. The higher it is, the more closely the model’s refined decision-making logic is aligned with human cognition.

[0215] The following indicators and They respectively represent the proportion of the "reliable component of interaction utility" in the sum of the positive utilities and the sum of the negative utilities of significant interactions of each order.

[0216] or

[0217] or

[0218] Where ∈ represents a small positive real number to prevent the fraction from being divided by 0. For each order interaction o, the sum of the positive interaction utilities of the significant interactions is calculated. The sum of the negative interaction utilities of the significant interactions Calculate the positive interaction utility of the "reliable component of interaction utility" and the negative interaction utility of the “reliable component of interaction utility” index and The higher it is, that is, for each order interaction o, the higher the proportion of the “reliable component of interaction utility” in the positive and negative utilities of significant interactions, indicating a higher degree of alignment between the model’s refined decision-making logic and human cognition.

[0219] The following indicators It represents the ratio of positive and negative offsets in each order of significant interaction effects.

[0220]

[0221] Among them, the indicator The higher it is, the higher the proportion of effective interactions remaining after the significant interaction utilities are offset. This indicator helps to understand whether low-order interactions or high-order interactions play a major role in the model output. A well-trained model tends to model low-order interactions.

[0222] The following indicators O reliable Represents the weighted average order of "reliable interactions" among the significant interactions.

[0223] or

[0224]

[0225] Among them, the weighted average order of "reliable interaction" is O reliable The lower the value, the more susceptible the model's fine-grained decision logic is to low-order interactions. On the contrary, the lower the value, the more susceptible the model's fine-grained decision logic is to low-order interactions. reliable The higher it is, the more susceptible the influence of the model's refined decision logic is to high-order interactions.

[0226] Therefore, as shown in Figure 4(a), an evaluation indicator based on interaction utility can be used to count the ratio of “reliable component of interaction utility” to “unreliable component of interaction utility”, characterize the difference between model decision logic and human cognition, and reflect the potential representation defects of the model.

[0227] In the above sub-step (b), as shown in FIG4(b), statistical graphs of significant interaction utilities of each order, reliable components of interaction utilities, and unreliable components of interaction utilities are respectively displayed, indicating the difference between the decision logic of the model and human cognition, and reflecting the potential representation defects of the model.

[0228] For each order interaction o, the sum of the positive interaction utilities of the significant interactions is shown separately. +(o), the sum of the negative interaction utilities of significant interactions - (o), the sum of the positive utilities of the reliable interaction components + (o), the sum of negative utilities of reliable interaction components - (o), the sum of the positive utilities of the unreliable interaction components + (o):=Salient + (o)-Reliable + (o), the sum of negative utilities of unreliable interaction components - (o):=Salient - (o)-Reliable - (o).

[0229] For each order interaction o, when the indicator and The higher it is, that is, the higher the proportion of reliable components of interaction utility in significant interactions, the higher the degree of alignment between the model's sophisticated decision-making logic and human cognition.

[0230] Therefore, as shown in Figure 4(b), statistical graphs of significant interaction utilities of each order, reliable components of interaction utilities, and unreliable components of interaction utilities can be used to characterize the differences between the model decision logic and human cognition and reflect the potential representation defects of the model.

[0231] Note that when calculating the indicators mentioned in all the above preferred examples, you can choose to add a small positive real number ∈ to the denominator to prevent the occurrence of "division by 0".

[0232] It should be noted that those skilled in the art should understand that the implementation functions of the modules shown in the above-mentioned implementation of the system for evaluating the degree of alignment between the fine decision logic of the artificial intelligence black box model and human cognition can be understood by referring to the above-mentioned description of the method for evaluating the degree of alignment between the fine decision logic of the artificial intelligence black box model and human cognition. The functions of the modules shown in the above-mentioned implementation of the system for evaluating the degree of alignment between the fine decision logic of the artificial intelligence black box model and human cognition can be implemented by a program (executable instruction) running on a processor, or by a specific logic circuit. If the system for evaluating the degree of alignment between the fine decision logic of the artificial intelligence black box model and human cognition in the embodiment of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the method of each embodiment of the present invention. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, etc., which can store program codes. Thus, the embodiments of the present invention are not limited to any specific combination of hardware and software.

[0233] All documents mentioned in the present invention are considered to be included in the disclosure of the present invention as a whole, so that they can be used as the basis for modification when necessary. In addition, it should be understood that the above are only preferred embodiments of this specification and are not used to limit the scope of protection of this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this specification should be included in the scope of protection of one or more embodiments of this specification.

Claims

1. A method for evaluating the degree of alignment between the fine decision logic of an artificial intelligence black box model and human cognition, characterized in that: The method comprises the following steps: (1) Provide a black box artificial intelligence model; Providing a black box artificial intelligence model to be analyzed, wherein the black box artificial intelligence model includes an artificial intelligence model pre-trained based on a certain data set; (2) Select input samples and perform recognition; Select an input sample for interactive utility calculation, and identify the input sample, so as to decompose the input sample into n input units, and combine the n input units to obtain 2 n The input sample is selected from the following group: table data, picture, text, voice or a combination thereof; (3) Calculate and combine "interaction utility"; In step (2), n The combination of the input units is respectively input into the black box artificial intelligence model, and the output of the black box artificial intelligence model is obtained; based on the output of the black box artificial intelligence model, the interaction utility between the input units is modeled, so as to obtain the "interaction utility" of the black box artificial intelligence model for each combination of the input units; and the output of the black box artificial intelligence model on a certain input unit combination is interpreted as the combination of its "interaction utility" on the input unit combination; (4) Roughly divide the "interaction utility"; According to whether each input unit is relevant to the output of the black box artificial intelligence model in human cognition, all input units are divided into "relevant input units" and "non-relevant input units" by using an algorithm or user definition; (5) Subdividing the relevant input units and performing further calculations Performing a secondary division on the relevant input units, further dividing the non-relevant input units into "irrelevant input units" and "mutually exclusive units", and dividing the "interaction utility" of the combination of all input units in step (3) into "reliable components of interaction utility" and "unreliable components of interaction utility" based on the "relevant input units", "irrelevant input units" and "mutually exclusive units"; (6) Use interaction-based evaluation indicators to evaluate the degree of alignment between the model’s decision logic and human cognition; According to the preset interaction-based evaluation indicators, the ratio of "reliable component of interaction utility" to "unreliable component of interaction utility" is calculated to evaluate the degree of alignment between the model's refined decision-making logic and human cognition; The order of step (1) and step (2) can be replaced at will or performed simultaneously.

2. The method according to claim 1, characterized in that The interaction utility can be divided into "AND interaction utility" and "OR interaction utility", which respectively represent the "AND relationship" between the input units modeled by the model and the "OR relationship" between the input units modeled by it.

3. The method according to claim 2, characterized in that The "reliable component of interaction utility" can be calculated as the interaction utility of a combination that includes "related input units" and does not include "mutually exclusive units"; the "unreliable component of interaction utility" can be calculated as the interaction utility of a combination that does not include "related input units" or includes "mutually exclusive units".

4. The method according to claim 3, characterized in that The "reliable component of interaction utility" and "unreliable components of interaction utility" The specific value can be calculated by the following formula: Among them, the set N of all input units is divided into a set R of "related input units", a set T of "unrelated input units" and a set M of "mutually exclusive units".

5. The method according to claim 2, characterized in that: The step (4) also The method comprises the following steps: evenly distributing the "or interaction utility" to each input unit included in the interaction, thereby obtaining the "or interaction utility reliable component", wherein the "or interaction utility reliable component" can be calculated as the interaction utility component evenly distributed to the "related input unit" in the "or interaction utility"; the "or interaction utility unreliable component" can be calculated as the interaction utility component evenly distributed to the "irrelevant input unit" and the "mutually exclusive unit" in the "or interaction utility", and is specifically obtained by the following formula: Among them, |S∩R| is the number of "related input units" in the combination S considered for the partition of "or interaction utility", and |S| is the number of input units in the combination S.

6. The method according to claim 3, characterized in that The reliable component of the or interaction utility can also be obtained by a division similar to that of the interaction utility. Specifically, it is calculated by the following formula: Among them, the set N of all input units is divided into a set R of "related input units", a set T of "unrelated input units" and a set M of "mutually exclusive units".

7. The method according to claim 2, characterized in that: The evaluation indicators include: calculating the strength of the reliable component and the unreliable component of the interaction utility or their proportion in the interaction utility and comparing them as indicators; visualizing statistical graphs of significant interaction utilities of each order, the reliable component of the interaction utility, and the unreliable component of the interaction utility and using them as indicators; calculating the proportion of the reliable component of the interaction utility in significant interactions as an indicator.

8. The method according to claim 7, characterized in that The significant interaction is defined as the set of interactions Ω greater than a threshold τ and and Ω 2r , can be calculated by the following formula: The threshold τ is set to the value of the absolute value of the Xth interaction utility after the absolute values ​​of all interaction utilities are sorted from large to small, where X is set according to user experience; Ω and is the set of significant interactions among the interactions, Ω or is the set of significant interactions in or interactions.

9. The method according to claim 8, characterized in that The “mutually exclusive unit” refers to an input unit that should not affect the model output in terms of human cognitive logic, or an input unit that is cognitively exclusive to the model output.

10. The method according to claim 1, characterized in that The proportion of the reliable component of the interaction utility in the significant interaction Calculate using the following formula:

Images