Methods and systems for characterization diagnosis and interpretation, model comparison, and training sample collection for black-box models

Abstract

The present invention relates to the field of machine learning technology, and discloses a method and system for characterization diagnosis, model comparison, and training sample collection for black box models. The method can automatically interpret the internal logic of the black box model and obtain an "AND-OR graph" model representing the internal logic of the black box model, thereby realizing characterization diagnosis, model comparison, and training sample collection for the black box model. The method includes: providing a certain black box model; inputting a sample into the black box model, wherein the sample contains input units of a certain dimension; based on the middle-level output features of the black box model, modeling the interaction between the input units of the sample, calculating the interaction strength between the combinations formed by the input units, and expressing the black box model as an "AND-ADD relationship" between the combinations of the input units; optimizing the benchmark value on each input unit in the interactive calculation so that the "AND-ADD relationship" obtained from the black box model is more concise; and establishing an "AND-OR graph" model expressing the internal logic of the black box model based on the above relationship.

Description

Technical Field

[0001] The present invention relates to the field of machine learning technology, and in particular to a method and system technology for characterization diagnosis, model comparison, and training sample collection for black box models. Background Art

[0002] Currently, many models with black-box properties have demonstrated strong performance in various fields, such as images and text. However, this black-box nature still restricts their widespread application in many sensitive areas. Numerous studies have focused on the interpretability of black-box models. However, these techniques lack objectivity and are unable to demonstrate the degree of interpretability. Furthermore, they are overly complex to explain. Specifically, existing technologies cannot objectively and concisely explain the internal logic of black-box models.

[0003] Therefore, it is an urgent problem to objectively and concisely explain the internal logic of the black box model and obtain a hierarchical diagram representing the internal logic of the black box model. Summary of the Invention

[0004] The purpose of the present invention is to provide a method and system for characterization diagnosis, model comparison, and training sample collection for black box models, which can automatically characterize the internal logic of the black box model and obtain an "AND-OR graph" representing the internal logic of the black box model.

[0005] The present invention discloses a method for interpreting a black box model, comprising the following steps:

[0006] (1) Select the model;

[0007] Selecting a black box model to be explained and analyzed, wherein the black box model includes a neural network model;

[0008] (2) Obtain the data required for modeling:

[0009] Select a sample and input it into the black box model to obtain a model output sample or a mid-level output feature sample based on the input sample, and use the input sample and the model output sample or the mid-level output feature sample as data for modeling, wherein the input sample includes input units of W dimensions, where W is a positive integer ≥5. In another preferred embodiment, W is 10-10000, more preferably 15-100;

[0010] (3) Establishing a complex “with-addition relationship”

[0011] Based on the middle-level output features or model output samples of the black-box model, the interaction and cooperation between the input units of the input sample are modeled to obtain the combinations between the input units, and the interaction strength of the combinations between the input units is calculated. Then, the black-box model is represented as an "and-add relationship" between the combinations between the input units according to the interaction strength;

[0012] (4) Establishing a streamlined “relationship with Canada”

[0013] Based on the “AND-ADD relationship” in step (3), by optimizing the benchmark values of each input unit in the interactive calculation, the combinations of input units with strong correlation in the “AND-ADD relationship” are further combined, and the combinations of unimportant input units are eliminated, so that the “AND-ADD relationship” is expressed more concisely and sparsely;

[0014] (5) Establishing an “AND-OR graph” model

[0015] Based on the simplified “AND-ADD relationship” described in step (4), an “AND-OR graph” model is established to express the internal logic of the black box model.

[0016] In another preferred embodiment, the step (c) further comprises the following steps:

[0017] The black box model is denoted as v, and the input includes n input units, which are represented as a set N = {1, 2, ..., n}. Specifically, the value of input unit n is generally greater than or equal to 5. The output obtained by the black box model on this input is denoted as v(N). For any subset of input units It is called the combination between input units. v(S) represents the output of the black box model when all the input units in a certain combination between input units S are given. In a black box model, the input units of the sample do not work alone, but interact with each other. Therefore, based on the Harsanyi dividend indicator, the interaction between input units is quantified, and the numerical contribution of the interaction between input units within the combination between input units S to the middle-level output features or model output is calculated, recorded as I(S), and the output of the black box model or the middle-level output feature v(N) is split into the sum of the interaction sizes of each combination between input units S, that is,

[0018]

[0019] in, s represents the number of input units in set S, and l represents the number of input units in set L.

[0020] In the splitting method, the interaction within each input unit combination S is triggered only when all input units in S are present. In other words, when any input unit in S is "absent", the I(S) term does not contribute to the black box mid-level output features or model output. Therefore, the input units in the input unit combination S form an "and" relationship. In addition, based on the method of splitting the black box model output: The output v(N) of the black-box model v is modeled as an “additive” relationship between the interactions I(S) between different input units. Therefore, the output v(N) of the black-box model on the entire input sample can be expressed as an “and-additive relationship” between the input units.

[0021] In another preferred embodiment, the step (d) further comprises the following steps:

[0022] Based on the aforementioned splitting of the black box model output v(N): The output v(N) corresponding to the input sample N can be modeled as 2 n The sum of the interactions I(S) between the input units. n In the interaction, the calculation method based on the above interaction size is: The interaction strength |I(S)| corresponding to some combinations S of input units is large, indicating that it has a significant impact on the black-box model's output v(N). Meanwhile, the interaction strength |I(S)| corresponding to other combinations S is small, indicating that these combinations S have only a minor impact or interference on the black-box model's output v(N). Therefore, to simplify the interpretation of the black-box model and make the "and-addition relationship" modeled by the entire black-box model more concise and sparse, the benchmark values for each input unit in the interaction calculation are optimized.

[0023] The reference value refers to the requirement to retain the original sample input values of the input units in S when calculating the output v(S) of the black box model on a certain combination S of input units in interactive calculations, and replace all input units in the complement of S (N\S) with their reference values to indicate the "non-existent" state of these input units.

[0024] In the aforementioned method for interpreting the black-box model, different I(S) values are calculated under different benchmark values, resulting in different "and-additive relationship" expressions and interpretations. These interpretations are rigorous interpretations of the black-box model from different perspectives. The optimization method aims to find a sparse and concise "and-additive relationship" expression by optimizing the benchmark value. Therefore, the benchmark value optimization method optimizes and updates the benchmark value by penalizing the interaction strength |I(S)| between input unit combinations that have only a small impact, while retaining the interaction strength between input unit combinations that have a large impact on the output v(N) of the black-box model, in order to find a sparse and concise way to model the black-box model.

[0025] In another preferred embodiment, the step (d) further comprises the following sub-steps:

[0026] The vector consisting of the reference values of all input units is recorded as b, where each dimension b iRepresents the baseline value of input unit i. By penalizing the interaction strength |I(S)| between input unit combinations S that only produce minor effects or interference, the baseline value b of each input unit in the input sample N is optimized. i , so that the interaction strength |I(S)| of the input unit combination with smaller interaction strength |I(S)| approaches 0, and sparse feature interaction is obtained. The loss function of the optimization benchmark value is:

[0027]

[0028] Among them, all input unit combinations S are sorted by the size of the interaction strength |I(S)|. Assume {S1, S2, ..., S m} is the m input unit combination with the smallest interaction strength, and r is defined as the ratio of these input unit combinations to all input unit combinations, that is, Select the r2 with the smallest interaction strength n Input units are combined to form a set Because of the collection Only the interactions with the smallest r·2 n Input unit combination, so It represents a set of input unit combinations S that have only a small impact or interference on the mid-level feature output or model output of the black box model.

[0029] In another preferred embodiment, the method further comprises the following steps:

[0030] First, an initial benchmark value is generated by taking the sample mean, random value, zero value, or any other method. The methods for generating the benchmark value include but are not limited to these options. Second, the benchmark value optimization step is repeated T1 times until the newly calculated loss function value decreases and converges; wherein, for each repetition, the loss function value is recalculated according to the optimization function, and the benchmark value is updated based on the gradient of the loss function value, and the interaction strength value of each input unit combination is calculated based on the updated benchmark value; wherein, the optimization function is:

[0031]

[0032] It should be noted that methods for making the "AND-ADD relationship" representation obtained by the entire black-box model more concise and sparse include, but are not limited to, the method for optimizing the baseline value described above. For example, "adversarial training" can also be performed on the black-box model to make the "AND-ADD relationship" representation obtained by the entire black-box model more concise and sparse.

[0033] In another preferred embodiment, the step (e) further comprises the following steps:

[0034] According to the above method of splitting the black box model output v(N): The method for optimizing the benchmark value results in a modeling of the black-box model's output v(N) based on a set of sparse "AND-additive relationships." Based on this set of "AND-additive relationships," the black-box model is modeled using an "AND-OR graph": the "AND" relationship represents the relationship between input units within each input unit combination S, and the "OR" relationship indicates that the black-box model's output v(N) can be decomposed into the sum of the interactions I(S) across all input unit combinations S. Therefore, an "AND-OR graph" is used to model the black-box model's output: the "AND-OR graph" contains at least three layers, consisting of several nodes and edges. The n leaf nodes at the bottom layer represent n input units; the intermediate "AND" nodes represent the "AND" relationships between input units within a certain input unit combination S. Each "AND" node is assigned a score (score = I(S), representing the numerical contribution of the interaction between the input unit combination corresponding to that node to the output features of the middle layer or the model output. The root node at the top layer is an "OR" node, representing the sum of the scores of all its child nodes. In other words, the output v(N) of the black-box model is ultimately expressed as the sum of the interactions I(S) of all input unit combinations S. It should be noted that when the number of input units n is too large, the number of layers in the resulting "AND-OR" graph model may also increase due to the exponential increase in the number of input unit combinations S.

[0035] In another preferred embodiment, the step (e) may further include the following steps:

[0036] After obtaining a certain “AND-OR graph” of the black box model, the expression of the “AND-OR graph” is further simplified by the following method: Since the different input unit combinations S often contain the same part, such as the input unit combinations {i, j, k}, {i, j, l}, {i, j, m, n} contain the same input unit {i, j}, these common input units are merged and the merged result is recorded as the feature unit λ ij ={i, j}, then the original input unit combinations {i, j, k}, {i, j, l}, {i, j, m, n} can use the common feature unit λ ij Simplify and obtain new feature unit representations, namely {λ ij ,k},{λ ij , l}, {λ ij , m, n}. Feature unit λ ij It can serve as an "AND node" in the middle layer of the "AND-OR graph," replacing the representation of the coexistence of input units {i, j}, and connecting to other nodes containing the feature unit, thereby simplifying the "AND-OR graph" modeling of the black box model. It should be noted that the method of simplifying the "AND-OR graph" modeling of the black box model is only one way to simplify the "AND-OR graph" modeling. Methods for simplifying the "AND-OR graph" modeling include but are not limited to the method described above.

[0037] In another preferred embodiment, the method further comprises the following sub-steps:

[0038] The set of input units and feature units is denoted as X. The units in the set X are used to represent the combination S between input units. When constructing the "AND-OR graph" model, the frequency of occurrence of each unit λ∈X is defined by the following formula:

[0039]

[0040] in, represents the set of combinations between input units contained in the “AND-OR graph”. Based on the above probability, the representation length of each feature unit is Based on this, the total representation complexity of the “AND-OR graph” model can be calculated as:

[0041]

[0042] Here, p(S)∝|I(S)| represents the frequency of the input unit combination S in the AND-OR graph. The following iterative method is used to merge the common input units contained in certain input unit combinations S as new feature units, which are incorporated into the feature unit set X. Based on this set, all input unit combinations are re-represented and the AND-OR graph is modeled, thereby reducing the overall complexity of the AND-OR graph model:

[0043] (1) Initially, the feature unit is the input unit, then X (0) =N={1, 2, 3, ..., n};

[0044] (2) Based on the interaction strength |I(S)| between the input unit combinations, the frequency p of each feature unit is calculated when the feature unit X(t) at the current moment is used to represent the input unit combination. (t) (λ), and calculate the total complexity of the current "AND-OR graph" model (t) ;

[0045] (3) For each subset of all input units N Calculate when L is used as a new feature unit and incorporated into X (t) get Afterwards, based on The total complexity of the "AND-OR graph" model when representing the combination of input units

[0046] (4) According to step (3), take the total complexity The lowest set of characteristic units make

[0047] (5) Repeat steps (2)-(4) T2 times and update the optimized X (t) .

[0048] A second aspect of the present invention provides a system for characterization diagnosis, model comparison, and training sample collection for black box models, characterized by comprising:

[0049] The input module is configured as a black box model to be interpreted and a sample containing input units of a certain dimension;

[0050] An input unit combination interaction calculation module is configured to model the interaction between the input units of the sample based on the output of the black box model to be interpreted, calculate the interaction strength of the input unit combination formed by the input units, and express the black box model as an "and-add relationship" between the input unit combinations;

[0051] A benchmark value optimization module is configured to optimize the benchmark value of each input unit in the interactive calculation to make the "AND-ADD relationship" expression obtained from the black box model more concise and sparse;

[0052] The "AND-OR graph" generation module is configured to establish an "AND-OR graph" model that expresses the internal logic of the black box model based on a set of sparse "AND-ADD relationships" on the black box model to be interpreted.

[0053] In another preferred embodiment, the input unit combination interaction calculation module is further configured as follows: the black box model is represented as v, the input includes n input units, represented as a set N = {1, 2, ..., n}. The output obtained by the black box model on the input is represented as v(N). For any subset of input units It is called the combination between input units. v(S) represents the output of the black box model when all the input units in a certain combination between input units S are given. In a black box model, the input units of the sample do not work alone, but interact with each other. Therefore, the Harsanyi dividend indicator is used to quantify the interaction between input units, and the numerical contribution of the interaction between input units within the combination between input units S to the middle-level output features and model output is calculated, denoted as I(S), and the output v(N) of the black box model is split into the sum of the interaction sizes of each combination between input units S, that is,

[0054]

[0055] in, s represents the number of input units in set S, and l represents the number of input units in set L.

[0056] In the splitting method, the interaction within each input unit combination S is triggered only when all input units in S are present. In other words, when any input unit in S is "absent", the term I(S) does not contribute to the black box model output. Therefore, the input units in the input unit combination S form an "and" relationship. In addition, based on the method of splitting the black box model output: The output v(N) of the black-box model v is modeled as an “additive” relationship between the interactions I(S) between different input units. Therefore, the output v(N) of the black-box model on the entire input sample can be expressed as an “and-additive relationship” between the input units.

[0057] In another preferred embodiment, the input unit combined interaction calculation module is further configured to: based on the aforementioned splitting of the black box model output v(N): The output v(N) corresponding to the input sample N can be modeled as 2 n The sum of the interactions I(S) between the input units. n In the interaction, the calculation method based on the above interaction size is: The interaction strength |I(S)| corresponding to some combinations S of input units is large, indicating that it has a significant impact on the black-box model's output v(N). Meanwhile, the interaction strength |I(S)| corresponding to other combinations S is small, indicating that these combinations S have only a minor impact or interference on the black-box model's output v(N). Therefore, to simplify the interpretation of the black-box model and make the "and-addition relationship" modeled by the entire black-box model more concise and sparse, the benchmark values for each input unit in the interaction calculation are optimized.

[0058] The sparse interaction between input units means that the interaction intensity |I(S)| of most combinations between input units is small (approaching 0) and has no effect on the network output; only the interaction intensity |I(S)| of a small number of combinations between input units is large, which has a significant impact on the network output.

[0059] The reference value refers to the requirement to retain the original sample input values of the input units in S when calculating the output v(S) of the black box model on a certain combination S of input units in interactive calculations, and replace all input units in the complement of S (N\S) with their reference values to indicate the "non-existent" state of these input units.

[0060] In the aforementioned method for interpreting the black-box model, different I(S) values are calculated under different benchmark values, resulting in different "and-additive relationship" expressions and interpretations. These interpretations are rigorous interpretations of the black-box model from different perspectives. The optimization method aims to find a sparse and concise "and-additive relationship" expression by optimizing the benchmark value. Therefore, the benchmark value optimization method optimizes and updates the benchmark value by penalizing the interaction strength |I(S)| between input unit combinations that have only a small impact, while retaining the interaction strength between input unit combinations that have a large impact on the output v(N) of the black-box model, in order to find a sparse and concise way to model the black-box model.

[0061] In another preferred embodiment, the benchmark value optimization module is further configured to:

[0062] The vector consisting of the reference values of all input units is recorded as b, where each dimension b i Represents the baseline value of input unit i. By penalizing the interaction strength |I(S)| between input unit combinations S that only produce minor effects or interference, the baseline value b of each input unit in the input sample N is optimized. i , so that the interaction intensity |I(S)| of the input unit combination S with a small interaction intensity |I(S)| approaches 0, and a sparse feature interaction is obtained. The loss function of the optimization benchmark value is:

[0063]

[0064] Among them, all input unit combinations S are sorted by the size of the interaction strength |I(S)|. Assume {S1, S2, ..., S m} is the m input unit combination with the smallest interaction strength, and r is defined as the ratio of these input unit combinations to all input unit combinations, that is, Select the r2 with the smallest interaction strength n Input units are combined to form a set Because of the collection Only the interactions with the smallest r·2 n Input unit combination, so It represents a set of input unit combinations S that have only a small impact or interference on the mid-level feature output or model output of the black box model.

[0065] In another preferred embodiment, the benchmark value optimization module is further configured to:

[0066] Repeat the benchmark value optimization step T1 times until the newly calculated loss function value decreases and converges; wherein, for each repetition, the loss function value is recalculated according to the optimization function, and the benchmark value is updated based on the gradient of the loss function value, and the interaction strength value of each combination of input units is calculated based on the updated benchmark value; wherein, the optimization function is:

[0067]

[0068] In another preferred embodiment, the “AND-OR graph” generation module is further configured to: according to the aforementioned method of splitting the black box model output v(N): The method for optimizing the benchmark value results in a modeling of the black-box model's output v(N) based on a set of sparse "AND-additive relationships." Based on this set of "AND-additive relationships," the black-box model is modeled using an "AND-OR graph": the "AND" relationship represents the relationship between input units within each input unit combination S, and the "OR" relationship indicates that the black-box model's output v(N) can be decomposed into the sum of the interactions I(S) across all input unit combinations S. Therefore, an "AND-OR graph" is used to model the black-box model's output: the "AND-OR graph" contains at least three layers, consisting of several nodes and edges. Among them, the n leaf nodes at the bottom layer represent n input units; the several "AND nodes" in the middle represent the "AND" relationship between the input units within a certain input unit combination S. Each "AND" node is assigned a score score = I(S), which represents the numerical contribution of the interaction between the input unit combination corresponding to the node to the middle-level output feature or model output; the root node at the top layer is an "OR node", which represents the sum of the scores of all its child nodes, that is, the output v(N) of the black box model is ultimately expressed as the sum of the interactions I(S) of all input unit combinations S.

[0069] In a preferred embodiment, the “AND-OR graph” generation module is further configured to: after obtaining a certain “AND-OR graph” of the black box model, further simplify the expression of the “AND-OR graph” by the following method: since the different input unit combinations S often contain a portion of the same input units, for example, the input unit combinations {i, j, k}, {i, j, l}, {i, j, m, n} contain the same input unit {i, j}), these common input units are merged, and the merged result is recorded as the feature unit λ ij ={i, j}, then the original input unit combinations {i, j, k}, {i, j, l\}, {i, j, m, n} can use the common feature unit λ ij Simplify and obtain new feature unit representations, namely

[0070] {λ ij ,k},{λ ij , l}, {λ ij , m, n}

[0071] . Feature unit λ ij It can be used as an "AND node" in the middle layer of the "AND-OR graph", replacing the expression when the input units {i, j} coexist, and connected to other nodes containing the feature unit, thereby simplifying the "AND-OR graph" modeling of the black box model.

[0072] In a preferred embodiment, the “AND-OR graph” generation module is further configured to: record the set of the aforementioned feature units as X, use the feature units in the set X to represent the combination S between input units, and when constructing the “AND-OR graph” model, the frequency of occurrence of each feature unit λ∈X is defined by the following formula:

[0073]

[0074] in, represents the set of combinations between input units contained in the “AND-OR graph”. Based on the above probability, the representation length of each feature unit is Based on this, the total representation complexity of the “AND-OR graph” model can be calculated as:

[0075]

[0076] Here, p(S)∝|I(S)| represents the frequency of the input unit combination S in the AND-OR graph. The following iterative method is used to merge the common input units contained in certain input unit combinations S as new feature units, which are incorporated into the feature unit set X. Based on this set, all input unit combinations are re-represented and the AND-OR graph is modeled, thereby reducing the overall complexity of the AND-OR graph model:

[0077] (1) Initially, the feature unit is the input unit, then X (0) =N={1, 2, 3, ..., n};

[0078] (2) Based on the interaction strength |I(S)| between the input units, the feature unit X at the current moment is calculated. (t) Represents the frequency p of each feature unit when input units are combined (t) (λ), and calculate the total complexity of the current "AND-OR graph" model (t) ;

[0079] (3) For each subset of all input units N Calculate when L is used as a new feature unit and incorporated into X (t) get Afterwards, based on The total complexity of the "AND-OR graph" model when representing the combination of input units

[0080] (4) According to step (3), take the total complexity The lowest set of characteristic units make

[0081] (5) Repeat steps (2)-(4) T2 times and update the optimized X (t) .

[0082] A third aspect of the present invention provides a device for interpreting a black box model, comprising:

[0083] a memory for storing computer-executable instructions; and

[0084] A processor is configured to implement the steps of the method described above when executing the computer-executable instructions.

[0085] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the steps in the method described above are implemented.

[0086] It should be understood that within the scope of the present invention, the above-mentioned technical features of the present invention and the technical features described in detail below (such as in the embodiments) can be combined with each other to form new or preferred technical solutions. Due to space limitations, they will not be listed here one by one.

[0087] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 is a flowchart of a method for interpreting a black box model according to a first embodiment of the present invention;

[0089] Figure 2 is a schematic diagram of calculating the interaction strength of a combination of input units on an input sample according to the present invention;

[0090] Figure 3 It is a schematic diagram of making the “AND-ADD relationship” expression obtained from the black box model more concise and sparse by optimizing the reference value of each input unit in the interactive calculation according to the present invention;

[0091] Figure 4 It is a schematic diagram of the result of obtaining an "AND-OR diagram" of the internal logic of the black box model to be interpreted based on a set of sparse "AND-ADD relationships" on the black box model to be interpreted according to the present invention;

[0092] Figure 5FIG. 4 is a schematic diagram of the system structure for explaining the black box model according to the second embodiment of the present invention. DETAILED DESCRIPTION

[0093] After meticulous and in-depth research, the inventors have developed for the first time a method and system for characterization diagnosis and interpretation, model comparison, and training sample collection for black-box models. The method of the present invention constructs a method and system for objectively and concisely explaining the internal logic of a black-box model. Based on the mid-level output features or model output of the black-box model to be explained, the interaction between the input units of the sample is modeled based on the Harsanyi dividend indicator, and the interaction strength of the combinations formed by the input units is quantified. The black-box model is represented as an "and-add relationship" between the combinations of input units, making the process easier to understand. Furthermore, by modeling the samples, the interaction strength between the combinations of input units is quantified, and the interaction strength between the combinations of input units that has only a slight impact on the mid-level output features or model output is penalized. This optimizes the benchmark value on each input unit in the interaction calculation, making the "and-add relationship" obtained from the black-box model concise and sparse. Finally, the AND / OR graph interprets and presents the combinations between input units. Frequently co-occurring combinations between input units are extracted and merged as separate AND nodes, optimizing the complexity of the AND / OR graph and simplifying the interpretation of the AND / OR graph derived from the black-box model. This also facilitates feature comparison after black-box model analysis, comparison between black-box models, and the collection of training samples.

[0094] General Methods

[0095] The present invention is used to interpret the black box model and obtain the combination of input units that has a greater impact on the model output or middle-level output features. First, a prepared input sample is input into the black box model, and the sample has input units of a certain dimension. Then, based on the middle-level output feature sample of the black box model or the output sample of the black box model, the interaction and coordination between the input units of the sample is modeled, and the interaction strength of the combination formed by the input units is calculated. The black box model is represented as an "and-add relationship" between the combinations of input units. Then, the benchmark value on each input unit in the interactive calculation is optimized, so that the "and-add relationship" obtained from the black box model is more concise and sparse. Finally, based on the sparse "and-add relationship" of the black box model, an "and-or graph" model of the internal logic of the black box model is established.

[0096] Typically, the method of the present invention comprises the following steps:

[0097] 1. Select a model

[0098] Select the black box model that needs to be explained

[0099] 2. Get data samples

[0100] An input unit of a certain dimension is selected as an input sample and input into the black box model. Then, the obtained middle-level output feature sample or black box model output sample is used as the output unit, which corresponds one-to-one with the input unit as the data sample.

[0101] 3. Establish a first-level “relationship with Canada”

[0102] According to the output unit, the coordination between the input units of the sample is modeled. Then, based on the middle-level output feature sample of the black box model or the output sample of the black box model, the interaction coordination between the input units of the sample is modeled, and the interaction strength of the combination formed by the input units is calculated. The black box model is represented as an "and-add relationship" between the combinations of input units. Generally speaking, this "and-add relationship" is more complicated.

[0103] 4. Establish a secondary “relationship with Canada”

[0104] Optimize the benchmark value of each input unit in the interactive calculation, so that the "and addition relationship" obtained from the black box model is more concise and sparse.

[0105] 5. Build an “AND-OR graph” model

[0106] Finally, based on the sparse "AND-ADD relationship" of the black box model, an "AND-OR graph" model of the internal logic of the black box model is established.

[0107] The main advantages of the present invention are:

[0108] 1) A method and system for objectively and concisely explaining the internal logic of black box models was constructed.

[0109] 2) Based on the mid-level output features or model outputs of the black-box model to be explained, the interaction between the input units of the sample is modeled using the Harsanyi dividend indicator. This quantifies the interaction strength of the combinations formed by the input units and represents the black-box model as an "and-add relationship" between the combinations of input units, making it easier to understand.

[0110] 3) The combinations between input units are interpreted and presented through the "AND-OR graph", and the frequently co-occurring combinations between input units are extracted and merged as separate "AND nodes", which optimizes the complexity of the "AND-OR graph" and makes the interpretation of the "AND-OR graph" obtained from the black box model concise.

[0111] 4) The “AND-OR diagram” is used to explain the combinations between input units and show the importance of different combinations between input units, which facilitates feature comparison after black box model analysis, comparison between black box models, and collection of training samples.

[0112] The present invention will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art further understand the invention, but are not intended to limit the invention in any way. It should be understood that those skilled in the art will be able to make various modifications and improvements without departing from the scope of the present invention. Such modifications and improvements are all within the scope of protection of the present invention.

[0113] Example

[0114] The first embodiment of the present invention relates to a method for interpreting a black box model, the process of which is as follows: Figure 1 As shown, the method includes the following steps:

[0115] In step 101: a black box model to be interpreted is provided.

[0116] Any black box model with input and output can be used as the input original black box model to be interpreted in the present invention, such as but not limited to convolutional neural networks, deep neural networks, etc. The present invention does not limit the internal structure of the black box model, that is, the black box model of the present invention can adopt various internal structures.

[0117] Next, step 102 is performed: a sample containing features of a certain dimension is input into the black-box model to be interpreted. The sample can be any data that meets the requirements of the black-box model to be interpreted. Optionally, the sample can be a table dataset, image dataset, or text dataset that meets the requirements of the black-box model to be interpreted.

[0118] After that, proceed to step 103: Based on the middle-level output features or model output of the black box model to be explained, model the interaction and coordination between the input units of the sample, calculate the interaction strength of the combination formed by the input units, and represent the black box model as an "and-add relationship" between the combinations of input units.

[0119] Optionally, step 103 can be further implemented as follows: the black box model is represented as v, and the input includes n input units, which are represented as a set N = {1, 2, ..., n}. The output obtained by the black box model on the input is represented as v(N). For any subset of input units It is called the combination between input units. v(S) represents the output of the black box model when all the input units in a certain combination between input units S are given. In a black box model, the input units of the sample do not work alone, but interact with each other. Therefore, the Harsanyi dividend indicator is used to quantify the interaction between input units, and the numerical contribution of the interaction between input units within the combination between input units S to the middle-level output features or model output is calculated, recorded as I(S), and the output v(N) of the black box model is split into the sum of the interaction sizes of each combination between input units S, that is,

[0120]

[0121] in, , s represents the number of input units in set S, and l represents the number of input units in set L.

[0122] In the splitting method, the interaction within each input unit combination S is triggered only when all input units in S are present. In other words, when any input unit in S is "absent", the I(S) term does not contribute to the black box mid-level output features or model output. Therefore, the input units in the input unit combination S form an "and" relationship. In addition, based on the method of splitting the black box model output: The output v(N) of the black-box model v is modeled as an “additive” relationship between the interactions I(S) between different input units. Therefore, the output v(N) of the black-box model on the entire input sample can be expressed as an “and-additive relationship” between the input units.

[0123] like Figure 2 As shown, in an optional embodiment of the present invention, given a 5-layer multi-layer perceptron black box model to be interpreted, the 2-dimensional input unit (edge change ratio, fundamental frequency, spectral flux, spectral roll off, spectral centroid, zerocrossing rate, short time energy, frame difference, motion, shot length) contained in it is calculated on an input sample on the UCI commercial table dataset. 10 = The interaction intensity |I(S)| on 1024 different input unit combinations S. The input unit combinations are ranked according to their interaction strength. 40 important feature interactions are selected for plotting. Red indicates that the interaction size I(S) of the input unit combination is positive, and blue indicates that the interaction size I(S) of the input unit combination is negative.

[0124] Then, step 104 is performed: by optimizing the reference value of each input unit in the interactive calculation, the expression of the "AND-ADD relationship" obtained from the black box model is made more concise and sparse;

[0125] Optionally, step 104 may be further implemented as follows: based on the aforementioned splitting of the black box model output v(N): The output v(N) corresponding to the input sample N can be modeled as 2 n The sum of the interactions I(S) between the input units.n In the interaction, the calculation method based on the above interaction size is: The interaction strength |I(S)| corresponding to some combinations S of input units is large, indicating that it has a significant impact on the black-box model's output v(N). Meanwhile, the interaction strength |I(S)| corresponding to other combinations S is small, indicating that these combinations S have only a minor impact or interference on the black-box model's output v(N). Therefore, to simplify the interpretation of the black-box model and make the "and-addition relationship" modeled by the entire black-box model more concise and sparse, the benchmark values for each input unit in the interaction calculation are optimized.

[0126] The reference value refers to the requirement to retain the original sample input values of the input units in S when calculating the output v(S) of the black box model on a certain combination S of input units in interactive calculations, and replace all input units in the complement of S (N\S) with their reference values to indicate the "non-existent" state of these input units.

[0127] In the aforementioned method for interpreting the black-box model, different I(S) values are calculated under different benchmark values, resulting in different "and-additive relationship" expressions and interpretations. These interpretations are rigorous interpretations of the black-box model from different perspectives. The optimization method aims to find a sparse and concise "and-additive relationship" expression by optimizing the benchmark value. Therefore, the benchmark value optimization method optimizes and updates the benchmark value by penalizing the interaction strength |I(S)| between input unit combinations that have only a small impact, while retaining the interaction strength between input unit combinations that have a large impact on the output v(N) of the black-box model, in order to find a sparse and concise way to model the black-box model.

[0128] Optionally, the method further comprises the following steps:

[0129] The vector consisting of the reference values of all input units is recorded as b, where each dimension b i Represents the baseline value of input unit i. By penalizing the interaction strength |I(S)| between input unit combinations S that only produce minor effects or interference, the baseline value b of each input unit in the input sample N is optimized. i , so that the interaction intensity |I(S)| of the input unit combination S with a small interaction intensity |I(S)| approaches 0, and a sparse feature interaction is obtained. The loss function of the optimization benchmark value is:

[0130]

[0131] Among them, all input unit combinations S are sorted by the size of the interaction strength |I(S)|. Assume {S1, S2, ..., S m} is the m input unit combination with the smallest interaction strength, and r is defined as the ratio of these input unit combinations to all input unit combinations, that is, Select the r2 with the smallest interaction strength n Input units are combined to form a set Because of the collection Contains the r·2 with the smallest interaction strength n Input unit combination, so It represents a set of input unit combinations S that have only a small impact or interference on the mid-level feature output or model output of the black box model.

[0132] In this embodiment, the criteria for setting r can be: 1) the interaction strength |I(S)| between input unit combinations that only produce a minor impact is sufficiently low, and 2) the "and-addition relationship" established for the black box model is sufficiently sparse and concise. Optionally, r is set between 60% and 98%. It should be noted that the value of r in the present invention can be set as needed and is not limited to these options.

[0133] Optionally, the method further comprises the following steps:

[0134] Repeat the benchmark value optimization step T1 times until the newly calculated loss function value decreases and converges; wherein, for each repetition, the loss function value is recalculated according to the optimization function, and the benchmark value is updated based on the gradient of the loss function value, and the interaction strength value of each combination of input units is calculated based on the updated benchmark value; wherein, the optimization function is:

[0135]

[0136] Optionally, in this embodiment, the criteria for setting T1 may be: 1) the modified baseline value does not exceed the reasonable range of sample input unit values; 2) the loss decreases and converges. Optionally, T1 is set between 20 and 200 times. The value of T1 in the present invention can be set as needed and is not limited to these options.

[0137] Then, proceed to step 105: based on a set of sparse “AND-ADD relationships” on the black box model to be explained, establish an “AND-OR graph” model that expresses the internal logic of the black box model.

[0138] Optionally, step 105 may be further implemented as follows:

[0139] According to the above method of splitting the black box model output v(N): The method for optimizing the benchmark value results in a modeling of the black-box model's output v(N) based on a set of sparse "AND-additive relationships." Based on this set of "AND-additive relationships," the black-box model is modeled using an "AND-OR graph": the "AND" relationship represents the relationship between input units within each input unit combination S, and the "OR" relationship indicates that the black-box model's output v(N) can be decomposed into the sum of the interactions I(S) across all input unit combinations S. Therefore, an "AND-OR graph" is used to model the black-box model's output: the "AND-OR graph" contains at least three layers, consisting of several nodes and edges. Among them, the n leaf nodes at the bottom layer represent n input units; the several "AND nodes" in the middle represent the "AND" relationship between the input units within a certain input unit combination S. Each "AND" node is assigned a score score = I(S), which represents the numerical contribution of the interaction between the input unit combination corresponding to the node to the middle-level output feature or model output; the root node at the top layer is an "OR node", which represents the sum of the scores of all its child nodes, that is, the output v(N) of the black box model is ultimately expressed as the sum of the interactions I(S) of all input unit combinations S.

[0140] Optionally, the method further comprises the following steps:

[0141] After obtaining a certain “AND-OR graph” of the black box model, the expression of the “AND-OR graph” is further simplified by the following method: Since the different input unit combinations S often contain some of the same input units, for example, the input unit combinations {i, j, k}, {i, j, l}, {i, j, m, n} contain the same input unit {i, j}), these common input units are merged, and the merged result is recorded as the feature unit λ ij ={i, j}, then the original input unit combinations {i, j, k}, {i, j, l}, {i, j, m, n} can use the common feature unit λ ij Simplify and obtain new feature unit representations, namely {λ ij ,k},{λ ij , l}, {λ ij , m, n}. Feature unit λ ij It can be used as an "AND node" in the middle layer of the "AND-OR graph", replacing the expression when the input units {i, j} coexist, and connected to other nodes containing the feature unit, thereby simplifying the "AND-OR graph" modeling of the black box model.

[0142] Optionally, the method may be further implemented as follows:

[0143] The set of input units and feature units is denoted as X. The units in the set X are used to represent the combination S between input units. When constructing the "AND-OR graph" model, the frequency of occurrence of each feature unit λ∈X is defined by the following formula:

[0144]

[0145] in, represents the set of combinations between input units contained in the “AND-OR graph”. Based on the above probability, the representation length of each feature unit is Based on this, the total representation complexity of the “AND-OR graph” model can be calculated as:

[0146]

[0147] Here, p(S)∝|I(S)| represents the frequency of the input unit combination S in the AND-OR graph. The following iterative method is used to merge the common input units contained in certain input unit combinations S as new feature units, which are incorporated into the feature unit set X. Based on this set, all input unit combinations are re-represented and the AND-OR graph is modeled, thereby reducing the overall complexity of the AND-OR graph model:

[0148] (1) Initially, the feature unit is the input unit, then X (0) =N={1, 2, 3,..., n};

[0149] (2) Based on the interaction strength |I(S)| between the input unit combinations, the frequency p of each feature unit is calculated when the feature unit X(t) at the current moment is used to represent the input unit combination. (t) (λ), and calculate the total complexity of the current "AND-OR graph" model (t) ;

[0150] (3) For each subset of all input units N Calculate when L is used as a new feature unit and incorporated into X (t) get Afterwards, based on The total complexity of the "AND-OR graph" model when representing the combination of input units

[0151] (4) According to step (3), take the total complexity The lowest set of characteristic units make

[0152] (5) Repeat steps (2)-(4) T2 times and update the optimized X (t) .

[0153] In this embodiment, the criteria for setting T2 can be: 1) the interaction intensity |I(S)| corresponding to the input unit combination S is sufficiently sparse, and 2) the total complexity of the "AND-OR graph" model is reduced. Optionally, T2 is set between 4 and 20 times. Figure 4As shown, in an optional embodiment of the present invention, T2=5. Figure 4 is based on Figure 2 The above iterative process is carried out 5 times for the input unit combination formed by the input units shown in the table to extract the results of the feature units. The rows in the table constitute a set of feature units. The line graph above the table uses the set of feature units to represent the interaction intensity |I(S)| of the input unit combination when the input unit combination S is represented. The horizontal axis in the figure represents different input unit combinations S, and the vertical axis represents the size of the interaction intensity |I(S)|. The red color in the table indicates that the interaction size I(S) of the input unit combination is positive, and the blue color indicates that the interaction size I(S) of the input unit combination is negative.

[0154] like Figure 5 As shown, it is based on Figures 2 to 4 The steps are as follows: inputting a sample in a tabular data set into a multi-layer perceptron model to be interpreted, and obtaining a schematic diagram of an "AND-OR graph" model through the above-mentioned settings: wherein the 5 leaf nodes at the bottom layer represent 5 input units; the several "AND nodes" in the middle represent the "AND" relationship between the input units within a certain input unit combination S, and each "AND" node is assigned a score score = I(S), which represents the numerical contribution of the interaction between the input unit combination corresponding to the node to the output features of the model output layer and the model output; the root node at the top layer is an "OR node", which represents the sum of the scores of all its child nodes, that is, the output v(N) of the black box model is ultimately expressed as the sum of the interaction I(S) of all input unit combinations S.

[0155] The second embodiment of the present invention relates to a system for interpreting a black box model, the structure of which is as follows: Figure 5 As shown, the system for interpreting the black box model includes an input module, an input unit combination interaction calculation module, a benchmark value optimization module and an "AND-OR graph" generation module.

[0156] Specifically, the input module is configured as a black box model to be interpreted and a sample containing input units of a certain dimension; the input unit combination interaction calculation module is configured to model the interaction and coordination between the input units of the sample based on the output of the black box model to be interpreted, calculate the interaction strength of the combination formed by the input units, and express the black box model as an "AND-ADD relationship" between the combinations of input units; the benchmark value optimization module is configured to optimize the benchmark value on each input unit in the interaction calculation so that the "AND-ADD relationship" expression obtained from the black box model is more concise and sparse; the "AND-OR graph" generation module is configured to establish an "AND-OR graph" model that expresses the internal logic of the black box model based on a set of sparse "AND-ADD relationships" on the black box model to be interpreted.

[0157] Optionally, the input unit combination interaction calculation module is further configured to: denote the black box model as v, and the input includes n input units, which are represented as a set N = {1, 2, ..., n}. The output obtained by the black box model on the input is represented as v(N). For any subset of input units It is called the combination between input units. v(S) represents the output of the black box model when all the input units in a certain combination between input units S are given. In a black box model, the input units of the sample do not work alone, but interact with each other. Therefore, the Harsanyi dividend indicator is used to quantify the interaction between input units, and the numerical contribution of the interaction between input units within the combination between input units S to the middle-level output features or model output is calculated, recorded as I(S), and the output v(N) of the black box model is split into the sum of the interaction sizes of each combination between input units S, that is,

[0158]

[0159] in, s is the number of input units in set S, and l is the number of input units in set L.

[0160] In the splitting method, the interaction within each input unit combination S is triggered only when all input units in S are present. In other words, when any input unit in S is "absent", the I(S) term does not contribute to the black box mid-level output features or model output. Therefore, the input units in the input unit combination S form an "and" relationship. In addition, based on the method of splitting the black box model output: The output v(N) of the black-box model v is modeled as an “additive” relationship between the interactions I(S) between different input units. Therefore, the output v(N) of the black-box model on the entire input sample can be expressed as an “and-additive relationship” between the input units.

[0161] Optionally, the input unit combined interaction calculation module is further configured to: based on the aforementioned splitting of the black box model output v(N): The output v(N) corresponding to the input sample N can be modeled as 2 n The sum of the interactions I(S) between the input units. n In the interaction, the calculation method based on the above interaction size is: The interaction strength |I(S)| corresponding to some combinations S of input units is large, indicating that it has a significant impact on the black-box model's output v(N). Meanwhile, the interaction strength |I(S)| corresponding to other combinations S is small, indicating that these combinations S have only a minor impact or interference on the black-box model's output v(N). Therefore, to simplify the interpretation of the black-box model and make the "and-addition relationship" modeled by the entire black-box model more concise and sparse, the benchmark values for each input unit in the interaction calculation are optimized.

[0162] The reference value refers to the requirement to retain the original sample input values of the input units in S when calculating the output v(S) of the black box model on a certain combination S of input units in interactive calculations, and replace all input units in the complement of S (N\S) with their reference values to indicate the "non-existent" state of these input units.

[0163] In the aforementioned method for interpreting the black-box model, different I(S) values are calculated under different benchmark values, resulting in different "and-additive relationship" expressions and interpretations. These interpretations are rigorous interpretations of the black-box model from different perspectives. The optimization method aims to find a sparse and concise "and-additive relationship" expression by optimizing the benchmark value. Therefore, the benchmark value optimization method optimizes and updates the benchmark value by penalizing the interaction strength |I(S)| between input unit combinations that have only a small impact, while retaining the interaction strength between input unit combinations that have a large impact on the output v(N) of the black-box model, in order to find a sparse and concise way to model the black-box model.

[0164] Optionally, the benchmark value optimization module is further configured to:

[0165] The vector consisting of the reference values of all input units is recorded as b, where each dimension b i Represents the baseline value of input unit i. By penalizing the interaction strength |I(S)| between input unit combinations S that only produce minor effects or interference, the baseline value b of each input unit in the input sample N is optimized. i , so that the interaction intensity |I(S)| of the input unit combination S with a small interaction intensity |I(S)| approaches 0, and a sparse feature interaction is obtained. The loss function of the optimization benchmark value is:

[0166]

[0167] Among them, all input unit combinations S are sorted by the size of the interaction strength |I(S)|. Assume {S1, S2, ..., S m} is the m input unit combination with the smallest interaction strength, and r is defined as the ratio of these input unit combinations to all input unit combinations, that is, Select the r2 with the smallest interaction strength n Input units are combined to form a set Because of the collection Only the interactions with the smallest r·2 n Input unit combination, so It represents a set of input unit combinations S that have only a small impact or interference on the mid-level feature output or model output of the black box model.

[0168] Optionally, the benchmark value optimization module is further configured to:

[0169] Repeat the benchmark value optimization step T1 times until the newly calculated loss function value decreases and converges; wherein, for each repetition, the loss function value is recalculated according to the optimization function, and the benchmark value is updated based on the gradient of the loss function value, and the interaction strength value of each combination of input units is calculated based on the updated benchmark value; wherein, the optimization function is:

[0170]

[0171] Optionally, the “AND-OR graph” generation module is further configured to: according to the aforementioned method of splitting the black box model output v(N): The method for optimizing the benchmark value results in a modeling of the black-box model's output v(N) based on a set of sparse "AND-additive relationships." Based on this set of "AND-additive relationships," the black-box model is modeled using an "AND-OR graph": the "AND" relationship represents the relationship between input units within each input unit combination S, and the "OR" relationship indicates that the black-box model's output v(N) can be decomposed into the sum of the interactions I(S) across all input unit combinations S. Therefore, an "AND-OR graph" is used to model the black-box model's output: the "AND-OR graph" contains at least three layers, consisting of several nodes and edges. Among them, the n leaf nodes at the bottom layer represent n input units; the several "AND nodes" in the middle represent the "AND" relationship between the input units within a certain input unit combination S. Each "AND" node is assigned a score score = I(S), which represents the numerical contribution of the interaction between the input unit combination corresponding to the node to the middle-level output feature or model output; the root node at the top layer is an "OR node", which represents the sum of the scores of all its child nodes, that is, the output v(N) of the black box model is ultimately expressed as the sum of the interactions I(S) of all input unit combinations S.

[0172] Optionally, the “AND-OR graph” generation module is further configured to: after obtaining a certain “AND-OR graph” of the black box model, further simplify the expression of the “AND-OR graph” by the following method: since the different input unit combinations S often contain a portion of the same input units, for example, the input unit combinations {i, j, k}, {i, j, l}, {i, j, m, n} contain the same input unit {i, j}), these common input units are merged, and the merged result is recorded as the feature unit λ ij ={i, j}, then the original input unit combinations {i, j, k}, {i, j, l}, {i, j, m, n} can use the common feature unit λ ij Simplify and obtain new feature unit representations, namely {λ ij ,k},{λ ij , l}, {λ ij , m, n}. Feature unit λ ij It can be used as an "AND node" in the middle layer of the "AND-OR graph", replacing the expression when the input units {i, j} coexist, and connected to other nodes containing the feature unit, thereby simplifying the "AND-OR graph" modeling of the black box model.

[0173] Optionally, the “AND-OR graph” generation module is further configured to: record the set of the aforementioned feature units as X, use the feature units in the set X to represent the input unit combination S, and when constructing the “AND-OR graph” model, the occurrence frequency of each feature unit λ∈X is defined by the following formula:

[0174]

[0175] in, represents the set of combinations between input units contained in the “AND-OR graph”. Based on the above probability, the representation length of each feature unit is Based on this, the total representation complexity of the “AND-OR graph” model can be calculated as:

[0176]

[0177] Here, p(S)∝|I(S)| represents the frequency of the input unit combination S in the AND-OR graph. The following iterative method is used to merge the common input units contained in certain input unit combinations S as new feature units, which are incorporated into the feature unit set X. Based on this set, all input unit combinations are re-represented and the AND-OR graph is modeled, thereby reducing the overall complexity of the AND-OR graph model:

[0178] (1) Initially, the feature unit is the input unit, then X (0) =N={1, 2, 3, ..., n};

[0179] (2) Based on the interaction strength |I(S)| between the input units, the feature unit X at the current moment is calculated. (t) Represents the frequency p of each feature unit when input units are combined (t) (λ), and calculate the total complexity of the current "AND-OR graph" model (t) ;

[0180] (3) For each subset of all input units N Calculate when L is used as a new feature unit and incorporated into X (t) get Afterwards, based on The total complexity of the "AND-OR graph" model when representing the combination of input units

[0181] (4) According to step (3), take the total complexity The lowest set of characteristic units make

[0182] (5) Repeat steps (2)-(4) T2 times and update the optimized X (t) .

[0183] The first embodiment is a method embodiment corresponding to the present embodiment. The technical details in the first embodiment can be applied to the present embodiment, and the technical details in the present embodiment can also be applied to the first embodiment.

[0184] It should be noted that those skilled in the art should understand that the implementation functions of the various modules shown in the above-mentioned embodiments of the system for interpreting black-box models can be understood with reference to the relevant description of the method for interpreting black-box models. The functions of the various modules shown in the above-mentioned embodiments of the system for interpreting black-box models can be implemented by programs (executable instructions) running on a processor, or by specific logic circuits. If the above-mentioned system for interpreting black-box models in the embodiment of the present invention is implemented in the form of software functional modules and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the method of each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a magnetic disk, or an optical disk. Thus, the embodiment of the present invention is not limited to any specific combination of hardware and software.

[0185] Accordingly, an embodiment of the present invention also provides a computer-readable storage medium having computer-executable instructions stored therein, which, when executed by a processor, implement various method embodiments of the present invention. Computer-readable storage media include permanent and non-permanent, removable and non-removable media that can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable storage media does not include transitory media such as modulated data signals and carrier waves.

[0186] In addition, an embodiment of the present invention further provides a system for interpreting a black box model, which includes a memory for storing computer-executable instructions, and a processor; the processor is used to implement the steps in the above-mentioned method implementation methods when executing the computer-executable instructions in the memory. The processor can be a central processing unit (Central Processing Unit, referred to as "CPU"), or other general-purpose processors, digital signal processors (Digital Signal Processor, referred to as "DSP"), application-specific integrated circuits (Application Specific Integrated Circuit, referred to as "ASIC"), etc. The aforementioned memory can be a read-only memory (read-only memory, referred to as "ROM"), a random access memory (random access memory, referred to as "RAM"), a flash memory (Flash), a hard disk or a solid-state drive, etc. The steps of the methods disclosed in the various embodiments of the present invention can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

[0187] It should be noted that in this patent specification, relational terms such as first and second are used solely to distinguish one entity or operation from another, and do not necessarily require or imply any actual relationship or order between these entities or operations. Furthermore, the terms "comprise," "include," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. Without further limitation, an element specified by the phrase "comprising a" does not preclude the presence of additional identical elements in the process, method, article, or apparatus comprising that element. In this patent specification, reference to performing an action in accordance with an element means performing the action in accordance with at least that element, including two situations: performing the action in accordance with that element alone, and performing the action in accordance with that element and other elements. Expressions such as "plurality," "multiple times," and "many" include "two," "twice," "two kinds," and "more than two," "more than two times," and "more than two kinds."

[0188] All documents mentioned in this invention are considered to be included in their entirety in the disclosure of this invention so that they can be used as a basis for modification when necessary. In addition, it should be understood that the above are only preferred embodiments of this specification and are not intended to limit the scope of protection of this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this specification should be included in the scope of protection of one or more embodiments of this specification.

Claims

1. A method for interpreting a black box model, characterized in that The method comprises the following steps: (1) Select the model; Selecting a black box model to be explained, wherein the black box model includes a neural network; (2) Obtain the data required for modeling: Select a sample and input it into the black box model to obtain a mid-level feature output or a black box model output sample based on the input sample, and use the input sample and the mid-level feature output or the black box model output sample as data for modeling, wherein the input sample includes input units of W dimensions, where W is a positive integer ≥ 5; (3) Determine the primary "and-add relationship" Based on the middle-level output features or model output samples of the black-box model, the interaction and coordination between the input units of the input sample is modeled to obtain a combination of the input units, and the interaction strength of the combination of the input units is calculated. Then, based on the interaction strength, the first-order "AND-ADD relationship" between the combinations of the input units of the black-box model is determined. Specifically, the interaction and coordination between the input units is quantified using the Harsanyi dividend indicator, and the numerical contribution of the interaction and coordination between the input units within the input unit combination S to the middle-level output features or model output is calculated, recorded as I(S), and the output v(N) of the black-box model is split into the sum of the interaction sizes of each input unit combination S. In the splitting method, the interaction within each input unit combination S is triggered if and only if all input units in S exist, so that the input units in the input unit combination S form an "AND" relationship, thereby determining the first-order "AND-ADD relationship" between the output v(N) on the input sample in the black-box model and the input units, that is, the output v(N) on the input sample in the black-box model is expressed as an "AND-ADD relationship" between the input units. (4) Establishing a secondary "with-addition relationship" Based on the "AND-ADD relationship" in step (3), by optimizing the reference values of each input unit in the interactive calculation, the combinations of input units with strong correlation in the "AND-ADD relationship" are further combined, and the combinations of unimportant input units are eliminated, so that the "AND-ADD relationship" is expressed more concisely and sparsely; specifically, the vector composed of the reference values of all input units is denoted as b, where each dimension b i Represents the baseline value of input unit i, and optimizes the baseline value b of each input unit of the input sample N by penalizing the interaction strength |I(S)| between input unit combinations S that only produce minor effects or interference. i , so that the interaction intensity |I(S)| of the input unit combination S with a small interaction intensity |I(S)| tends to 0, and sparse interaction between input units is obtained, thereby determining the secondary "and-addition relationship" between the output v(N) on the input sample in the black box model and the input unit; The sparse interaction between input units means that the interaction intensity |I(S)| of most combinations of input units is small, that is, close to 0, and has no effect on the network output; only a small number of combinations of input units have a large interaction intensity |I(S)|, which has a significant impact on the network output; (5) Establishing an "AND-OR graph" model Based on the secondary "AND-addition relationship" described in step (4), an "AND-OR graph" model is established to express the internal logic of the black box model. Specifically, the "AND-OR graph" contains at least three layers, consisting of a number of nodes and edges, wherein the n leaf nodes at the bottom layer represent n input units; the middle number of "AND nodes" represent the "AND" relationship between the input units within a certain input unit combination S, and each "AND" node is assigned a score score = I(S), which represents the numerical contribution of the interaction between the input unit combination corresponding to the node to the middle layer output feature or model output; the root node at the top layer is an "OR node", which represents the sum of the scores of all its child nodes, that is, the output v(N) of the black box model is ultimately expressed as the sum of the interaction I(S) of all input unit combinations S; The sparse interaction between input units means that the interaction intensity |I(S)| of most combinations between input units is small (approaching 0) and has no effect on the network output; only the interaction intensity |I(S)| of a small number of combinations between input units is large, which has a significant impact on the network output.

2. The method according to claim 1, characterized in that In step (4), the input unit combination analysis is performed based on the reference value, that is, in the interactive calculation, when calculating the output v(S) of the black box model on a certain input unit combination S, the original sample input values of the input units in S need to be retained, and all input units in the complement set N\S of S need to be replaced with their reference values to indicate the "non-existent" state of these input units, and then the input unit combination S is analyzed.

3. The method according to claim 1, characterized in that The "reference values of each input unit in the optimization interactive calculation" include: By penalizing the interaction strength |I(S)| of input unit combinations that have only a small impact, the interaction strength of input unit combinations that have a greater impact on the output v(N) of the black box model is retained, thereby optimizing the update of the benchmark value.

4. The method according to claim 3, characterized in that The step (4) comprises: First, generate an initial benchmark value by taking the sample mean, a random value, zero, or any other method; Secondly, repeat the benchmark value optimization step until the latest calculated loss function value decreases and converges; For each repetition, the loss function value is recalculated according to the optimization function, the reference value is updated based on the gradient of the loss function value, and the interaction strength value of the combination between each input unit is calculated based on the updated reference value.

5. The method according to claim 1, wherein The step (5) includes: after obtaining a certain "AND-OR graph" of the black box model, further simplifying the expression of the "AND-OR graph" by the following method: since the combination S between different input units often contains a part of the same input units, these common input units are merged, and the merged result is recorded as the feature unit λ ij ={i, j}, then the original input unit combinations {i, j, k}, {i, j, l}, {i, j, m, n} can use the common feature unit λ ij Simplify and obtain new feature unit representations, namely: {λ ij ,k},{λ ij , l}, {λ ij , m, n}; characteristic unit λ ij It can be used as an "AND node" in the middle layer of the "AND-OR graph", replacing the expression when the input units {i, j} coexist, and connected to other nodes containing the feature unit, thereby simplifying the "AND-OR graph" modeling of the black box model.

6. The method according to claim 1, characterized in that The black box models include: autonomous driving information processing black box model, medical image processing black box model, intelligent financial black box model, image retrieval black box model, intelligent security black box model, military application black box model, etc.

7. A system for characterization diagnosis, model comparison, and training sample collection for black box models, characterized by: include: The input module is configured as a black box model and a sample containing input units of a certain dimension; The input unit combination interaction calculation module is configured to model the interaction and coordination between the input units of the sample based on the middle-level output features or model output of the black-box model, calculate the interaction strength of the input unit combination, and express the black-box model as an "and-add relationship" between the input unit combinations. Specifically, the interaction and coordination between the input units is quantified using the Harsanyidividend indicator, and the numerical contribution of the interaction and coordination between the input units within the input unit combination S to the middle-level output features or model output is calculated, recorded as I(S), and the output v(N) of the black-box model is split into the sum of the interaction sizes of each input unit combination S. In the splitting method, the interaction within each input unit combination S is triggered if and only if all input units in S exist, so that the input units in the input unit combination S form an "and" relationship, thereby determining the first-level "and-add relationship" between the output v(N) on the input sample in the black-box model and the input units, that is, the output v(N) on the input sample in the black-box model is expressed as an "and-add relationship" between the input units. The benchmark value optimization module is configured to optimize the benchmark values of each input unit in the interactive calculation to make the "AND-ADD relationship" expression obtained from the black box model more concise and sparse; and The "AND-OR graph" generation module is configured to establish an "AND-OR graph" model that expresses the internal logic of the black-box model based on a set of sparse "AND-addition relationships" on the black-box model. The "AND-OR graph" contains at least three layers, consisting of a number of nodes and edges. Among them, the n leaf nodes in the bottom layer represent n input units; the several "AND nodes" in the middle represent the "AND" relationships between the input units within a certain input unit combination. Each "AND" node is assigned a score to represent the numerical contribution of the interaction between the input unit combination corresponding to the node to the middle-layer output feature or model output; the root node in the top layer is an "OR" node, which represents the sum of the scores of all its child nodes. That is, the output of the black-box model is ultimately represented as the sum of the interactions between all input unit combinations.

Images