A method for measuring the sparse interaction utility of black-box artificial intelligence models.

Abstract

This application discloses a method and system for interpreting sparse interaction utility modeled by a black-box artificial intelligence model, relating to the field of machine learning technology. The method and system can automatically analyze the interaction distribution modeled by the model. Implementation of the method and system includes the steps of providing data that is essential for evaluation; predicting the data using a black-box model and obtaining the model's prediction results; modeling the interaction effects between sample input units based on the output of the black-box model, calculating the interaction intensity of the combinations formed between the input units, representing the black-box model as "and additive relationships" and "or additive relationships" between combinations of input units; and optimizing the "and additive relationships" and "or additive relationships" to make them more sparse. The advantage of the present invention is that it provides a quantification method for interpreting interactions modeled by a black-box artificial intelligence model, enabling a more sparse and concise interpretation of interactions compared to conventional studies.

Description

[Technical Field]

[0001] The present invention relates to the field of machine learning technology, and more specifically to a method and system for interpreting sparse interaction utility modeled by a black-box artificial intelligence model. [Background technology]

[0002] Currently, deep learning is widely applied in various fields and demonstrates excellent performance, but users and researchers struggle to analyze the black box nature of artificial intelligence models. While existing interaction-based interpretation methods can quantitatively analyze the interaction utility generated by combinations of units and attributes in the model input, under conditions that ensure reliability, they cannot guarantee sparsity or conciseness of the interpretation results. However, sparsity and conciseness in the interpretation results are essential conditions for interpretability methods using artificial intelligence.

[0003] Therefore, reliably interpreting the interaction utility modeled by black-box models while simultaneously ensuring their sparsity and conciseness is an urgent issue that needs to be addressed in the field of interpretability. [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] The object of the present invention is to invent a method and system for interpreting sparse interaction utility modeled by a black-box artificial intelligence model, which can automatically analyze the interaction distribution modeled by the model. [Means for solving the problem]

[0005] The method includes the following steps: (1) Model selection step A black-box artificial intelligence model is selected for analysis, and this black-box AI model includes a model that has been pre-trained on a specific dataset. (2) Steps to obtain the data to be evaluated The data to be evaluated is collected and preprocessed. The data format may be tabular data, images, text, audio, etc., but it must be compatible with the model in step (1). The data to be evaluated includes a certain number of input units.

[0006] (3) Steps to model the "and addition relationship" and "or addition relationship" between input units. The data to be evaluated in step (2) is input into the model in step (1), and the output of the model is obtained. Based on the output of the black-box model, the interaction between input units is modeled, a set consisting of different input units is obtained, and the interaction strength of each set is calculated. The black-box model is represented as a combination of "and additive relationships" and "or additive relationships" between combinations of input units.

[0007] (4) Step to optimize the combination of "and addition relation" and "or addition relation" Based on the "and addition relationship" and "or addition relationship" described in step (3), the combination of "and addition relationship" and "or addition relationship" is optimized so that all expressions of the two types of relationships become more concise and sparse. In a preferred example, step (1) further includes the following steps: A black-box artificial intelligence model is trained as the model to be analyzed based on a specific dataset. This model can optionally be a deep neural network.

[0008] In a preferred example, step (2) further includes the following substeps: (a) Collect the data to be analyzed. Optionally, if the model in step (1) can be used for image classification, the data to be analyzed can be an image. (b) Preprocess the data to be analyzed. Optionally, in order to adapt it to the model input, the data to be analyzed can be processed as downsampling or upsampling, normalization, etc.

[0009] In a preferred example, step (3) further includes the following substeps: (3a) Calculate the output value of the black-box model under the condition that the data to be analyzed is arbitrarily masked. (3b) Calculate the "and interaction" in the data to be analyzed modeled by the black-box model. (3c) Calculate the "or interaction" in the data to be analyzed modeled by the black-box model.

[0010] In the above substep (3a), the black-box type artificial intelligence model is represented as v, the data sample x to be analyzed includes n input units, and is represented as the set N = {1, 2,..., n}, where the number of input units is generally a positive integer of 5 or more. Optionally, each dimension in the input data can be regarded as an input unit, or the input data can be divided into multiple regions, and each region can be regarded as an input unit.

[0011] Any subset S ⊆ N of the input units is called a combination between the input units, and v(S) represents the output of the black-box model when a specific combination S between the input units is masked and the input units in N\S are masked. In particular, when the data to be analyzed is input into the model without masking, the obtained output is represented as v(N), and when the analysis data is completely masked and input into the model, the obtained output is

Number

[0012] When calculating the output v(S) of the black-box model for a specific combination S of input units, the original input values of the analysis target samples are maintained in the input units included in S, but it is necessary to replace the input units in the complement N\S of all S with reference values. In particular, when the reference value is b, each input unit of the masked sample x S is defined by the following formula,

Equation

[0013] In this way, v(S) can be calculated as the output value obtained by the input model from the sample x S Optionally, the reference value can be set to the average value, random value, zero value, etc. of the sample set. Furthermore, the output v(S) of the black-box model calculated based on the reference value b can be specifically expressed as v(S|b), and when there is no ambiguity, v(S|b) can be abbreviated as v(S).

[0014] In the above sub-step (3b), by calculating "and interaction", the contribution value of the and relationship between any combination of input units to the output of the black-box model is evaluated and measured. "And interaction" refers to the additional utility generated by the output of the black-box model when the units in the combination are triggered (i.e., not masked) simultaneously. In the black-box model, it is necessary to understand that input units do not contribute to the output of the model alone and often interact with each other to affect the output of the model.

[0015] Therefore, when S⊆N, I AND (S) is defined as the "and interaction" corresponding to the combination S between input units. When all input variables in S are triggered, I AND (S) is triggered and accumulated in the output of the black-box model. Specifically, IAND (S) can be calculated by the following formula. [Number] Here, s = |S| and l = |L|. Optionally, different "and interaction" values can be calculated based on different reference values in step (2), and the and interaction calculated based on the reference value b is I AND Marked as I(S│b). When no ambiguity is generated, it should be understood that I(S│b) is abbreviated as I(S). AND I(S│b) as I AND (S).

[0016] The above "and interaction" can be divided into the sum of the outputs v(T) of the black box model in this combination of input units for any combination T⊆N of arbitrary predetermined input units and all triggered and interactions I(S), and satisfy the following formula. AND (S).

[0017] JPEG2026512532000005.jpg93170

[0018] In the above sub-step (3c), by calculating the "or interaction", the or relationship between any combination of input units evaluates and measures the contribution value of the output of the black box model. The "or interaction" represents the additional utility generated by the output of the black box model when at least one unit in the combination is not triggered (i.e., not masked). In the black box model, it should be understood that input units often do not contribute to the output of the model alone and interact with each other to affect the output of the model.

[0019] Therefore, when S⊆N, I ORLet (S) be defined as an "or interaction" corresponding to a combination S between input units, and if at least one input variable in S is triggered, then I OR (S) is triggered and accumulated in the output of the black box model. Specifically, I OR (S) can be calculated by the following formula:

number

[0020] Here, s=|S| and l=|L|. The above "or interaction" refers to the situation where, for any given combination of input units T⊆N, the output v(T) of the black box model in this combination of input units is the result of all triggered or interaction I OR It can be divided into the sum of (S) and satisfy the following equation:

[0021] JPEG2026512532000007.jpg81170

[0022] In a preferred example, step (4) further includes the following substeps: (4a) Based on the "and interaction" and "or interaction" in step (3), the output of the black box model is interpreted as a combination of "and addition relationships" and "or addition relationships" between combinations of input units. (4b) Based on the combination of "and addition relation" and "or addition relation" in substep (4a), the combination of "and addition relation" and "or addition relation" is further optimized so that the expression of the two types of relations becomes more concise and sparser.

[0023] In substep (4a), based on the "and interaction" and "or interaction" in step (3), the output v(T) of the black box model for any combination of input units T is interpreted as a combination of "and addition" and "or addition". Selectively, based on different reference values ​​in step (3), the output v(T) of the black box model is interpreted as a combination of "and addition based on different reference values" and "or addition". Generally, m different reference values ​​b (1) ,…,b (m) If such a condition exists, the output v(T) of the black-box model for any combination of input units is interpreted as shown in the following equation.

number

[0024] Therefore, based on the interpretation of the black-box model described above, the model's output v(N) in a complete sample is

number

number

number

[0025] JPEG2026512532000012.jpg70170

[0026] JPEG2026512532000013.jpg111170

[0027] In particular, in one embodiment, when only one type of reference value is selected in substep (a) above, i.e., when m=1, the output of the black box model can be interpreted as a combination of the "and addition relation" and the "or addition relation" shown in the following equation.

[0028] JPEG2026512532000014.jpg73170

[0029] JPEG2026512532000015.jpg133170

[0030] In a preferred example, the labeling of the combinations of input units in steps (3) and (4) above may further include the following steps: The output v(N) corresponding to the input sample N is 2 n Combinations and interaction between individual input units I AND and 2 n In combinations or interactions between individual input units OR It is possible to model this, and using the following method, 2 n The combinations between the n input units are labeled. For example, if n=3, N={i,j,k} is set, and preferably, all combinations between the input units can be labeled as shown in Table 1.

[0031] [Table 1] For example, when n=5, N={i,j,k,l,m} can be set, and preferably, all combinations between input units can be indicated as shown in Table 2.

[0032] [Table 2]

[0033] A second aspect of the present invention provides a system for interpreting sparse interaction utility modeled by a black-box artificial intelligence model, and includes the following modules: (1) The input module consists of one pre-trained black-box artificial intelligence model and one set of data to be analyzed. (2) The calculation module is configured to calculate the "and addition relationships" and "or addition relationships" between data input units modeled by the model, based on the model and data in the input module, and to optimize the combination of "and addition relationships" and "or addition relationships" so that all representations of the two types of relationships are more concise and sparse. (3) The output module is configured to output combinations of "and addition relation" and "or addition relation" optimized by the calculation module. [Effects of the Invention]

[0034] It should be understood that, within the scope of the present invention, new or preferred technical solutions can be constructed by combining the above-described technical features of the present invention with the technical features specifically described below (e.g., in the examples). Due to space limitations, this will not be repeated here.

[0035] Further features, purposes, and advantages of the present invention will become clearer by reading the detailed description of non-limiting embodiments with reference to the attached drawings below.

[0036] The main advantages of this invention are as follows: (1) The present invention provides a method and system that can automatically analyze the "and addition relationship" and "or addition relationship" inside a black box model. (2) The methods and systems disclosed in the present invention can ensure the accuracy of the interpretation results, that is, based on the “and interaction” and “or interaction” presented in the present invention, the output of the black box model can be accurately divided into the sum of the interaction utilities between different input units. (3) The methods and systems disclosed in the present invention can ensure the brevity of the interpretation results, that is, based on the optimization of the “and addition relationship” and “or addition relationship” presented in the present invention, a more sparse and concise interaction interpretation can be obtained. (4) The methods and systems disclosed in the present invention have broad universality and generality, that is, they can be applied to most black-box models, including but not limited to deep neural networks.

[0037] The specification of this application contains a large number of technical features, distributed across each technical solution, and listing all possible combinations of the technical features (i.e., technical solutions) of this application would make the specification excessively long. To avoid this problem, each technical feature disclosed in the above summary of the invention of this application, each technical feature disclosed in the following embodiments and examples, and each technical feature disclosed in the drawings can be freely combined with each other to form a variety of new technical solutions (all of which are deemed to have already been described herein), provided that it is not technically impossible to implement. For example, one example discloses feature A+B+C, and another discloses feature A+B+D+E, where features C and D are equivalent technical means that perform the same function, and technically only one of them can be used, not both at the same time, and if feature E is technically combined with feature C, the A+B+C+D solution should not be considered explained because it is technically impossible to implement, but the A+B+C+E solution should be considered explained. [Brief explanation of the drawing]

[0038] The following briefly introduces the drawings that may be used in the description of the examples or prior art. Clearly, the drawings in the following description represent only a few examples of the present invention, and those skilled in the art can obtain other drawings based on these without any creative effort. [Figure 1] This is a schematic flowchart illustrating the interpretation of a sparse interaction utility method modeled by a black-box artificial intelligence model based on the first embodiment of the present invention. [Figure 2] Figure 2(a) is a schematic diagram of the sample to be analyzed and the input variables described in the present invention; Figure 2(b) is the initial utility distribution of the "and additive relationship" and "or additive relationship" obtained according to the present invention; and Figure 2(c) is the final utility distribution of the "and additive relationship" and "or additive relationship" obtained according to the optimization of the present invention. These results are based on tabular data. [Figure 3] These are utility values ​​of some of the "and interactions" and "or interactions" obtained according to the optimization of the present invention, and the results are based on tabular data. [Figure 4] Figure 4(a) is a schematic diagram of the sample to be analyzed and the input variables described in the present invention; Figure 4(b) is the initial utility distribution of the "and additive relationship" and "or additive relationship" obtained according to the present invention; and Figure 4(c) is the final utility distribution of the "and additive relationship" and "or additive relationship" obtained according to the optimization of the present invention. These results are based on image data. [Figure 5] These are utility values ​​of some of the "and interactions" and "or interactions" obtained according to the optimization of the present invention, and the results are based on image data. [Figure 6] This is a schematic diagram illustrating the structure of a sparse interaction utility system modeled by a black-box artificial intelligence model based on a second embodiment of the present invention. [Modes for carrying out the invention]

[0039] Through careful and thorough research, the inventors have developed, for the first time, a method and system for interpreting sparse interaction utility modeled by a black-box artificial intelligence model.

[0040] General method Typically, the present invention includes the following steps: (1) Select a model. Select a black-box artificial intelligence model to be analyzed, which is a model that has been pre-trained on a specific dataset. (2) Obtain the data to be evaluated. The data to be evaluated is collected and preprocessed. The data format may be tabular data, images, text, audio, etc., but it must be compatible with the model in step (1). The data to be evaluated includes a certain number of input units.

[0041] (3) Model the "and addition relationship" and "or addition relationship" between input units. Input the data to be evaluated in step (2) into the model in step (1) and obtain the output of the model. Based on the output of the black box model, model the interaction between input units, obtain sets consisting of different input units, and calculate the interaction strength of each set. Represent the black box model as a combination of "and addition relationship" and "or addition relationship" between combinations of input units.

[0042] (4) Optimize the combination of "and addition" and "or addition" relationships. Based on the "and addition" and "or addition" relationships described in step (3), optimize the combination of "and addition" and "or addition" relationships so that all expressions of the two types of relationships become more concise and sparse.

[0043] Examples To further clarify the purpose, technical solutions, and advantages of this application, embodiments of this application will be described in more detail below, in conjunction with the accompanying drawings. A first embodiment of the present invention relates to a method for interpreting sparse interaction utility modeled by a black-box artificial intelligence model, the flow of which is shown in Figure 1, and the method includes the following steps.

[0044] Step 101: Train a black-box artificial intelligence model as the model to be analyzed based on a specific dataset. The model can be a deep neural network. In one example, the model is a 5-layer perceptron model (MLP-5) pre-trained on the UCI Wireless Indoor Localization dataset, which is a deep neural network. In another example, the model is a ResNet-18 model pre-trained on the CelebA dataset, which is a deep neural network.

[0045] Next, we proceed to step 102, which can be further divided into the following two substeps: (a) Collect the data to be analyzed. If the model in step (1) can be used for image classification, the data to be analyzed may be images.

[0046] (b) Preprocess the data to be analyzed. The data to be analyzed can be processed by downsampling, upsampling, normalization, etc., as selectable, in order to adapt it to the model input described above.

[0047] In one embodiment, as shown in Figure 2 (left), the data is selected as arbitrary data from the UCI Wireless Indoor Localization dataset and includes seven input variables, each representing the 1st to 7th Wi-Fi signal strength values, denoted as signal 1, signal 2, ..., signal 7. In another embodiment, as shown in Figure 4 (left), the model is selected as arbitrary data from the CelebA dataset and is divided into nine input variables such as hair, forehead, left eye, right eye, nose, left cheek, right cheek, mouth, and chin.

[0048] Next, we proceed to step 103, which can be further divided into the following three substeps: (a) Calculate the output value of the black box model under the condition that the above data to be analyzed is arbitrarily masked, (b) Calculate the “and interactions” in the data under analysis modeled by the black box model, (c) Calculate the "or interactions" in the data being analyzed, which are modeled by the black-box model.

[0049] In substep (a) above, the black-box artificial intelligence model is represented as v, and the data sample x to be analyzed contains n input units and is represented as the set N={1,2,...,n}, where the number of input units is generally between 5 and 20. Optionally, each dimension in the input data can be considered an input unit, or the input data can be divided into multiple regions, with each region considered an input unit. An arbitrary subset S⊆N of input units is called a combination of input units, and v(S) represents the output of the black-box model when a given specific combination S of input units is masked and the input units in N\S are masked. In particular, when the data to be analyzed is input to the model without masking, the obtained output is represented as v(N), and when the data to be analyzed is input to the model with a complete mask, the obtained output is,

number

[0050] When calculating the output v(S) of a black-box model for a specific combination of input units S, the original input values ​​of the sample under analysis are maintained for the input units included in S, but all input units in the complement N\S of S must be replaced with reference values. In particular, if the reference value is b, the masked sample x S The values ​​of each dimension are defined by the following formula:

number

[0051] In this way, v(S) is sample x S This can be calculated as the output value obtained by the input model. Selectively, the reference value can be set to the mean of the sample set, a random value, zero, etc. Furthermore, the output v(S) of the black box model calculated based on the reference value b can be specifically expressed as v(S|b), and if no ambiguity is generated, v(S|b) can be abbreviated as v(S). In one embodiment, the reference value b is set as the mean of different samples in the sample set.

[0052] In substep (b) above, the "and interactions" are calculated to evaluate and measure the contribution of the and relationships between any combination of input units to the output of the black-box model. "And interactions" refer to the additional utility generated by the output of the black-box model when the units in the combination are simultaneously triggered (i.e., not masked). It is important to understand that in a black-box model, input units often do not contribute to the model's output individually, but rather interact with each other to influence the model's output. Therefore, if S⊆N, I ANDLet (S) be defined as an "and interaction" corresponding to the combination S between input units, and if all input variables in S are triggered, then I AND (S) is triggered and accumulated in the output of the black box model. Specifically, I AND (S) can be calculated by the following formula:

number

[0053] Here, s=|S| and l=|L|. Selectively, different “and interaction” values ​​can be calculated based on different reference values ​​in step (2), and the and interaction calculated based on reference value b is I AND It is denoted as (S│b). If no ambiguity is generated, I AND (S│b) to I AND It is important to understand that this is abbreviated as (S).

[0054] The above "and interaction" refers to the fact that for any given combination of input units T⊆N, the output v(T) of the black box model in this combination of input units is all triggered and interaction I AND It can be divided into the sum of (S) and satisfy the following equation:

[0055] JPEG2026512532000021.jpg91170

[0056] In substep (c) above, the “or interaction” is calculated to evaluate and measure the contribution of any combination of input units or relationships to the output of the black-box model, where “or interaction” represents the additional utility generated by the output of the black-box model if at least one unit in the combination is not triggered (i.e., not masked). It is important to understand that in a black-box model, input units often do not contribute to the model's output individually, but rather interact with each other to influence the model's output.

[0057] Therefore, if S⊆N, I OR Let (S) be defined as an "or interaction" corresponding to a combination S between input units, and if at least one input variable in S is triggered, then I OR (S) is triggered and accumulated in the output of the black box model. Specifically, I OR (S) can be calculated by the following formula:

number

[0058] JPEG2026512532000023.jpg82170

[0059] In one embodiment of the present invention, the values ​​of "and interaction" and "or interaction" are calculated using the formulas for "and interaction" and "or interaction," respectively, based on the MLP-5 model in step 101. As shown in Figure 2 (center), a distribution chart is shown in descending order of the utility values ​​of "and interaction" and "or interaction" for different input unit combinations. Figure 3 (left) shows some of the utility values ​​of "and interaction" and "or interaction," and the figure is sorted in descending order of the absolute value of the utility values. (This embodiment is based on tabular data.)

[0060] In another embodiment of the present invention, the values ​​of "and interactions" and "or interactions" are calculated using the formulas for "and interactions" and "or interactions," respectively, based on the ResNet-34 model in step 101. As shown in Figure 4 (center), a distribution chart shows the utility values ​​of "and interactions" and "or interactions" for different input unit combinations, sorted in descending order. Figure 5 (left) shows some of the utility values ​​of "and interactions" and "or interactions," sorted in descending order of absolute utility values. (This embodiment is based on image data.)

[0061] Next, we proceed to step 104, which can be further divided into the following substeps: (a) Based on the "and interaction" and "or interaction" in step (3), the output of the black box model is interpreted as a combination of "and addition relationships" and "or addition relationships" between combinations of input units. (b) Based on the combination of "and addition relation" and "or addition relation" in substep (a), the combination of "and addition relation" and "or addition relation" is further optimized so that the expression of the two types of relations becomes more concise and sparser.

[0062] In substep (a), based on the "and interaction" and "or interaction" in step (3), the output v(T) of the black box model for any combination of input units T is interpreted as a combination of "and addition relation" and "or addition relation". In this embodiment, in substep (a) above, only one type of reference value is selected, and the output of the black box model is interpreted as a combination of "and addition relation" and "or addition relation" shown in the following equation.

[0063] JPEG2026512532000024.jpg58170

[0064] JPEG2026512532000025.jpg125170

[0065] In one embodiment of the present invention, the "and addition relationship" and "or addition relationship" calculated in Figure 2 (center) are optimized based on the method described above to obtain a more sparse interaction utility. As shown in Figure 2 (right), the figure shows a distribution of the "and interaction" utility and the "or interaction" utility in descending order for different input unit combinations, and compared to the result in Figure 2 (center), the distribution of the "and interaction" utility and the "or interaction" utility is more sparse. Figure 3 (right) shows some of the utility values ​​of the "and interaction" and the "or interaction", and the figure shows the absolute values ​​of the utility values ​​in descending order, and compared to Figure 3 (left), the sparsity of the "and interaction" and the "or interaction" is similarly improved.

[0066] In another embodiment of the present invention, the "and addition relation" and "or addition relation" calculated in Figure 4 (center) are optimized based on the method described above to obtain a more sparse interaction utility. As shown in Figure 4 (right), the figure shows a distribution of the "and interaction" utility and "or interaction" utility for different input unit combinations sorted in descending order, and the distribution of the "and interaction" utility and "or interaction" utility is more sparse compared to the result in Figure 4 (center). Figure 5 (right) shows some of the utility values ​​for "and interaction" and "or interaction," sorted in descending order of absolute value, and similarly improves the sparsity of "and interaction" and "or interaction" compared to Figure 5 (left).

[0067] A second embodiment of the present invention provides a system for interpreting sparse interaction utility modeled by a black-box artificial intelligence model, the structure of which is shown in Figure 4, and the system includes the following modules.

[0068] (1) The input module consists of a pre-trained black-box artificial intelligence model and a set of data to be analyzed. (2) The calculation module calculates the "and addition relationships" and "or addition relationships" between data input units modeled by the model, based on the model and data in the input module, and optimizes the combination of "and addition relationships" and "or addition relationships" so that all representations of the two types of relationships are more concise and sparse.

[0069] (3) The output module is configured to output combinations of "and addition relation" and "or addition relation" optimized by the calculation module.

[0070] It should be noted that those skilled in the art will understand that the functions implemented by each module shown in the embodiment of the system for interpreting sparse interaction utilities modeled by the black-box artificial intelligence model described above can be understood by referring to the description of the method for interpreting sparse interaction utilities modeled by the black-box artificial intelligence model described above. The functions of each module shown in the embodiment of the system for interpreting sparse interaction utilities modeled by the black-box artificial intelligence model described above can be implemented by a program (executable instruction) executed on a processor, or by specific logic circuits. The system for interpreting sparse interaction utilities modeled by the black-box artificial intelligence model described above in the embodiment of the present invention may also be implemented in the form of a software function module and, if sold or used as an independent product, may be stored on a computer-readable storage medium. Based on this understanding, the parts of the technical solutions of the embodiment of the present invention that are essentially or contribute to the prior art may be embodied in the form of a software product, which is stored on a storage medium and includes some instructions that enable a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the method of each embodiment of the present invention. The aforementioned storage media include various media capable of storing program code, such as USB flash drives, removable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, embodiments of the present invention are not limited to any specific combination of hardware and software.

[0071] Correspondingly, embodiments of the present invention further provide computer-readable storage media in which computer-executable instructions are stored and executed by a processor, thereby realizing embodiments of the methods of the present invention. Computer-readable storage media, including persistent and non-persistent, removable and non-removable media, can store information using any method or technique. The information may be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital purpose disc (DVD) or other optical storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or any other non-transmission media that can be used to store information accessible by a computing device. In accordance with the definitions herein, a computer-readable storage medium does not include transient computer-readable media such as modulated data signals and carrier waves.

[0072] Furthermore, embodiments of the present invention further provide a system for interpreting sparse interaction utility modeled by a black-box artificial intelligence model, comprising a memory for storing computer executable instructions and a processor, the processor configured to perform the steps of each embodiment of the method when executing the computer executable instructions in the memory. The processor may be a Central Processing Unit (CPU), another general-purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), etc. The memory may be read-only memory (ROM), random access memory (RAM), flash memory, a hard disk, or a solid-state drive, etc. The steps of the methods disclosed in each embodiment of the present invention can be performed by a hardware processor or directly implemented by a combination of hardware and software modules in the processor.

[0073] It should be noted that in the invention documentation of this patent, relational terms such as "first" and "second" are used solely to distinguish one entity or operation from another, and do not necessarily require or imply an actual relationship or order between these entities or operations. Furthermore, the terms "include," "incorporate," or any other variant thereof are intended to include non-exclusive inclusion, thereby meaning that a process, method, article, or device containing a set of elements includes not only those elements but also other elements not explicitly enumerated, or elements specific to such process, method, article, or device. Without further limitation, an element limited to the phrase "including one" does not preclude the presence of other identical elements in a process, method, article, or device containing that element. In the invention documentation of this patent, when an action is performed based on an element, it means that the operation is performed based on at least that element, including two situations: when the operation is performed based solely on that element, and when the operation is performed based on that element and other elements. Expressions such as "multiple," "multiple times," and "multiple types" include "two," "two times," "two types," and "two or more," "two or more times," and "two or more types."

[0074] All documents referenced herein are deemed to be included in the disclosure of the present invention as a whole and may be used as grounds for modification as necessary. Furthermore, it should be understood that the foregoing are merely preferred embodiments of this specification and are not intended to limit the scope of protection herein. Any modifications, equivalent substitutions, alterations, etc., made within the spirit and principles of one or more embodiments herein are intended to be included within the scope of protection of one or more embodiments herein.

Claims

1. A method for measuring the sparse interaction utility of a black-box artificial intelligence model, The following steps are included: (1) Step of providing a model The analysis provides a black-box artificial intelligence model, which includes an artificial intelligence model pre-trained on a specific dataset. (2) Steps to acquire the data to be evaluated Based on the aforementioned black-box artificial intelligence model, evaluation data compatible with the artificial intelligence model is acquired, where the evaluation data includes n input units, where n is a positive integer ≥ 4, and where steps (1) and (2) can be performed interchangeably or simultaneously. (3) A step of calculating the “and interaction” and “or interaction” between the input units. The evaluation target data is input to the black-box artificial intelligence model, and the output of the black-box artificial intelligence model is obtained, and the interaction between the input units is modeled based on the output of the black-box artificial intelligence model, thereby obtaining a set consisting of different input units, recording it as a combination S of input units (or evaluation set S), and calculating the "and interaction" and "or interaction" of each combination S of input units. (4) Steps to measure the utility of sparse interaction The method is characterized in that, based on the "and interactions" and "or interactions" between combinations of input units S obtained in step (3), the "and addition relationships" and "or addition relationships" between all combinations of input units S are calculated, and the output of the black box model is expressed as a combination of the "and addition relationships" and "or addition relationships" between combinations of input units, and is used to interpret the sparse interaction utility of the black box artificial intelligence model.

2. The aforementioned method, (5) The method further includes the steps of optimizing the "and addition relationship" and the "or addition relationship", increasing the degree of sparseness of the combination of the optimized "and addition relationship" and the "or addition relationship", and interpreting the sparse interaction utility of the black-box artificial intelligence model using the optimized "and addition relationship" and the "or addition relationship". The method according to claim 1.

3. "and interaction" in step (3) above AND (S) is calculated via the following formula: [Number 13] Here, s = |S| and l = |L|, where s is the number of input units in a specific combination S of input units, and is recorded as s = |S|, l is the number of input units in a combination L between input units, and is recorded as l = |L|, where the combination L between input units is an arbitrary subset of the combination S of input units, and the method for calculating v(L) in the formula is as follows, where n input variables are recorded as N = {1, 2, ..., n}, and v(L) holds only the input units in the combination L between input units in the data to be evaluated, masking the other input units to a reference value b, where the reference value b is a quantity having the same dimensions as the data to be evaluated, where the reference value b can be selected as the mean, random value, zero value, etc. of the input sample set, and the masked data to be evaluated x L The output value obtained by inputting the data into the black box model is recorded as v(L), specifically the masked evaluation data x L Each input unit is characterized by being defined by the following formula. The method according to claim 1. [Number 14]

4. "or interaction" in step (3) above OR (S) is calculated via the following formula: [Number 15] Here, s = |S| and l = |L|, where s is the number of input units in a particular combination S of input units, and is recorded as s = |S|, and l is the number of input units in a combination L between input units, and is recorded as l = |L|, and the combination L between input units is an arbitrary subset of the combination S of input units. The method according to claim 1.

5. In step (4) above, the output v(T) of the black box model is interpreted as a combination of "and addition relation" and "or addition relation" via the following equation: 【Number 16】 Here, the output v(N) of the black box model is interpreted as the sum of the utility of the "and interaction" of group m and the utility of the "or interaction" of group 1, b (１) , ..., b (m) This is expressed as m reference values, where m is the number of reference values, and the reference values ​​m can be selected as the mean, random values, zero values, etc., and the "and interaction" of each group of these reference values ​​is 2 n "and interaction" utility including combinations between individual input units [Number 17] This includes, where the total utility of the “or interaction” in that group is 2 n Utility of "or interaction" within combinations of individual input units [Number 18] is included, and the m different reference values can be selected as an average value, a random value, a zero value, etc. ∀ in the formula is expressed as "arbitrary", and e in the formula S is 1 when i ∈ S, and (e T ) i = 1; otherwise, (e T ) i = 0, satisfying the condition that Λ i∈S (e T ) i represents whether all units in the combination S of the input units are triggered. The result obtained by "and" operating on the elements after the ∧ symbol is Λ i∈S (e T ) i = Π i∈S (e T ) i . ∨ i∈S (e T ) i represents whether at least one unit in the combination S of the input units is triggered. The result obtained by "or" operating on the elements after the ∨ symbol is ∨ i∈S (e T ) i = 1 - Π i∈S [1 - (e T ) i , characterized in that The method according to claim 1.

6. In step (5) above, a more sparse interaction relationship is obtained through the following equation, [Number 19] Here, [Number 20] And, [Math 21] And here, ||・|| １ L is the vector １ - Represents the norm, where m in the above formula is the number of selected reference values, and the reference values ​​include the mean, random values, and zero values ​​of the input sample set. The method according to claim 2.

7. The formula described in claim 6 can be simplified by the following method: [Number 22] It is defined as and p （1） +...+p (m) +p (or) = 0 can be satisfied, and the above p （1） , ..., p (m) , p (or) This represents the partitioning of the neural network output v, and [Number 23] It is defined as follows, where q is a vector with small absolute values ​​in each dimension, representing the deviation resulting from interpreting the model's output as an "and additive relation" and an "or additive relation," and therefore, [Number 24] If this is the case, the above optimization problem can be transformed into the following form: [Number 25] Here, τ i This is a pre-set threshold, preferably, [Number 26] The range of the value of m is 1 to 10. The method according to claim 6.

8. Step (3) above is, If the data to be analyzed is masked by any of the input units, the step of calculating the output value of the black box model is further characterized in that the analysis data is masked by any of the input units. The method according to claim 1.

9. The data to be evaluated is selected from a group consisting of grid data, images, text, audio, or a combination thereof. The method according to claim 1.

10. A system for interpreting the sparse interaction utility modeled by a black-box artificial intelligence model, (1) An input module configured to consist of one pre-trained black-box artificial intelligence model and one set of data to be analyzed, (2) A calculation module configured such that, based on the model and data in the input module, it calculates the "and addition relationship" and the "or addition relationship" between data input units modeled by the model, and optimizes the combination of the "and addition relationship" and the "or addition relationship" so that all representations of the two types of relationships become more concise and sparse, (3) A system for interpreting sparse interaction utility modeled by the black-box artificial intelligence model, comprising an output module configured to output combinations of "and addition relationships" and "or addition relationships" optimized by a computation module.

Images