Explaining neural networks using logic

Abstract

The explanation engine has a set of interconnected modules configured to evaluate layers in a hierarchical architecture of a machine-based reasoning process using machine learning. The set of modules cooperate to support an explanation of how the machine-based reasoning process reached a reported result, both a final / top-level result and corresponding intermediate output results. A messaging module of the explanation engine can collect the top-level result and one or more intermediate output results from the intermediate layers of the machine-based reasoning process. To communicate how the machine-based reasoning process reached its reported result in a communication, the multiple reasoning layers are associated with terminology used in at least one of i) the problem to be solved and ii) the domain related to the problem.

Description

[Technical field]

[0001] cross reference This application claims priority under 35 U.S.C. § 119 to U.S. Provisional Patent Application No. 63 / 030,699, entitled "NEURAL NETWORK EXPLANATION USING LOGIC," filed May 27, 2020, the disclosure of which is incorporated herein by reference in its entirety.

[0002] TECHNICAL FIELD Embodiments of this disclosure relate generally to artificial intelligence based inference engines. [Background technology]

[0003] Today, the problem of understanding the output from computational reasoning is typically addressed through post-hoc explanations, which may not match what the system actually does, leading to a mismatch between the explanation provided and the actual factors that cause the system to run. Summary of the Invention

[0004] Provided herein are descriptions of various methods, apparatus and systems for an artificial intelligence based inference engine and its inference process.

[0005] The explanation engine has a set of cooperating modules configured to evaluate layers in a hierarchical architecture of a machine-based reasoning process using machine learning. The set of modules cooperate to support an explanation of how the machine-based reasoning process reached a reported result, both a final / top-level result and a corresponding intermediate output result. A messaging module of the explanation engine can collect one or more intermediate output results and a top-level result from the intermediate layers of the machine-based reasoning process. To communicate how the machine-based reasoning process reached its reported result in a communication, the multiple reasoning layers are associated with terms used in at least one of i) the problem to be solved, and ii) the domain related to the problem.

[0006] These and many other embodiments are discussed. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram of one embodiment of an exemplary explanation engine having a set of modules for evaluating layers in a hierarchical architecture of a machine-based reasoning process.

[0008] [Diagram 2] FIG. 1 illustrates one embodiment of an exemplary inference engine configured to decompose its machine-based inference process into divisible layers that provide intermediate output results to other layers and a top-level result.

[0009] [Diagram 3] FIG. 1 illustrates one embodiment of a block diagram of exemplary layers in a hierarchical architecture of a machine-based reasoning process created by an inference engine using terminology used in at least one of: i) the problem being solved; and ii) the domain related to the problem.

[0010] [Figure 4]FIG. 1 is a block diagram of one embodiment of an exemplary machine-based reasoning process and an explanation engine module configured to interact with the machine-based reasoning process to crawl a decomposition of the machine-based reasoning process, collect information, and then report intermediate output results from each layer of the reasoning process and explain top-level results in terms of the intermediate output results.

[0011] [Diagram 5] FIG. 1 is a block diagram of one embodiment of an exemplary problem-specific terminology, where the terminology assigned to each layer of the machine-based reasoning process can be obtained directly from a user-provided documented description of the problem and / or extracted from a domain-specific database.

[0012] [Figure 6] FIG. 13 is a block diagram of one embodiment of an example communication generated by a messaging module that works in conjunction with an ablation module to trace back each intermediate layer of the machine-based reasoning process and record the factors considered and how important they were in arriving at the top-level result from the machine-based reasoning process.

[0013] [Figure 7] FIG. 13 is a block diagram of an embodiment of another page in a communication generated by a messaging module that describes potential errors or improvements in the machine-based inference process.

[0014] [Figure 8] FIG. 1 is a diagram of several electronic systems and devices communicating with each other in a network environment, according to one embodiment of an explanation engine in cooperation with an inference engine.

[0015] [Figure 9] FIG. 1 illustrates an embodiment of one or more computing devices that may be part of a system associated with an explanation engine that cooperates with the inference engine and its associated models discussed herein.

[0016] While the design may be subject to various modifications, equivalents, and alternative forms, specific embodiments thereof have been shown by way of example in the drawings and will now be described in detail. It is to be understood that the design is not limited to the particular embodiments disclosed, but on the contrary, is intended to encompass all modifications, equivalents, and alternative forms employing the particular embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] In the following description, numerous specific details may be described, such as examples of specific data signals, referenced components, frame numbers, etc., to provide a thorough understanding of the design. However, it will be apparent to one of ordinary skill in the art that the design may be practiced without these specific details. In other instances, well-known components or methods have not been described in detail, but rather shown in block diagrams, to avoid unnecessarily obscuring the design. Furthermore, reference may be made to specific numbers, such as a first server. However, the specific reference to numbers should not be interpreted as a literal order, but rather that the first server is different from the second server. Thus, the specific details described may be merely examples. Those specific details may vary from the spirit and scope of the design, but may still be contemplated to be included within the spirit and scope of the design. The term "coupled" is defined to mean directly connected to the component, or indirectly connected to the component through another component.

[0018] FIG. 1 illustrates a block diagram of one embodiment of an exemplary explanation engine having a set of modules for evaluating layers in a hierarchical architecture of a machine-based reasoning process.

[0019] The explanation engine 100 comprises a set of cooperating modules that evaluate layers of a hierarchical architecture (including layers implemented in neural networks, logic rules, linear layers, etc., and any combination of these) of a machine-based reasoning process that uses machine learning to support detailed explanations of how the machine-based reasoning process arrived at a reported result, both the top-level result and corresponding intermediate output results.

[0020] The explanation engine 100 has a terminology module configured to accept input of terminology for the problem to be solved by the inference engine and the machine-based inference process, provided by at least one of: i) a problem to be solved from a user; ii) a description of a preferred approach to solving the problem from the user; and iii) a database of known terminology specific to the domain related to the problem. Note that terminology can refer to the understanding of the meaning and appropriate use of linguistic constructs such as terms, symbols, acronyms, etc. used in the subject domain and / or the described problem to be solved. The terminology module of the explanation engine 100 also matches the terminology (e.g., words, symbols, acronyms, etc.) to the domain and specific problem at hand, layer by layer in the hierarchical architecture of the machine-based inference process from the inference engine.

[0021] It should be noted that the explanation engine 100 uses a single / common explanation methodology across multiple different domains and different types of problems, thereby enabling any of its users to match explanations generated by the inference engine and its machine-based inference process with explanations they receive from generated communications such as reports, emails, etc. The generated communications contain further information that fully explains how the machine-based inference / learning arrived at the output results in language specific to the subject domain or problem at hand.

[0022] The modules of the explanation engine 100 attempt to provide an explanation of what decisions and factors led to a final / top-level result from a machine-based reasoning (e.g., a computer-generated machine learning reasoning) by providing one or more intermediate decisions made to arrive at a generated final / top-level result. The intermediate decisions of how the machine learning reasoning arrived at its generated result are included in the communication from the explanation engine 100 to support a user in understanding why a particular final / top-level result was provided by an automated machine learning system using machine-based reasoning. Thus, the explanation engine 100 can enable a person to better utilize the top-level and intermediate results and recognize when a result is incorrect, and can increase a user's confidence in the generated top-level result of the machine-based reasoning.

[0023] The ablation module of the explanation engine 100 can perform an ablation cycle to remove and / or otherwise alter a layer of the machine-based inference process, and then go back through the inference tree / layers created by the inference engine to determine the impact of the layer on the final / top-level result. The ablation module can include comparison and loss functions to aid in the decision.

[0024] The crawlback module of the explanation engine 100 is configured to backtrack through each step of the machine-based reasoning process (eg, a decision process) and collect data for each tier in the reasoning process.

[0025] The crawlback module cooperates with the messaging module to collect and communicate the top-level results and one or more intermediate output results from the intermediate layers of the machine-based inference process. The messaging module may also include a comparison function and a loss function.

[0026] The modules of the explanation engine 100 work together to provide the ability for systems with machine-based reasoning processes using machine learning to more efficiently support humans in their work, allowing users / humans to have more confidence in the output of such systems, and assisting in the generation of a broader range of automated reasoning adaptations.

[0027] FIG. 2 illustrates one embodiment of an exemplary inference engine configured to decompose its machine-based inference process (e.g., decision process) into divisible layers / intermediate steps that provide intermediate output results to other layers and a top-level result.

[0028] An inference engine 210, such as Deep Adaptive Semantic Logic (DASL) by SRI International, generates layers and layer flows of a machine-based inference process. The inference engine 210 builds hierarchical inference trees / processes of individual neural networks, logic rules, linear layers, etc. into a hierarchical structure, such as a tree structure or other hierarchical structure (see, e.g., FIG. 3).

[0029] The exemplary inference engine 210 may have a theory module, a language module, a model representation module, and other exemplary modules, each performing a unique function in incorporating the problem from the user, converted into layers, into a hierarchical architecture of logic and machine-based inference processes implemented in software.

[0030] The converter module can take the input of what the user wants and any rules, in human terms, supplied by the user via the theory module, which is then converted via additional modules in the inference engine 210 into a machine-based inference process implemented via logic rules, linear layers, neural networks, and any combination of these.

[0031] The theory module of the inference engine 210 provides the conversion of user words into logic rules that machine learning can understand and use, and is configured to create linear layers, neural networks, and logic rules that combine multiple complex outputs, for example, built on the DASL framework implementing larger theories using DASL and evaluating smaller sub-theories used in the inference process. The inference engine 210 creates layers of the machine-based inference process as neural networks, logic rules, linear layers, etc., and performs specific inference functions in the layers. See, for example, different functional layers in the inference process layers, such as method logic, associated Obs filter, ∀ indicator, alert, etc. in FIG. 3.

[0032] In the language model and model representation module, the inference engine 210 compiles the user-documented domain-specific language corresponding to the modified version of the first-order logic into neural networks or the like, then wires / implements the flow connections together, and then creates an overall hierarchical architecture of a machine-based inference process that the learner algorithm module can train. Each layer can be trained individually to perform its function and generate its inferred intermediate output results.

[0033] When the code for the current problem is executed, each layer / function block produces its inferred intermediate output, which is taken into account in the top-level result. The intermediate output results from all layers of inference go into the final layer of inference, such as the warning layer in Figure 3, and the top-level result is produced.

[0034] The inference engine 210 can use a library of neural networks to compile neural networks, such as a linear layer, and select an appropriate neural network for the layer's functions. The inference engine 210 can build logical rules using simple logic from a domain-specific database of known terms and known rules for the user and / or the domain. Each layer of the machine-based inference process calculates intermediate output results until the final layer produces a top-level result.

[0035] The inference engine 210 can build the architecture of the machine inference process to solve the problem, and can also build the modules of the explanation engine at the same time as the tool extracts and formats the explanation from the machine inference process. Thus, the explanation engine can work with an inference engine 210 that is configured to decompose its machine-based inference process into decomposable layers that provide intermediate output results to other layers to ultimately determine the top-level results from the machine-based inference process, as opposed to another type of inference engine 210 that is configured to simply create one omnibus neural network compiled as a black box that outputs its final decision. The explanation engine can work with the inference engine 210 to allow a user to query what the intermediate output results are for each layer / step in the machine-based inference process, and what happens if the intermediate output results are changed.

[0036] The explanation engine's ablation module runs an ablation cycle that removes layers / blocks from the inference process and compares the ablated results to the true results to determine the influence of the layer on the true top-level results from the machine-based inference process.

[0037] The crawlback module can also extract outputs and explanations from the inference engine 210 by first querying the inference engine 210. The terminology module can format the explanations into terms understandable by the user. The explanation engine then explains the results of each layer in terms of the contributions of each lower layer based on the ablation. Thus, the network of machine-based inference processes explains larger logic theory outputs in terms of domain-specific smaller logic theory outputs and in terms related to the problem at hand. The main advantage of the explanation engine is its ability to explain the results generated by the inference engine 210 with machine-based inference processes in terms that the user understands.

[0038] Again, the hierarchical architecture of the inference process using machine-based learning can use layers in the inference created using low-level neural networks, logic rules, linear layers, etc. In one embodiment, a linear layer can be used to learn the average rate (or average value, etc.) of correlation between the output and the input. For example, if x and y are positively correlated ⇒ w is positive and an output result is generated. In one embodiment, the neural network can be a set of algorithms intended to recognize patterns and interpret data through clustering or labeling, recognize underlying relationships in a set of data, and generate an output result. Typically, the construction of a neural network is used to solve more complex sub-theories / layers having multiple variables or other complexity challenges. A set of logic rules can be translated from user-supplied rules and / or domain-specific known rules into the structure of the created machine-based inference process. Typically, the logic rules can be used to solve simpler and / or easier sub-theories / layers. The logic rules can also be used for other simple decisions in the inference process such as which inference layer flows to another inference layer.

[0039] It should be noted that machine learning can employ many exemplary types of neural networks, such as feedforward neural networks, radial basis function neural networks, Kohonen self-organizing neural networks, recurrent neural networks, convolutional neural networks, etc., and any combinations thereof. Other types of related networks are transformers, generative adversarial networks, restricted Boltzmann machines, etc.

[0040] In one embodiment, further details regarding how the exemplary inference engine 210 translates a user's problem description into layers that make up the hierarchical architecture of a machine-based inference process can be found in U.S. Patent Application No. 16 / 611,177, filed November 5, 2019, entitled "DEEP ADAPTIVE SEMANTIC LOGIC NETWORK," U.S. Patent Application Publication No. 2020 / 0193286, published June 18, 2020, which is incorporated herein by reference.

[0041] FIG. 3 illustrates one embodiment of a block diagram of exemplary layers in a hierarchical architecture of a machine-based reasoning process created by an inference engine using terminology used in at least one of: i) the problem being solved; and ii) the domain related to the problem.

[0042] In this example, the layers in the hierarchical architecture of the machine-based inference process 320 are: related Obs filter layer, indicator predictor layer, method logic layer, weighting and summing layer, warning layer, ∀ method layer, ∀ indicator layer, ∀ warning layer, true warning layer and loss layer. The warning layer in this example produces the final top-level result of the score. The example layers are derived from the user's documented description (see, e.g., FIG. 5) and / or other terminology in the problem domain.

[0043] FIG. 4 shows a block diagram of one embodiment of an exemplary machine-based reasoning process 320 and an explanation engine module configured to interact with the machine-based reasoning process 320 to crawl a decomposition of the machine-based reasoning process 320, collect information, and then report intermediate output results from each layer of the reasoning process and explain top-level results in terms of the intermediate output results.

[0044] 3 and 4, the terminology module 450 is configured to crawl the generated hierarchical architecture of the machine-based reasoning process 320 (e.g., an inference tree) created by the inference engine and then associate i) problem-specific terms and / or related subject area-specific terms with ii) each of the layers that make up the hierarchical architecture of the machine-based reasoning process 320.

[0045] For this purpose, the true observation layer is created by the inference engine to collect all true observations that need to be input to solve the problem from FIG. 5 as an example, and to determine the presence of a dangerous situation based on the observations. The terminology module 450 can assign labels of the layers created by the inference engine to be true observations, which are derived from the problem description submitted by the user. The output results of the true observation layer can be provided as input to the inference layer, which is labeled with the relevant Obs (observation) filter layer. The relevant Obs (observation) filter layer is assigned a name based on the problem and the subject domain. The relevant Obs filter layer can be constructed as a simple linear layer to take inputs from the true observation layer and filter relevant observations for the problem of determining the presence of a dangerous situation based on observations from observations that have not been found to be useful. Similarly, the indicator predictor layer can be constructed as a neural network that takes inputs from the relevant Obs filter layer and the ∀ indicator layer. In many cases, the flow of the inference layer between different blocks of the layer can be implemented through logic rules that can be obtained from i) directly supplied by the user in the description of the problem statement, ii) implicit to the subject matter domain extracted from a specific subject matter domain database, or iii) even based on past or expert inputs that have been found to be generally accurate / useful. The rest of the inference layer is similar to an inference engine that creates the implementation of that inference layer / step, and a terminology module 450 that maps labels to that layer, which are derived from the problem description from the user and / or extracted from a database. Typically, in this example, the output results from the previous layer flow to the subsequent lower layer. Finally, in this example, the warning layer / step takes input from the weighting and summing layer above, which takes into account the inputs and intermediate output results from other inference layers, and then outputs the inferred top-level results of the output.

[0046] The explanation engine and its modules can then perform their functions to help a user understand how the exemplary hierarchical architecture of the machine-based reasoning process 320 arrived at its output for the problem of determining the existence of a hazardous situation based on observations. The explanation engine can explain the results of each layer in terms of the contribution of each of the layers based on ablation, and crawl back to collect the contributions and intermediate output results.

[0047] The inference engine builds this hierarchical inference tree / process of individual neural networks, logic rules, and linear layers into a hierarchical structure, such as a tree structure, and runs a machine-based inference process 320 that is run by a combination of machine learning and logic rules, and then captures the output results of each intermediate output result from the layers that make up that hierarchical inference process, puts these output results into language used by people in that field of work, and creates an explanation of what the intermediate inferences were in terms (e.g., words, symbols, etc.) that a user can easily understand.

[0048] To this end, a user can provide a documented description of a problem, which can be fed into an inference engine such as DASL, which then uses machine learning to compile a machine-based inference process 320 into neural networks, linear layers, logic rules, etc., and then the explanation engine does its job.

[0049] The terminology module 450 can receive input of problem terms and domain-specific terms from words, symbols, etc. that a user has provided as input to the explanation engine and / or that are derived from a database of known rules and domain-specific words. The theory module of the inference engine can cooperate with the terminology module 450 to provide the terminology.

[0050] The crawlback module 460 cooperates with the messaging module 480. The explanation engine crawlback module 460 is configured to crawl a decomposition of the machine-based reasoning process 320 to collect intermediate output results from multiple layers of the reasoning process, and then report and explain top-level results in terms of the intermediate output results.

[0051] The crawlback module 460 also works in conjunction with the ablation module 470 to trace back each intermediate layer of the machine-based inference process 320 built by the inference engine and record the factors being considered and how important they were in arriving at the top-level result from the machine-based inference process 320, which is then presented in a communication from the messaging module 480.

[0052] The explanation engine has an ablation module 470 configured to remove each intermediate layer of the machine-based inference process 320 one at a time, run the inference engine on the results, and then return them for the next ablation cycle to remove a different layer. Each time, the results from the ablation cycle of removing a layer are compared to the initial final result of the system with the inference process using machine learning when all layers of the inference process are taken into account in the decision. Based on the comparison, the messaging module 480 of the explanation engine can determine how much impact each individual layer directly associated with the words / terms used in the user-written explanation has on the determination of the final result of the system with the inference process using machine learning when all layers are taken into account in the decision. The user can trigger and instruct the ablation module 470 by asking a question / query to the explanation engine. The ablation module 470 removes a layer from the inference flow by changing the inputs, such as zero weights for the layer. The explanation engine has an ablation module 470 configured to modify the output results from a layer from the machine-based reasoning process 320 by changing the inputs to that layer and record the new output results from that layer of the machine-based reasoning process 320 and the new top-level results.

[0053] The ablation module 470 performs one or more ablation cycles that vary the inputs to a layer of the machine-based inference process 320 created by the inference engine, and determines and records the impact of that layer on the top-level results. Thus, the ablation module 470 can vary the input data to a layer and explain the results of each layer in the inference process in terms of the contribution of each lower layer based on the ablation (removal / analysis) of one layer at a time (as determined analytically or through small experiments).

[0054] The messaging module 480 is then configured to obtain all of the output results of the ablation cycle and the data generated using these to generate a report result of the impact of each tier of the machine-based reasoning process 320 on the communications generated by the messaging module 480.

[0055] The explanation engine messaging module 480 is configured to 1) extract intermediate output results from each layer of the machine-based reasoning process 320 created by the inference engine, and 2) cooperate with the terminology module 450 to associate the intermediate output results from each layer with terminology obtained from at least one of i) the subject matter domain, and ii) problem-specific terminology presented in the generated communication.

[0056] The messaging module 480 and / or the ablation module 470 can use comparison and loss functions to determine the impact of changing the inputs and / or removing layers of inference. In this example (see FIG. 6), the messaging module 480 breaks down the top-level results (e.g., scores) through the ablation of the method, indicator, and alert layers in the example generated communication.

[0057] FIG. 5 shows a block diagram of one embodiment of an exemplary problem-specific terminology, where the terminology assigned to each layer of the machine-based reasoning process can come directly from a user-provided documented description of the problem and / or be extracted from a domain-specific database.

[0058] A user can submit a problem 525 (in this case just an example problem) to the theory module of the inference engine.

[0059] for example: "The goal is to determine, on the basis of observations, in an explainable manner, the existence of a hazardous situation. The inference engine compiled the machine-based inference process decomposed into sub-theories / layers and explained its inferences: High level warnings (e.g. enemy advance) are broken down by possible avenues - land, sea, air, etc. Each method (eg, ground attack) is described in terms of the low-level indicators of that attack - mobilization, transportation, materiel concentration, etc. Each indicator is assumed to be a function of a subset of all potentially relevant observations, and this function is trained as a neural network. The warnings are explained through methods, the methods are explained through indicators, and the indicators are finally explained through observations."

[0060] Thus, the inference engine solves this problem 525 posed by the user by building a problem-specific high-level flow of inference layers to capture the machine-based inference process within the layers used by machine learning (e.g., neural networks, linear layers, logic rules) and associates domain-specific names with the layers using understandable terminology.

[0061] FIG. 6 illustrates a block diagram of one embodiment of an example communication generated by a messaging module that works in conjunction with an ablation module to trace back each intermediate layer of the machine-based reasoning process and record the factors being considered and how important they were in arriving at the top-level outcome from the machine-based reasoning process.

[0062] The messaging module of the explanation engine generates the resulting communication 635 using problem-specific logical connectives and linear weights that fuse the outputs of the low-level neural networks, the logical rules and / or the outputs of the linear layers.

[0063] An exemplary communication 635 may present intermediate results from multiple tiers along with their problem and / or domain specific names, as well as a final top-level result, e.g., “Score,” and a score for the intermediate output results.

[0064] Communication 635 presents the layer scores prior to any ablation cycle and the impact of the ablation cycle, thus presenting each layer's contribution to the output "score" from the top level results from the alert layer. "1A. By the following<enemy name> Attacks by: 1A-1. Air attacks are predicted (3.8 / 5.0, False without), and 1A-2. Ground attack expected (2.5 / 5.0, False without). 1A-1i. The following predict air attacks (p=.75): refueling of combatants (p=.75, 0.0 without) and mobilization of aircrew (p=.8, 0.0 without) and (withdrawal of diplomatic missions (p=1.0, .08 without) or severing diplomatic relations (p=.1, .75 without)). 1A-1ii. Combatant fuel injection (.75) was demonstrated by: Aerial refueling aircraft was observed (.45 without) Increase in fuel truck count (.55 without) Increased activity in fuel depots (.6 without).”

[0065] Note that the reasoning layer outputs numbers, such as scores, that are in the language of machine-based reasoning. However, the terminology module assigns terms from any of the domains relevant to the problem, and the particular problem at hand, for each layer in the hierarchical architecture of the machine-based reasoning process created by the inference engine, allowing a human user to match descriptions with terms that the human user can understand in the communications 635 generated by the messaging module. Thus, the user can understand the results in terms of a particular problem or domain, based on the words and format in which the communications are generated.

[0066] In the above example, the ablation module runs an ablation cycle, zeroing out aspects of the inputs into a given layer, and runs the inference engine to see what the new values ​​are for the intermediate results from the functional layers during the ablation cycle and / or the new top-level results / scores.

[0067] The messaging module then takes all the results of the ablation cycle and the data generated with these and generates a generated communication of the impact of each functional block / layer on the final result from the inference engine.

[0068] The explanation engine can be used with any system that has a machine-based reasoning process (e.g., using machine learning), such as in military applications, autonomous vehicle applications, virtual assistant applications, etc., to explain why a virtual assistant arrived at its recommendation (e.g., the top result).

[0069] Note that one or more parts of the explanation engine can be coded in a Python file.

[0070] FIG. 7 illustrates a block diagram of one embodiment of another page in a communication generated by a messaging module that describes potential errors or improvements in the machine-based inference process.

[0071] The messaging module can review the data collected by the crawlback and ablation modules and then use some artificial intelligence algorithm to account for potential errors or improvements in the machine-based inference process and incorporate this in the communication 735. Thus, the generated results can be presented with either 1) additional pre-hoc information when similar learning examples are not available, but still suggesting how this learning can be performed differently, and 2) additional ad-hoc information based on past / historical learning examples that the user may want to change the weights, other learning parameters, and learning algorithms differently than currently implemented. This additional information can optionally be indicated in the communication. For example, the messaging module can provide the following example errors and / or suggestions made in the communication 735: □ Train an explainable network as usual, but add additional networks in parallel at a given layer (e.g., add a separate network for each method that predicts using indications); Add a second round of training freeze on explainable networks, training only new black-box networks that are unexplainable; and □Using ablation, we obtain explanations based on new black-box networks.

[0072] Note that the modules of the explanation engine work together to replace black-box results by using at least an ablation of each of the network inputs to understand the impact of the layers on the inference process and make suggestions on its structure.

[0073] network FIG. 8 illustrates a diagram of several electronic systems and devices communicating with each other in a network environment, according to one embodiment of an explanation engine in cooperation with an inference engine. The network environment 800 has a communication network 820. The network 820 may include one or more networks selected from an optical network, a cellular network, the Internet, a local area network ("LAN"), a wide area network ("WAN"), a satellite network, a fiber network, a cable network, and combinations thereof. In one embodiment, the communication network 820 is the Internet. As illustrated, there may be multiple server computing systems and multiple client computing systems connected to each other via the communication network 820. However, it should be understood that, for example, a single client computing system may also be connected to a single server computing system. Thus, any combination of server computing systems and client computing systems may be connected to each other via the communication network 820. As discussed, the inference engine may use such a network to provide training data for creating and training a neural network. The explanation engine in cooperation with the inference engine may also reside and be implemented in this network environment, for example in the cloud platform of the server 804A and database 806A, in the local server 804B and database 806B, on devices such as the laptop 802D, in smart systems such as the smart car 802D, in particular the cloud platform server 804A, in particular in devices such as the laptop 802D where the two systems communicate and cooperate with each other, and other similar platforms.

[0074] A communication network 820 can connect one or more server computing systems selected from at least a first server computing system 804A and a second server computing system 804B to each other and further to at least one or more client computing systems. The server computing system 804A may be, for example, one or more server systems 220. The server computing systems 804A and 804B can each optionally include organizational data structures such as databases 806A and 806B. Each of the one or more server computing systems can have one or more virtual server computing systems, and the multiple virtual server computing systems can be implemented by design. Each of the one or more server computing systems can have one or more firewalls to protect data integrity.

[0075] The at least one or more client computing systems may be selected from a first mobile computing device 802A (e.g., a smartphone having an Android-based operating system), a second mobile computing device 802E (e.g., a smartphone having an iOS-based operating system), a first wearable electronic device 802C (e.g., a smart watch), a first portable computer 802B (e.g., a laptop computer), a third mobile computing device or a second portable computer 802F (e.g., a tablet having an Android or iOS-based operating system), a smart device or system incorporated in a first smart automobile 802D, a smart device or system incorporated in a first smart bicycle 802G, a first smart television 802H, a first virtual reality or augmented reality headset 804C, etc. Client computing system 802B may be, for example, one of one or more client systems 210, and any one or more of the other client computing systems (e.g., 802A, 802C, 802D, 802E, 802F, 802G, 802H, and / or 804C) may include, for example, software applications or hardware-based systems in which artificial intelligence may be trained and / or deployed. Each of the one or more client computing systems may have one or more firewalls to protect data integrity.

[0076] It should be understood that the use of the terms "client computing system" and "server computing system" is intended to indicate a system that generally initiates a communication and a system that generally responds to a communication. For example, a client computing system can generally initiate a communication and a server computing system generally responds to a communication. No hierarchy is implied unless explicitly stated. Both functions may be within a single communication system or device, in which case the client-server and server-client relationships can be viewed as peer-to-peer. Thus, if the first portable computer 802B (e.g., a client computing system) and the server computing system 804A can both initiate and respond to a communication, their communication can be viewed as peer-to-peer. In addition, the server computing systems 804A and 804B include circuitry and software that allows them to communicate with each other over the network 820. The server 804B can, for example, send simulator data to the server 804A.

[0077] Any one or more of the server computing systems can be a cloud provider. The cloud provider can install and run application software in the cloud (e.g., a network 820 such as the Internet), and the cloud users can access the application software from one or more of their client computing systems. In general, a cloud user with a cloud-based site in the cloud cannot single-handedly manage the cloud infrastructure or platform on which the application software runs. Thus, the server computing system and its organizational data structures can be a shared resource, and each cloud user is given dedicated use of a certain amount of the shared resource. Each cloud user's cloud-based site can be given dedicated space and bandwidth of a virtual amount in the cloud. Cloud applications can differ from other applications in their scalability, which can be achieved by copying tasks to multiple virtual machines at run time to meet changing work demands. A load balancer distributes work across a set of virtual machines. This process is transparent to the cloud users who only see a single access point.

[0078] The cloud-based remote access can be coded to utilize protocols such as Hypertext Transfer Protocol ("HTTP") to engage in request and response cycles with applications on a client computing system, such as a web browser application resident on the client computing system. The cloud-based remote access can be accessed by a smartphone, desktop computer, tablet, or any other client computing system, anytime and / or anywhere. The cloud-based remote access is coded to engage in 1) request and response cycles from all web browser-based applications, 3) request and response cycles from a dedicated online server, 4) direct request and response cycles between native applications resident on the client device and the cloud-based remote access to another client computing system, and 5) combinations thereof.

[0079] In one embodiment, the server computing system 804A may include a server engine, a web page management component or direct application component, a content management component, and a database management component. The server engine may perform basic processing and operating system level tasks. The web page management component may handle the creation and display or routing of web pages or screens related to receiving and serving digital content and digital advertisements via a browser. Similarly, the direct application component may interface with client apps residing on the user's device. A user (e.g., a cloud user) may access one or more of the server computing systems using a uniform resource locator ("URL") associated therewith. The content management component may handle most of the functions in the embodiments described herein. The database management component may include storage and retrieval tasks related to databases, queries to databases, and storage of data.

[0080] In one embodiment, the server computing system may be configured to display information in windows, web pages, etc. Applications, including any program modules, applications, services, processes, and other similar executable software, when executed on, for example, the server computing system 804A, may cause the server computing system 804A to display windows and user interface screens in a portion of the display screen space.

[0081] Each application has code written to perform the functions that the software components are coded to perform, such as presenting fields to obtain details of desired information. Algorithms, routines, and engines, for example in the server computing system 804A, can obtain information from the presented fields and place the information in an appropriate storage medium, such as a database (e.g., database 806A). A comparison wizard can be written to reference the database and can utilize such data. The application may be hosted, for example, on the server computing system 804A, or may be provided to a specific application or browser, for example, on the client computing system 802B. The application then provides a window or page that allows for the input of details.

[0082] Computing Systems 9 illustrates a diagram of one embodiment of one or more computing devices that may be part of a system associated with an explanation engine that cooperates with the inference engine and its associated models discussed herein. The computing device 900 may include one or more processors or processing units 920 for executing instructions, one or more memories 930-932 for storing information, one or more data entry components 960-963 for accepting data entry from a user of the computing device 900, one or more modules including a management module, a network interface communication circuitry 970 for establishing communication links for communicating with other computing devices external to the computing device, one or more sensors, where output from the sensors is used to detect certain trigger conditions and generate one or more pre-programmed actions in response thereto, a display screen 991 for displaying at least a portion of the information stored in the one or more memories 930-932 and other components. It should be noted that the portions of this system implemented in software 944, 945, 946 may be stored in one or more memories 930-932 and executed by one or more processors 920.

[0083] The system memory 930 includes computer storage media in the form of volatile and / or non-volatile memory such as read only memory (ROM) 931 and random access memory (RAM) 932. These computing machine-readable media may be any available media that can be accessed by the computing system 900. By way of example and not limitation, applications of computing machine-readable media include storage of information such as computer-readable instructions, data structures, other executable software, or other data. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other tangible medium that can be used to store the desired information and that can be accessed by the computing device 900. Transient media such as wireless channels are not included as machine-readable media. Communication media typically embody computer-readable instructions, data structures, other executable software, or other transport mechanisms and include any information delivery media.

[0084] The system further includes a basic input / output system 933 (BIOS), containing the basic routines that help to transfer information between elements within the computing system 900, such as during start-up, and is typically stored in ROM 931. RAM 932 typically contains data and / or software that is immediately accessible to and / or presently being operated on by the processing unit 920. By way of example, and not limitation, RAM 932 may include a portion of operating system 934, application programs 935, other executable software 936, and program data 937.

[0085] The computing system 900 may also include other removable / non-removable, volatile / non-volatile computer storage media. By way of example only, the system includes solid-state memory 941, which is typically connected to the system bus 921 via a non-removable memory interface such as interface 940, and a USB drive 951, which is typically connected to the system bus 921 by a removable memory interface such as interface 950.

[0086] A user can enter commands and information into the computing system 900 through input devices such as a keyboard, touch screen, or software or hardware input buttons 962, a microphone 963, a pointing device such as a mouse, trackball or touchpad, and / or a scrolling input component. These and other input devices are often connected to the processing unit 920 through a user input interface 960 that is coupled to the system bus 921, but may be connected by other interface and bus structures, such as a parallel port, a game port, or a universal serial bus (USB). A display monitor 991 or other type of display screen device is also connected to the system bus 921 through an interface, such as a display interface 990. In addition to the monitor 991, computing devices may also include other peripheral output devices, such as speakers 997, vibrators 999, and other output devices, which may be connected through an output peripheral interface 995.

[0087] The computing system 900 can operate in a networked environment using logical connections to one or more remote computers / client devices, such as a remote computing system 980. The remote computing system 980 may be a personal computer, a mobile computing device, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements previously described in relation to the computing system 900. The logical connections can include a personal area network (PAN) 972 (e.g., Bluetooth), a local area network (LAN) 971 (e.g., Wi-Fi), and a wide area network (WAN) 973 (e.g., a cellular network), but also other networks such as a personal area network (e.g., Bluetooth). Such network environments are common in offices, enterprise-wide computer networks, intranets, and the Internet. A browser application can reside in the computing device and can be stored in memory.

[0088] When used in a LAN networking environment, the computing system 900 is connected to the LAN 971 via a network interface 970, which may be, for example, a Bluetooth or Wi-Fi adapter. When used in a WAN networking environment (e.g., the Internet), the computing system 900 typically includes some means for establishing communications over the WAN 973. For mobile telecommunications technologies, for example, a wireless interface, which may be internal or external, may be connected to the system bus 921 via the network interface 970, or other appropriate mechanism. In a networked environment, other software illustrated in conjunction with the computing system 900, or portions thereof, may be stored in a remote memory storage device. By way of example and not limitation, the system has remote application programs 985 residing on the remote computing device 980. It will be appreciated that the illustrated network connections are examples and other means of establishing a communications link between computing devices that may be used.

[0089] As discussed, the computing system 900 may include a mobile device having a processing unit 920, memory (e.g., ROM 931, RAM 932, etc.), a built-in battery for powering the computing device, an AC power input for charging the battery, a display screen, and built-in Wi-Fi circuitry for communicating wirelessly with remote computing devices connected to a network.

[0090] It should be noted that the design may be implemented in a computing system such as the one described with respect to the present specification, however, the design may be implemented in a server, a computing device dedicated to message processing, or in a distributed system in which different parts of the design are implemented in different parts of a distributed computing system.

[0091] In some embodiments, the software used to facilitate the algorithms discussed herein may be embedded in a non-transitory machine-readable medium. A machine-readable medium includes any mechanism for storing information in a form readable by a machine (e.g., a computer). For example, a non-transitory machine-readable medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium, an optical storage medium, a flash memory device, a digital versatile disk (DVD), an EPROM, an EEPROM, a flash memory, a magnetic or optical card, or any type of medium suitable for storing electronic instructions.

[0092] It should be noted that the applications described herein include, but are not limited to, programs that are part of software applications, mobile applications, and operating system applications. Some portions of this specification are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, but not necessarily, these physical quantities take the form of electrical or magnetic signals that can be stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, primarily for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like. These algorithms may be written in a number of different software programming languages, such as C, C++, HTTP, Java, Python, or other similar languages. Also, the algorithms may be implemented in lines of code in software, configured logic gates in software, or a combination of both. In one embodiment, logic consists of electronic circuitry that follows the rules of Boolean logic, software that contains patterns of instructions, or any combination of both. Any portion of an algorithm implemented in software may be stored in an executable format in a portion of memory and executed by one or more processors.

[0093] It should be noted, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically indicated otherwise, and as is evident from the preceding discussion, discussions utilizing terms such as "processing" or "calculating" or "computing" or "determining" or "displaying" throughout this specification will be understood to refer to the actions and processes of a computer system or similar electronic computing device that manipulates data represented as physical (electronic) physical quantities in the computer system's registers and memory and converts it into other data that is similarly represented as physical quantities in the computer system's memory or registers or other such information storage, transmission or display device.

[0094] Many functions performed by electronic hardware components can be duplicated by software emulation. Thus, software programs written to achieve those same functions can emulate the functions of the hardware components in the input-output circuitry. Thus, provided herein is one or more non-transitory machine-readable media configured to store instructions and data that, when executed by one or more processors on a computing device of the aforementioned system, cause the computing device to perform the operations outlined herein. In one embodiment, the modules can be implemented in i) software - algorithms, routines, apps, etc., in conjunction with memory and electronic hardware such as CPU / GPU, ii) logic gates configured to receive inputs, perform desired functions, and output results, iii) other forms of electronic circuitry, and / or iv) combinations thereof.

[0095] References herein to an "embodiment," "example," or the like indicate that the described embodiment or example may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Such phrases may not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described with respect to one embodiment, it is believed to be within the knowledge of one of ordinary skill in the art to affect such feature, structure, or characteristic with respect to other embodiments, whether or not explicitly indicated.

[0096] Although the foregoing designs and embodiments thereof have been provided in considerable detail, the intent of this application is not to limit the designs and embodiments provided herein. Additional adaptations and / or modifications are possible, and in their broader aspects, these adaptations and / or modifications are also encompassed. Thus, departures may be made from the foregoing designs and embodiments without departing from the scope provided by the following claims, which, when properly interpreted, are limited only by the claims.

Claims

1. an explanation engine having a set of cooperating modules configured to evaluate layers in a hierarchical architecture of a machine-based reasoning process that uses machine learning to support explanations of how the machine-based reasoning process arrived at a reported result, both a top-level result and a corresponding intermediate output result; a messaging module of the explanation engine configured to collect one or more intermediate output results and the top-level result from intermediate layers of the machine-based reasoning process, in order to communicate how the machine-based reasoning process arrived at its reported result in a communication, the multiple reasoning layers being associated with terminology used in at least one of: i) the problem to be solved; and ii) a domain related to the problem; An apparatus comprising:

2. 2. The apparatus of claim 1, wherein the explanation engine has a terminology module configured to assign terminology from either i) the domain related to the problem and ii) the particular problem to be solved to multiple layers in the hierarchical architecture of the machine-based reasoning process provided by an inference engine, such that a user can understand results in terms of the particular problem or domain based on how the communication is generated.

3. the explanation engine has an explanation engine terminology module configured to accept input of terms for the problem to be solved provided by at least one of: i) a description of the problem to be solved, ii) a description of a preferred approach to solving the problem from a user, and iii) a database of known terms specific to the domain related to the problem; 2. The apparatus of claim 1, wherein the terminology module is configured to crawl the hierarchical architecture of the machine-based reasoning process created by an inference engine and then: i) associate terms specific to the problem to be solved and / or terms specific to a related subject area supplied by the user with ii) a number of layers that make up the hierarchical architecture of the machine-based reasoning process.

4. the explanation engine is configured to cooperate with a first inference engine configured to decompose its machine-based inference process into divisible layers that provide intermediate output results to other layers to determine top-level results from the machine-based inference process, as opposed to a second inference engine configured to create an omnibus neural network that is compiled simply as a black box that outputs its final decision; 2. The apparatus of claim 1, wherein the explanation engine is configured to cooperate with the first inference engine to allow a user to query what the intermediate output results are for each layer of the machine-based inference process and what would happen if the intermediate output results were changed.

5. 2. The apparatus of claim 1, wherein the explanation engine has a crawlback module configured to cooperate with an ablation module to trace the intermediate layers of the machine-based inference process constructed by an inference engine and record the factors being considered and how important they were in reaching the top-level result from the machine-based inference process.

6. 2. The apparatus of claim 1, wherein the explanation engine has a crawlback module configured to cooperate with the messaging module, the crawlback module of the explanation engine configured to crawl a decomposition of the machine-based reasoning process to collect and then report on and explain the top-level result in terms of the intermediate output results from the multiple layers of the reasoning process.

7. 2. The apparatus of claim 1, wherein the explanation engine comprises an ablation module configured to modify the intermediate output results from a layer of the machine-based reasoning process by changing inputs of that layer and to output new intermediate output results and new top-level results from that layer of the machine-based reasoning process.

8. an ablation module configured to perform one or more ablation cycles that vary inputs to a layer of the machine-based inference process created by an inference engine to determine an impact of that layer on the top-level result and record the impact; 2. The apparatus of claim 1, wherein the messaging module is configured to obtain results of the ablation cycles and data generated therewith to generate a report result of the impact of each layer of a machine-based inference process on the communications generated by the messaging module.

9. 2) cooperate with a terminology module to associate the intermediate output results from the multiple layers with terminology obtained from at least one of: i) a subject matter domain related to the problem; and ii) problem-specific terminology used in the problem to be solved.

10. When executed by one or more processors, the explanation engine performs the following operations: having an explanation engine having a set of modules evaluate layers in a hierarchical architecture of a machine-based reasoning process that uses machine learning to support explanations of how the machine-based reasoning process arrived at a reported result of both a top-level result and a corresponding intermediate output result; causing a messaging module of the explanation engine to collect the top-level result and one or more intermediate output results from intermediate layers of the machine-based reasoning process, each reasoning layer being associated with terminology used in at least one of: i) the problem being solved, and ii) a domain related to the problem, in order to communicate how the machine-based reasoning process arrived at its reported result in a communication; A non-transitory computer readable medium comprising executable instructions for causing a computer to perform operations including:

11. 1. A method for describing machine-based reasoning, comprising: configuring an explanation engine having a set of modules to evaluate layers in a hierarchical architecture of a machine-based reasoning process that uses machine learning to support explanations of how the machine-based reasoning process arrived at a reported result, both a top-level result and a corresponding intermediate output result; configuring a messaging module of the explanation engine to collect one or more intermediate output results from intermediate layers of the machine-based reasoning process and the top-level result, each reasoning layer being associated with terminology used in at least one of: i) the problem being solved; and ii) a domain related to the problem, in order to communicate how the machine-based reasoning process arrived at its reported result in a communication; A method comprising:

12. configuring a terminology module of the explanation engine to assign, for each layer in the hierarchical architecture of the machine-based reasoning process provided by an inference engine, terms from domains relevant to the problem and any of the particular problems at hand, thereby enabling a user to match explanations with terms that the user can understand in communications generated by the messaging module; The method of claim 11 further comprising:

13. configuring a terminology module of said explanation engine to accept input of terms for the problem to be solved provided by at least one of: i) a description of the problem to be solved; ii) a description of a preferred approach to solving said problem from a user; and iii) a database of known terms specific to a domain related to said problem; configuring the terminology module to crawl the hierarchical architecture of the machine-based reasoning process produced by the inference engine and to associate the problem-specific and / or domain-specific terms with each of the layers of the hierarchical architecture of the machine-based reasoning process; The method of claim 11 further comprising:

14. configuring said explanation engine to cooperate with a first inference engine configured to decompose its machine-based inference process into divisible layers that provide intermediate output results to other layers to determine top-level results from said machine-based inference process, as opposed to a second inference engine configured to create an omnibus neural network that is compiled simply as a black box that outputs its final decision; configuring the explanation engine in cooperation with the first inference engine to allow a user to query what the intermediate output results are for each layer of the machine-based inference process and what would happen if the intermediate output results were changed; The method of claim 11 further comprising:

15. 12. The method of claim 11, further comprising configuring a crawlback module of the explanation engine, in cooperation with an ablation module, to trace back each intermediate layer of the machine-based reasoning process built by an inference engine and record the factors considered and how important they were in reaching the top-level result from the machine-based reasoning process.

16. 12. The method of claim 11, further comprising configuring a crawlback module of the explanation engine in cooperation with the messaging module, the crawlback module of the explanation engine configured to crawl a decomposition of the machine-based reasoning process to collect and then report intermediate output results from each layer of the reasoning process and explain a final top-level result in terms of the intermediate output results.

17. 12. The method of claim 11, further comprising configuring an ablation module of the explanation engine to remove each intermediate layer of the machine-based reasoning process, one at a time, and assessing the impact on the top-level result from the machine-based reasoning process.

18. 12. The method of claim 11, further comprising configuring an ablation module of the explanation engine to modify an output result from a layer of the machine-based reasoning process by changing an input of that layer, and to output a new output result from that layer of the machine-based reasoning process and a new top-level result.

19. configuring an ablation module to perform one or more ablation cycles that vary inputs to a layer of the machine-based inference process produced by an inference engine to determine an impact of that layer on the top-level results and record the impact; configuring said messaging module to obtain all results of said ablation cycles and data generated therewith to generate a report of the impact of each layer of a machine-based inference process on the communications generated by said messaging module; The method of claim 11 further comprising:

20. 12. The method of claim 11, further comprising configuring the messaging module of the explanation engine to: 1) extract the intermediate output results from each layer of the machine-based reasoning process produced by an inference engine; and 2) cooperate with a terminology module to associate the intermediate output results from each layer with terminology obtained from at least one of: i) the problem being solved; and ii) a domain related to the problem, wherein the terminology assigned to each layer of the machine-based reasoning process is obtained directly from a user-provided documented description of the problem and / or extracted from a domain-specific database.

Images