First-order logic neural network with bidirectional inference
Patent Information
- Application Number
- JP2022567200
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-05-13
- Filing Date
- 2021-04-13
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2041-04-13
Smart Images

Figure 0007577132000136 
Figure 0007577132000137 
Figure 0007577132000138
Abstract
Description
[Technical field]
[0001] The present disclosure relates generally to machine learning and neural networks, as well as apparatus, systems, and methods for modeling logic and logical neural networks capable of inference in any direction. [Background technology]
[0002] Neuro-symbolic reasoning systems aim to bridge the large gap that currently exists between principled deductive reasoning via any of the various systems of formal logic and data-driven, gradient-optimized neural network architectures, two of the most studied areas of artificial intelligence. Both of these frameworks suffer from several important strengths and shortcomings. In particular, formal logic is interpretable, verifiable, and broadly generalizable, whereas undecidable cases are computationally intensive, require extensive domain expert input, and can be derailed by even minor inconsistencies. Neural networks, on the other hand, perform well on raw and / or noisy data, require little human configuration, and can run efficiently in parallel. However, the negative aspects of neural networks are their requirement for large training datasets, their vulnerability to adversarial attacks, and their uninterpretable black-box nature. Summary of the Invention
[0003] An aspect of the present disclosure is a logical neural network, and method of operation, that integrates two disciplines: (1) principled deductive reasoning via any of various systems of formal logic, and (2) data-driven, gradient-optimized neural network architectures, that exploits the strengths of both disciplines while mitigating their shortcomings.
[0004] One aspect of the present invention is a neural network architecture, and in particular a logical neural network (LNN), that implements systems, methods, and computer program products in which neurons model rigorously defined concepts of weighted fuzzy logic or classical logic.
[0005] In a further aspect, the LNN architecture is arranged in one-to-one correspondence with operations in a knowledge base (KB) of logical expressions, and such a neural network is capable of reasoning in any direction, i.e., via normal evaluation, modus ponens, conjunction elimination, and all relevant inference rules.
[0006] In further aspects, the LNN systems, methods, and computer program products operate to behave equivalently to reasoning on systems of classical logic formulas (using an established set of inference rules), while at the same time always being differentiable with respect to the formulas expressed so that they can constantly change between different forms.
[0007] In a further aspect, to meet the requirements of both behaving equivalently to performing inference on a system of classical logical formulas and always being differentiable with respect to the formulas expressed in a way that can constantly change between different forms, the LNN implements a weighted fuzzy logic that provides plausible inference rules and intuitive mathematical behavior such that it behaves similarly to a rectified linear unit (ReLU) type activation function mathematically defined as y=max(0,x).
[0008] Furthermore, the LNN systems, methods, and computer program products define a form of continuous classical logic that operates through the enforcement of constraints and is amenable to automated learning that is guaranteed to behave strictly classically given classical inputs.
[0009] Further provided are systems, methods, and computer program products that provide the capability to produce neuro-symbolic fusion, which is an interpretable, verifiable neural network that is informed by any available domain knowledge, yet is resilient to inconsistencies, and that can take advantage of big data, yet has the capability in the absence of big data.
[0010] According to one embodiment, a computer-implemented method is provided that includes receiving, with one or more hardware processors, an input query and an input proposition used to infer a logical truth based on the input query; configuring, using the one or more hardware processors, a logical neural network (LNN) structure including one or more propositional neurons that represent the input proposition according to a system of logical expressions corresponding to inference rules and one or more neurons that represent logical connectives; evaluating, using the LNN structure, the logical inference based on the input query; and, in response to the evaluation, outputting an output truth value that either proves or does not prove the logical inference within an aggregated boundary.
[0011] In a further embodiment, a computer implemented system is provided that includes one or more hardware processing units configured as a logical neural network (LNN) structure including a plurality of neurons and connector edges, the plurality of neurons and connector edges being in one-to-one correspondence with a system of logical expressions, and performing a method for performing logical inference, wherein one neuron of the plurality of neurons exists for each corresponding logical connective in each expression or subexpression of the system of logical expressions, the one neuron being connected to one or more linking connective edges that provide input information including operands of the logical connective and information further including parameters configured to implement a truth function of the logical connective. the plurality of neurons includes one or more link connector edges corresponding to an expression that provides information proving a bound on the truth value of the corresponding proposition and further includes a parameter configured to aggregate a tightest bound; and the one or more hardware processors are configured to receive an input query or logical inference, evaluate the logical inference based on the input query or logical inference using the LNN structure, and in response to the evaluation, output a truth value that proves or does not prove the logical inference within the aggregated tightest bound.
[0012] In a further aspect, a computer program product for performing operations is provided, the computer program product including a storage medium readable by a processing circuit and storing instructions for execution by the processing circuit to perform a method, the method being the same as that shown above.
[0013] The objects, features, and advantages of the present disclosure will become more clearly apparent when the following description is taken in conjunction with the accompanying drawings. [Brief description of the drawings]
[0014] [Figure 1] FIG. 1 illustrates a system and method for constructing and using a Logical Neural Network (LNN) in accordance with an embodiment of the present invention. [Figure 2A] FIG. 13 is a diagram showing a knowledge graph tree structure illustrating a use case of LNN. [Figure 2B] FIG. 2B illustrates backwards propagation inference computations in the knowledge graph tree structure of the LNN use case of FIG. 2A. [Figure 2C] FIG. 2C illustrates an example inverse activation function used to compute for inverse inference of the use case's syntax tree for the downward traversal path shown in FIG. 2B. [Diagram 3] FIG. 2B illustrates the input of expressions and an example formulated query in the knowledge graph tree of the LNN for the use case shown decomposed in FIG. 2A. [Figure 4] FIG. 2 illustrates a syntax tree representing an exemplary first-order logic neural network (Net) in accordance with an embodiment. [Diagram 5] FIG. 1 illustrates the methods of prediction and bidirectional inference used to evaluate propositional LNNs and first-order logic LNNs according to an embodiment in which recurrent neural nets contain neurons in a one-to-one correspondence with equations that represent the system being modeled. [Figure 6A] FIG. 6 illustrates an exemplary method of forward pass or upward pass evaluation of the two-way inference method of FIG. 5. [Figure 6B] 6 illustrates an exemplary method of backward-pass or downward-pass evaluation of the bidirectional inference method of FIG. 5. [Figure 6C] 6 illustrates an exemplary method of aggregation in the two-way inference method of FIG. 5. [Figure 7A]FIG. 2 illustrates an exemplary LNN structure organized as a syntax tree graph structure representing an exemplary logic program. [Figure 7B] FIG. 2 is an exemplary diagram illustrating weighted logical neurons (e.g., disjunction neurons) of an exemplary LNN including logical activation functions. [Figure 8] FIG. 1 illustrates an exemplary embodiment of a pipeline implementation of a Deep Thinking Question Answering (DTQA) system. [Figure 9] FIG. 1 illustrates an LNN graph formed based on a query according to an exemplary DTQA implementation. [Figure 10] FIG. 1 illustrates a schematic diagram of an exemplary computer or processing system that may configure and implement a logical neural network in accordance with embodiments herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0015] Symbolic logic is a method of encoding knowledge using symbols and logical operators. The central element of a symbolic system is a knowledge base (KB), which is a set of logical representations. These representations are expressed using one of many logical knowledge representation languages, including the most widely used languages of first-order logic, the simplest and least expressive of propositional logic. Logical expressions in a knowledge base are also called axioms when they are considered given without being derived from other representations. Axioms represent facts about a subject domain and the rules that govern that domain. After knowledge is captured in a knowledge base, it can be used to answer questions / queries through logical inference to derive new representations from the knowledge contained in the knowledge base. Answering a query q using a knowledge base KB involves entailment.
number
[0016] This disclosure provides systems, methods, and computer program products for constructing and using logical neural networks ("LNNs"). A logical neural network (LNN) is a neural network ("net") that has a one-to-one correspondence with systems of logical expressions, where evaluation is equivalent to logical inference. This is significant because LNNs, for the first time, simultaneously integrate a number of important properties from both neural nets (learning) and symbolic logic (inference). They incorporate domain knowledge by providing a system that is fully differentiable and symbolically interpretable, supports complex first-order-logic (FOL) expressions, and follows the open-world assumption. LNNs extend the input of traditional neural nets by incorporating rules and facts into a graph structure.
[0017] Neural network architectures model individual neurons using rigorously defined concepts of weighted fuzzy or classical logic. Such neural networks are arranged in a one-to-one correspondence with the operations in a knowledge base (KB) of logical expressions and are capable of reasoning in any direction, i.e., via normal evaluation, modus ponens, conjunction elimination, and all relevant inference rules.
[0018] The main design goal of LNNs is that their evaluation behaves equivalently to inference on systems of classical logic formulas (using an established set of inference rules). At the same time, it is an important benefit that LNNs are always differentiable, and indeed that the formulas represented can change continuously between different forms. To satisfy both of these seemingly incompatible requirements, a rigorously defined notion of weighted fuzzy logic is introduced that provably provides plausible inference rules and intuitive mathematical behavior that happens to behave similarly to the rectified linear unit (ReLU) in the current neural network literature. Furthermore, the enforcement of constraints defines a form of continuous classical logic that is amenable to automated learning, while still guaranteeing that it behaves exactly classically given classical inputs.
[0019] In an embodiment, an LNN is a recurrent feed-forward neural net that includes (a) neurons arranged to match a system of equations, (b) neuron weights constrained to ensure classical logic behavior, and (c) groups of neurons that share associated weights to perform inference. An LNN is distinct from other neural networks because the neurons generate boundary pairs (tensors of), and alternatively, pairs of neurons with matching weights that exist for both boundaries.
[0020] LNN systems make predictions as all neural nets do, i.e., they take input data and generate predicted outputs. The policies used to compute the predictions are different because they are structurally implemented to match a system of logical expressions, and the computations are constrained to be equivalent to performing inference using that system of logical expressions. The computations are highly interpretable and structured to take advantage of inputs (e.g., knowledge created by experts, existing knowledge bases) and are capable of performing complex inferences.
[0021] LNNs are represented as structured graphs with nodes and interconnecting edges between the nodes that establish positive and negative correlations between truth values, and any node can be an input or an output, but the truth values of an LNN do not need to carry a probabilistic interpretation and do not require costly normalization. Special cases of LNNs can be developed that present results that include a probabilistic interpretation.
[0022] FIG. 1 illustrates an exemplary method 100 for configuring, training, and performing inference using a logical neural network ("LNN" or "net") structure 150. As referred to herein, the term "inference" refers to the computation of (boundaries on) truth values for propositions, predicates, and expressions based on knowledge of (boundaries on) truth values for other propositions, predicates, and expressions. Through multi-way inference, the inference performed by the LNN is iterative in nature in the sense that tightening the boundaries on a particular truth value may enable subsequent inference of tighter boundaries on other truth values. The LNN is configured to implement weighted real-valued logic in the form of a neural network that can be constrained to behave equivalently to classical logic (or fuzzy logic). In an embodiment, the boundaries on the truth values of expressions and propositions can be initialized such that inference (i.e., evaluation of the neural network) produces results in the form of boundaries on truth values computed at specific neurons according to available observations (thereby tolerating missing data).
[0023] The LNN 150 computes two types of inferences: forward inference and backward inference. With respect to the neuron's operands or propositions A and B, the forward inference and backward inference include, but are not limited to, the following types:
[0024] Involutional negation: A=1-~A
[0025] (1) conjunctions containing (B&~(A&B)) → ~A, (2) displacements: A → ~(B&(A&B)), and (3) tautologies: A → (B → (A&B)), where "&" is the logical AND operator.
[0026] Disjunctive syllogism:
number
number
[0027] Modus ponens: (A&(A→B))→implication(B) Implications (A) (1) Modus Torrence: (~B&(A→B))→~A (2) Rearrangement: A → (~B&(A → B)), and (3) Tautology: A → ((A → B) → B)
[0028] Typically, a logical neural network 150 combines the properties of neural networks (i.e., learning) with those of symbolic logic (i.e., knowledge and reasoning), specifically, it is a neural net that has a one-to-one correspondence with a system of logical expressions whose evaluation is equivalent to logical inference. The LNN structure is a neural network structure of nodes (or neurons) and connector edges, where the nodes are configured to receive inputs according to the type of inference problem being solved. Structurally, an LNN is a graph that contains the syntax trees of expressions in a represented knowledge base (KB), connected to each other through nodes added for each proposition. Specifically, there is one neuron for each unique proposition that appears in any expression, in addition to one neuron for each logical connector that appears in each expression. All neurons return pairs of values in the range [0,1] that represent upper and lower bounds on the truth values of their corresponding subexpressions and propositions. Neuron connections are managed using explicit graph structures for navigation, traversal, shortest path finding, and use in general graph theory formulations.
[0029] FIG. 7A illustrates an exemplary LNN structure configured as a syntax tree (knowledge graph) structure 700 illustrating an exemplary logic program that includes the following logic statements or formulas: A=B&C, B=C&(~D)&E, and D=E
[0030] In FIG. 7A, the nodes labeled D and E are propositional neurons that provide inputs (e.g., facts about the world). The nodes labeled A and B are logical connective neurons that implement logical functions. In an embodiment, there is one neuron for each logical connective in the formula, with inputs linking to neurons corresponding to the operands of the connective, and parameters configured to implement the truth function of the connective. As shown in FIG. 7A, upper and lower bounds (e.g., U ) in the range [0,1] (i.e., uncertainty bound ∈ [0,1]) are calculated recursively in forward and backward inference passes to obtain / update the degree of truth or uncertainty of the particular logical inference being evaluated. A , L A , U B , L B ) is associated with each node and connector. In an embodiment, for each proposition, there is one neuron with inputs linking to neurons corresponding to expressions that establish bounds on the truth value of the proposition, and parameters configured to aggregate the tightest such bounds.
[0031] FIG. 7B illustrates, for example, a logic function (e.g., a disjunction neuron).
number
[0032] Returning to FIG. 1, the LNN structure 150 is configured to include nodes (neurons) each configured with a corresponding activation function 153 connected to it, which determines how to compute the corresponding node (e.g., whether and how a particular node is activated based on whether that neuron's input is relevant to the type of prediction problem the model is solving). The activation function can be linear or nonlinear, and the output of each neuron can be a value in the range from 1 to 0.
[0033] As further shown in Figure 1, the LNN is configured to include inference rules 157, which are functions that represent and implement logical forms that receive premises, parse the syntax of the premises, and infer and return one (or more) conclusions from the premises. The LNN is configured as a logical theorem prover that (a) uses the entire set of inference rules associated with each connective and (b) focuses on proving truth values in subformulas that appear in the network rather than new formulas. The LNN differs from other theorem provers because (a) truth values are continuous and (b) connectives have weights. A comprehensive set of inference rules implemented within the LNN structure includes, but is not limited to, rules of negation (e.g., introduce negation, eliminate negation, introduce double negation, eliminate double negation), rules of conditionals (e.g., introduce conditionals, modus ponens (eliminate conditionals), modus tollens), rules of conjunctions (e.g., introduce conjunctions, simplify conjunctions), rules of disjunctions (e.g., introduce disjunctions, simplify disjunctions, disjunctive syllogisms), and rules of biconditionals (e.g., introduce biconditionals, eliminate biconditionals).
[0034] As further shown in FIG. 1, the LNN structure 150 of the network nodes is configured to receive both “trusted” formulas 120 used for computation and inference, including neuron weights and parameters that do not need to be learned and can be used as is, and “untrusted” formulas 130 that can contain information that may cause inconsistencies coming from non-expert or machine-generated sources. Such formulas can be a system of formulas obtained before training from an ontology (a set of rules that are considered true that describe how something works, such as the Cyc® Ontology and knowledge base (a trademark of Cycrop, Inc.)). In an embodiment, based on formulas obtained from an existing logical knowledge base, the components of the logical neural network are initialized, in particular, neural inputs and parameters are selected to match the syntax and behavior of the various connectives that make up the formulas borrowed from the knowledge base, and initial bounds on the truth values of the formulas and propositions are selected to match the borrowed facts.
[0035] In the embodiment shown in FIG. 1, the trusted formula 120 includes one or more expert-created formulas or inferences 125, and logical constraints 128, which are constraints specified or declared according to specific domain knowledge implemented during training. These constraints are further input as components of queries executed on the model to determine inputs that satisfy the constraints. Additionally, the untrusted formula 130 includes one or more non-expert formulas or inferences 135, and inputs 138 provided by an inductive logic programming (ILP) / natural language processing (NLP) machine. After these sources are configured in the LNN network structure 150, the LNN can compute from the inferences / formulas already applied, and thus the LNN is used as a computational device.
[0036] Using standard neural network tools, the LNN goes through a learning phase 140 in which it receives LNN training input 160, further informed by a knowledge graph 145 containing knowledge graph data 145 fed by a knowledge base (KB) (e.g., a knowledge base or vocabulary resource such as DBpedia, WordNet, etc., providing unstructured and structured cross-domain knowledge data). The DBpedia knowledge base responds to queries by accessing a Resource Description Framework (RDF) representation of the extracted information (e.g., structured RDF triples on the order of billions). Exemplary training input 160 includes ground truth boundaries on propositions (nodes) containing data presented in the form of observed training facts 168, including any annotated ground truth statements, such that logical truths can be inferred by the LNN given the training facts, knowledge, and inference rules, and training queries 165, including one or more formulas. During the training phase, an evaluation step is performed that includes recursive forward and backward inference passes of the initial syntax tree programmed using initial conditions, such as the placement of the logical formula, the weighted importance values assigned to the propositions and / or predicates, and certain truths. To obtain results, the inference evaluation is continuously performed, and based on the results, there is an associated loss and gradient measure associated with the loss function (i.e., a collection of partial derivatives of the loss function with respect to each of the model's parameters). Based on the gradient results after each convergence (after the evaluation is completed), the individual LNN model parameter values can be updated or fine-tuned to improve the score. In an embodiment, when aggregating the tightest bounds for the propositions, the importance weights of each occurrence of each proposition in each formula are reused as the basis for the importance weights of the corresponding proofs.
[0037] A loss function 170 that is input to the LNN to establish an appropriate loss (e.g., least squared error) guides the learning process. However, in the case of the LNN 150, logical consistency can also be used as the loss. That is, based on the knowledge graph data input, the training fact data input, and the formula, if logical inconsistencies exist (i.e., upper and lower bounds are exceeded), those logical inconsistencies are penalized and there is an error associated with any of the inconsistencies.
[0038] During training, the parameters of the model are fine-tuned / re-tuned to minimize the measurement of the loss function of the error in which the model participates, in order to improve the measurement of the performance of the LNN model. During training, optimization is performed to minimize the loss. In an embodiment, at the beginning of each pass of inference, the truth values presented as known (i.e., measured) serve as inputs that are facts (TRUE). There remain unknown (UNK) values that need to be updated during the forward and backward traversals of the tree.
[0039] After the learning / training process 140 has been performed, a new LNN model 180 containing optimized weights 175 that avoid inconsistencies is obtained and used to process queries and make inferences. Thus, as shown in FIG. 1, the optimized LNN model 190 is used during an inference stage 193, where the LNN model 190 receives one or more prediction inputs 180 in the form of input prediction queries 185 and input facts 188 (e.g., queries and facts received from a remote computing device 110 associated with a user in the outside world), as well as knowledge graph data 147 used with the model. In response to the input queries 185, input facts 188, and knowledge graph data 147, the optimized LNN model 190 generates an inference prediction output 195 (e.g., everything that the LNN predicts during the inference stage), which can be fed back into any system 105 using the optimized LNN model. The LNN model can further generate a query result 198 (i.e., an answer to the posed query). For example, in response to a posed query such as "Does this photo contain a cat?", the LNN model outputs an answer for receipt by and / or presentation to a user via a computer or mobile device display 110.
[0040] The key innovations that distinguish LNNs from other neural networks are: adapted neuron activation functions that are constrained to match the truth functions of the connectives that they represent; results expressed in terms of bounds on truth values to distinguish between known, unknown, and contradictory states; and two-way inference, which allows x → y to be evaluated on its own, in addition to being able to, for example, given x, prove y, or similarly, given ¬y, prove ¬x.
[0041] As mentioned before, structurally, an LNN is a graph containing the syntax trees of all expressions in the represented KB, connected to each other through nodes added for each proposition. Specifically, there is one neuron for each logical connective that appears in each expression, plus one neuron for each unique proposition that appears in any expression. All neurons return pairs of values in the range [0,1] that represent upper and lower bounds on the truth value of their corresponding subformulas and propositions. Neurons corresponding to logical connectives take as input the outputs of neurons corresponding to their operands, and their activation functions are configured to match the truth function of the connective.
[0042] Activation Function
[0043] There are many candidate activation functions that can correspond to the classical truth functions of the corresponding connectives of neurons, and these activation functions differ in how strictly they treat the inputs as being between 0 and 1. For example, min{x,y} is the conjunction of real values
number
[0044] Domain and range. An activation function must return values in the range [0,1] for inputs in the range [0,1].
[0045] Monotonicity. Activation functions for conjunctions and disjunctions must be monotonically increasing with respect to their operands, and activation functions for implications must be monotonically decreasing with respect to their antecedents and monotonically increasing with respect to their consequents.
[0046] Neurons in an LNN return upper and lower bounds on truth values rather than a scalar result. This is achieved by applying a selected activation function twice as necessary: conjunction and disjunction both compute a lower bound from lower bounds on the inputs and an upper bound from upper bounds on the inputs, while implication computes a lower bound from an upper bound on the antecedent and a lower bound on the consequent, and vice versa. This simplistic approach to bounds requires the use of monotonic functions.
[0047] Some useful, but not essential, properties of activation functions are:
[0048] 1. DeMorgan's Laws. Conjunctions and disjunctions should be related via DeMorgan's laws. That is, real-valued disjunctions
number
number
[0049] 2. Implication through the remainder (residuum). Implication is the remainder of a conjunction, or more specifically,
number
number
[0050] 3. Commutativity and associativity. Conjunctions and disjunctions should be commutative and associative (allowing for rearrangement of parameters, if any), and the antecedents of implications should be commutative, i.e., (x1 → (x2 → y)) = (x2 → (x1 → y)).
[0051] Each of the above aims to preserve the behavior of inference that is intuitive to human observers and to the creators of the formulas used by LNNs. These properties are:
number
number
[0052] Triangular or t-norms and their associated t-conorms and residuals are natural choices for activation functions in LNNs because they are already well-behaved for classical inputs and have well-known inference properties. Logic defined in terms of such functions is denoted t-norm logic. Common examples of these include:
[0053] [Table 1]
[0054] Of these, only Lukasiewicz's identity is well known.
number
number
[0055] Weighted activation function
[0056] The t-norm and related functions presented above meet the requirements for an activation function for an LNN, but do not provide adjustable parameters for model optimization and training purposes. It is therefore useful to introduce the concept of importance weighting, whereby neuronal inputs with larger (non-negative) weights have more influence on the neuron's output, and intuitively, neuronal inputs with zero weights have no influence. In general, weighted activation functions can take many forms.
[0057] Importance weighting is an important feature of LNNs, but is not required for them to be useful in certain situations. LNNs using non-rigidly weighted activation functions can still infer with partial confidence in the constituent expressions, and can infer real-valued truth values, provided that the input truth value is somewhere in the range [0,1]. Also, importantly, the differentiability of LNNs allows them to pass the gradient computation to any system that uses the output (e.g., loss function) and to any system that generates the input (e.g., another neural network tasked with extracting predicate data from differently structured data such as images or text).
[0058] Use Cases of Propositional LNN
[0059] Referring to FIG. 1, an exemplary logical neural net 150 is formed based on an initial observation and theorization phase in which observations are recorded and facts and rules about those observations are theorized. In the use case of FIG. 2A, during these phases, it may be observed that an entity that is a dog, having a tail and whiskers, is also chasing. It may be observed that there are no other devices, such as laser pointers, that the entity is chasing. Further observations may be the observation of an entity that is a cat, having a tail and whiskers, and the observation that the cat is chasing and there are no lasers. From these observations, it may be theorized that (1) cat.nand.dog, e.g., an entity can be either a cat or a dog, but not both, (2) implies.cat, e.g., an entity that has a tail, whiskers, and chases a laser implies a cat, or an entity that has a tail, whiskers, and chases only a laser implies a cat. Alternatively, a rule of cats, e.g., cat.rule, that all cats have tails, whiskers, and chase only lasers, may be theorized.
[0060] The construction of the LNN proceeds to the logic programming stage, where corresponding predicate and / or propositional elements (e.g., dog, tail, whiskers, chase, laser (or ~laser), etc.) are defined, and logical formulas using these elements that correspond to theories or rules are proposed, e.g., (1) ~(cat&dog), (2) tail&whiskers&(laser→chase)→cat, (3) tail&whiskers&(laser→chase)&(chase→laser)→cat.
[0061] For an exemplary theorized cat rule, the following logical formula may be devised:
[0062] Cat → tail & whiskers & (laser → chase) & (chase → laser)
[0063] In an embodiment, the logical neural network is constructed as a neural net program. During the programming stage, this stage further includes converting the corresponding propositional elements and the proposed logical formula into a programmed LNN structure using a programming language for graphic user interface applications. In the case of an exemplary proposed implies.cat theory, the statements in the Python programming language include: g['cat.nand.dog'] = Not(And('cat','dog')) g['Laser->Chase'] = Implies('Laser', 'Chase') g['Chasing->Laser'] = Implies('Chasing', 'Laser') g['cat.feats']=And('tail','whiskers','laser->chase','chase->laser') g['implies.cat'] = Implies('cat.feats', 'cat')
[0064] A further LNN construction stage then involves mapping the LNN's programmed statements into a graph structure (e.g., a knowledge graph or a syntax tree).
[0065] FIG. 2A illustrates a use case of a logical neural network ("net") 200 formed according to the methods herein. In general, LNNs are described in terms of first-order logic, and the following use case first describes an LNN restricted to the scope of propositional logic. As shown in FIG. 2A, there is one neuron 220 per proposition / predicate 202, in addition to one neuron 210 per logical connective or edge 212. Each generated neuron 210 has a one-to-one correspondence with the system of logical expressions 205, 208, where evaluation is equivalent to logical inference. To generate a logical neural network, a computing system receives inputs (e.g., knowledge graph / predicted input data / facts) for use in outputting truth values (represented by upper and lower bounds), e.g., at one neuron, typically one per query expression. Neurons 210 are arranged to match the syntax tree of the expression, with propositional / predicate neurons 220 both providing proven truth values and iteratively aggregating. In general, negation (and first-order quantifiers) are pass-through nodes that do not contain parameters. As shown in the figure, certain neurons 210 provide uncertainty bounds 213 with values between 0 and 1. Each neuron 210 includes an adapted activation function 211, whose computation results in a bound on the truth value. Furthermore, a method of bidirectional reasoning is employed.
[0066] As shown in FIG. 2A, the use case knowledge graph (net) 200 includes a syntax tree graph structure 204 configured to correspond to a logical formula 205 at the time of LNN model creation, as illustrated in symbolic logic reasoning. A second syntax tree structure 207, also constructed to correspond to an input logical formula 208 at the time of model creation, is shown in FIG. 2A. The input logical formulas 205, 208 are rules about the world and can be created by experts or generated in an ad-hoc manner (e.g., extracted by an ILP or NLP device). The formulas 205, 208 can be known formulas or parts of known formulas (i.e., sub-formulas) and can be subject to change or fine-tuned at the time of model creation and / or prediction or inference. That is, certain inputs to a formula or inputs to a sub-formula (e.g., operands) can be weighted based on their importance and thus assigned importance weights that become part of the structure of the LNN. In this way, each operand of each combinator in a formula is or can be assigned an importance weight, allowing them to be changed unless each formula or weight is optimized.
[0067] For the use case, in subgraph 218, the "cat" node can be weighted with twice the importance of "dog," and is thus a modified version of the disjunction's "OR" connective, skewing the results in the sense that "cat" now has more influence than "dog" in the expression.
[0068] Each of the syntax tree graphs 204, 207 for each formula is constructed based on an initial list of input propositions 202 with defined truth values (e.g., true or false), including the “whiskers” 202A, “tails” 202B, “laser pointers” 202C, “chases” 202D, “cats” 202E1, 202E2, “dogs” 202F, and “pets” 202D present in the formula. Alternatively, these inputs can be predicates, e.g., statements with truth values that depend on two or more variables. Logical formulas 205, 208 are applied, resulting in the evaluation of each inference or implication (e.g., “whiskers ∧ tails ∧ (laser pointers → chases))) → cats”) and the evaluation of inferences or implications (e.g., “cats ∨ dogs) → pets”). In FIG. 2A, the subformulas are shown as corresponding subgraphs 218 (e.g., “cats ∨ dogs”). No inference rules are shown, but activation functions 211 present at particular nodes 210 are shown based on a particular selected logic (e.g., Lukasiewicz logic). Activation functions 211 corresponding to particular logical connectives (e.g., nodes 210A, 210B, 210C present in expressions 205, 208) are shown. The logical connectives of the exemplary classical propositions employ Boolean operations, including, for example, AND, OR, implication, negation, and logical equivalence, and employ truth values as 1 or 0.
[0069] During the learning process, as shown in step 140 of FIG. 1, a forward inference pass is performed, where at each node, the expression is evaluated first to find a truth value. In the use case of FIG. 2A, the implication is evaluated first. If laser means chase, then chase is true even if laser is unknown (which may have a high lower bound, using fuzzy values), and this is sufficient to evaluate the "laser means chase" node as true (so the lower bound may be a high lower bound (e.g., 0.95)). The information that "tail" and "whiskers" are true causes the cat.features node to have a high lower bound, and this high lower bound is pushed forward to the implies.cat node. Since the truth value of "cat" is unknown, essentially the inference is calculated as "true means unknown", so its truth value cannot be evaluated at this point, and the calculated upper and lower bounds remain loose (e.g., cannot yet be evaluated as true or false). However, in this example, since "implies.cat" is given as true initially, the strictest bound is retained, so the bounds are not changed. Similarly, in the upward pass through other parts of the graph, given that the particular "cat" is unknown and "dog" is true, the strictness of the bounds in the conjunction operation remains unknown (both lower and upper bounds) and insufficient to give a tight bound, and the bounds in the negation operation remain loose. Thus, for "cat.nand.dog", the bounds are computed loosely (lower lower bound, higher upper bound), but given the truth value, the bounds do not need to be tightened. Thus, in the forward pass, the model has learned cat.features implies and laser.implies chase as true.
[0070] During the backward (downward) inference pass, a recursive depth-first traversal of the tree is performed from each root to each of the leaves, executing the reverse inference rules along the way and tightening the upper and lower bounds. In the example backward (downward) inference pass from the cat node (=true), we assume that the system has now learned in the forward pass that cat.features is true, so we then use modus polence to determine that the expression "cat" is true.
[0071] In one aspect, fuzzy logic is employed that provides degrees of truth in the range of 0 to 1, i.e., values in the range of 0 to 1 are fuzzy mixes between true and false. In an embodiment, an alpha, α, is defined as a threshold value of truth, e.g., in the range of 0.5<α≦1. Any evaluated truth value is thus constrained such that values above α are considered "true" and any truth value less than the amount of 1-alpha is considered "false."
[0072] The activation function 211 is a particular function at each neuron 210 for computing the output of that neuron, chosen to represent a particular logical operation, or a set of activation functions that represent logical operators consistent with a particular logic scheme (e.g., Lukasiewicz logic). For example, as shown in sub-formula 218, the activation function is a disjunctive function that is evaluated as the addition (sum) of the inputs. In the "unweighted" case shown in FIG. 2A, the disjunctive activation function at node 210C provides an output that is fixed to be a value of 0 or 1.
[0073] In an embodiment, for real-valued logical connectives, an exemplary activation function computes a truth value subject to the following constraints:
[0074] Conjunction (AND):
[0075]
number
[0076] where p and q are operands and the constraint is that the sum of the two operand value inputs minus one is clamped below by zero.
[0077] In the disjunction (OR) case:
[0078]
number
[0079] That is, the sum of the two operand input values is clamped up by 1.
[0080] In the case of implication (→):
[0081]
number
[0082] That is, it is similar to a disjunction, but one of the inputs is negated (1-p). As an example, the activation function is a conjunctive function, as shown in node 210A. In the "unweighted" case shown in Figure 2A, the activation function of a conjunctive function following Lukasiewicz logic is evaluated as the addition (sum) of the inputs minus the number of elements plus one.
[0083] In embodiments, other activation functions following other logic schemes can be employed. Furthermore, different logistic functions can be used as activation functions in different parts of the LNN. For example, a "weighted" Lukasiewicz logic scheme for logical connectives is employed, where the activation function calculates the truth value as follows:
[0084] Weighted Lukasiewicz logic
[0085] The weighted Lukasiewicz t-norm for two variables and n variables (n-ary) used in logical AND is given as follows:
[0086]
number
[0087] where I is the input set, β is a non-negative bias term, and w is a non-negative weight. i , and an input x in the range [0,1] i By De Morgan's laws, the weighted Lukasiewicz t-conorms for 2 and n variables used in logical OR are:
[0088]
number
[0089] In either case, all w i =β=1, the unweighted Lukasiewicz norm is obtained. If any of these parameters are omitted, their assumed value is 1. For integer weights k, this form of weighting is used for each unweighted norm (e.g.,
number
number
[0090] The bias term β can be adjusted by adjusting β to give the classically equivalent equations x → y, ¬y → ¬x, and
number
[0091]
number
[0092] It is given as follows:
[0093]
number
[0094] In each of the above constraints, β is a bias term that establishes the operation, and the weight w i represents importance, and w x is the weight for the input operand x, and w y are the weights for the input operand y. The bias term β can have a value of 1, but can also have another value. The fixed interior is a representation or formula of the hyperplane, i.e., the dot product between the weight (input) vectors plus an offset.
[0095] In the weight of the consequent
number
number
number
number
[0096] The Lukasiewicz norm is the function of the input x i Together with weight w i When changing the order of, it is commutative and the bias term β≦min{1,w i :i∈I}. Furthermore, the Lukasiewicz norm is associative if 1≦β≦min{w i For classical inputs, i.e., for all i∈I, we return classical results, i.e., results in the set {1,0}. This clearly requires β=1 to have both associative and classical behavior, which is also not a requirement of LNNs. In fact, β≦w i constraining i to w effectively removes i from the input set I when the constraint no longer applies. i If it becomes 0, there is a problem.
[0097] Parameter Semantics
[0098] Weight w i does not have to sum to 1, so the weights w iis best interpreted as absolute importance, as opposed to relative importance. As mentioned before, in the conjunction case, an increased weight amplifies the distance of each input from 1, while in the disjunction case, an increased weight amplifies the distance of each input from 0. A decreased weight has the opposite effect, to the extent that an input with a weight of 0 has no effect on the outcome at all.
[0099] The bias term β is best interpreted as continually changing the "difficulty" of satisfying an operation. In weighted Lukasiewicz logic, this can even be transformed from one logical connective to another (e.g., from logical AND to logical OR). β and w i Constraints imposed on can, for example, ensure that if the inputs are assumed to be within a certain distance from 1 or 0, then the operations performed at each neuron match the corresponding connectors in the represented expression.
[0100] Activation functions for aggregation
[0101] In contrast to neurons related to connectors in a set of expressed expressions, neurons related to atoms require a slightly modified set of activation functions to function. Specifically, these activation functions must aggregate the truth-value boundaries found for the various computations identified in the atomic proof. For example,
number
number
number
number
[0102] The simplest means of aggregating proven truth values is to return the maximum of the lower bounds of the input truth values and the minimum of the upper bounds of the input truth values. However, it may be desirable to employ weighting of importance in aggregation as well. To this effect, one may aggregate the bounds of proven truth values using either the weighted Lukasiewicz norm or the weighted Godel norm (but note that the Lukasiewicz norm is highly prone to amplifying multiple sources of weak evidence into strong evidence for truth or falsity). This is done by computing the OR of the real values of the input lower bounds and the AND of the real values of the input upper bounds. To be clear, the activation function for aggregation differs from that for combinators in that combinators use the same norm for both upper and lower bounds, whereas aggregation uses the opposite norm for each bound.
[0103] The weights of the various proofs involved in the aggregation may be learned in a similar manner to the weights for the operands of a combinator. Alternatively, the weights used in the formula that serves as the atomic proof may be reused as weights for the proof itself. In an embodiment, the weights of the various proofs involved in the aggregation may be learned in a similar manner to the weights for the operands of a combinator. Alternatively, the weights used in the formula that serves as the atomic proof may be reused as weights for the proof itself. For example,
number
number
number
number
[0104] In principle, there are many candidate activation functions that can correspond to the classical truth functions of the corresponding connectives of neurons, and these activation functions differ in how strictly they treat inputs between 0 and 1. The choice of activation function is important because it interacts with the differentiability of the LNN as well as whether various tautologies in classical logic (e.g., p→q≡¬p∨q) can be proven over the entire range [0,1].
[0105] Triangular norms or t-norms as well as their t-conorms and residuals are natural choices for such activation functions, as they have been thoroughly investigated in the fuzzy logic literature. They already behave well for classical inputs and have well-known inference properties. However, t-norms usually do not bear useful parameterizations that can serve as targets for neural net optimization. Therefore, for AND activation functions, the weighted Lukasiewicz t-norms for 2 and n variables are defined as follows:
[0106]
number
[0107] Here, the input set I, the bias term β, and the weights w i , and input x i β and all w iFor integer weights k, this form of weighting is equivalent to the standard Lukasiewicz t-norm (e.g.,
number
[0108] For an OR activation function, the associated weighted t-conorm is
[0109]
number
[0110] These are bounded to be associative under reasonable constraints on the weights, and to be weighted residuals (i.e., IMPLIES).
[0111]
number
[0112] That is, it has been shown to include classical tautologies over the entire range [0,1] precisely. For LNNs, use cases that receive an existing knowledge base defined with the assumption that it will be evaluated according to a classical logic scheme can be evaluated using these weighted fuzzy logic schemes.
[0113] Many other tautologies are also possible, such as associative (with reasonable constraints), negation (¬p=1-p), double negation / elimination (¬¬p=p), entailment tautologies,
number
[0114] inference
[0115] Inference refers to the entire process in which truth boundaries for (sub)formulas and atomic units are calculated based on initial knowledge, ultimately resulting in a prediction made in a neuron on the queried formula or other outcome of interest. LNNs characterize this process using multiple passes over the represented formula, propagating the tightened truth boundaries between neurons until the calculation inevitably converges. Thus, in addition to the important upward pass of inference, where formulas calculate truth boundaries based on the available bounds for their subformulas, there is a downward pass that allows prior beliefs in the truth or falsity of the formula to inform the truth boundaries for the propositions or predicates used. In an embodiment, a simplified procedure of evaluation is implemented in which the calculation is managed using a priority queue or similar data structure, such that the method iteratively (re)calculates the truth boundaries for each neuron popped from the queue, and then pushes all neurons that receive the popped neuron as input into the queue only if it is observed that the bounds are tightened.
[0116] Two-way inference
[0117] In addition to computing truth bounds for each combinator's output based on its inputs, the LNN infers truth bounds for each of the combinator's inputs according to the combinator's inverse activation function with respect to that input and the known truth bounds for the combinator's output and other inputs. The downward pass of inference performs all such inverse computations in a depth-first traversal across all the neurons representing the syntax tree of each expression.
[0118] Depending on the type of connectives and which inputs are involved, such computations correspond to well-known inference rules of classical logic: in embodiments, bidirectional inference rules for Lukasiewicz's activation functions are propagated, which are equivalent to inference rules in classical logic forms (e.g., modus ponens and related forms below).
[0119]
number
[0120] Here,
number
[0121] The exact nature of these computations depends on the set of activation functions selected (e.g., weighted Lukasiewicz logic or weighted Goldel logic). For example, if implication is defined as a residual, then modus ponens is performed via the t-norm of the logic (i.e., AND). The remaining inference rules follow a pattern similar to that prescribed by the inverse function of the upward computation.
[0122] In the following, U and L denote the upper and lower bounds, respectively, found in the neuron corresponding to the formula indicated by the subscript, e.g., L x→y is a lower bound on the truth value of the set of expressions x → y, while U x is an upper bound on the truth value for x only. Computing the bound on ¬ is trivial: L ¬x ≧¬U x =1-U x , L x ≧¬U ¬x =1-U ¬x , U ¬x ≦¬L x =1-L x , U x ≦¬L ¬x=1-L ¬x
[0123] The use of inequalities above acknowledges that tighter bounds on the values may be available from other sources. For example, y and x → y are both L ¬y or , and the more stringent of the two will apply.
[0124] Typically, using two-way inference, uncertainty is handled in both the facts and the formulas (rules), and lower and upper truth bounds are applied to both the facts and the rules, which affect how the facts are updated or how the rules can be learned to be false. Thus, in two-way inference, both rules can be trusted, the learned "new" fact (or facts) can be trusted, and the truth value of the rule can be inferred or learned. For example, with respect to a fact that is believed or trusted, the rule can be learned, may be incorrect, and can be removed. In addition to learning the truth values of both the facts and the rules (by upper and lower bounds) by feedforward and backward pass inference, the LNN system can learn how much to weight the facts in their rules. That is, facts can have gradual changes in truth or degrees of certainty or uncertainty, and the weights associated with the arrows in the backpropagation path can be adjusted. Thus, in an embodiment, the truth values (upper and lower bounds) can be adjusted, and the weights associated with the backpropagation arrows can be adjusted. In an embodiment, refined and new rules triggered by observations are extracted from the trained LNN for completion of the knowledge base used to initialize the neural network or for discovery of new knowledge. In a further embodiment of inverse inference, a weighted inverse inference technique is implemented such that if the importance weight is less than the average value, the lower bound regresses to 0 and the upper bound regresses to 1 in proportion to 1 minus the ratio of the importance weight of the input to the average value of the importance weights of all inputs of the neuron.
[0125] Inference rules in weighted Lukasiewicz logic.
[0126] In weighted Lukasiewicz logic,
number
[0127]
number
[0128] Meanwhile, the lower bound calculation with respect to y (i.e., modus ponens) is:
[0129]
number
[0130] The lower bound calculation on x (i.e., the modus torsion) is:
[0131]
number
[0132] The introduced condition is due to the fixation used in the double bounded ReLU. x→y = 1 and L x→y = 0. To understand why these occur, observe, for example, that whenever x ≤ y in the unweighted case, x → y can return 1 for many different values of x and y. Thus, U x→y = 1, no upper bound on y or lower bound on x can be inferred. The condition in the other direction works similarly, although it is only achievable if β>1.
[0133]
number
number
[0134]
number
[0135] On the other hand, the downward bound calculation for conjunctive syllogisms and conjunction elimination is as follows:
[0136]
number
[0137] The downward bound computation for disjunctive syllogisms and disjunction elimination is as follows:
[0138]
number
[0139] Note that the above downward computation is not necessarily implemented as multiple operations in series, but can be computed using a form of implication of n-ary variables, or equivalently, a form of disjunction and conjunction of n-ary variables including negation.
[0140] Regression Algorithms and Evaluation
[0141] In an embodiment, to perform an inference task using an LNN, a syntax tree is traversed to obtain the truth value at a particular output node, given known truth values at the input nodes.
[0142] FIG. 5 illustrates a method 500 for evaluating propositional LNNs and first-order logic LNNs according to an embodiment in which the recurrent neural net includes neurons in a one-to-one correspondence with the expressions representing the system being modeled, with the neurons arranged as a syntax tree of the expressions. The method 500 controls the overall method of prediction or bidirectional reasoning, and 503 illustrates a first step of initializing neurons corresponding to the roots of the propositions and expressions, which includes initiating truth value boundaries. This corresponds to an operation that includes obtaining data value inputs from the outside world to initialize the neurons with observed truth values. All neurons inside the expressions are initialized with unknown truth values, while other neurons related to the entire expressions trusted in the network may include truth values that are learned and / or assumed to be "1". For the proposition 202 in the example of FIG. 2A, these initial truth values can be observed values, e.g., "whiskers" are known to be greater than 0.7, "tails" are known to be greater than 0.8, while "laser pointer" has been observed to be less than 0.2, for example. The propositions including starting boundaries represent the input data.
[0143] The traversal of the knowledge graph in the LNN is iterative, i.e., repeated in both directions, and while the system is not converged, forward passes, backward passes, and evaluation of truth-value aggregates on propositions / predicates are performed.
[0144] Step 506 in Figure 5 illustrates expression-by-expression processing, where neurons are evaluated forward in paths from the leaves of the syntax tree to the root of each expression, while annotating each neuron in the upward pass with the resulting truth-value bounds after performing logical operations.
[0145] FIG 6A is an exemplary method 600 for evaluating neurons in a forward pass of a knowledge graph from a leaf to the root. As shown in FIG 6A, each node j (neuron) is evaluated in the upward pass. The first test 605 is whether the node is an atomic node, and if so, the upper and lower bounds are evaluated and returned. If not, a loop is performed at 608 to evaluate a value L for each operand i. i,j and U i,j is calculated. Next, the argument β j , w j , L j , U j Assuming that the activation function f j () is evaluated. f j (β j ,w j ,L j ,U j ) is returned and stored for later use by the program calling the logical neural net. While the bounds calculated at each neuron are stored, the truth values L j ,U j is propagated upwards through the graph. If there is already a tightest usable bound from a previous computation step, then that tightest usable bound is retained. That is, for a proposition that can be proven in many ways, the tightest such bound becomes the smallest upper bound any computation can return, and the largest lower bound any computation can return.
[0146] Each node uses operands A and B, and each operand / operator A
number
[0147] Negation: L ~A =~UA U ~A =L A
[0148] Conjunction (forward reasoning) L A&B =L A &L B U A&B =U A &U B
[0149] Disjunction (forward reasoning)
[0150]
number
[0151] Implication (forward inference) L A→B =U A →L B
[0152] U A→B =L A →U B
[0153] The inverse function is L A =U A = 0, it becomes false and L A =U A = 1, the lower limit L A becomes 0, and U A If L is 1, the truth value is unknown. A U A If it is larger than L A >U A If so, then a contradiction exists.
[0154] Further examples of reverse forward passes through the knowledge graph are shown in Appendix A.
[0155] Returning to Figure 5, after reaching the root of the expression, at 509 the system uses a reverse calculation to backtrack to each leaf and update the bounds of the subexpressions based on the stored bounds (and the initial bounds of the expression). That is, the backward pass goes back down through the graph to the predicates or propositions and propagates inference based on known truth values.
[0156] FIG 6B is an exemplary method 610 for evaluating neurons backwards on a backward pass of the knowledge graph from the leaves to the root. As shown in FIG 6B, on the downward pass, each node j and its current upper bound U and lower bound L are evaluated. The first decision 615 is whether the node is an atomic node, and if so, the upper and lower bounds are recorded. If not, a loop is performed at 618 to evaluate the value L for each operand i. i,j and U i,j The upper and lower limits are calculated using the argument β j , w j , L -i,j , U -i,j Inverse activation function given L and U
number
number
[0157] Each node uses operands A and B, and each operand / operator A
number
[0158] Negation: L A =~U ~A U A =~L A
[0159] Conjunction: A→(B→(A&B)) L A&B L if >0 A =U B →L A&B , otherwise 0 U A =L B →U A&B
[0160] Disjunction:
[0161]
number
[0162] Implication (B):A&(A→B))→B L B =L A &L A→B U A→B < 1 then U B =U A &U A→B , otherwise 1
[0163] Implication(A):A→((A→B)→B) U A→B L if < 1 A =U A→B →L B , otherwise 0 U A =L A→B →U B
[0164] Typically, in the backward pass, the calculations are the inverse of the activation function calculations performed in the forward pass, and the results are expressed in terms of bounds.
[0165] An exemplary recursive (bidirectional) backward propagation (backward pass) corresponding to the exemplary LNN 200 shown in FIG. 2A is shown in FIG. 2B. While the system is not converged, a backward traversal of the knowledge graph occurs to update the truth values of various propositions. In the LNN, all paths from the root node 215 to all leaves are traversed, and at each node, the reverse inference rules are evaluated downwards. In an unguided embodiment, a depth-first traversal is performed going through the paths in the syntax tree from the root to the leaves for each expression. For example, the first path evaluated is a modus torrens edge 242 that attempts to disprove a conjunction based on knowledge of “cat”, followed by a conjunction syllogism edge 242A that attempts to disprove “whiskers” if the conjunction is false. Then, the evaluation of the next conjunction syllogism can be performed at edge 242B, then 242C for the sub-expression “laser pointer means chasing” at node 210B, and so on.
[0166] In an embodiment, information is back-propagated along the following backward paths to update the inverse function calculations in tree 204: the path indicated by arrows 242, 242A, where the truth value (i.e., upper and lower bounds) of predicate “whiskers” 202A is updated by the inverse activation function calculation in conjunctive neuron 210A; the path indicated by arrows 242, 242B, where the truth value (i.e., upper and lower bounds) of predicate “tail” 202B is updated by the inverse activation function calculation in conjunctive neuron 210A; the path indicated by arrows 242, 242B, where the truth value (i.e., upper and lower bounds) of predicate “laser pointer” 202C is updated by the inverse activation function calculation in conjunctive neuron 210A and implication neuron 210B; 10B, a path indicated by arrows 242, 242C, 242D, where the truth value (upper and lower bounds) of the predicate “chase” 202D is updated by the inverse activation function calculations in the conjunctive neuron 210A and the implication neuron 210B, a path indicated by arrows 242, 242C, 242E, where the truth value (upper and lower bounds) of the predicate “cat” 202E1 is updated by the inverse activation function calculations in the implication neuron 215, and a path indicated by arrow 243, where the truth value (upper and lower bounds) of the predicate “cat” 202E1 is updated by the inverse activation function calculations in the implication neuron 215. Similarly, the following backward paths are traversed to perform the inverse function calculations in the syntax tree 207: That is, those paths are the path indicated by arrows 272, 272A, in which the truth value (upper and lower bounds) of the predicate “cat” 202E2 is updated by an inverse activation function calculation in the disjunction neuron 210C, the path indicated by arrows 272, 272B, in which the truth value (upper and lower bounds) of the predicate “dog” 202F is updated by an inverse activation function calculation in the disjunction neuron 210C, and the path indicated by arrow 273, in which the truth value (upper and lower bounds) of the predicate “pet” 202G is updated by an inverse activation function calculation in the implication neuron 225.
[0167] Further examples of backward passes through knowledge graphs are shown in Appendix A.
[0168] 2C illustrates an example of an inverse activation function used to compute for the reverse inference of the syntax tree of the use case with respect to the downward traversal path shown in FIG. 2B. For example, a backward traversal of the syntax tree along paths 242, 242C, 242E updates the truth value (upper and lower bounds) of the predicate "chase" 202D by performing a computation of the inverse activation function 281 (using equal weighting) in the root implication neuron 215, performing a computation of the inverse activation function 282 (using equal weighting) in the conjunctive neuron 210A, and performing a computation of the inverse activation function 283 in the implication neuron 210B, using the strictest bounds available at each node. Similarly, a backward traversal of the syntax tree along paths 272, 272A updates the truth value (upper and lower bounds) of the predicate "cat" 202E2 by performing a computation of the inverse activation function 291 (using equal weighting) in the implication neuron 225, and performing a computation of the inverse activation function 292 in the disjunction neuron 210C.
[0169] Referring again to FIG. 5, we proceed to 512 where the system aggregates the tightest bounds computed at the leaves for each proposition / predicate. For the aggregation at 512, min and max can be used to aggregate the upper and lower bounds proven for each proposition, although smoothed versions of min and max may be preferred to spread gradient information across multiple proofs. Alternatively, when targeting classical logic, conjunctions and disjunctions (possibly smoothed themselves) can be used to aggregate the bounds of propositions. In that case, there is an opportunity to reuse the weights of propositions from each proof to limit the impact of proofs in which the proposition plays only a small role.
[0170] 6C illustrates a method 620 for aggregating an atomic node j. As shown in 625, the atomic proof i, its current upper bound U j , and the lower bound L j Tuple of (L i, j ,U i, j ), a FOR loop is implemented.
[0171] Next, in 628, for node j, j =maxL j , U j =maxU j According to the,strictest bound U for each leaf, j , L j The aggregate of is calculated.
[0172] In the use case of the LNN 200 of FIG. 2A, values are aggregated across all predicates, and in this use case, aggregation is a "no operation" since there is only one proof in the network other than the predicate "cat". In this example, upon convergence, the predicates "cat" 202E1, 202E2 are the intended result (the predicate of interest) since two different means of inferring whether something is a "cat" are provided in the network. The updated predicate "cat" that provides the tightest bound is the aggregated truth output. Typically, in an LNN, a predicate or proposition serves as a candidate for a bound proof no matter where it appears in any expression.
[0173] Of the bounds computed on the propositions, the method aggregates the tightest bounds, i.e., selects the largest lower bound and the smallest upper bound. In the exemplary LNN model 200 shown in FIG. 2A, whether something is a “cat” can be proven by two methods, for example, based on tails, whiskers, laser pointers, or based on pets and dogs. An exemplary aggregation step may evaluate one of these proofs that something is a “cat” as, for example, a lower bound of 0.6, and aggregate the other proofs as, for example, a lower bound of 0.7. Using the tightest bound (e.g., max), the proof of the proposition “cat” associated with the aggregated value 0.7 becomes the deciding factor.
[0174] Thus, given the exemplary task of predicting whether something is a "cat", the end result will be both bounds of that proposition; for example, if the lower bound is high, something is inferred to be a "cat", if the upper bound is low, something is inferred not to be a "cat", and if the lower and upper bounds are loose, a response is returned that it is known that it is not possible to determine whether something is a "cat". If the upper and lower bounds are found to cross, then a state of inconsistency exists in the system of equations.
[0175] In use, after every evaluation (i.e., forward and backward reasoning passes), there is a set of one or more neurons in the LNN that act as outputs that can be examined and used when needed. The neurons that answer the task of predicting whether something is a "cat" become the final outputs in this use case of FIG. 2A.
[0176] 5, proceeding to 515, a determination is made as to whether the bounds have converged when the aggregated value has not changed significantly (e.g., everything that can be proven based on the given information has been proven, e.g., a measure of the change in the truth bounds of the last pass relative to the current pass is less than 0.01, etc.). In an embodiment, due to the nature of LNN computation, the truth value is expressed in terms of upper and lower bounds, so that the upper bound can decrease or the lower bound can increase (the largest lower bound is 1 and the smallest upper bound is 0). As a result, the aggregated truth value converges to a fixed value or within an amount of epsilon (ε) from the fixed value, depending on the activation function of the neuron.
[0177] If the bounds have not converged, the method returns to step 506 and repeats step 506 in a forward pass and a backward pass via 512. In an embodiment, tightening the bounds is monotonic so oscillations cannot occur.
[0178] After convergence of the bounds is detected at 512, the process proceeds to 520 where the method performs a check of the computed bounds on a particular proposition or formula (i.e., a proposition or formula that represents the prediction / query of the model).
[0179] As proposed in step 520, a predicted result is obtained by inspecting the output of one or more neurons, as is done in the case of a conventional neural network, except that different neurons may serve as inputs and results for different queries, and indeed the results of one query may possibly be used as inputs for another query. In addition, an existing LNN model may be optionally extended with neurons representing new expressions to serve as new queries.
[0180] More complete working examples corresponding to the LNNs in Figures 2A-C are shown in the examples labeled #1, #2, and #3 provided in Appendix A.
[0181] Two-way inference
[0182] Logical expressions can be evaluated directly, but can also be used to infer the truth values of other related expressions and propositions. The LNN performs such inferences when the inferences relate to subexpressions and propositions present in the represented syntax tree. In addition to computing truth-value boundaries for each combinator output based on various inputs, the LNN infers truth-value boundaries for each of the combinator neuron's inputs according to the combinator neuron's inverse activation function for that input and known truth-value boundaries for the combinator's output and other inputs. Depending on the type of combinator and the inputs involved, such calculations correspond to well-known inference rules of classical logic, including modus ponens, modus tollens, disjunctive syllogisms, and many others, such as:
number
[0183]
number
number
[0184] L ¬p =¬U p =1-U p , L p =¬U ¬p =1-U ¬p , U ¬p =¬L p =1-L p , U p =¬L ¬p =1-L ¬p
[0185]
number
[0186]
number
[0187] The backward computation of the upper bound is similar to implication, but
number
[0188]
number
[0189]
number
[0190] The calculation of the lower bound in the reverse direction is similar, where
number
[0191]
number
[0192]
number
[0193] The introduced condition is due to the fixation used in the double bounded ReLU.
number
number
number
number
[0194] In an embodiment, the LNN thus infers upper and lower bounds for each neuron's input (or equivalently, the operands of a logical connective) based on the neuron's inverse activation function with respect to the input, as well as known bounds on the neuron's output and the neuron's other inputs.
[0195] For these exemplary inverse upper and lower bound calculations, the subscripts p and q represent which part of the formula the truth value pertains to. Thus,
number
number
number
[0196] More specifically, for an exemplary inverse upper bound, the expression
number
[0197] In general, the LNN approach thus infers upper and lower bounds for each input of a neuron (or equivalently, the operands of a logical connective) based on the inverse activation function of the neuron with respect to the input, and known bounds for the neuron's output and other inputs of the neuron. The LNN approach further involves using each occurrence of each proposition in each formula as a proof of the bounds of the proposition, computed recursively and / or dynamically as necessary, by first computing bounds for each of the (sub-)formula's outputs and other inputs.
[0198] Furthermore, the activation functions and boundary calculations of each neuron with respect to the inputs, instead of having true inverse correlations, are smoothed, fixed, or otherwise transformed versions, or a combination thereof, of some other function and its inverse, respectively.
[0199] Furthermore, neurons corresponding to propositions may aggregate bounds using smoothed, fixed, or otherwise transformed versions of these functions, or combinations thereof, instead of the traditional min and max, including but not limited to activation functions that are the same as the activation functions for the logical connectives AND and OR for aggregating upper and lower bounds, respectively.
[0200] For any system of logic (e.g., a system other than the Lukasiewicz logic system), in general, if the conjunction is "min", the disjunction is "max", and the implication involves multiple alternatives (not A or B) as candidates, then the values of min, max, and not A or B each have a functional inversion (the reverse inference of that system of logic).
[0201] Referring to FIG. 2A, which uses the exemplary net 200 for computational purposes, a user, via a computer system, may submit various queries to the LNN model, asking various questions of the same model 200. In the case of this exemplary net, the user may wish to predict whether something is a "cat", i.e., given the input, "cat" is what is required to be proven, and the neuron relating to "cat" provides a corresponding output, e.g., the value at that neuron provides the output. To predict whether something is a "cat", data relating to each of the propositions 202 may be presented as part of the prediction process. However, in another embodiment using the exemplary trained net 200, it may be desirable to provide a different output (e.g., "pet"), and a different set of inputs may be provided.
[0202] FIG. 3 illustrates an exemplary implementation for injecting additional expressions into an existing trained LNN, such as the exemplary net 200 of FIG. 2A. As shown in FIG. 3, a new expression / subexpression 250 of constituent neurons containing a conjunction connective 251 and a negation connective 252 is shown to add the constraint that something cannot be both a cat and a dog. The added syntactic logic in the injected expression 250, when inserted into the network 200, includes an indicated truth value 253, where the degree of truth can be a learned parameter (e.g., learned during an optimization pass). In an embodiment, given a data set of various combinations of truth values of known propositions, for each training pass across each combination, the optimization randomly selects a subset of the known propositions to serve as observations and uses the withheld propositions as queries.
[0203] An additional exemplary new expression 260 is shown that has been created to include a conjunction connective 261 and a negation connective 262 to provide further logical syntax for processing queries that the initial LNN 200 was not configured to process (i.e., "Is there something that doesn't have a tail but is a pet?"). After several passes moving the truth value through the network, the truth value can be propagated into the query's syntax tree representing a sub-expression 260 that can provide a value output 263 (e.g., a definitive true, false, or undetermined response).
[0204] In this use case of FIG. 3, this insertion of a new formula 260 into an existing LNN is a "created query" that the LNN is used to provide an output for. A query is a formula that can ask something and is inserted into an LNN. This exemplary created query does not have any particular truth value and is added to evaluate whether it is proven that there exists a pet that does not have a tail based on the input. This created query is the insertion of neurons corresponding to the specified query formula into the existing logical neural network such that inference (i.e., evaluation of the neural network containing the inserted formula) produces a result in the form of a truth value bound computed by the neurons for the query formula.
[0205] Additionally, each one of the nodes 210 in the LNN computes upper and lower bounds on the truth values. These upper and lower truth values are continuous variables at each node and are passed throughout the network. These bounded truth variables are calculated at the node at prediction or inference time, calculated based on the inputs and with respect to learned importance weights, and so are not structurally part of the LNN that was initially constructed. For example, the net 200 may be programmed with inputs of certainty (e.g., something has whiskers, something has a tail, something is a pet, or something is not a dog), but it may be unclear whether a laser pointer is present. Thus, there may be enough input information to conclusively prove that something is a "cat," but the LNN 200 returns that it cannot determine that it is a cat.
[0206] First Order Logic
[0207] According to further embodiments, the LNN supports complex first-order logic (FOL) expressions that implement first-order logic connectives employing universal and existential quantifiers (e.g., "for all" and "exists"). Thus, rather than characterizing propositions as pairs of values (e.g., upper and lower bounds), in an embodiment of an LNN that implements first-order logic processing, predicates are characterized in terms of sparse, but vectors, matrices, or even larger structures of value pairs, and the LNN's inference operations take into account the sparsity of the data. In an embodiment, a keyed table data structure representation of sparse matrices is used. For predicates, since the LNN's syntax tree contains matrix-valued neurons, the LNN's inference operations perform a tensor product or table join operation to return the entire table of values. For example, for activation functions in the interior neurons (e.g., conjunction / disjunction), the LNN performs the join along with the normal function, and for each row that is the result of the join, it also performs a calculation based on the inputs for that row. In an embodiment, the table of truth bounds for (sub-)expressions and predicates includes specified default bounds for missing key combinations, but in many cases the default bounds for (sub-)expressions are not necessarily selected to match the result given the default bounds of the inputs.
[0208] First-Order Logic Neural Networks
[0209] To extend LNNs with first-order logic inference, scalar propositions become predicates and are materialized as tables or equivalently tensors, or as neurons replicated for each justification. That is, LNNs are extended to first-order logic by replacing proposition neurons with neurons representing predicates and introducing nodes for universal quantification and existential quantification. Instead of pairs of bounds, every neuron returns a table uniquely keyed as above by a group of columns for unbounded variables that appear in the represented subexpression or predicate. The activation functions of the neurons are then modified to perform a join across all columns for shared logical variables, computing truth bounds on the relevant rows as before. The inverse activation functions are modified similarly, but to aggregate the tightest bounds, the results must also be reduced across any columns for variables that are absent in the corresponding subexpression of the target input. In the special case where tables are keyed by consecutive integers, these calculations are equivalent to an element-wise broadcast operation on sparse tensors, where each tensor dimension is associated with a different logical variable. Additionally, predicates grounded over all the same variables may be linked to the same tensor structure to allow easy lookup.
[0210] First-order logic LNN processing utilizes universal and existential quantifiers, and includes, but is not limited to, reduction by min or max, inverse inference to propagate the associated bounds downwards, and the possibility of adopting other bounds by skolemization, where a group of new keys (columns) is introduced for each quantified variable, along with bounds, in addition to the usual FOL variable columns. In an embodiment, special neurons corresponding to universal and existential quantification are introduced to eliminate columns of certain keys (i.e., variables) by reducing the bounds of truth values for matching combinations of keys in other columns, using min and max, respectively. For example, in an embodiment, given the tensor-type neurons in the LNN first-order logic processing, the activation function of the neuron has changed so that it performs a combination. The inverse of the combination may require a reduction, i.e., an operation to remove a column from a table by selecting the maximum of all of the lower bounds, or the minimum of all of the upper bounds, or the inverse of these, depending on which operation is being performed. Each of these operations involves matching values in the column, performing an operation on the matched rows, and possibly removing the column by performing a reduction (e.g., ignore) on the column.
[0211] Like negation, quantifiers are parameter-free pass-through nodes that eliminate a particular key sequence by reducing it using min or max as necessary. However, inverse inference of quantifiers is more complicated in that it requires skolemization: each existential quantifier broadcasts a known upper bound to all key values associated with the sequence (i.e., the variable) and a known lower bound to the group of new key values identified by each combination of key values associated with any containing universal quantifier sequence, and vice versa for universal quantifiers.
[0212] To make aggregation operations tractable, embodiments need to limit the number of key values participating in the calculation while leaving other key value combinations sparse (i.e., with default bounds). This is achieved by filtering predicates in favor of calculations toward smaller contents whenever they can only include contents related to specific key values referenced in the query or involved in joins with other tables. In this model, the results of quantifiers and other reductions may not be severe because many truth values remain uncomputed, but they are normal nonetheless. When a predicate has a known truth value for all key values (i.e., because the predicate makes a closed-world assumption), the method uses different bounds for the sparse values of the predicate and for the sparse values of the connectives containing the predicate, such that the sparse values of the connectives are the result of the sparse values of the input.
[0213] It is necessary to guide the evaluation of neurons towards rules that are more likely to produce useful results while minimizing the number of key values participating in the computation. The first opportunity in this sense is to shorten a computation when it cannot produce tighter bounds than those previously stored at a particular neuron. In addition, the method exploits the graph structure of neurons to prioritize evaluations on rules that contain shorter paths to the query and to visited rules that contain recently updated bounds.
[0214] With the goal of not feeding too much data into the FOL LNN, a stepwise grounding decision is implemented to use the groundings of neurons in the input network to determine the effective minimization of the amount of a neuron's grounding set based on its set of related operations. For a conjunctive neuron, first the number of groundings of its inputs are counted individually, and the input with the fewest groundings is selected, then it is checked whether those groundings are contained in the other inputs, and if so, that grounding is kept. For the conjunctive case, this involves performing an intersection operation on the input set with stepwise data access from performing the count, taking the groundings of one operand, and finally performing the containment check.
[0215] Managing variables
[0216] Logical predicates, neurons, and quantifiers each have a defined set of variables according to their arity, which specifies the number of constants in the justification tuple. The arity of the predicates depends on the knowledge base that supplies the truth values to the LNN and can typically include a variety of zero-variable (propositional) predicates, one-variable predicates, and two-variable predicates. Logical neurons collect variables from a set of input operands in order of appearance at initialization, and these operands can include predicates, neurons, and quantifiers. Variables are collected only once from operands that define repeated occurrences of a particular variable in two or more variable positions, unless otherwise specified. Logical expressions can also be defined using arbitrary variable placements across the constituent nodes. Variable mapping operations transform the justifications to enable the search for truth values in adjacent nodes.
[0217] Although a quantifier can contain multiple variables and justifications when partial quantification is required for only a subset of variables from the underlying operands, quantification is typically performed on a single variable to generate the truth value of the proposition associated with the quantifier output. In the case of partial quantification, the greatest lower bound of the justifications from the quantified subset of variables is selected for the existential quantification and assigned to a unique justification consisting of the remaining variables, while the least upper bound is used for the universal quantification. In the case of existential partial quantification, the truth justifications of the quantified subset of variables form an argument that is stored under the justifications of the subset of remaining variables, so that it can be recalled that the justifications satisfy the
[0218] Variable Bindings
[0219] Variable binding assigns a specific constant to a variable of a predicate, neuron, or quantifier, usually as part of the reasoning task. A variable can be bound only in a subset of its occurrences in a logical formula, but the procedure for generating reasoning usually propagates the binding to all occurrences. Thus, a variable needs to be preserved even when bound in order to interact with other occurrences of the variable in the logical formula to perform combine operations. There can be implicit equality of constants referenced by the same identifier, but the same represented entity can also be denoted by different constants.
[0220] Classification of variables
[0221] Variables in classified first-order logic can have a type assigned to a particular use case, just as a particular variable can have different types assigned in different use cases in different parts of a logical formula. Two predicates involving a shared variable can each specify a different expected type for the use of the variable. Types are typically ontology class identifiers, allowing taxonomic and hypernymy relationships to be identified between two types, such as stating that a particular type is a subtype of another type.
[0222] Type consistency in a logical formula means that a particular variable has a single type for all occurrences of that variable. Even if one type assignment is a direct subtype of another, type equivalence is required at all variable occurrences, since an entity correctly classified in one occurrence may be improperly specialized in another. Groundings that are not available in predicates, neurons, or quantifiers are usually assumed to be unknown according to the open-world assumption, and are assigned maximally loose truth-value bounds. However, if at least one of the grounding constants has an assigned type that is not transitively subtyped of the expected variable's type, a false value can be assigned.
[0223] Processing Equations in LNN
[0224] A general characterization of the processing of equations within an LNN is as follows.
[0225] (1) Equations are a special kind of predicate that interacts strongly with a set of second-order formulas; (2) mathematically, the truth values of equational predicates can be treated in the same way that any other truth values are treated, but there is a computational motivation to treat them specially; (3) proper treatment of FOL functions and Skolemization follows from proper treatment of equations.
[0226] equation
[0227] Partial equalities are fuzzy logic facts, but we can at least assert that (X=X)≡1 and (X=Y)≡(Y=X). Equalities are translated to truth values with (higher order) metarules such as: ∀P,X,Y.((X=Y)→(P(X)≡P(Y))) ∀Q, X,Y.((X=Y)→∀Z.(Q(X,Z)≡Q(Y,Z))) ∀Q, X,Y.((X=Y)→∀W.(Q(W,X)≡Q(W,Y)))...
[0228] All such rules are assumed to have a truth value of exactly 1, and the operations are not weighted (but still fuzzy). Logical equivalence can be a two-way implication, and the truth values on both sides must be equal. It can be reasonable to include ∀Q,X,Y.((X=Y)→(Q(X,X)≡Q(Y,Y))) and possibly ∀Q,X,Y.((X=Y)→(Q(X,Y)≡Q(Y,X))) as special cases. Simply including an implication to (X=Y)
number
[0229] By weakening the equivalence of both implications,
number
[0230]
number
[0231] Observe that f(A;f(A)) ≡ (f(A) = f(A)) ≡ 1. Similarly, choosing a function as the predicate, the above gives
number
[0232] Similarly for higher arity. ∀g,X,Y.((X=Y)→∀Z.(g(X,Z)=g(Y,Z))) ∀g,X,Y.((X=Y)→∀W.(g(W,X)=g(W,Y)))
[0233] As with regular predicates, it can be useful to include: ∀g,X,Y.((X=Y)→(g(X,X)=g(Y,Y))) and ∀g,X,Y.((X=Y)→(g(X,Y)=g(Y,X)))
[0234] The above comes with the same motivations and cautions.
[0235] We can then find the truth for a complex construction such as Q(f(X),g(f(Y),Z)) by combining the above rules as follows: Q(f(X),g(f(Y),Z)) ←Q(U,V) ←(U=f(X)) ←(V=g(W,Z)) ←(W=f(Y))
[0236] or equivalently,
[0237]
number
[0238] For the same result rationale (say X, Y, and Z), there can be multiple intermediate rationales, e.g. U, V, and W in this case, and as usual the most stringent computed bound applies.
[0239] All of the above can be done in both directions; i.e., a known equality proves that the truth values of the predicates must match, but the truth values of the known non-matching predicates proves an inequality.
[0240] Processing Functions
[0241] With regard to the specific justification of all arguments of a function (i.e., the assignment of constants to all arguments), the application of the function may itself act as a new constant that specifies the return value, e.g. literally f(cat123) or even g(cat123,f(dog456)), where f(dog456) is again a new constant. Thus, when a function is called with the same constant arguments, it returns the same "value." When a function is called with different arguments that may or may not refer to the same things, its return value may be proven to be equal to some other return value, following the above rules for equality. Functions are partial in nature: some argument configurations do not easily emerge, and insistence on the existence of return values for all of a function should be avoided.
[0242] Skolemization
[0243] Skolemization is necessary to handle both quantifiers, but for opposite bounds. Existential quantifiers pass upper bounds to all possible variable groundings, but (necessarily) pass lower bounds only to a single constant returned by the function of all enclosing variables groundings, and the reverse is true for universal quantifiers. Enclosing quantifiers that are themselves skolemized are not considered enclosing variables, allowing the functions introduced by skolemization to be simpler. Note that which quantifiers are skolemized depends on which bounds are being computed. Another consequence is that the computation of upper and lower bounds under quantifiers involves different tensor dimensions. These functions can then be treated like any other function, replacing the skolemized variables wherever they occur. The skolem functions themselves should be uniquely labeled (e.g. exists789) for each quantifier in each formula. The return value of the Skolem function is then identified as above by its application to the grounded arguments (i.e. the surrounding variables), e.g., if there exists ∀X.(cat(X) → ∃Y.(...)), then exists789(cat123). As a minor note, even if the result of the function is always unique, there may be no constant that satisfies the quantifier (though some logics include a separate operator for "there is exactly one", ∃!). Thus, Skolemization is normal, but not logically equivalent, and nevertheless does not cause problems in either fuzzy or classical logic. Skolemization is only required for downward reasoning. However, unless one introduces further extensions such as closed-world predicates, upward inference of existential quantifiers aggregates lower bounds using max but fails to tighten upper bounds, and the reverse is true for universal quantifiers, so in the case of closed-world predicates one can definitively conclude that no other justifications can yield truth values other than 0 or 1.
[0244] implementation
[0245] While mathematically correct, a huge number of computations are required to handle any equality, function application, or quantifier (on one side of the bound). There is furthermore the notion of equality classes: constants connected by some path of non-zero lower bound equality edges. Not all constants in the same equality class necessarily have non-zero equality with each other; for example, in Lukasiewicz logic, (x=y)≧5 and (y=z)≧5 cannot strict (x=z). That is, it is not immediately clear whether there is a (useful) definition of inequality classes: constants connected by some path of unitless upper bound equality edges.
[0246] For a particular equality class, there is a set of fully normalized truth values assigned to each edge that can be found by repeated application of transitive equality. Whenever an equality edge is updated by other means, it becomes necessary (eventually) to renormalize the equality class. Note that equality classes can be fused; a proven inequality (i.e., an upper bound less than 1) does not split an equality class. The normalized equality class can then propagate the truth values of the stricted equalities to predicates, including functional predicates. Functional predicates can exhibit cyclic updates if their results feed back to the equality classes that affect them; for example, functions that can return arguments that propagate equalities through such cycles can be considered part of the normalization.
[0247] It is important to know when to normalize and to what extent. It is possible to normalize on demand and then mark the equality class as no longer needing to be updated. Similarly, predicates on members of an equality class can be updated on demand and marked as up-to-date with respect to said class. As stated, on any update to an equality edge or any update to a predicate on a member of an equality class, each set of values must be marked as stale.
[0248] First-Order Inference
[0249] Reasoning on logical neurons involves forward and backward execution of the relevant logical connectives for a given set of justifications, while reasoning on quantifiers involves reduction operations in case of partial quantification and the creation of new justifications. The provided justifications may not be usable for all participating operands of the reasoning operation, in which case a search attempt adds previously unavailable justifications to operands with unknown truth values, according to the open world assumption. When proofs for unavailable justifications are provided to a node, the aggregation of these proofs also assumes a maximally loose starting bound.
[0250] Computational and memory considerations for large universes of discourse involving many constants should be considered, and steps may be taken to avoid storing reasoning with unknown bounds. However, inference, which is the primary means by which reasoning is propagated through logical expressions to enable theorem proving, may be able to avoid storage in some cases. In particular, negation may be viewed as a pass-through operation, where inference is instead performed on the underlying operands or descendants that are also not negated. If approached simply instead, negation may require adding data to all false reasoning lists or reasoning missing from the underlying operands, according to the closed-world assumption, and storing those data as true.
[0251] An inference context includes input operands and output operations, where the input operands are used in forward inference to compute a proof of the output, or all but one of the input operands and the output are used to compute a proof of the remaining input. If any participant in the inference context has a non-obvious justification, it is possible to derive a truth value in the inference context that is also non-obvious within real-valued logic. Thus, each participant in the proof generation can add their justification to the set of inference justifications. A particular inference justification is used as is for the operands of other participants that have the same variable configuration as the original operand. In the case of disjoint variables not present in the inference justifications, the overlapping variables are first searched for a match with all disjoint variable values used in the conjunction to create an expanded set of inference justifications. If no overlapping variables exist or a match is found, the overlapping variables can be assigned according to the inference justifications, and the disjoint variables are taken to cover the set of all observed combinations.
[0252] The set of relevant justifications from the context of real-valued reasoning can be a meaningfully expanded set, especially when there are disjoint variables. However, guided reasoning can be used to expand the minimal reasoning justification set, which contains only the justifications relevant to the target proof. LNNs can obtain the target proof using a combination of goal-driven backward chaining and data-driven forward chaining, where backward chaining is used as a means to propagate justifications in a search for a known truth value, which can then be used in forward chaining to infer the goal. Implication-based rules typically require backward reasoning, in the form of modus torrens to propagate justifications to the antecedent via backward chaining, and modus ponens to perform forward chaining to help compute the target proof in the consequent. This chaining process in both directions continues until the target justifications in the consequent are no longer unknown, or until the inference does not produce a tougher proof.
[0253] acceleration
[0254] Since tightening the bounds is monotonic, the order of evaluation does not change the final result. As a result, consistent with traditional theorem provers, computations can undergo significant speedups depending on the order in which the bounds are updated.
[0255] For such aggregations to be tractable, it is necessary to limit the number of key values that participate in the computation while leaving other key value combinations sparse (i.e., with default bounds). This is achieved by filtering predicates in favor of computations that tend toward smaller contents whenever they can only include contents related to specific key values that are referenced in the query or that participate in joins with other tables. In this model, the results of quantifiers and other reductions may not be severe because many truth values remain uncomputed, but they are nevertheless normal. When a predicate has a known truth value for all key values (i.e., because the predicate makes a closed-world assumption), different bounds are used for the sparse values of the predicate and for the sparse values of the combinators that contain the predicate, such that the sparse values of the combinators result in sparse values of the inputs.
[0256] It is necessary to guide the evaluation of neurons towards rules that are more likely to produce useful results while minimizing the number of key values participating in the computation. The first opportunity to this effect is to shorten a computation when it cannot produce tighter bounds than those previously stored for a particular neuron. In addition, the graph structure of neurons is exploited to prioritize the evaluation of rules that contain shorter paths to the query and to visited rules that contain recently updated bounds.
[0257] While the formulas and neurons shown in Figures 2A-3 operate on propositions (e.g., list 202) and pass values as upper and lower bound pairs (i.e., the values are scalar pairs), this functionality is extended as shown in Figure 4, described below, to enable the syntactic network to implement first-order logic inference. That is, instead of operating on propositions that have values as upper and lower bound pairs, the LNN is configured to process predicates (e.g., statements whose truth values depend on variables).
[0258] To process the predicates, neurons receive and pass along larger data structures or objects in the form of tables (e.g., tensors) of upper and lower bounds. Furthermore, the variables are the primary keys or dimensions of the tensors. The activation functions of neurons perform a combination, or equivalently, an element-wise broadcast. Inverse inference also involves a reduction operation.
[0259] The computation of activation functions in neurons is arranged to be tensor operations or combinations. For example, for a column in a table representing a particular variable that corresponds to the way a particular predicate is used, the operation between the two sets of tables is a combination on one of the columns and the matching row (the row to be combined), and the same activation function is called. Thus, activations in neurons generate new tables that can be of different dimensions, i.e., as the computation proceeds upwards through the syntax tree graph, the dimensions of the table can become larger and larger until a quantifier (e.g., "for all" (∀) or "exists" (∃)) is reached, in which case a reduction occurs for one of the dimensions. In first-order logic processing, the "for all" quantifier acts as a conjunction (AND) that applies to all possible evaluations of the variable, while "exists" acts as a disjunction (OR) that applies to all possible evaluations of the variable.
[0260] Guided Inference
[0261] In an embodiment, since tensors can be sparse and higher order, first-order logic LNN processing implements a backward chaining type algorithm in which inference patterns of first-order logic algorithms are induced based on the query, first examining which predicates and which constants in those predicates are involved in answering a particular query. The process then expands outward from that observation, incorporating only the data necessary to continue those calculations. In this way, a universe of discourse management or "guided inference" type process is implemented such that during each computation pass, decisions are made as to what data needs to be incorporated to update tensors (tables) to ensure that it is feasible to implement first-order logic in an LNN, for example by providing a mechanism for filtering out irrelevant work to save computation time and memory resources.
[0262] In particular, for universe of discourse management, the first-order logic neural network is configured to provide computations that function to reduce the number of justifications that must be loaded / computed to answer a query. Universe of discourse management includes managing which constants in the table are used to ensure that too much data is not loaded into the table to prove the query of interest. Meta-learning and heuristics are used to configure the first-order logic network to use guided inference to reduce the number of formulas visited per regression evaluation (e.g., to inform which formulas should be traversed first). Evaluation shortening is implemented, including, for example, prioritizing computations on neurons that are more likely to be on a critical path toward tightening the bounds on the particular neuron of interest, including, but not limited to, considering the distance of the shortest path of neurons to the neuron of interest. Thus, if there is a node on the graph that provides an answer to the query, the shortest distance path (length of the edge on the graph) to evaluate first within the LNN is found, or a switch in another algorithm informs which path to evaluate next. One or more first-order logic networks are configured using predicate defaults defined that allow for closed-world assumptions with respect to the particular predicate without representing too many facts.
[0263] Further use of LNNs in a Semantic Query Language for Databases (e.g., SPARQL) application for searching and manipulating data stored in the Resource Description Framework (RDF) format is presented in Appendix A.
[0264] FIG. 4 illustrates an exemplary syntax tree representing a first-order logic neural network (net) 400 containing logical expressions in the form of inference rules 412, 414, and 416 implementing first-order logic to answer the use case question "Was Natalie Portman born in the United States?" created by a human. For illustrative purposes, the inference is a conclusive "no", making the open-world assumption. In the illustrative embodiment, facts are selected from DBpedia. As shown in the exemplary syntax, the predicate "bornIn(·.·)" 402 is of arity 2, the predicate "partOf(·.·)" 404 is of arity 2, and the predicate "typeCountry(·)" 406 is of arity 1. The predicate "bornIn" 402 establishes a relationship between a person and the city in which they were born and is retrieved from DBpedia, the predicate "partOf" 404 is a more general relationship that applies to establish the hierarchy (e.g., which state or country a city is in, or which country a state is in), and the predicate "typeCountry" 406 is a general type predicate (e.g., "is something a country"). A further predicate "equals" 408 is used in this example simply to indicate whether one constant is the same as another constant.
[0265] In this example, the predicate bornIn() instantiates the open-world assumption because it relates everyone to every place. However, a concrete predicate such as "the astronaut on the Apollo 13 mission" is a closed-world predicate.
[0266] First, the variables A, B, and X, as well as the nodes that appear in the formula, can assume any value, including the United States, Israel, Jerusalem, etc., but can also include other possible values not mentioned. Each step of the inference illustrated in Figure 4 selects a particular formula and performs a calculation on a different set of candidate values for A, B, and X (if present). Given that the query provides clues (Natalie Portman and US) and the relation "bornIn", bornIn(Natalie Portman, US) is queried directly, which returns nothing (and therefore is not true) since it is not in the database. Other possible solutions could be to first query bornIn(Natalie Portman, any place) or bornIn(any person, US). These sets could be singular, but a more detailed explanation of these assignments follows.
[0267] In the first-order logic process illustrated in FIG. 4, a universe management algorithm is invoked to provide an inference induction (e.g., heuristic) that indicates to first perform a query using the smallest possible fill-in (e.g., in this use case, bornIn(Natalie Portman, anywhere)). That is, a first observation is made by fetching data about Natalie Portman from a knowledge base (e.g., DBpedia) to determine in which city the person Natalie Portman was born. Exemplary fact data 403 is found that the person Natalie Portman was born in the city Jerusalem, and everything else related to the predicate bornIn() remains sparse. Using this observation, the following operation is performed for all of the hits that could be selected for Jerusalem, with respect to the predicate partOf(). Thus, the second observation causes a search of the knowledge base to find data relating the country in which the city Jerusalem is located (405). Thus, in use case 400, the method first considers X=Natalie Portman, A=Jerusalem, and B=the country of Israel.
[0268] Inference rule 414 is first provided to computationally evaluate the logic (b(X,A)∧p(A,B))→b(X,B) stating that if Natalie Portman was born in a first location (Country A) and the first location (Country A) is part of a second location (Country B), then Natalie Portman was born in a second location (Country B). Given rule 414, to computationally prove that person Natalie Portman was born in Israel, i.e., the syntax tree 424 of the corresponding LNN is traversed, i.e., processed backwards, as indicated by arrows 423, 427, by implementing the logic to evaluate predicate data 403 “bornIn(Natalie Portman, Jerusalem)” AND predicate data 405 “partOf(Jerusalem, Israel)”, thereby yielding the resulting predicate 425 as “bornIn(Natalie Portman, Israel)”.
[0269] However, an additional step is required to prove that person Natalie Portman was not born in the United States, as requested in the original query. Thus, the method next considers both A∈{Israel, United States} and B∈{Israel, United States} and computes a result at each combination of these values. Considering multiple combinations allows for proof both that the United States is not part of Israel and that Israel is not part of the United States. That is, an inference rule 416 is provided to computationally evaluate the logic (c(A) ∧ c(B) ∧ p(A,B)) → (A=B) which states that if A is a country and B is a country and one of these countries is part of the other one of these countries, then both of these countries must be the same (i.e., A=B). Given the rule 416, the syntax tree 426 of the programmed corresponding LNN is traversed backwards, as indicated by arrows 430, 433, 436, allowing for conclusive proof whether the country represented by the predicate typecountry(United States) is equal to typecountry(Israel). That is, by implementing logic to evaluate proposition 435 expressing predicate data 407 “typecountry(USA)” AND predicate data 409 “typecountry(Israel)” AND Israel ≠ US, predicates 444 ¬partOf(Israel, US) and 445 ¬partOf(USA, Israel) are produced.
[0270] Finally, the method considers X=Natalie Portman and A,B∈{Israel, United States} and computes the results at each combination of values. If A=B, then it is understood that the country is part of itself, so the inference does not show anything useful and makes the formula necessarily true, but both cases where A≠B (i.e., A=Israel and B=USA or A=USA and B=Israel) are sufficient to prove ¬bornIn(Natalie Portman, United States). In these computations, inference rules 412 are provided to computationally evaluate the logic (b(X,A)∧b(X,B))→(p(A,B)∨p(B,A)) which states that if Natalie Portman was born in a first place (country A) and Natalie Portman was born in a second place (country B), then either the first place is contained in the second place or, conversely, the second place is contained in the first place. Thus, given the rule 412, the corresponding programmed syntax tree 422 is traversed backwards, as indicated by arrows 450, 453, 456, making it possible to conclusively prove that the person Natalie Portman was not born in the USA, i.e. by implementing the logic of evaluating the predicate data 444 "¬partOf(Israel, USA)", data 445 "¬partOf(USA, Israel)", given the evaluated predicate 425 "bornIn(Natalie Portman, Israel)", yields a conclusive result ¬bornIn(Natalie Portman, USA), which is the answer to the processed query "Was the person Natalie Portman born in the USA?".
[0271] The inference pattern as illustrated in the exemplary first-order logic net use case 400 of Figure 4 does not assume that the knowledge base data set is closed-world, i.e., it may not be conclusively found within the knowledge base set that simply saying that Natalie Portman was born in the United States does not mean that Natalie Portman was not born in the United States. Thus, the inference paths constructed as rules 412, 414, and 416 are programmed as corresponding syntax trees in the first-order LNN such that traversal in response to a query computationally proves the fact that Natalie Portman was born somewhere other than the United States.
[0272] The computational pattern in an LNN is similar to a feedforward neural network, where an input layer passes inputs to one or more internal or hidden layers, which compute based on each input and whose outputs are passed on, ultimately arriving at a single output. However, the LNN structure is implemented as a recurrent neural net, and in addition to the pattern of a traditional feedforward neural network, some of the outputs are looped back to become inputs for the next pass, thereby providing a bidirectional inference structure. For example, in the exemplary first-order logic net use case 400 of FIG. 4, predicate input information or knowledge graph data 402, 404, 406 (e.g., provided by DBpedia) goes through a feedforward pass of evaluation (traverses upwards through the tree) until each formula 412, 414, 416 is reached. At that point, the result of the feedforward pass of computation is the new truth value of the formula, and in this example, all results are assumed to be true (a tighter bound) before traversing in the reverse direction of inference. In the backward pass, the resulting values or learned parameters (e.g., values of new information) of the feedforward pass are passed backwards from the root to each leaf. In bidirectional inference, every edge of the knowledge graph (syntax tree) contains a forward computation and a backward computation. The backward pass of computation is equivalent to many of the various inference rules as employed in classical and fuzzy logic.
[0273] Typically, in an LNN, a predicate or proposition serves as a candidate for bounds proof no matter where it appears in any expression. The use case 400 in Figure 4 is complex in the sense that the predicate "bornIn(·.·)" 402 and the predicate "partOf(·.·)" 404 are each used simultaneously in many different places (containing many uses), and therefore there are an increased number of ways to provide values for those predicates.
[0274] For the use case of Fig. 4, classical "modus ponens" inference rules are implemented as backward-pass information flow steps 423, 427 in a backward traversal of the syntax tree 424 representing the formula 414, where the learned facts b(X,A) and p(A,B) and their conjunction provide the assumed truth value of the whole formula, i.e., this structure provides the truth b(X,B) (modus ponens inference). Given the first-order logic process, the conjunction of the b(X,A) and p(A,B) nodes 403, 405 of the LNN, the method performs joins (e.g., dot products or inner joins) where the variables match, and for the variables that do not match (outer joins), all combinations are obtained. Thus, for all X, A, and B, a truth value is obtained that indicates both whether someone was born in one place and whether that place was part of another place, which is (if implemented without any induction) a fairly large three-dimensional tensor that is very sparse. Thus, computational guidance is provided to indicate interest in only some of these combinations, and some values can remain undetermined (e.g., these values can remain loose because they do not affect the well-being of the system). Furthermore, in the second step of reasoning, as a backward traversal of the syntax tree 426 of the exemplary LNN of FIG. 4 in equation 416, information flow steps 430, 433, 436 implement the "modus torrens" inference rule, followed by the conjunctive syllogism inference rule. That is, the method first proves that the entire conjunction c(Israel) ∧ c(US) ∧ p(Israel,US) (and the conjunction with Israel and US swapped) is false, and then proves that since c(Israel) and c(US) are both true, thereby p(Israel,US) and p(US,Israel) are both false. Finally, in the third step of inference, as expression 412 traverses backward the syntax tree 422 of the example LNN of FIG. 4, information flow steps 450, 453, 456 implement example modus tollens inference rules, followed by conjunctive syllogism inference rules.That is, first the conjunction b(Natalie Portman, Israel) ∧ b(Natalie Portman, USA) is proven to be false, and then, since b(Natalie Portman, Israel) is true, b(Natalie Portman, USA) is proven to be false.
[0275] As explained, the downward inference used to propagate the bounds downward is modified in the first-order logic processing of LNN. For example, in the case of existential quantifiers, the upper bound is propagated downward, while in the case of universal quantifiers, the lower bound is propagated downward. Skolemization is required to propagate the lower bound downward in the case of existential quantifiers and the upper bound downward in the case of universal quantifiers. In the use case situation of Figure 4, a new constant candidate (e.g., Portman, Jerusalem, USA) is added to the universe of discourse and a specific skolemization rule (e.g., for every person there exists a place x and that person was born in that place) is adopted. In the example, for Natalie Portman there exists a place x with respect to this rule, and this (x for Natalie Portman with respect to this rule) is introduced as a constant that can be proven equal to some other constant.
[0276] Discussion Area Management
[0277] For two-way inference, the order of evaluation is determined by a "universe of discourse management" method, which in an embodiment depends on what is known from the query. In the use case of Figure 4, given an initial query (e.g., "Natalie Portman was born in the United States"), first syntax tree 424 is traversed, then syntax tree 426, and finally syntax tree 422, which are evaluated in that order in example 400 of Figure 4.
[0278] Additionally, for bidirectional inference, the LNN can be further programmed to provide a backward evaluation pattern that includes efficient update paths (e.g., traversing all of the neurons of the network and returning to all of the neurons). In an embodiment, universe of discourse management provides a mechanism for controlling the depth of inference, i.e., the number of backward passes through the network. The "universe of discourse management" method further provides the ability to computationally determine whether a particular expression needs to be evaluated, for example, based on whether there is something to be learned or not from evaluating the expression. This is because the LNN structure knows which constants are currently being addressed and which predicates need to be known about the constants. Knowing the LNN structure, certain expressions can be queued to be evaluated to increase learning.
[0279] study
[0280] As mentioned above, an existing knowledge base of rules can be transformed into a network of neurons. Without any subsequent training, such a network can perform inference as specified by the rules. Meanwhile, ground truth observations can be presented and the constrained optimization can be tuned to make the network more tolerant to uncommon cases and noisy data. In an embodiment, constrained optimization is used to learn the parameters of each of the neurons of the LNN to minimize a certain loss function while preserving the classical properties of the corresponding logical connectives, in particular such that a neuron corresponding to AND produces an output that is sufficiently true if all participating inputs are sufficiently true, produces an output that is sufficiently false if any of the participating inputs are sufficiently false, and vice versa for OR. In an embodiment, some of the neurons of the LNN have fixed parameters, i.e., the equations corresponding to these neurons still participate in inference and loss, but act as components of the ground truth while only participating in the learning insofar as they affect the optimal parameter configuration of other neurons. In embodiments, ground truth equations (i.e., neurons with fixed parameters) can be used that become inconsistent if undesirable inference behavior occurs, thereby allowing simple loss functions to prioritize arbitrarily specific, yet interpretable, logic requirements.
[0281] LNNs are trained using a gradient-based constrained optimization method that seeks to minimize a defined loss function. The Frank-Wolf algorithm is well suited for this purpose since all constraints are linear with respect to the weights and bias terms, but other methods are also applicable. In Frank-Wolf constrained optimization, the objective, subject to the neuron constraints M, is minimized by calculating the network's loss E according to:
[0282]
number
[0283] When borrowing from an existing KB structure, the LNN may be limited to adjusting the parameters of inputs that already participate in the formula. On the other hand, to perform rule induction, the method involves building a speculative network that links many propositions to multiple layers of connectives across multiple formulas. One consequence of the constraints is that for lower values of the truth "α" threshold, the operation can support a small number of operands with non-zero weights, thereby limiting the complexity of the rules. Both KB structure borrowing and rule induction may be used simultaneously when training the model. In an embodiment, in addition to a (non-negative) slack variable per constraint, a slack penalty is introduced that is directly proportional to both the slack variable and the (increasing function of) the importance weights of the inputs that define the constraint, thereby allowing the optimization to move inputs in or out of the participating set, allowing further control over how the parameters of the learned neurons should classically behave.
[0284] Depending on the particular problem being solved, different loss functions may be used. For example, an LNN configured to predict binary outcomes may use mean squared error (MSE) as usual, but may use s i w i In addition to a slack penalty that varies with , we should also include a contradiction penalty that varies with the degree to which the boundaries are crossed at each neuron. Alternatively, an arbitrarily complex logical loss function can be constructed using the contradiction penalty by introducing new expressions into the model that become contradictory if undesirable inference behavior occurs. In particular, the parameters of these introduced expressions should not be tuned but instead left in their default state (e.g., all 1s) so that the optimizer cannot override the logical loss function. Other loss functions of interest penalize all bounds becoming tighter in order to favor inference parsimony and any combination of the above.
[0285] In an embodiment, the loss function E includes a term that varies with the degree of discrepancy observed at each neuron (e.g., a function of the neuron's upper and lower bounds that increases with the amount of computed boundary crossing). An exemplary loss function E is:
[0286]
number
[0287] Another learning paradigm that LNNs support is learning arbitrary tasks, characterized by attempting to predict arbitrary inputs given any subset of other inputs. Given a data set of various combinations of the truth values of known propositions, this is done by randomly selecting, for each pass through each combination, a subset of the known propositions to serve as observations, and then attempting to predict the withheld propositions.
[0288] constraints
[0289] Constraints on the parameters of a neuron are derived from the truth table of the operation the neuron is intended to model, as well as from established ranges of "true" and "false" values. Subject to a threshold of truth 1 / 2<α≦1, continuous truth values are considered true if they are greater than α and false if they are less than 1-α. Accordingly, a truth table for, say, a two-variable AND would suggest a set of constraints given that:
[0290] [Table 2]
[0291] Corresponds to the following table:
[0292] [Table 3]
[0293] More generally, a conjunction of n variables has a constraint of the form ∀i∈I, w i ≧0 ∀i∈I, β-w i ·α ≦1-α (1)
[0294]
number
[0295] On the other hand, a disjunction of n variables has a constraint of the form ∀i∈I, w i ≧0 ∀i∈I, 1-β+w i α ≧α (3)
[0296]
number
[0297] Intuitively, equation (1) requires that the conjunction returns false (i.e., maximally true) if any of the inputs are false, even if the other input is 1, while equation (2) requires that the conjunction returns true if all inputs are true, and the opposite is true for disjunctions.
number
[0298] The result of these constraints is that the evaluation of the LNN is guaranteed to behave classically, i.e., produce results within established ranges of true and false at all neurons, provided that all inputs themselves are within these ranges. In an embodiment, the constraints are enforced by tracking regions of acceptability, and parameters are improved by moving in the direction of the gradient of the loss function while being pushed away from components perpendicular to the crossed boundaries.
[0299] Slack Variables
[0300] It is desirable to allow the weight of an input to reduce to exactly 0, thereby effectively removing that input from I. To make this possible, we need to introduce a slack variable for each weight and allow each constraint in equation (1) or equation (3) to be violated as the weight reduces to 0.
[0301] ∀i∈I, s i ≧0 ∀i∈I, β-w i α-s i ≦1-α (1*) ∀i∈I, 1-β+w i ·α+s i ≧α(3*)
[0302] These are s i =0 or w i = 0, the original constraint remains consistent. i w i The optimization can be encouraged to select such a parameterization by updating the training loss function to include a penalty term that varies with . The coefficient on this penalty term controls what the classical learned operation should be, with the exact classical behavior being restored if the optimization reduces the penalty term to 0.
[0303] Logical consistency: Contradiction Loss
[0304] If the boundaries are crossed, this means that no single truth value can be assigned to the neuron. Main driver of optimization: enforcing logical consistency.
[0305] Optimization Problem
[0306] The contradiction loss is similar to the hinge loss, and an alternative is the squared hinge loss. In addition to or instead of the square regularization weight decay, a linear regularization loss can be used.
[0307]
number
[0308]
number
[0309]
number
[0310] Depending on the particular problem being solved, different loss functions may be used. For example, an LNN configured to predict binary outcomes may use mean squared error as usual, but may use s i w iIn addition to the aforementioned slack penalty, which varies with , we should also include a contradiction penalty that varies with the degree to which the boundaries are crossed at each neuron. Alternatively, an arbitrarily complex logical loss function can be constructed using the contradiction penalty by introducing new expressions into the model that become contradictory if undesirable inference behavior occurs. Of course, in particular, the parameters of these introduced expressions should not be tuned but instead left in their default state (e.g., all 1s) so that the optimizer cannot override the logical loss function. Other loss functions of interest penalize all bounds becoming tighter in order to favor inference parsimony and any combination of the above.
[0311] Another learning paradigm that LNNs support is learning arbitrary tasks, characterized by attempting to predict arbitrary inputs given any subset of other inputs. Given a data set of various combinations of the truth values of known propositions, this is done by randomly selecting, for each pass through each combination, a subset of the known propositions to serve as observations, and then attempting to predict the withheld propositions.
[0312] Further example learning use cases using LNNs for semantic tagging applications are presented in Appendix A.
[0313] Gradient-transparent clamping
[0314] Real-valued logic typically constrains truth values to a defined range, which requires the use of clamping operations on the results. Automatic differentiation provides clamping that can perform value clamping, but it also invalidates the associated gradients, potentially invalidating learning of the involved parameters. Gradient-transparent clamping addresses this by fully recovering gradients outside the acceptable region by asserting clamping while preserving the original gradient information, utilizing computations separate from the gradient-tracking computation graph.
[0315] Fixation in Neural Networks
[0316] Smooth inhibited activation
[0317] Sigmoid (1+e -x ) -1 and tanh(e x -e -x ) / (e x +e -x ), ensures that the output is within an inhibited range (usually [-1,1] or [0,1]) while ensuring differentiability over the entire domain. Activation of inhibited neurons ensures that values in the neural network do not become too large and that there is some interpretability in the neuron's output, such as in cases of binary classifiers or real-valued logic.
[0318] Smooth suppressed activation functions f(), similar to sigmoid and tanh, have a gradient
number
[0319] ReLU
[0320] The rectified linear unit (ReLU) max(0;x) has a one-sided saturation of zero, with an associated gradient of 0 at x<0 and linearity with a gradient of 1 elsewhere. The ReLU is able to address the problem of vanishing and exponential gradients because its gradient can only be 0 or 1. However, this activation suffers from the "vanishing ReLU" problem, where if the preactivation distribution is restricted to the negative region, the input gradient does not propagate through the ReLU, preventing the ReLU from learning. Various forms of leaky and parameterized ReLUs have been proposed to address the lack of gradient in the negative region.
[0321] It can be shown that a lower bound on the output of the ReLU is useful if neurons are considered as concept detectors that give a certain degree of presence of a feature by linearity, but only a uniform description of its absence by rectification. The ReLU lower bound can also prevent correlated neurons in some cases by not propagating negative values that correspond to the degree of absence of the learned feature. The ability to stop signal propagation can also reduce noise and duplication of irrelevant information in the network.
[0322] The computational advantage of ReLU during training comes from the sparsity of the gradient introduced by the zero gradient, which means that the gradient computation needs to operate on only a subset of neurons. In addition, the two possible gradients of ReLU are constant scalars, so there is no computation involved required to determine its gradient.
[0323] Information and gradient dropout
[0324] Neural networks are universal function approximations and, if equipped with sufficient power and a sufficiently large set of neurons and interconnections between them, can embody functions of arbitrary complexity. Since determining the exact power required for any function can be an intractable problem, neural networks are usually overpowered or greatly augmented beyond a series of hyperparameter optimizations.
[0325] The problem with overfitting is that a trained neural network cannot generalize because it cannot reach the desired performance on data it has not seen before. Redundancy or excess capacity in a neural network allows overfitting, and the input training data can be memorized to optimize a stated objective. To prevent overfitting, various regularization methods have been proposed, such as data augmentation and shedding, which introduces random variations in the input data in the case of data augmentation, or randomly removes neurons during training epochs.
[0326] Constrained activation functions, such as sigmoid and ReLU, that can saturate to zero outputs and gradients, can be thought of as performing a combination of information and gradient shedding. Constrained activation functions can act as regularization methods, effectively removing capabilities or decoupling certain neurons subject to the condition of zero saturation. The empirical success of shedding and ReLU is attributed to regularizing the effects of sparsity that these methods introduce.
[0327] Gradient transparent fixed
[0328] Fixation in automatic differentiation
[0329] A lower bound on ReLU max(0,x), performed by the clamp, min, or max functions of an automatic differentiation system, typically disconnects x from this node in the computation graph and replaces x with a new node that evaluates to 0, with no previous computation history attached. This is because even if x was involved in determining a fixed outcome, the loss gradient
number
[0330] Decoupling arguments so that they are no longer represented in the calculations when clamping can introduce inaccuracies in the interpretation of the computed gradient. If the subdifferential with respect to 0 is set to the right-hand discontinuity such that 0 gives a linear gradient of 1, then one can say that ReLU reduces to negative values at a linear rate in the case of gradient descent. So, for x=0, the expected result of gradient descent with a step of s=-0.1 could be x'=x+s=-0.1, assuming a linear gradient, but this update cannot be realized because x is already clamped at the starting value. So the non-zero gradients provided give no guidance as to when clamping can occur, and show that even the usual application of clamping can result in incorrect interpretation of the gradient, especially when linearity of the function is assumed.
[0331] The gradients, both inside and outside the fixed region, provided by the automatic differentiation, are then interpreted, and there is a reliance on the learning update mechanism to manage the step size and momentum to navigate the anomalous loss situations introduced by the fixation. Fixation can also inform the learning of the gradients, indicating that the loss can be changed by updating the parameters, even if fixation is activated with a small parameter step size. Fixation can also indicate that if the output is currently fixed, there is no learning possible, such that no learning will occur, even if fixation is disabled for small parameter steps in the right direction.
[0332] Information omissions and gradient transparency
[0333] A suppressed activation function that saturates at 0 can be said to perform regularization according to certain preactivation conditions by stopping the propagation of dynamic information and gradients. Techniques for information and gradient shedding are particularly relevant for neural networks that have redundant capabilities as a means of introducing sparsity to prevent overfitting and address the problem of vanishing gradients.
[0334] In the case of LNNs as a one-to-one mapping of the syntax tree of a logic program, there are significant differences from regular neural networks, where hidden neurons are not necessarily related to external concepts and the width of the hidden layer can be set to any size. Thus, the problem of dealing with overfitting and redundancy is not a major concern in LNNs, but logical neurons and sub-networks can be copied to expand the parameter capabilities. The motivation for introducing gradient sparsity via ReLU in the case of regular neural networks does not easily apply to LNNs.
[0335] The vanishing gradient problem can be completely avoided in LNNs for certain loss terms that can be constrained per neuron so that a shorter interpretation of the gradient can be computed. However, it can become even more complicated for losses combined with supervised task-specific objectives that are computed simultaneously across different batch dimensions that represent alternative universes of discourse for truth assignment. Even in the case of LNNs, smooth activations should still be avoided to address the vanishing gradient problem.
[0336] The information dropout due to fixation in real-valued logic helps to keep truth values within an acceptable range of logic interpretability. Because Lukasiewicz logic aggressively applies fixation to ensure that the logic is sound, the output values of logic operations need to be suppressed. However, it is not necessary that the associated gradients outside the fixed region are also suppressed. In fact, decoupling gradients outside the fixation prevents learning for a significant portion of the operating range of the logic neuron's activation. Gradient-transparent fixation provides information dropout or value fixation while leaving the gradient unaffected, such that any gradient calculation effectively ignores the fixation. The advantage of this is that the main purpose of fixation is still met, i.e., to fix the output value, but the entire output region maintains the gradient as if the fixation had not been applied. The "vanishing ReLU" problem can also be solved using gradient-transparent fixation, since learning can receive gradients throughout the entire operating range, such that gradient descent always has an optimization direction available for the parameters.
[0337] The perceived error that gradient transparency fixation introduces is of exactly the same nature as the error that would also be produced by fixing the output values. In particular, gradients inside the bounds can show that the output will change even if the fixation is applied immediately in the update step and this is not yet reflected in the gradients. Similarly, gradients outside the bounds imposed by gradient transparency fixation can show a change in the output, but then after the update step the fixation still applies. To ensure robust learning under these circumstances, both with and without gradient transparency, the learning optimizer should manage the step size and momentum.
[0338] Gradient transparent fixed
[0339] Automatic differentiation usually involves cutting off the computed node when clamping an output and replacing it with a new node that represents the bounds that are applied. By removing the computed node, we also remove the trace of its participation in the current output, which is undesirable since this participation was actually part of the computation. The idea then is to keep the inputs to the clamping operation as part of the resulting node in the computation graph. This idea requires the definition of a new clamping function that performs the clamping of values while preserving the gradients as if the clamping had not been applied.
[0340] x min For fixed lower bounds at x min To give the output of x <x min The clamping is performed when x-min(0,x * -x min ), x * denotes a copy of x that has been detached from the computation graph and therefore has no history. *By obtaining ,value clamping can still be applied without causing the destructive interference that would otherwise occur if tracked copies of the same nodes were used. Automatic differentiation tools usually allow for the duplication of isolated values, and various features exist that allow for the copying and reuse of values. The superscript (1) indicates that the magnitude of the gradient within the clamped region is not changed, in contrast to the linear rescaling performed by gradient supplantation, described later. min(x max ,x) (1) =x-max(0,x * -x max ), (5) max(x min ,x) (1) =x-min(0,x * -x min ), (6)
[0341] Then, the clamping can be replaced with a gradient transparent clamping to recover the gradient over the entire output range while still providing the intended value clamping. The condition here is that the parameter update strategy should consider the possibility that there will be no output change in case of insufficient step size, even though the gradient in the clamped region tells us that a change in output was expected. Again, this negative aspect is similar to the difficulty of interpreting the gradient faced when clamping values and being close to the borders of the interior of the acceptable region, stepping outside, where a change in output was also expected but not realized when the clamping was applied immediately.
[0342] Fixation with Fuzzy Logic
[0343] Real unit interval
[0344] The continuous t-norm of fuzzy logic performs binary operations on the real unit interval [0,1], which represents the logic of infinite values, where the truth value can be interpreted as an ambiguous mixture between true and false. To ensure that the truth value stays within the interpretable range of [0,1], it is necessary to clamp the results of fuzzy logic operations, such as those of Lukasiewicz logic. The min() and max() functions clamp the values computed for all Lukasiewicz logic operators, which would otherwise cause the results to fall outside the range of acceptable truth values in [0,1]. For forward inference of Lukasiewicz conjunctions, we constrain the output by clamping and provide the associated clamped gradient, as follows:
[0345]
number
[0346] True input x i If w = 1, i The gradient corresponding to the output in the term
number
[0347] Gradient transparent fixed
[0348] Note that any fixation applied will usually invalidate the gradient information so that the learning does not receive any useful gradient information, since the gradient will be non-zero only inside the real unit interval. This is problematic given that, depending on the parameters, a large part of the operating range of the logical conjunction is subject to fixation. The solution to this problem is to only perform value fixation and not change the gradient through gradient transparent fixation. In this case, both inside and outside the fixed region, the output gradient of β is always 1, and w i If , the output slope is always
number
[0349] The proposed parameterization of the logical neurons, in particular, allows for a range of 0 ≤ w, usually without loss of functionality. i ≦1 and 0≦β≦1+Σ i w i , it works well and is constrained. This is because any parameter configuration from these constraints can allow the range of useful functions of the constrained ReLU to be accessed. As a result, the learning optimizer can set a robust update step size and constrain the parameter updates, which means that the presence of transparent gradients from parameters in fixed situations requiring larger update steps can be feasibly handled. The inconsistency loss Σ j max(0,L j (β,W)-U j (β,W)) also has a lower bound L j is the upper limit U j We include a fix with the intention that the loss term should only be enabled if it is higher than the loss term. If there is no inconsistency, we do not want to make parameter updates based on this loss. Therefore, a fix can be performed such that if there is no inconsistency, the gradient is also fixed and zero gradient is set. This means that a normal fix can be performed when learning should not be performed, especially with regard to the loss term.
[0350] Backward reasoning
[0351] Backward inference on a weighted Lukasiewicz conjunction can be determined by first unfixing its output, which is
number
number
number
[0352]
number
[0353] x'=min(x max Unfixing x',x is simply max(x',x), and similarly x'=max(x' min For x',x), we have min(x',x) as the unfixing operation. Here, the value of x is recalculated using gradient tracking, but during fixation where the value of x is isolated in the subtraction. Note that disabling gradient tracking is not required for unfixing, as was the case for fixation, since there is no subtraction that can disable the gradient or its destructive interference. Then, unfixing of the extreme-constrained output can follow as follows:
[0354]
number
[0355] An inverse function can then be determined using the output of the conjunction and the known values of all but one of the inputs, i, after which backward reasoning can compute the values of the remaining target inputs, j. The proofs computed for the target inputs are then also fixed to the real unit interval before proof aggregation is performed.
[0356]
number
[0357] Conjunctive syllogism
number
[0358]
number
[0359] Here, weighted Lukasiewicz implication is generally defined as follows:
[0360]
number
[0361] Note that there exists a negation of partial conjunctions, which also involves swapping lower and upper bounds in the backward inference computation. The unfixed gradient obtained over the entire operating range using gradient transparency fixation is computed as follows:
[0362]
number
[0363] Weight less than 1 0≦w jIf ≤ 1, the gradient of all backward inference
number
number
[0364] Gradient Substitution
[0365] In an embodiment, the activation functions of neurons are purposefully modified to ensure desirable properties including logical consistency, properties of inference and learning such as monotonicity, classical logic, and maintenance of non-zero gradients.
[0366] The introduction of the truth threshold α allows for non-zero gradients in the regions where fixation is normally performed. The fitted piecewise linear activation function provides the range [0,1-α] to represent the fixed false region of the logical conjunction and therefore has a positive non-zero gradient associated with the classical false value region. This gradient expresses that a positive change in the input of the function leads to a positive increase in the output truth value, even if an insufficient step size can still result in a classical false output. By adjusting the truth threshold 2 / 3≦α≦1, the magnitude of the gradient in the classical region can be changed, allowing for more conservative gradients to be obtained for these regions that can make the learning update more accurate.
[0367] In contrast, gradient transparency fixation provides more aggressive gradients in the classical domain compared to piecewise linear activation, since it utilizes existing gradients of the same magnitude as the unfixed domain. The gradient replacement approach in gradient transparency fixation can guarantee gradient magnitudes comparable to piecewise linear activation, but does not need to support classical symmetry in the range [α,1] associated with true. Output values do not need to be relaxed by α, such that false is still only at 0 outputs, and any gradient can be provided within the fixed domain using gradient transparency fixation.
[0368] Basic gradient substitution modifies gradient-tracked values by resizing the gradients using a provided scalar according to a specified condition. In gradient transparent fixation, the addition of gradient substitution can change the effective gradient magnitude where the value fixation is applied. Then, in the case of fixed regions, a smaller gradient can be selected so that a suppressed ReLU can show arbitrarily resized gradients in the rectification region to enable more accurate learning updates. Gradient substitution based on magnitude modification separates the gradient-tracked values x from the computation graph for x. * In addition to the above, we use indicators or Boolean condition functions that are readily available in the automatic differentiation library, and therefore adapted gradient-transparent functions are applied as follows:
[0369] f α (x,x max )=(x≦x max )+α(x max <x)、 g α (x,x min )=α(x <x min )+(x min ≦x), min(x max ,x) (α) =(x-max(0,x * -x max / f α (x,x max)))f α (x,x max ), max(x min ,x) (α) =(x-min(0,x * -x min / g α (x,x min )))g α (x,x min )
[0370] Usual fixation, which involves setting the gradient to zero within the fixed region, uses gradient substitution with a scaling factor of a = 0. Zero-scaled gradient transparent fixation min(x max ,x) (0) and max(x min ,x) (0) Matches.
[0371] So, in summary, the features of LNNs include: 1. immediate interpretability as a system of logical expressions; 2. ability to incorporate domain knowledge by seeding existing KBs; 3. support for compound expressions in first-order logic, as opposed to simply Horn clauses, for example; 4. explicit adherence to the open-world assumption by tracking both upper and lower bounds on truth-values; 5. ability to assign weights to each operand of each connective within each expression, as opposed to simply including the weight for the entire expression; 6. ability to constrain logic to be well-formed and to behave exactly classically; 7. support for novel neuronal training paradigms, including contradiction-based logical losses and learning arbitrary tasks; 8. tractable training and evaluation run-times, as compared to restricted Boltzmann machines and Markov logic networks.
[0372] In a non-limiting implementation, LNNs can be configured as neural symbolic systems that can be applied to fields including data science, ontology learning, training and evaluation in simulators, as well as cognitive learning and reasoning, models of visual and language processing, or any model that combines logical reasoning and neuronal learning.
[0373] Moreover, as a versatile neural network architecture, LNNs are broadly applicable to many AI tasks, including both supervised and unsupervised learning. In particular, LNNs are well suited for natural language processing tasks in that they reflect the linguistic superposition of solid syntactic rules and fuzzy word meanings. As such, LNNs are suitable target representations for semantic and syntactic analysis, simultaneously offering the possibility of immediate inference based on the input sentence and subsequent refinement given additional information. Finally, LNN architectures may be seamlessly integrated into existing neuronal solutions, extending already highly capable machine vision and / or text processing systems with the ability to infer.
[0374] FIG. 8 illustrates an embodiment of a pipeline 800 of a Deep Thinking Question Answering (DTQA) system. The input 802 to the pipeline is a question text that is first parsed using an Abstract Meaning Representation (AMR) parser module 805. The AMR is a directed, acyclic graph (DAG) that represents, for example, the sentence "who is doing what to whom". Each node in the graph represents a concept, while the edges represent relationships between concepts. A canonical example of an edge is a subject or object relationship between concepts. As described herein with reference to FIG. 4, an exemplary input question text is "Was Natalie Portman born in the United States?" The output 808 of the AMR parser, i.e., the AMR DAG for each question, includes entities and frames (relationships), which may not be mapped / linked to corresponding structures in the knowledge base (KB) 850.
[0375] Thus, in the pipeline, an entity extraction and linking module 810 and a relationship extraction and linking module 812 are implemented. The entity extraction and linking module 810 is configured to extract entities and relationships and enrich the semantic representation obtained by the AMR parsing with explicit links to entities and concepts appearing in a knowledge base (e.g., DBPedia). The relationship extraction and linking module 812 links those entities and relationships to the KB. The linked entities / relationships are then mapped to their corresponding nodes in the AMR. The inputs to the entity extraction and linking module 810 are the AMR parsing and the question text, and the input to the relationship extraction and linking module 812 is the AMR parsing including the linked entities. The output of these modules is an Extended AMR (E-ARM) 820 that includes both the entities and relationships linked to the KB.
[0376] Regarding the Entity Extraction and Linking Module 810, in an embodiment, the system jointly applies entity extraction and entity disambiguation techniques to achieve high accuracy. First, a list of candidate entities is created based on (1) the entities in the AMR parsing, (2) the named entities and concepts extracted, for example, using the Watson Natural Language Understanding API, and (3) the noun phrases constructed based on the parsing. The actual linking to DBpedia is based on a set of similarity-based matching algorithms or rules 860.
[0377] For each candidate entity, DBpedia's search API is used to identify a subset of possible matching entities (either ontology classes or resources) for the candidate entity. Matching rules are then applied to identify higher precision matches. Matching rules 860 are based on a combination of features constructed from the input query itself (e.g., entity names, co-occurring entities), DBpedia's candidate entities (e.g., entity names and aliases, characteristics), and the semantic similarity of DBpedia subgraphs of the query and the matching candidate entities within a common vector space.
[0378] With regard to the Relation Extraction and Linking Module 812, in an embodiment, the task is implemented to enrich the AMR representation by linking frames and edges in the graph to their corresponding (semantically equivalent) relations in the KB. When mapping frames from the ARM graph to an RDF / OWL (Web Ontology Language) KB such as DBpedia, the inherent mismatch between the n-variable argument ProBank frames used in AMR and the two-variable predicates in the ontology is overcome by combining and expanding all arguments before removing unlikely combinations. For example, from the frames, predicates encoded by arguments can be generated and these predicates can be linked to ontology relations. To map the AMR predicates to KB relations, a parallel corpus is created, for example using a subset of Wikipedia sentences. For each sentence, an ARM graph is generated and KB triples are extracted from DBpedia in a weakly supervised manner. The mapping is then generated using a method inspired by statistical machine translation. Furthermore, the context around the entities and mentions of relations in the question can help identify the corresponding knowledge base elements. To exploit these context clues, a unigram language model approach is employed to build a context model for each relation predicate. For each predicate in the knowledge base, all sentences mentioning instances of the predicate are extracted from the underlying text corpus and a unigram language model is built, which is then used to calculate the probability of the predicate being mentioned in the question. Then, the top N relations (e.g., N=5) based on these probabilities are selected as the output of module 812.
[0379] As further shown in the pipeline system 800, the E-AMR 820 is then converted by an AMR to logic translator module 825 into a formal first-order logic representation 827, both of which have greater expressiveness and allow the methodology to be generalizable rather than specific to a particular query language (e.g., the SPARQL RDF query language).
[0380] In an embodiment, a logical formalism for this task is used to create a bridge between AMR and SPARQL that can represent both predicate and procedural knowledge. The formalism supports two-variable predicates that are ubiquitous in linked open data, as well as higher order functional predicates to support aggregation and manipulation of sets of variable bindings. The formalism is not restricted to the SPARQL query language. The formalism follows the same syntax rules as the OpenCyc and NextKB projects, allowing support for a wide range of semantics, from standard logical operators to DBpedia predicates and functions that emulate SPARQL constructs.
[0381] AMR performs very well in identifying the meanings and semantic roles of verbs. ARM provides an important foundation for rule-based translation from AMR to logic. In the implementation, the following problems are addressed: (1) identifying (unbounded) variables in the targets of queries, (2) identifying queries that require reordering of sets of variable bindings, and (3) resolving URIs of KB entities to variables in AMR.
[0382] The AMR representation of questions and imperative statements is used to identify the target variables. In AMR, questions are represented using nodes belonging to the amr-unknown class. Intuitively, if a question is converted to a fill-in-the-blank statement, an instance of the amr-unknown class is used to represent the blank. In such cases, the AMR-to-logical translator 825 replaces the amr-unknown node (or an adjacent node) with an unbounded variable. For imperative statements, the AMR-to-logical translator replaces the direct object role of the top-level command (ARG1) with an unbounded variable.
[0383] A question requiring reordering can be detected by the presence of superlatives (i.e., AMR most and least classes) and quantity Probank frames (i.e., have-quant-91, have-degree-91). The AMR to logic translator 825 must determine the AMR variable that corresponds to the quantity of interest and the reordering direction (ascending for "least" and descending for "most"). The entity and relationship linking results are essential inputs for the AMR to logic translator 825 because they provide the required KB-specific vocabulary. However, even with accurate entity and relationship linking results, there may be differences in the granularity of the entities and relationships in the KB and AMR representations. For example, an acceptable AMR parsing for "composers of the Titanic soundtrack" may not include an edge directly connecting "composer" to "Titanic" but may include a path of two edges between "composer" and "Titanic" through "soundtrack". In contrast, in the KB, there may be a direct link between composer and film. To mitigate such problems, the AMR to logical translator 825 generates an initial logical interpretation of the query and looks for alternative AMR variables that correspond to the entity link results in order to minimize the number of unbounded variables (and maximize the use of the entity link results) in the final logical representation.
[0384] The logical expression may then be transformed by the logical to SPARQL module 830 into SPARQL, which queries DBpedia KB 850 for answers. The logical to SPARQL module 830 is responsible for transforming the logical queries generated by the AMR to logical module into equivalent SPARQL queries 835, which can be executed against any RDF store (e.g., a triple store 870 RDF database for storage and retrieval of triples (e.g., data entities consisting of subject-predicate-object) via semantic queries). There is a one-to-one mapping from constructs in the logical expression to constructs in SPARQL. The logical to SPARQL module 830 transforms the input logical expressions into SPARQL constructs using rule-based transformations, and then generates output SPARQL queries 835, for example using the Jena query builder.
[0385] The formal first-order logic representation 827 output by the AMR to logic translator module 825 is used to obtain a query generalization representation 839 that can be mapped to a corresponding graph syntax tree LNN 840 according to the methods described herein. The LNN 840 operates to extract predicate justifications via fine-grained SPARQL integration and performs multi-way inference to answer the question ASK, for example, according to open-world assumptions.
[0386] For example, using the pipeline system 800 of FIG. 8 on the input question text "Was Natalie Portman born in the United States?", Natalie Portman from the AMR parse is mapped to dbr:Natalie_Portman, United States from the AMR parse is mapped to dbr:United_States (both of which are named entities in DBpedia), the prefix dbo: represents http: / / dbpedia.org / ontology / , and the prefix dbr: represents http: / / dbpedia.org / reference / . "born in" from the AMR parse is mapped to the relationship dbo:birthplace. For this particular "objection inclusion" query, a generalization of the query is formed according to the following formula:
[0387] pred(s, y) = ∃ x (pred(s,x) ∧ isPartOf(x,y))
[0388] For this example input question text, the logical expression of the query is ∃ x (birthplace(Natalie_Portman,x) ∧ isPartOf(x,US))
[0389] This results in the syntax tree of the LNN in Figure 4.
[0390] Trainable auxiliary rules 860 for determining whether a target contains a known object assume the most specific ontology assignment.
[0391] As described herein, the logical neural network architecture 840 provides neurons that model the rigorously defined concepts of weighted fuzzy or classical first-order logic. The LNN 840 is arranged in one-to-one correspondence with the operations in the system of logical formulas and is capable of reasoning in any direction, i.e., via normal evaluation or inverse reasoning, such as modus ponens, modus torrens, conjunction elimination, and all related inference rules. The LNN explicitly accepts open-world assumptions by tracking both upper and lower bounds on truth values. This is achieved using special neuronal activation functions and computational patterns such that proven truth value boundaries propagate from each of the formula inputs to each of the other inputs, and these truth value boundaries are aggregated and used again in the regression computation. A trained LNN model constrained by sufficient penalties is guaranteed to converge with classical reasoning behavior. Looser constraints can be used to handle incomplete formulas, sparse formulas, and other sources of uncertainty in the ground truth.
[0392] In an embodiment, meronym relationships between entity types (e.g., isPartOf) can be used to link paths between query triples and expanded entities. Such query expansion allows more chances to reach negative answers to Boolean questions. An open-world LNN can continue reasoning after an initial null result by binding lemmas, for example, to check whether the retrieved entity is part of the entity provided by the query.
[0393] Referring to FIG. 8, given the bogus question ASK “Was Natalie Portman born in the United States?”, a knowledge base search (Request for facts) 832 finds the predicates (facts) 834 “bornIn(Natalie Portman, Jerusalem)” and “type(USA, Country)” and “type(Israel, Country)”. Given Jerusalem as the place of birth, the United States as the place of birth remains unknown. Then, a forward chain involving predicates such as dbo:isPartOf concludes that Jerusalem is part of Israel (isPartOf(Jerusalem,Israel)) via a holonym relationship. Israel is the more suitable entity to answer the query (880) since it is of the same special ontological type as the United States (i.e., dbo:Country). Then, a multi-way recursive inference proves that the retrieved extended place of birth (Israel) is not part of the queried object (United States) by the axiom of containment (effectively, different countries cannot contain each other).
[0394] FIG. 9 illustrates an LNN graph 900 formed according to a query implementation of an exemplary DTQA system.
[0395] Given the natural language question "What are all the actors who have starred in films directed by and starring William Shatner?" Upon execution of the AMR parsing and the applied entity and relationship extraction and linking logic, the following corresponding JSON (Java(R)script Object Notation) code has been generated: Logic: {'pred_or_fn': 'select', 'arg0': '?z', 'arg1': {'pred_or_fn': 'and', 'arg0': {'pred_or_fn': 'and', 'arg0':{'subject': '?x', 'predicate':'http: / / dbpedia.org / ontology / director', 'object':'http: / / dbpedia.org / resource / William_Shatner'}, 'arg1':{'subject': '?x', 'predicate':'http: / / dbpedia.org / ontology / starring', 'object':'http: / / dbpedia.org / resource / William_Shatner'}}, 'arg1': {'pred_or_fn': 'or', 'arg0':{'subject': '?x', 'predicate':'http: / / dbpedia.org / ontology / starring', 'object': 'z'}, 'arg1':{'subject': '?x', 'predicate': 'http: / / dbpedia.org / property / starring', 'object': '?z'}}}}
[0396] The following Python code is generated to represent the logic statements for mapping to the LNN: Exists(And(And(('dbo:director', x, (y,'dbr:William_Shatner')), ('dbo:starring', x, (y,'dbr:William_Shatner')),lnn=lnn), Or(('dbo:starring ', x, z) ('dbp:starring', x, z),lnn=lnn)), dim=z).
[0397] As shown in Figure 9, the LNN graph 900 formed is a mapping of syntax trees corresponding to the Python inference statements of these LNNs. The LNN 900 is a tree of existential quantifications (∃ z ) as answer output 950, which provides film actor "z" appearing in film "x" that contains "William Shatner" as director "y" and also contains "William Shatner" as the lead. LNN 900 includes logical nodes and conjunction edges, with node 920 being the first conjunctive node and node 930 being the conjunction (disjunction) node, both of which operate on predicate facts (justification), and nodes 920, 930 providing the answer output (∃ z ) 950.
[0398] To obtain global justification, fact requests are generated as Global SPARQL statements according to the following logic. SELECT DISTINCT * WHERE { VALUES ?ydbr:William_Shatner} {{{{?x dbo:director ?y} {?x dbo:starring ?y}} .{{?xdbo:starring ?z} UNION {?x dbp:starring ?z}}}}}
[0399] When accessing DBpedia, given the global grounding 910 “y” = dbr:William_Shatner”, the ontology relation provides the entities of films that William_Shatner directed, which is the global grounding “x” = dbr:Groom_Lake_(film)”. Then, in a fine-grained sense, the following SPARQL statement obtains the grounding of the predicate on “dbo:director” 901: SELECT DISTINCT * WHERE { VALUES ?x{dbr:Groom_Lake_(film)}? VALUES ?y {dbr:William_Shatner} {?x dbo:director ?y}}
[0400] Then, the following SPARQL statement obtains the justification for the predicate on "dbo:starring" 904: SELECT DISTINCT * WHERE { VALUES ?x{dbr:Groom_Lake_(film)}? VALUES ?z {dbr:William_Shatner dbr:Dan_Gauthierdbr:Tom_Towles …} {?x dbo:starring ?z}}
[0401] Furthermore, the following SPARQL statement obtains the justification of the predicate on "dbp:starring" 906: SELECT DISTINCT * WHERE { VALUES ?x{dbr:Groom_Lake_(film)}? VALUES ?z {dbr:William_Shatner dbr:Dan_Gauthierdbr:Tom_Towles …} {?x dbp:starring ?z}}
[0402] The corresponding predicate justifications representing actor "z"=dbr:William_Shatner, dbr:Dan_Gauthier, and dbr:Tom_Towles, etc. are obtained.
[0403] Next, a forward inference pass is performed to evaluate the logic at the nodes in the LNN graph 900, in the order 901, 904, 906, 920, 930, 940, 950. From the child groundings at the dbo:director node 901 and the dbo:starring node 904, the respective (x,y) values 902 (x=dbr:Groom_Lake_(film), y=dbr:William_Shatner) and (x,y) values 903 (x=dbr:Groom_Lake_(film), y=dbr:William_Shatner) are obtained, and all the operators are
number
number
[0404] Assume the following conjunction (forward inference) calculation: L A&B =L A &L B U A&B =U A &U B
[0405] The equations of logical neural network inference at the conjunctive node 920 are formulated as follows: U & =f(tw A (1-U A )-w A (1-U B )) L & =f(tw A (1-L A )-w B (1-L B ))
[0406] From these, the truth boundary 923 at the conjunction node 920 of the LNN graph 900 is updated as follows:
[0407]
number
[0408] The proof converges to show (1) a higher lower bound, (2) a lower upper bound, and (3) monotonically strict.
[0409] Continuing with the next disjunction node in the ordering of the forward inference path, from the child justifications at dbo:starring node 904 and dbp:starring node 906, each (x,z) value 905 (x=dbr:Groom_Lake_(film), z=dbr:William_Shatner) and (x,z) value 907 (x=dbr:Groom_Lake_(film), z=dbr:Dan_Gauthier dbr:Tom_Towles…) are obtained, and all operators
number
number
[0410] Assume the following disjunctive (forward inference) calculation:
[0411]
number
[0412] The logical neural network inference equations at disjunction node 930 are evaluated as follows:
[0413]
number
[0414] From these, the truth boundary 933 at the disjunction node 930 of the LNN graph 900 is updated according to:
[0415]
number
[0416] The proof converges to show (1) a higher lower bound, (2) a lower upper bound, and (3) monotonically strict.
[0417] Next, a backward inference pass is performed in the following order to evaluate the logic at the nodes in the LNN graph 900: 950, 940, 920, 930, 901, 904, 906.
[0418] The self-justification at the conjunction node 920 includes x=dbr:Groom_Lake_(film), y=dbr:William_Shatner, and all operators are
number
[0419] The backward inference equations of the logical neural network at the dbo:director node 901 are evaluated according to the following: U A =f(1-t+w B (1-L B)+w & U & ) L A =f(1-t+w B (1-U B )+w & L & )
[0420] Here, the upper and lower bounds are
number
[0421] From these, we can also perform backward inference of the logical neural network at the dbo:starring node 904 (i.e., x = dbr:Groom_Lake_(film), y = dbr:William_Shatner).
number
[0422] For proof aggregation, the truth value bounds 923 at the conjunctive node 920 of the LNN graph 900, the child justification node 901 (dbo:director), and the child justification node 904 (dbo:starring) are updated according to the following:
[0423]
number
[0424] This is the value
number
[0425] This proof aggregates to show (1) a higher lower bound, (2) a lower upper bound, and (3) monotonically strict.
[0426] The self-justification at node 904, dbo:starring, is
number
[0427] The backward inference of the logical neural network at the dbo:starring node 904 is evaluated according to the following:
[0428] U B =f(1-t+w A (1-L A )+w & U & )
[0429] L B =f(1-t+w A (1-U A )+w & L & )
[0430] Here, the upper and lower bounds on the dbo:starring node 904 are
number
[0431] In the case of proof aggregation, the truth bounds 929 at the dbo:starring node 904 are values that comply with the following:
[0432]
number
[0433] This proof aggregates to show (1) a higher lower bound, (2) a lower upper bound, and (3) monotonically strict.
[0434] Continuing with the ordering of the backward inference path to the next disjunction node 930, the child justification at the disjunction node includes predicates at dbo:starring node 904 and dbp:starring node 906, which are the respective (x,z) values 905 (x=dbr:Groom_Lake_(film), z=dbr:William_Shatner) and (x,z) values 907 (x=dbr:Groom_Lake_(film), z=dbr:Dan_Gauthier dbr:Tom_Towles...), and all operators
number
[0435] Given the following backward inference computation for disjunctions:
[0436]
number
[0437] The logical neural network inference equations at dbo:starring node 904 are evaluated according to: U A =f(tw B L B -w & (1-U & )) L A =f(tw B U B +w & (1-L & ))
[0438] Here, the upper and lower bounds on the dbo:starring node 904 are
number
[0439] Given the previous computation from the conjunction node 920, the bounds of the dbo:starring node 904 at the disjunction node are
number
number
[0440] From this, the truth boundary at dbo:starring node 904 in LNN graph 900 is updated according to:
[0441]
number
[0442] Next, assume the following backward inference computation for disjunctions:
[0443]
number
[0444] The logical neural network inference equations at dbp:starring node 906 are evaluated according to: U B =f(tw A L A -w & (1-U & )) L B =f(tw A U A +w & (1-L& ))
[0445] Here, the upper and lower bounds at the dbp:starring node 906 are
number
number
[0446]
number
[0447] This proof aggregates to show (1) a higher lower bound, (2) a lower upper bound, and (3) monotonically strict.
[0448] FIG. 10 shows a schematic diagram of an exemplary computer or processing system in which the system may be implemented in one embodiment. This computer system is merely one example of a suitable processing system and is not intended to suggest any limitation as to the scope of use or functionality of the method embodiments described herein. The illustrated processing system may be operated in numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, or configurations, or combinations thereof, suitable for use with the processing system illustrated in FIG. 10 may include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of these systems or devices.
[0449] A computer system may be described in the general context of instructions executable by a computer system, such as program modules being executed by the computer system. Typically, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The computer system may be executed in a distributed cloud computing environment where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media, including memory storage devices.
[0450] Components of a computer system may include, but are not limited to, one or more processors or processing units 12, a system memory 16, and a bus 14 that couples various system components including the system memory 16 to the processor 12. The processor 12 may include modules 30 that perform the methods described herein. The modules 30 may be programmed into integrated circuits of the processor 12 or loaded into the memory 16, the storage device 18, or the network 24, or a combination thereof.
[0451] Bus 14 may represent one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures, examples of which include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnects (PCI) bus.
[0452] The computer system may include a variety of computer system readable media. Such media may be any available media that can be accessed by the computer system and may include both volatile and nonvolatile media, removable and non-removable media.
[0453] System memory 16 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. The computer system may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 18 may be provided for reading from and writing to a non-removable, non-volatile magnetic medium (e.g., a "hard drive"). Although not shown, a magnetic disk drive may be provided for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive may be provided for reading from and writing to a removable, non-volatile optical disk, such as a CD-ROM, DVD-ROM, or other optical medium. In such examples, each may be connected to bus 14 by one or more data media interfaces.
[0454] The computer system may communicate with one or more external devices 26, such as a keyboard, pointing device, display 28, one or more devices that allow a user to interact with the computer system, or any device that allows the computer system to communicate with one or more other computing devices (e.g., network cards, modems, etc.), or any combination thereof. Such communication may occur through an input / output (I / O) interface 20.
[0455] Additionally, the computer system may communicate with one or more networks 24, such as a local area network (LAN), a general wide area network (WAN), or a public network (e.g., the Internet), or a combination thereof, via a network adapter 22. As shown, the network adapter 22 communicates with other components of the computer system via a bus 14. It should be understood that other hardware and / or software components, not shown, may be used with the computer system, including, but not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archive storage systems.
[0456] The present invention may be a system, method, and / or computer program product at any possible level of technical detail of integration. The computer program product may include a computer readable storage medium including computer readable program instructions for causing a processor to perform aspects of the present invention.
[0457] A computer readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes portable floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridges in grooves on which the instructions are recorded, and any suitable combinations of these. As used herein, computer-readable storage media should not itself be construed as being ephemeral signals such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted over wires.
[0458] The computer readable program instructions described herein can be downloaded from a computer readable storage medium to each computing device / processing device or to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof) that may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface within each computing device / processing device receives the computer readable program instructions from the network and forwards the computer readable program instructions for storage on a computer readable storage medium within each computing device / processing device.
[0459] The computer readable program instructions for carrying out the operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk®, C++, and procedural programming languages such as the “C” programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partially on the user's computer as a standalone software package, partially on the user's computer and on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, to carry out aspects of the invention, electronic circuitry including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer readable program instructions to customize the electronic circuitry by utilizing state information of the computer readable program instructions.
[0460] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks included in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.
[0461] These computer readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, produce means for performing the functions / operations specified in the blocks of the flowcharts and / or block diagrams. These computer readable program instructions may be stored on a computer readable storage medium and capable of directing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer readable storage medium on which the instructions are stored comprises an article of manufacture containing instructions for performing aspects of the functions / operations specified in the blocks of the flowcharts and / or block diagrams.
[0462] Computer readable program instructions may be loaded into a computer, other programmable data processing apparatus, or other device such that the instructions, which execute on the computer, other programmable apparatus, or other device, perform the functions / acts specified in the blocks of the flowcharts and / or block diagrams, thereby causing a series of operable steps to be performed on the computer, other programmable apparatus, or other device to generate a computer-implemented process.
[0463] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of instructions, comprising one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions shown in the blocks may occur in a different order than the order shown in the figures. For example, two blocks shown in succession may in fact be executed simultaneously as one step, or may be executed substantially simultaneously in a partially or fully overlapping manner in time, or may be executed in reverse order, depending on the functionality involved. It is also noted that each block in the block diagrams and / or flowchart illustrations, as well as combinations of blocks included in the block diagrams and / or flowchart illustrations, may be implemented by a dedicated hardware-based system that executes the specified functions or operations, or executes a combination of dedicated hardware and computer instructions.
[0464] The terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural unless expressly indicated otherwise. As used herein, the term "or" is an inclusive operator and can mean "and / or" unless the context expressly or clearly indicates otherwise. It will be further understood that the terms "comprise", "comprises", "comprising", "include", "includes", "including", or "having", or combinations thereof, when used in this patent specification, can indicate the presence of stated features, integers, steps, operations, elements, or components, or combinations thereof, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or groups thereof, or combinations thereof. As used herein, the phrase "in an embodiment" does not necessarily refer to the same embodiment, although it may. As used herein, the phrase "in one embodiment" may refer to the same embodiment, but not necessarily the same embodiment. As used herein, the phrase "in another embodiment" may refer to a different embodiment, but not necessarily the same embodiment. Furthermore, embodiments and / or elements of the embodiments can be freely combined with each other unless they are mutually exclusive.
[0465] Corresponding structures, materials, acts, and equivalents of all means or steps and functional elements within the scope of the claims below are intended to include any structures, materials, or acts for performing functions in combination with other claimed elements, if any, as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or to limit the invention to the disclosed form. It will be apparent to those skilled in the art that many changes and modifications are possible without departing from the scope of the invention. The embodiments have been selected and described in order to best explain the principles and practical application of the invention and to enable others skilled in the art to understand the invention in terms of various embodiments with various modifications as may be suitable for the particular use contemplated.
Claims
1. receiving, at one or more hardware processors, an input query and an input proposition used to infer a logical truth based on the input query; using said one or more hardware processors to configure a logical neural network (LNN) structure including one or more propositional neurons representing said input propositions and one or more neurons representing logical connectives according to a system of logical formulas corresponding to inference rules; evaluating logical inferences based on the input query using the LNN structure; and in response to the evaluation, outputting an output truth value that either proves or does not prove the logical inference within a consolidated boundary.
2. The system of formulas for the LNN structure is obtained from an ontology knowledge base, and constructing the LNN structure includes:
2. The computer-implemented method of claim 1, comprising: selecting neuron inputs and parameters for the one or more propositional neurons and logical connective neurons to match the syntax and behavior of connectives that make up the system of logical expressions from the knowledge base; and selecting initial bounds on the truth values of the selected expressions and propositions to match the input facts.
3. 2. The computer-implemented method of claim 1, further comprising: configuring, at each logical connective neuron, using the one or more hardware processors, a respective activation function used to compute the truth of a logical connective of the logical formula.
4. said evaluating logical inferences; 4. The computer-implemented method of claim 3, further comprising: using the one or more hardware processors to perform a forward inference pass, evaluating the representation of the LNN structure using an activation function to calculate the truth value at each neuron, the truth value indicating upper and lower bounds for a logical expression or sub-expression of an expression of the system.
5. said evaluating logical inferences; performing a backward (downward) inference pass using said one or more hardware processors and evaluating inverse inference rules during said downward inference pass; aggregating the truth values at propositional neurons and logical connective neurons during the backward inference pass.
6. said evaluating reverse inference rules during said downward inference pass; 6. The computer-implemented method of claim 5, further comprising: computing an inverse activation function at a logical connective neuron to update the truth values; and aggregating the truth values obtained in the downward inference passes.
7. 7. The computer-implemented method of claim 6, wherein the LNN structure is organized as a syntax tree, and the backward (downward) inference pass is performed using a recursive depth-first traversal of the syntax tree performed from one or more root nodes to each of the leaves of the root.
8. 6. The computer-implemented method of claim 5, wherein the aggregating the truth values with propositional neurons and logical connective neurons comprises obtaining the tightest upper and lower bounds by selecting the greatest lower bound and the least upper bound among the obtained truth values.
9. The one or more propositional neurons represent input predicates including n-dimensional tables or tensor expressions, n being a non-negative number, and the method further comprises: providing a neuron with a table of truth-value boundaries uniquely keyed by a group of columns for unbounded logical variables appearing in the corresponding subexpression or predicate; and modifying the activation functions of the neurons to perform combinations across all columns for shared variables while computing truth boundaries at associated rows.
10. said evaluating the logical inference to obtain a truth value; performing a combination between a table representation of known boundaries of the output of the neuron and a table representation of known boundaries of other inputs of said neuron; computing truth bounds on the associated rows of the tensor representation; and reducing across any columns for which a logical variable is absent in a corresponding sub-expression of the target input to aggregate the tightest such bound.
11. 4. The computer-implemented method of claim 3, further comprising initializing parameters of each activation function of each neuron to provide a weighting of importance across the inputs, outputs, or both inputs and outputs of the neuron.
12. 4. The computer-implemented method of claim 3, wherein the logical connectives are constructed according to a weighted Lukasiewicz logic scheme.
13. A computer-implemented system comprising one or more hardware processing units configured as a logical neural network (LNN) structure including a plurality of neurons and connector edges, the plurality of neurons and connector edges of the LNN structure being in one-to-one correspondence with a system of logical expressions, the system performing a method for performing logical inference; a neuron of the plurality of neurons exists for a corresponding logical connective in each formula or sub-formula of the system of logical formulas, the one neuron including one or more link connective edges providing input information including operands of the logical connective and information further including parameters configured to implement a truth function of the logical connective, each one neuron of a corresponding logical connective includes a corresponding activation function for providing a calculation, the calculation of the activation function returning a pair of values indicating upper and lower bounds for the formula or sub-formula of the system of formulas or returning a truth value of a proposition; a neuron of the plurality of neurons exists for a corresponding proposition of the expression or subexpression of the expressions of the system, the one neuron including one or more link connector edges corresponding to an expression providing information that proves a bound on the truth value of the corresponding proposition and further including a parameter configured to aggregate a tightest bound; the one or more hardware processors: receiving an input query or logical inference; using said LNN structure to evaluate logical inferences based on said input queries or logical inferences; A computer-implemented system configured to, in response to the evaluation, output an output truth value that either proves or does not prove the logical inference within a consolidated tightest bound.
14. a logic knowledge base including a set of lexical information organized according to said logic formulas or sub-formulas, said one or more hardware processors further comprising: and further configured to initialize each of the plurality of neurons using information based on a logical formula from the logical knowledge base, the initializing comprising: selecting input information and parameters to match the syntax and behavior of the connectors that make up the system of expressions from the logic knowledge base; and selecting initial bounds on truth values of formulas and propositions to be consistent with facts from the knowledge base.
15. the one or more hardware processors:
14. The computer-implemented system of claim 13, further configured to initialize bounds on truth values of expressions and propositions according to available observations, such that the evaluation of the neural network outputs results in the form of bounds on truth values computed at particular neurons.
16. the one or more hardware processors:
14. The computer-implemented system of claim 13, further configured to insert additional neurons into the logical neural network corresponding to a specified query expression, such that the evaluation of the neural network outputs a result in the form of a truth-value boundary computed by a neuron for the query expression.
17. the one or more hardware processors:
14. The computer-implemented system of claim 13, further configured to perform one or more of inferring upper and lower bounds for each input of a neuron or inferring upper and lower bounds for each operand of an existing neuron of a logical connective, the inference being based on an inverse activation function of the neuron with respect to the inputs and known bounds of the output of that neuron and other inputs of the neuron.
18. the one or more hardware processors: training the logical neural network using training data; 14. The computer-implemented system of claim 13, further configured to: extract rules, new rules triggered by observations from a trained neural network, or both rules and new rules, corresponding to expressions, to complete a logical knowledge base used to initialize the neural network or to discover new knowledge.
19. the one or more hardware processors: using each occurrence of each proposition within each formula or subformula as a proof of the bounds of said proposition; and performing calculations to recursively or dynamically evaluate the LNN, wherein bounds on each of the containing expression's or sub-expression's outputs and other inputs are first calculated.
20. The one or more hardware processors configure the LNN using a plurality of predicate neurons, each predicate neuron including an activation function, the plurality of predicate neurons are used to evaluate first order logic, and each predicate neuron is configured to emit a table of truth-value bounds uniquely keyed by a group of columns for unbounded logical variables appearing in the corresponding expression or sub-expression or predicate; 20. The computer implemented system of claim 17, wherein the LNN evaluates first order logic by configuring the one or more hardware processors to perform one or more joins across all columns of a table for shared variables while computing truth bounds at associated rows in activation functions of predicate neurons.
21. A processor comprising: configuring one or more hardware processing units as a logical neural network (LNN) structure including a plurality of neurons and connector edges, the plurality of neurons and connector edges of the LNN structure being in one-to-one correspondence with a system of logical expressions and configured to perform logical inference; a neuron of the plurality of neurons exists for a corresponding logical connective in each formula or sub-formula of the system of logical formulas, the one neuron including one or more link connective edges providing input information including operands of the logical connective and information further including parameters configured to implement a truth function of the logical connective, each one neuron of a corresponding logical connective includes a corresponding activation function for providing a calculation, the calculation of the activation function returning a pair of values indicating upper and lower bounds for the formula or sub-formula of the system of formulas or returning a truth value of a proposition; a neuron of the plurality of neurons exists for a corresponding proposition of the expression or subexpression of the expressions of the system, the one neuron including one or more link connector child edges corresponding to an expression providing information proving a bound on the truth value of the corresponding proposition and further including a parameter configured to aggregate a tightest bound; and the method further comprising: Receiving a query or inference; and configuring said LNN structure based on said queries or logical inferences; and evaluating the LNN structure and returning an output truth value that either proves or does not prove a proposition or logical inference within a consolidated tightest bound.
22. The processor, and initializing each of the plurality of neurons using information based on a logical formula from a logical knowledge base, the logical knowledge base including a set of lexical information organized according to the logical formula or sub-formula, the initializing further comprising: selecting input information and parameters to match the syntax and behavior of the connectors that make up the system of expressions from the logic knowledge base; and selecting initial bounds on truth values of formulas and propositions to be consistent with facts from the knowledge base.
23. The processor, 23. The program of claim 22, further comprising initializing bounds on truth values of expressions and propositions according to available observations, and wherein said evaluation of said neural network outputs results in the form of bounds on truth values computed at particular neurons.
24. The processor, 23. The program of claim 22, further comprising inserting additional neurons into the logical neural network corresponding to a specified query expression, and wherein the evaluation of the neural network outputs a result in the form of a truth-value boundary computed by the neurons for the query expression.
25. The processor, 25. The program of claim 24, further comprising performing one or more of the following: inferring upper and lower bounds for each input of a neuron, or inferring upper and lower bounds for each operand of an existing neuron of a logical connective, the inference being based on an inverse activation function of the neuron with respect to the inputs, and known bounds for the output of that neuron and other inputs of the neuron.
Citation Information
Patent Citations
Hierarchical fuzzy control method
JP1994102907A
Object Recognition For Security Screening and Long Range Video Surveillance
US20120243741A1
Deep adaptive semantic logic network
WO2018208813A1
