Learning interpretable differentiable logic networks

Abstract

The present disclosure provides a method for training an interpretable differentiable logic network (DLN) for classification tasks. The method includes initializing a DLN and iteratively training the DLN by alternating between two phases for a predetermined number of iterations. In a first phase, neuron functionality parameters are learned while fixing neuron connections such that a link between any two neurons is either 0 or 1. In a second phase, neuron connection parameters are learned while fixing neuron functionalities such that each neuron performs only one discrete logic operation among all possible operations. The DLN comprises a ThresholdLayer to convert continuous inputs to binary values, at least one LogicLayer to perform binary logic operations, and a SumLayer to aggregate outputs for classification.

Description

LEARNING INTERPRETABLE DIFFERENTIABLE LOGIC NETWORKSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Application No. 63 / 647,141, titled Learning Interpretable Differentiable Logic Networks, filed May 14, 2024, which is hereby incorporated by reference in its entirety.STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

[0002] This invention was made with government support under Grant No. CNS-1907831 awarded by the National Science Foundation. The government has certain rights in the invention.FIELD OF INVENTION

[0003] The present disclosure relates to machine learning techniques for classification tasks, and more particularly to a method and system for training interpretable differentiable logic networks that perform binary logic operations.BACKGROUND

[0004] Neural networks have become a cornerstone of modem machine learning, driving advancements in various fields including healthcare, finance, and natural language processing. Their ability to capture complex relationships within high-dimensional data has made them invaluable tools for solving a wide range of problems. Traditional neural networks typically consist of layers of interconnected nodes that process and transform input data through a series of mathematical operations to produce an output.

[0005] While neural networks have demonstrated impressive performance across many applications, they are often characterized as "black box" models due to the difficulty in interpreting their decision-making processes. This lack of interpretability can be problematic in domains where understanding the reasoning behind predictions is crucial, such as medical diagnostics or financial risk assessment. Additionally, neural networks tend to require significant computational resources for both training and inference, which can limit their deployment on edge devices or in resource-constrained environments.

[0006] Another challenge associated with traditional neural networks is their tendency to overfit the training data, potentially leading to poor generalization on unseen examples. Thisissue often necessitates careful regularization techniques and extensive hyperparameter tuning to achieve optimal performance. Furthermore, the continuous nature of neural network computations can make them less suitable for tasks that inherently involve discrete or logical decision-making processes.

[0007] It has been appreciated that a method is needed that overcomes one or more of these problems.SUMMARY

[0008] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description.

[0009] In a first aspect, a method for training an interpretable differentiable logic network for classification tasks is provided. The method comprises: initializing a differentiable logic network (DLN); and iteratively training the DLN by alternating, for a predetermined number of iterations, between: a first phase of learning neuron functionality parameters for the DLN while fixing neuron connections such that a link between any two neurons is either 0 or 1; and a second phase of learning neuron connection parameters for the DLN while fixing neuron functionalities such that each neuron performs only one discrete logic operation among all possible operations.

[0010] This two-phase iterative training approach enables the DLN to leam both the optimal logic operations for each neuron and the optimal connections between neurons, resulting in a network that can perform complex classification tasks while maintaining interpretability.

[0011] The DLN may comprise: a ThresholdLayer configured to convert continuous inputs to binary values; at least one LogicLayer configured to perform binary logic operations; and a SumLayer configured to aggregate outputs to determine classification.

[0012] This layered architecture allows the DLN to process continuous input data through discrete logic operations, enabling efficient and interpretable decision-making for classification tasks.

[0013] The first phase may comprise: applying a Softmax function to weights associated with logic operators in the at least one LogicLayer; and updating the weights using gradient descent.

[0014] By using a Softmax function and gradient descent, the method enables differentiable training of discrete logic operations, allowing the network to leam optimal logic functions for each neuron.

[0015] The second phase may comprise: applying a Softmax function to weights associated with neuron connections; and updating the weights using gradient descent.

[0016] This approach allows the network to leam optimal connections between neurons in a differentiable manner, facilitating the discovery of efficient network architectures.

[0017] The ThresholdLayer may comprise trainable bias and scale parameters for each neuron.

[0018] Trainable parameters in the ThresholdLayer enable the network to leam optimal thresholds for converting continuous inputs to binary values, improving the network's ability to handle diverse input data.

[0019] The method may further comprise: concatenating outputs of the ThresholdLayer with inputs to intermediate LogicLayers.

[0020] This concatenation technique allows the network to maintain access to raw input features throughout the network, potentially improving classification performance and interpretability.

[0021] The method may further comprise: applying a Straight-Through Estimator during training to sharpen function outputs while enabling gradient flow.

[0022] The use of a Straight-Through Estimator helps maintain the discrete nature of logic operations during training while still allowing gradients to flow, improving the network's ability to leam effective discrete representations.

[0023] Training the DLN may comprise: searching over a subset of possible logic operators and neuron connections.

[0024] By limiting the search space to a subset of possible operations and connections, the method can improve training efficiency while still allowing for sufficient expressivity' in the learned network.

[0025] The subset may comprise eight logic operators and eight possible input connections for each neuron.

[0026] This specific subset size provides a balance between computational efficiency and network expressivity, allowing for effective learning of complex classification tasks.

[0027] The method may further comprise: simplifying the quantized discrete logic mles of the DLN using symbolic mathematics techniques.

[0028] Simplification of the learned logic rules can improve the interpretability of the network's decision-making process and potentially reduce computational requirements during inference.

[0029] In a second aspect, a non- transitory computer readable storage medium is provided. The non-transitory computer readable storage medium contains instructions that, when executed by one or more processing units, causes the one or more processing units to perform the method of training an interpretable differentiable logic network for classification tasks.

[0030] This aspect enables the implementation of the training method on various computing devices, facilitating widespread adoption and use of the interpretable differentiable logic network approach.

[0031] In a third aspect, a system for performing classification tasks is provided. The system is configured to implement the method of training an interpretable differentiable logic network and comprises a differentiable logic network (DLN) comprising: a ThresholdLayer configured to convert continuous inputs to binary values: at least one LogicLayer configured to perform binary logic operations; and a SumLayer configured to aggregate outputs to determine classification.

[0032] This system architecture enables efficient implementation of the interpretable differentiable logic network, allowing for practical deployment in various classification applications.

[0033] Each neuron in the at least one LogicLayer may be configured to receive two binary inputs and output a binary value based on a learned logic operation.

[0034] This binary input-output configuration for LogicLayer neurons ensures that the network maintains interpretable discrete logic operations throughout its structure.

[0035] The SumLayer may be configured to: compute a weighted sum of inputs from the at least one LogicLayer; and determine a classification based on the weighted sum.

[0036] This aggregation approach in the SumLayer allows the network to combine the outputs of multiple logic operations to make a final classification decision, enabling complex decision-making while maintaining interpretability.

[0037] The ThresholdLayer may comprise: a plurality of neurons, each neuron associated with a continuous input feature and configured to output a binary value based on a learned threshold.

[0038] This structure for the ThresholdLayer enables effective conversion of continuous input features to binary values, facilitating the processing of real-world data through the logicbased network.

[0039] The DLN may be configured to output interpretable logic rules for classification decisions.

[0040] This capability to output interpretable logic rules provides transparency in the decision-making process, which is crucial for applications where understanding the reasoning behind classifications is important.BRIEF DESCRIPTION OF FIGURES

[0041] Embodiments of the invention will be described, by way of example, with reference to the following drawings, in which:

[0042] Figure 1 is a schematic showing a simplified DLN example. It takes input samples and binarizes continuous variables through a ThresholdLayer. It then passes the binary vector to layers of two-input Boolean logic operators. Finally, it counts logic rule triggers to determine the sample’s class.

[0043] Figure 2 is a table (Table I) listing of real-valued binary logic operations.

[0044] Figure 3 is a schematic of a DLN training flowchart.

[0045] Figure 4 is an algorithm (Algorithm 1) of a skeleton training process for DLNs.

[0046] Figure 5 is a table (Table II) providing a summary of example DLN trainable parameters and feedforward functions during training and inference, where "x" denotes input and "y" denotes output of each layer.

[0047] Figure 6 is an algorithm (Algorithm 2) of Phase I training.

[0048] Figure 7 is an algorithm (Algorithm 3) of Phase II training.

[0049] Figure 8 is an algorithm (Algorithm 4) of inference.

[0050] Figures 9 A and 9B are schematic illustrations of a two-phase training algorithm, where Phase I (9A) involves discretize and fix the connections between neurons while training their parameters within ThresholdLayer and LogicLayer, and Phase II (9B) includes holding the neuron operations constant and focusing on optimizing the connections between them; key details are highlighted for clarification.

[0051] Figure 10 is a graph showing an example training process of a ThresholdLayer neuron.

[0052] Figure 11 is a chart showing example training of a LogicLayer neuron, using a logarithmic scale.

[0053] Figure 12 is a table (Table III) providing characteristics of the post-processed datasets and the average balanced-class test accuracy of models.

[0054] Figure 13 is a table (Table IV) providing average number of operations for inference, assuming Float 16 for floating-point and Inti 6 for integer.

[0055] Figure 14 is a table (Table V) providing average number of model parameters and disk space in bytes, assuming Floatl6 for numerical values and Intl6 for indices.

[0056] Figures 15A-15F are, when combined, a visualization of the decision-making process of a DLN for predicting cirrhosis disease; it achieves a balanced-class test accuracy of 59.3%. The DLN uses 6 continuous features and 4 categorical features, selected from an original set of 10 continuous features and 5 categorical features.

[0057] Figures 16A-16D are. when combined, a visualization of the decision-making process of a DLN for predicting heart failure disease; it achieves a balanced-class test accuracy of 70.5%. The DLN uses 4 continuous features and 2 categorical features, selected from an original set of 6 continuous features and 5 categorical features

[0058] Figures 17A-17C are. when combined, a visualization of the decision-making process of a DLN for predicting liver disease; it achieves a balanced-class test accuracy of 72.5%. The DLN uses 5 continuous features, selected from an original set of 9 continuous features and 1 categorical feature.

[0059] Figure 18 is a table (Table VI) providing average balanced-class test accuracy percentage of different methods and their ranks based on the number of basic logic-gate operations.

[0060] Common reference numerals are used throughout the figures to indicate similar features.DETAILED DESCRIPTION

[0061] The present disclosure relates to methods and systems for training and implementing interpretable differentiable logic networks for classification tasks. A differentiable logic network (DLN) is a type of machine learning model that combines the interpretability of logic-based systems with the trainability of neural networks.

[0062] In some examples, a method for training a DLN includes initializing the network and iteratively training it through alternating phases. The DLN may be initialized with random weights and a predefined architecture comprising multiple layers designed to perform logical operations in a differentiable manner.

[0063] A system for performing classification tasks using a DLN may include various components configured to implement the training method and execute inference. The system may comprise a differentiable logic network with specialized layers for processing input data, performing logical operations, and aggregating results to determine classifications.

[0064] The DLN approach aims to address limitations of traditional neural networks by providing more interpretable decision-making processes while maintaining competitive classification performance. By leveraging differentiable approximations of discrete logical operations, DLNs enable gradient-based optimization techniques to be applied to logic-based models.

[0065] A differentiable logic network (DLN) comprises multiple layers designed to perform logical operations in a differentiable manner. FIG. 1 illustrates an example architecture of a DLN. The DLN includes a ThresholdLayer, at least one LogicLayer, and a SumLayer.

[0066] A ThresholdLayer is configured to convert continuous inputs to binary values.

[0067] In FIG.9A and FIG.9B, the graph consists of these components from left to right:

[0068] 1. Two squares, the top one represents a continuous input and the bottom one represents a categorical input;

[0069] 2. A ThresholdLayer with four threshold neurons and one virtual neuron;

[0070] 3. A LogicLayer with six logic neurons that their outputs are labeled from xo to xs;

[0071] 4. A LogicLayer with four logic neurons that their outputs are labeled from zo to zs; and

[0072] 5. A SumLayer with three sum neurons.

[0073] Therefore, xo - xs are outputs of the first LogicLayer, which also makes them inputs to the second LogicLayer. zo - Z3 are outputs from the second LogicLayer. which also makes them inputs to the SumLayer. Each of these nodes may, independently, be configured to output a binary value based on a learned threshold applied to its corresponding continuous input.

[0074] At least one LogicLayer may be configured to perform binary logic operations. A LogicLayer receives binary inputs from the previous layer and applies learned logical operations to produce binary outputs. The LogicLayer may contain multiple neurons, each performing a specific logical function on its inputs.

[0075] A SumLayer is configured to aggregate outputs to determine classification. The SumLayer may include multiple output nodes. Each output node may compute a weighted sum of inputs from the preceding LogicLayer and determine a classification based on this sum. For example, in FIGS. 9A and 9B, zo - Z3 are outputs from the second LogicLayer, which also makes them inputs to the SumLayer.

[0076] The combination of these layers allows the DLN to process continuous input data, perform logical reasoning, and produce classification outputs in a manner that can be trained using gradient-based optimization techniques while maintaining interpretability.

[0077] The ThresholdLayer of the differentiable logic network (DLN) may be configured to convert continuous inputs to binary values. The ThresholdLayer may include a plurality of neurons, each neuron associated with a continuous input feature and configured to output a binary value based on a learned threshold.

[0078] Each neuron in the ThresholdLayer includes trainable bias and scale parameters. A threshold value may be used as an initial value for the bias parameter of a neuron. The bias value may represent an example of a learned bias parameter for a neuron after training.

[0079] During the training phase, the ThresholdLayer uses a shifted and scaled sigmoid function to enable differentiability7. The output y of a neuron for an input x can be expressed as:

[0080] where s is the scale parameter and b is the bias parameter.

[0081] During inference, the ThresholdLayer switches to using a Heaviside function to produce binary outputs. The output y of a neuron for an input x can be expressed as: yt— Heaviside(.$i ■ (x, — b;)).

[0082] To initialize the bias values of the ThresholdLayer neurons, tree-based binning may be employed. This approach uses decision tree algorithms to determine initial threshold values that effectively separate the input data into binary categories.

[0083] The ThresholdLayer's ability to convert continuous inputs to binary values while maintaining differentiability during training allows the DLN to process real-valued data in a manner compatible with logical operations in subsequent layers.

[0084] The LogicLayer of the differentiable logic network (DLN) is configured to perform binary logic operations. FIG. 2 illustrates various logic operations that may be implemented in the LogicLayer.

[0085] The LogicLayer comprises multiple neurons, each configured to receive two binary inputs and output a binary value based on a learned logic operation. During training, the LogicLayer uses real-valued logic operations to enable differentiability, while during inference, it switches to binary logic operations.

[0086] The LogicLayer may implement various types of logic operations, including:

[0087] 1. A false operation 0, which always outputs a binary 0 regardless of inputs.

[0088] 2. An and operation 1, which outputs 1 only if both inputs are 1.

[0089] 3. A negated implication operation 2, which outputs 1 if the first input is 1 and the second input is 0.

[0090] 4. An input operation 3, which passes through the first input.

[0091] 5. A negation of the converse implication operation 4, which outputs 1 if the inputs are not(B implies A) (e.g., outputs 1 when the first input is 0 and second input is 1).

[0092] 6. An output operation 5, which passes through the second input.

[0093] 7. An exclusive or operation 6, which outputs 1 if exactly one input is 1.

[0094] 8. An or operation 7, which outputs 1 if at least one input is 1.

[0095] 9. A negated or operation 8, which outputs 1 only if both inputs are 0.

[0096] 10. A negated exclusive or operation 9, which outputs 1 if the inputs are the same.

[0097] 1 1. A negated output operation 10, which outputs the negation of the second input.

[0098] 12. A converse implication operation 11, which outputs 1 if the inputs are the B implies A (e.g., it outputs 0 when the first input is 0 and the second input is 1).

[0099] 13. A negated input operation 12, which outputs the negation of the first input.

[0100] 14. An implication operation 13, which outputs 0 only if the first input is 1 and the second input is 0.

[0101] 15. A negated and operation 14, which outputs 0 only if both inputs are 1.

[0102] 16. A true operation 15, which always outputs a binary 1 regardless of inputs.

[0103] During training, each neuron in the LogicLayer is associated with a logic weight for each possible logic operation. These weights are used to compute a weighted sum of the real-valued logic operations, allowing for gradient-based optimization.

[0104] The real- valued logic operations used during training are continuous approximations of the binary logic operations. For example, the real-valued AND operation may be implemented as the product of the two inputs, while the real-valued OR operation may be implemented as the sum of the inputs minus their product.

[0105] During inference, the LogicLayer quantizes the learned weights to select a single binary logic operation for each neuron. This quantization process ensures that the final network produces interpretable logic rules while maintaining the performance gained through gradientbased training.

[0106] The combination of these logic operations in the LogicLayer allows the DLN to perform complex logical reasoning on the binary outputs from the ThresholdLayer, enabling the network to learn and represent intricate decision boundaries for classification tasks.

[0107] The SumLay er of the differentiable logic network (DLN) is configured to aggregate outputs from the LogicLayer to determine classification. FIG. 1 illustrates an example architecture of a DLN, including the SumLay er.

[0108] The SumLay er is configured to compute a weighted sum of inputs from the at least one LogicLayer. Each neuron in the SumLay er receives inputs from neurons in the precedingLogicLayer. These inputs are multiplied by corresponding weights and summed to produce an activation score for the neuron.

[0109] The SumLayer is configured to determine a classification based on the weighted sum. The activation scores of the neurons in the SumLayer represent the likelihood of the input belonging to different classes. The class corresponding to the neuron with the highest activation score may be selected as the final classification output.

[0110] During training, the SumLayer uses probabilistic connections to enable gradientbased optimization. Each connection between a LogicLayer neuron and a SumLayer neuron is associated with a weight. These weights are trained using backpropagation to optimize the classification performance of the DLN.

[0111] During inference, the SumLayer switches to using binary connections. The trained weights are thresholded to create binary connections, where weights above a certain threshold are set to 1, and those below are set to 0. This binarization process allows the SumLayer to produce interpretable logic rules for classification decisions.

[0112] The SumLayer may include multiple output nodes, such as the first output node, the second output node, the third output node, and the fourth output node. Each of these output nodes may correspond to a different class in the classification task. The binary outputs from the LogicLayer neurons are aggregated by these output nodes to determine the final classification.

[0113] By combining the binary outputs from the LogicLayer with learned weights and thresholding, the SumLayer enables the DLN to make classification decisions based on interpretable logical rules while maintaining the ability to be trained using gradient-based optimization techniques.

[0114] The training process for a differentiable logic network (DLN) involves an iterative approach that alternates between two phases for a predetermined number of iterations. FIG. 3 illustrates an overview of this two-phase training process.

[0115] In the first phase, the DLN learns neuron functionality parameters while fixing neuron connections. During this phase, a Softmax function is applied to weights associated with logic operators in the LogicLayer. The weights are then updated using gradient descent. FIG. 9A shows examples of different logic operations that may be learned during this phase, including the false operation 0, the and operation 1, the negated implication operation 2, and the true operation 15.

[0116] The second phase focuses on learning neuron connection parameters while fixing neuron functionalities. In this phase, a Softmax function is applied to weights associated withneuron connections. These weights are also updated using gradient descent. FIG. 9B depicts the process of learning connection parameters between neurons.

[0117] To enable more efficient training, the DLN searches over a subset of possible logic operators and neuron connections. This subset typically comprises eight logic operators and eight possible input connections for each neuron. FIG. 2 illustrates a selection of logic operators that may be included in this subset, such as the negated equivalence operation 4. the output operation 5, and the exclusive or operation 6.

[0118] During both phases of training, a Straight-Through Estimator is applied to sharpen function outputs while enabling gradient flow. This technique helps maintain binary-like behavior during the forward pass while allowing gradients to flow during backpropagation.

[0119] To enhance the network's ability to capture complex relationships, the output of the ThresholdLayer is concatenated to multiple intermediate LogicLayers. This concatenation allows the network to maintain access to the original binarized inputs throughout the logical reasoning process.

[0120] The training process involves various weights and parameters. A threshold parameter is used to determine the cutoff for binarization. A logic operator weight is used to determine the logic functionality strength for logic neurons. A connection weight represents the strength of connections between neurons.

[0121] By alternating between learning neuron functionalities and connections, the DLN can effectively leam to perform complex logical operations while maintaining interpretability. This training process allows the network to adapt both its internal logic and its connectivity structure to best fit the classification task at hand.

[0122] After training, the differentiable logic network (DLN) undergoes a simplification process to extract interpretable logic rules for classification decisions. This process involves quantizing the discrete logic rules and applying symbolic mathematics techniques to simplify the resulting expressions.

[0123] FIG. 14 illustrates the relative sizes of different model parameters after training. The DLN parameters, represented by various element labels such as the logic weight, the connection weight, and the sum weight, are quantized to discrete values. This quantization step converts the continuous weights learned during training into binary values, effectively selecting specific logic operations and connections in the network.

[0124] Once the network parameters are quantized, symbolic mathematics techniques are applied to simplify the resulting logic expressions. In some examples, the SymPy library may be used to perform this simplification. SymPy provides tools for symbolic mathematicsmanipulation, allowing for the reduction of complex logical expressions into simpler, equivalent forms.

[0125] The simplification process may involve combining redundant logic operations, eliminating unnecessary connections, and reducing the overall complexity of the network while maintaining its classification performance. For instance, a series of the and operation 1 and the or operation 7 may be simplified into a single logical expression that captures the same decision boundary.

[0126] FIGS. 15A-15F, together, show an example of the simplified logic rules extracted from the DLN. The figure includes a first output node Class_l, a second output node Class_2, and a third output node Class_3, each representing a different class in the classification task. The connections between the input features and these output nodes represent the simplified logic rules for making classification decisions.

[0127] For example, the logic rule for the first output node Class_l may be expressed as a combination of thresholded input features and logical operations. This rule might involve the negated implication operation 2 applied to two binarized input features, followed by the or operation 7 with another thresholded feature.

[0128] The simplified logic rules provide a clear and interpretable representation of the DLN's decision-making process. Unlike traditional neural networks, where the decision process may be opaque, the DLN's simplified rules allow for direct interpretation of how input features contribute to classification decisions.

[0129] FIGS. 17A-17C, together, demonstrate how these simplified logic rules can be visualized for easier interpretation. The figure may show a decision tree-like structure where each node represents a logical operation or a thresholded input feature, and the paths from the root to the leaves represent the logic rules for different classes.

[0130] By configuring the DLN to output these interpretable logic rules, the model provides both high classification performance and explainable decision-making. This combination of accuracy and interpretability makes the DLN particularly suitable for applications where understanding the reasoning behind classifications may be as significant as the classifications themselves.

[0131] The differentiable logic network (DLN) demonstrates competitive performance in terms of accuracy, computational efficiency, and model size when compared to other machine learning methods. FIG. 12 illustrates a comparison table III that showcases the performance metrics of DLNs alongside other classification algorithms.

[0132] In terms of accuracy, DLNs achieve comparable or superior results to traditional machine learning methods across various datasets. The DLN architecture, comprising a ThresholdLayer, LogicLayer, and SumLayer, enables effective learning of complex decision boundaries while maintaining interpretability.

[0133] FIG. 13 presents an operation comparison table IV that highlights the computational efficiency of DLNs relative to other methods. The table includes an operation rank score 8.0 for various algorithms, where a lower score indicates higher efficiency. A decision tree rank 1.0 represents the most computationally efficient method, followed by a differentiable network rank 2.1 for DLNs. This demonstrates that DLNs offer substantial computational advantages over more complex models such as neural networks.

[0134] The operation comparison table IV also shows the number of high-level operations and basic logic gate-level operations required for inference across different datasets. For example, the bank operation count 316K and bank gate count 106M illustrate the computational requirements for a banking dataset. The relatively low operation counts for DLNs across various datasets underscore their efficiency.

[0135] FIGS. 15A-17C depict the decision-making process of various DLNs, showcasing its efficient architecture. The figures includes input nodes, logic nodes, and output nodes. These components w ork together to process input data through logical operations, enabling efficient inference.

[0136] The compact size of DLNs contributes to their efficiency. The logic weight, connection weight, and sum weight represent the learned parameters of the network, which are typically fewer in number compared to traditional neural networks. This reduced parameter count leads to smaller model sizes and faster inference times.

[0137] FIG. 18 may provide additional insights into the performance characteristics of DLNs across different datasets or tasks. The figure could illustrate how DLNs maintain high accuracy while requiring fewer computational resources compared to other methods.

[0138] When comparing DLNs to other machine learning approaches, the operation comparison table IV in FIG. 13 provides a comprehensive overview. The table includes rankings for various methods, such as a random forest (RF) rank 5.0, a Naive Bayes (NB) rank 3.4, an adaptive boost (AB) rank 4.9, a neural network (MLP) rank 7.0, and a Deep Differentiable Logic Gate Network (LGN) rank 4.7. These rankings demonstrate that DLNs offer a favorable balance between accuracy and computational efficiency.

[0139] The efficiency of DLNs stems from their use of binary logic operations in the LogicLayer. Operations such as the and operation, or operation, and exclusive or operation canbe implemented using simple logic gates, resulting in faster computation compared to the floating-point arithmetic used in traditional neural networks.

[0140] Furthermore, the ThresholdLayer of the DLN, with its learned threshold value and bias value, enables efficient binarization of input features. This binarization process reduces the complexity of subsequent computations in the LogicLayer and SumLayer.

[0141] The SumLayer, which computes an activation score for classification, aggregates the binary outputs from the LogicLayer efficiently. This approach contrasts with the more computationally intensive operations found in fully connected layers of traditional neural networks.

[0142] In summary, DLNs offer a compelling combination of accuracy, computational efficiency, and model compactness. The architecture of DLNs, leveraging binary operations and efficient feature processing, enables them to achieve competitive performance while maintaining interpretability and requiring fewer computational resources compared to many traditional machine learning methods.

[0143] The differentiable logic network (DLN) comprises multiple layers that work together to perform classification tasks. FIG. 1 illustrates an example architecture of a DLN, showing the interaction between the ThresholdLayer, LogicLayer, and SumLayer.

[0144] The process begins with the ThresholdLayer, which receives continuous input features and converts them into binary values. FIG. 9B depicts an example of a ThresholdLayer with multiple nodes, each corresponding to an input feature. The ThresholdLayer applies learned thresholds to each input, effectively binarizing the data.

[0145] The binary' outputs from the ThresholdLayer are then passed to the LogicLayer. The LogicLayer consists of neurons that perform binary logic operations on their inputs. These neurons leam to implement various logical functions during the training process, allowing the network to capture complex relationships in the data.

[0146] To enhance the network's ability to reason about the original input features throughout the logical processing, the outputs of the ThresholdLayer may be concatenated with inputs to intermediate LogicLayers. This concatenation allows subsequent layers to have direct access to the binarized input features, potentially improving the network's ability to leam relevant patterns.

[0147] After processing through one or more LogicLayers, the binary outputs are fed into the SumLayer. The SumLayer aggregates these inputs, computing weighted sums to produce activation scores for each class. The class with the highest activation score is typically selected as the final classification output.

[0148] FIG. 9 A shows an example of how the different layers may be connected in a DLN architecture. The flow of data from the input features through the ThresholdLayer, LogicLayer(s), and finally to the SumLayer illustrates the sequential processing that occurs during both training and inference.

[0149] The interaction between these layers enables the DLN to process continuous input data, perform logical reasoning on binarized features, and produce classification outputs. By leveraging differentiable approximations of discrete operations during training, the DLN can leam complex decision boundaries while maintaining interpretability in its final form.

[0150] Example.

[0151] FIG. 1 illustrates a sample DLN, with inputs of Continuous features 110 and Categorical features 120. The DLN consists of three types of layers: a ThresholdLayer (with ThresholdLayer neurons 130) at the input to binarize continuous input features into binary 0 / 1 values using threshold functions, multiple LogicLayers (with LogicLayer neurons 140) consisting of two-input logic operators for performing logic operations, and a SumLayer (with SumLayer neurons 150) at the end to aggregate the output from the last LogicLayer to determine the class. One can directly extract logic rules from the network, thereby making its predictions interpretable. The disclosed DLN is similar to conventional MLPs in terms of input and output: both accept samples with continuous and one-hot categorical features and output probability predictions for each class. However, a DLN differs from MLPs in three major ways: it transforms input into binary values at the first step; instead of conducting matrix multiplications between MLP hidden layers, it performs logic operations within the network; and the connections in LogicLayers are very sparse, primarily because each neuron accepts only two inputs.

[0152] The disclosed training objectives are twofold: determining which function each neuron should implement and establishing how the neurons should be connected. Neuron function optimization and neuron connection optimization can be divided into two separate tasks because they optimize different sets of parameters. Therefore, the training process is divided into two phases: one for optimizing the internals of neurons (referred to as Phase I) and another for optimizing the connections between neurons (referred to as Phase II). However, both problems are discrete, making common NN training optimizers like gradient descent unsuitable for the task.

[0153] The disclosed methodologies relax the discrete search spaces to make them continuous and differentiable. Details of this are described below; a coarse summary is that discrete search is relaxed among candidates into a probabilistic mixture of candidates.

[0154] A high-level overview of the DLN training workflow can be seen in FIG. 3 and its algorithmic summary in Algorithm 1 (FIG. 4).

[0155] Model parameters are iteratively updated in the two phases in an alternating manner. After initialization (step 310 in FIG. 3), Phase I training is conducted for E epochs (step 320), followed by Phase II training for E epochs (step 330). The combination of Phase I and Phase II training is continued for T iterations (step 340). As Algorithm 1 suggests, during one phase, the trainable parameters of the other phase are frozen, and a hard assignment to its related functions is performed. Thus, in Phase I, the neuron connections are fixed, and argmax is taken over connection parameters; specifically, the link between any two neurons is either 0 or 1 and nothing in between. In Phase II, the neuron functions are frozen and hard-assigned so that each neuron performs only one discrete logic operation among all possible operations. After freezing and quantizing the parameters related to the other phase, one can relax the discrete parts related to the under-training phase to make optimization differentiable. The algorithm updates relevant parameters solely through gradient descent.

[0156] Once training is complete, the functions and connections of neurons are both discretized to hard binary states, as FIG. 1 demonstrates. Generally, this discretization leads to a quantization loss; however, this loss is typically small compared to the generalization loss because most components converge well. In some cases, discretization even reduces overfitting and helps with generalization.

[0157] Table II (FIG. 5) summarizes the trainable parameters for each layer type and their usages in the two training phases and inference. The forward computation equation for each layer is presented and, for simplicity', the output function at the neuron level, i.e., the function to calculate the z-th output value yi from the input vector x. The parameters involved in training are separated by phases and are applied differently between Phase I. Phase II. and inference. For the ThresholdLayer, the Heaviside function is used when its parameters are not under training and use the scaled, shifted Sigmoid function is used when they are under training. In LogicLayer, weights are trained to search for optimal neuron functions and connections. Each weight acts as a learnable probability logit during its training phase to make the network differentiable. During the other phase or inference, the parameter is quantized to its corresponding maximum probability. In SumLayer, w eights represent the connection strengths between the neurons of the last hidden layer and the output neurons. They are quantized to binary' values during inference.

[0158] The Phase I training process is presented in Algorithm 2 (FIG. 6). the Phase II training process in Algorithm 3 (FIG. 7), and the inference process in Algorithm 4 (FIG. 8).The processes for the two training phases are quite symmetric because they employ the same procedure; it is just that the layers switch their functions between the two modes. For clear illustration, a mini-batch size of 1 is employed, and the layer output computation is showcased at the neuron level, i.e., it is shown how the / -th output value is computed. For practical implementation, a larger batch size is used during training and the layer output is computed with the help of vectorization. In the two training algorithms, the forward pass to highlight details of how to make the forward propagation differentiable is emphasized. Gradient-based backpropagation is conducted automatically as long as the forward steps are differentiable. During the forward pass, the input sample goes through the DLN, layer by layer; each layer takes the output of the previous layer as its input. Depending on the type of the layer and the phase it is in, each neuron computes its output based on the equations in the pseudo-code, which is discussed in more detail below.

[0159] A. Logic Operations

[0160] Neurons in LogicLayer act as binary logic operators, which are inherently non- differentiable. To overcome this obstacle, real-valued operations are employed during the training phase, as outlined in Table I. This approach effectively softens the discrete logic operations, making them both continuous and differentiable. For inference, it is reverted to the standard binary' 0 / 1 logic operations. SoftLogic* is used to denote the real-valued version of the A- th logic operator in Table I and HardLogic / c to denote its binary operation. Consider the example of an AND logic gate, which is described first in the table. For this gate, the soft (real- valued) and hard (binary) outputs are obtained as follows. For the soft output, the real-valued function is:

[0161] real-valued AND(c / . 6) = SoftLogici(a, b) = a b,

[0162] where a, b e [0, 1] are continuous real numbers. For the hard output, the discretized function is:

[0163] binary AND(a, b) = HardLogici(

[0164] w here a, b £ {0, 1 } are discrete binary numbers, and indicator functions are applied to them.

[0165] To ensure gradient flow between layers, fuzzy7logic expressions are utilized throughout the training process, regardless of the phase type. As show n in line 14 of Algorithm 2 (FIG. 6) and line 14 of Algorithm 3 (FIG. 7), SoftLogic is used in both training Phase I and Phase II. Line 9 of Algorithm 4 (FIG. 8) indicates that HardLogic is used for inference.

[0166] B. Phase I: Determining Neuron Functions

[0167] FIG. 9A illustrates the process of determining the functions of neurons. In this phase, the connections between neurons are discretized and frozen, allowing one to focus on training the neuron parameters. DLNs consist of three types of layers: ThresholdLayer, LogicLayer, and SumLayer. Correspondingly, there are three types of neurons. Neurons in the SumLayer (SumLayer neurons 150 in FIG. 1) simply aggregate the outputs from the preceding LogicLayer (LogicLayer neurons 140 in FIG. 1) that connect to them; therefore, they do not require any parameters to describe their function. During this phase, the disclosed approach focuses on training the parameters for neurons in ThresholdLayer (ThresholdLayer neurons 130 in FIG. 1) and LogicLayer (LogicLayer neurons 140 in FIG. 1). In the following subsections, we detail the training algorithms for each layer type.

[0168] 1) ThresholdLayer: The purpose of the ThresholdLayer is to binarize continuous variables using threshold functions. This layer takes a continuous vector as input and outputs a binary vector. Each neuron takes one scalar input and outputs a scalar value. The output binary vector can later be concatenated with one-hot encoded categorical feature values to form the final binary vector for the next layer. Each continuous input value is connected to multiple neurons to map it to a finite set of values, an approach similar to binning but not limited to mutually exclusive bins. Each neuron takes in one scalar and serves as a threshold function with trainable bias and scale. During training, the non-differentiable threshold function is softened by substituting it with a shifted and scaled sigmoid function:

[0169] fix) = Sigmoidty •

[0170] where b represents the cutoff and s denotes the slope of the input passed to the Sigmoid function. This modification enables gradient flow to both b and During Phase I of training, both b and s are trainable. During Phase II or inference, the soft threshold function is quantized to the normal threshold function:

[0171] F(x) = Heavisi dety • (x~b))

[0172] and freeze parameters b and s. In Fig. 9A the training mode of the ThresholdLayer is illustrated, while the inference mode is illustrated in FIG. 9B (ThresholdLayer functions the same way in both Phase II and during inference). In Phase I, the highlighted neuron applies a shifted and scaled Sigmoid function to its input. After forward and backward propagations, gradients suggest a positive increase in s (which results in steeper scaling) and a positive increase in b (which results in a rightward shift of the threshold function). When in Phase II, the function of this neuron is quantized and fixed to a Heaviside function and is no longer trainable. The Phase I training pseudo-code is shown for the ThresholdLayer in lines 6 to 8 inAlgorithm 2, the Phase II training pseudo-code in lines 6 to 8 in Algorithm 3, and the inference pseudo-code in line 2 in Algorithm 4.

[0173] In experiments, the input data is preprocessed by applying Min-Max Scaling to continuous variables and one-hot encoding to categorical variables, ensuring that all input values range between 0 and 1. Four or six neurons are allocated to each continuous input and the starting slope is initialized to 2, a value not too flat but still smooth in the beginning. Here, b is initialized for each neuron by the bin edges of calculated decision trees. Hence, a decision tree is fit using training data, by specifying the maximum number of leaf nodes. It has been observed that most neurons in the ThresholdLayer converge during training, with some converging to values within the 0 and 1 range while others do not. Neurons whose biases converge to below 0 or over 1 act as constant Boolean False or True. It was found that out-of- range convergence leads to good feature selection. DLNs with fixed (i.e., non-trainable) ThresholdLayers underperform DLNs with trainable ThresholdLayers and are larger in size.

[0174] FIG. 10 shows an example of a neuron’s subsampled training process. It plots the resulting neuron function based on the b and s values at the sampled epochs. In the beginning, it starts with a certain cutoff and a small slope, thus initially acting as a smooth function. As the epochs progress, both the cutoff and slope converge, with the bias shifting to the left and the slope increasing, making the neuron increasingly resemble a threshold function.

[0175] 2) LogicLayer: As indicated in Table I. a two-input neuron can implement 222=16 Boolean functions. Next, the goal is to determine which of the 16 classes each neuron should belong to. This can be treated as a classification problem.

[0176] During training, each neuron is considered a weighted sum of all 16 operators. This weight is trainable. To make sure that the weights associated with the operators sum to 1, a Softmax function is applied to the weights. Let a 16-valued weight vector w denote the weights for a sample logic neuron. Thus, w, corresponds to the z-th Boolean operator. In this phase, connections to neurons are fixed, with each logic neuron receiving inputs from exactly two input neurons. Given scalar inputs a and b, the output y of the logic neuron is given by the sum of logic operator mixtures:

[0177] y = Efcl=0[Softmax(

[0178] Lines 11 to 14 of Algorithm 2 calculate the weights and sum up the 16 logic function outputs.

[0179] During Phase II or inference, the summation is quantized by selecting the one operator that has the largest corresponding weight. The index of this operator is found and employ solely its operation as the logic neuron’s function:

[0180] k = arg max Wy( SoftLogicfe(a, b), if training,

[0181] y (HardLogiCfe(a)) otherwise

[0182] In Algorithm 3, line 11 finds the index, and line 14 performs the real-valued logic operation. In the inference algorithm, Algorithm 4, line 5 finds the index, and line 9 performs the binary' logic operation. FIGS. 9A-9B illustrate the training mode of the LogicLayer neuron function (9A) and the inference mode (9B). Neuron function training only occurs in Phase I; hence, neurons perform the same function in Phase II and during inference. In Phase I. the output of a neuron is a weighted sum of all 16 logic functions, and this weight is trainable. When not under training, the function of the neuron is frozen and attached to one type of operator.

[0183] In practice, with the aid of the search for neuron connections in Phase II, it was found that most neurons can converge to a single operator. In FIG. 11, a typical example of a logic neuron’s subsampled training process is shown. The x-axis displays the epoch number and the y-axis shows the softmaxed weight distribution of the 16 operators on a logarithmic color scale. As illustrated in the figure, this neuron rapidly converges to the fourth (index no. 3, shown in yellow) logic operator.

[0184] C. Phase II: Determining Connections.

[0185] FIG. 9B illustrates the training process for determining the connections between neurons. In this phase, the functions of neurons are fixed; that is, the parameters of the threshold functions for threshold neurons and the selection of logic operators for logic neurons. For the ThresholdLayer, the input connections are fixed after initialization. In the setup, each continuous input feature is connected to several threshold neurons, and each neuron takes only one input feature. Therefore, the ThresholdLayer does not have trainable parameters in Phase II.

[0186] Next, it is described how connections to the LogicLayers and SumLayer are determined.

[0187] 1) LogicLayer: Each neuron in a LogicLayer has two inputs, denoted as a and b.The objective is to determine which two input values, among in_dim input values from the previous layer’s output, should be linked to a and b. This task is divided into two sub-goals:determining the source of a and determining the source of b. During training, the connections are softened by representing them as weighted sums of all inputs. The disclosed method is showcased for finding the first input, a, to the logic neuron. The method for finding a connection to b is the same. Let x G Ukin-dimrepresent the input of the logic neuron under study. In a LogicLayer, there are out_dim neurons; suppose one picks a random neuron for study. By design, the input a of this neuron must be one of the elements in x. Hence, the goal is to find the index of that element. This is a discrete optimization problem. During Phase II training, the problem is relaxed through weighted connections to all elements in x and reduce the problem to training a weight vector u 6to leam the weights linking the logic neuron to its input.A Softmax function is used to make sure that the weights sum to 1. The equation for the relaxed a is then a probabilistic mixture of x:

[0188] [SoftmaxCu)]

[0189] This approach allows each neuron to be non-uniformly connected to all neurons from the preceding layer during training. In practice, the neuron leams the importance of incoming connections and typically converges to the most relevant one. During Phase I or inference, the connections are quantized by selecting the most heavily weighted input neuron, resulting in

[0190] k = arg max Uy,

[0191] a — xk.

[0192] FIG. 9 A illustrates fixed connections to logic neurons, with a using dashed lines and b used dotted lines. Each neuron takes only one input as a and one as b. Connections are discretized and fixed during both Phase I training and inference. Lines 12 and 13 in Algorithm 2 show discrete link selection during Phase I training. Lines 6 and 7 in Algorithm 4 show the same argmax selections during inference.

[0193] In FIG. 9B, an example of a connection weight matrix during training is highlighted. In this phase, the neuron’s incoming a (again, dashed lines) is connected to all its inputs, and the value of the final a is a weighted sum of all these candidates. It is expected that gradient descent steps will make the weight concentrate on one link in the end. Lines 12 and 13 in Algorithm 3 show the relaxed, differentiable link search for logic neurons during Phase II training.

[0194] 2) SumLayer: The SumLayer has a number of neurons equal to the number of classes, |C|, with each neuron representing a class. This setup is similar to the output layer of an MLP. Each neuron in the SumLayer connects to a subset of neurons from the previouslayer and sums up their output scores. These scores are normalized, then used as logits, and later passed to the Cross-Entropy Loss function for training. To maximize the expressiveness of the network, nodes in the last hidden layer are allowed to connect to multiple neurons in the SumLayer.

[0195] Consequently, this setup translates to a classification problem for each (input, output) neuron pair: determine whether they should connect or not. A matrix S G n^m_dimx|c| is trained, and Sigmoid(S ) is treated as the probability that the / -th input neuron should connect with the / -th output class. In this training phase, for any output neuron, probabilistic connections from all inputs are summed up to enable differentiability7, as illustrated in FIG. 9B.

[0196] Suppose one picks the z-th output neuron to study. Mathematically, one can compute its output w from input x G Wn-dimusing:

[0197] Sigmoid(S;;i) ■ xy.

[0198] Line 17 of Algorithm 3 implements this equation.

[0199] When not in training mode, i.e., in Phase I or in inference, the output logit w is the binary7sum of inputs that have connection probability above a threshold, namely sum-th, 3S illustrated in FIG. 9A. Mathematically,

[0201] The pseudo-code for this equation is shown in line 17 of Algorithm 2 and in line 12 of Algorithm 4. Binary connections are imposed to SumLayers to maximize interpretability, ftum-th may be any appropriate value. In some embodiments, ftum-th may be 0.6-1.0, such as 0.7-0.9. It was found that setting ftum-th to 0.8 works well, in general.

[0202] D. Searching over Subspaces

[0203] In LogicLayers, for each neuron, the full search space has a size of 16 for the neuron type and a size of in_dim for the incoming link. However, full search is often not necessary. For example, any logic circuit can be constructed using just NAND gates. Selecting a subset of gates simplifies the training objective and helps with the vanishing gradient problem, as it reduces the weight dimension in Softmax layers.

[0204] One could select only logic gates that exhibit good gradient flow. From Table I, it can be seen that the OR gate has better gradient flow than the AND gate because inputs A and B are between 0 and 1. A similar situation applies to link search: there are many paths in a network that can yield good local optima. Fortunately, searching over a subspace can be flexibly implemented, thanks to the properties of the Softmax function. One just needs to assign -co to the weight entries that one wishes to turn off the search for; their Softmaxprobabilities will then be zero. Thus, they will never be active during training and inference, and there is no gradient flow to them.

[0205] One can sort the gate types by logic completeness and gradient flow, from high to low: NOR, NAND, XOR, XNOR, OR, AND, A, B, NOT A, NOT B, B IMPLIES A, A IMPLIES B, NOT (A IMPLIES B), NOT (B IMPLIES A), False, True. One can just include the first k candidates from the list and mask out the others. To obtain a subset of the link space, one can randomly select k incoming candidates and mask out the weights of the other candidates by assigning themIn practice, it was found that using a subset of 8 logic operators and a subset of 8 links works the best. Thus, in this example, this setting was used as the default.

[0206] The results of not using subsets are presented, using a subset size of 16 for link search, and using a subset size of 4 for both gates and links search, in FIG. 18.

[0207] E. Using Straight-Through Estimators

[0208] By design, every layer in a DLN employ s either a Sigmoid or Softmax function. This enables differentiability of the network. However, although Sigmoid and Softmax functions soften the step and argmax functions, they also weaken output activations. After encountering layers of Sigmoid and Softmax functions, the layer outputs tend towards the average and become more uniform, making the gradient descent method harder to work with. To sharpen the function outputs while enabling backpropagation of gradients at the same time, the Straight-Through Estimator (STE) can be employed. During the forward pass, STE applies the non-differentiable function (e.g., threshold, argmax, etc.) normally, but during the backward pass, it bypasses the non-differentiable function by treating its derivative as 0, allowing gradients from the subsequent layer to pass through unchanged. Mathematically, assume / is the discretized output value and g is the soft value; the final output h then is h = detach( / -g)+g, where detach is an operator that detaches the value from the backpropagation graph. Thus, the value / is passed to h but only g is counted for calculating gradients. This enables the output to attain the same value as the discrete function during the forward pass, and the same gradient as the soft function during the backward pass.

[0209] In practice, STE was found to be useful for DLN training and have set its use as the default. Results for the case when STE is not used is presented in FIG. 18.

[0210] F. Concatenating Inputs

[0211] Some inputs are brought closer to the output layer by concatenating the ThresholdLayer to every’ intermediate LogicLayer. i.e., from the second layer to the second last. The concatenated ThresholdLayer may either be the same layer reused at multiple placesor different ThresholdLayers in each place. Experimental results show that using different input layers is better than using the same layer, and both approaches are better than not concatenating. Multiple input concatenations are set as the default and the results of other mentioned methods are presented in FIG. 18.

[0212] G. Model Simplification

[0213] After training, logic expressions are obtained from the network and they are simplified using SymPy. This was found generally beneficial for reducing model size because the trainable ThresholdLayer acts as a feature selector. Other simplification methods such as pruning are explored, and the results are presented in the ablation studies section in FIG. 18. Based on the results, it was found that SymPy is sufficient for model simplification.

[0214] IV. EXPERIMENTS

[0215] The disclosed methods are tested and compared with seven other methods on 20 tabular datasets. The seven methods tested are k-nearest neighbors (KNN), decision tree (DT), random forest (RF), Gaussian Naive Bayes (NB), AdaBoost (AB), MLP, and tree-initialized DDLGN (abbreviated to LGN in our tables). The characteristics of the datasets after preprocessing are listed in Table III (FIG. 12). Their sample sizes range from hundreds to thousands, and their number of classes ranges from two to 26, which is typical for tabular datasets.

[0216] A. Experimental Setup

[0217] The datasets are first preprocessed. This involves removing samples with too many missing values, one-hot encoding categorical features, and scaling continuous features using Min-Max scalers to ensure they range between 0 and 1. For each (dataset, model) pair, four experiments are conducted, each with a different random seed. Each experiment consists of three steps: hyperparameter search, training, and evaluation. 32 parameter sets are sampled for every hyperparameter search and the set with the best cross-validated balanced-class test accuracy is selected. The number of folds ranges from two to four, with datasets containing fewer samples having more folds.

[0218] The Optuna search algorithm is employed for neural network-based models (i.e., MLP, DDLGN, DLN) and random sampling is used for traditional methods (i.e.. KNN, DT. RF, NB, AB). The scikit-leam package is used for data processing and experiments with traditional methods. NN models are trained using PyTorch. Ray is used to run parallel hyperparameter searches.

[0219] B. Accuracy

[0220] The predictive power of the models is measured using balanced-class accuracy. Table III (FIG. 12) presents the average results. DLN achieved the best average rank of 2.3 and NN-based methods generally outperform traditional methods. Another observation is that bagging and boosting help, as random forest and AdaBoost outperform the decision tree, as expected. DDLGN requires binary inputs; hence, continuous features need to be binned before passing to the network. In the original work, DDLGN bins were selected manually for tabular data and uniformly for image data. It was found that, compared to uniform thresholds, treebased binning improves DDLGN accuracy by a significant margin. Thus, results are presented for tree-initialized DDLGNs.

[0221] C. Efficiency

[0222] Efficiency is measured from two perspectives: inference cost and disk space cost. For inference computation cost, the number of high-level operations (OPs) and the corresponding number of basic logic gate-level operations are provided in Table IV (FIG. 13). High-level operations include addition, comparison, multiplication, division, logarithm, etc. For each high-level operation, depending on the input type (e g., float32. int64). one can compute its total number of operations in basic hardware logic gates. For example, to perform addition of two n-bit inputs, the least significant bit (LSB) is fed to a half-adder and the other bits to full-adders. A half-adder can be implemented with 5 NAND gates and a full-adder with 9 NAND gates. Hence, adding two / ?-bi t inputs requires 5+9*( / 7 I ) N AND operations. One can similarly convert all high-level operations to basic 2-input logic operations. One can count AND / OR / NAND / NOR as one logic operation, XOR / XNOR as three, and NOT as 0. The default number types were used when training models, i.e., float32 for NNs and float64 for most traditional models. Since most models can maintain inference accuracy even after downgrading the datatype to floatl6, it was assumed that floating-point numbers have a data ty pe of floatl6 and integers have a data type of inti 6 when presenting the counts in Table IV. DLN would have a higher advantage with larger datatypes because the only place it needs floating-point operations is when comparing continuous features with threshold values in the ThresholdLayer. Thus, this advantage of DLN is muted in Table IV. The table shows that decision trees are the most computationally efficient method and DLN is the second best. Logic-based networks are orders of magnitude more efficient than multiplication-based networks.

[0223] Table V (FIG. 14) presents the number of parameters and the required disk space for storage. Trees need to store threshold values and indices of features and child nodes. DLNs need to store the functionality of neurons and indices of their incoming links. It was assumedthat each float / int requires 1 bits and each index also needs 16 bits. On an average, decision trees are the smallest in size and naive Bayes the second smallest. The average rank of DLNs is 3.3, which is the smallest among NNs and substantially smaller than vanilla MLPs.

[0224] D. Interpretability

[0225] DLNs are interpretable by nature and their interpretability7is reinforced by the feature selection power of ThresholdLayers. FIGS. 15A-15F, 16A-16D, and 17A-17C illustrate the decision-making process of DLNs. Take the DLN in FIGS. 15A-15F as an example. It achieves a balanced-class test accuracy of 59.3% on the Cirrhosis dataset and makes decisions based on 6 continuous features and 4 categorical features, chosen from 10 continuous and 5 categorical features. ClassO. Classi, and Class2 represent the three class types. Yellow boxes represent features and diamonds represent logic operators. The DLN binarizes inputs based on comparisons for continuous features and equality checks for categorical features. Logic operators receive binary inputs and output binary7values. Class nodes receive binary scores from either feature boxes or logic operators and aggregate the scores. A link to a class node represents a logic rule. The multiplier * on the right of the link to the class node represents the strength of the associated rule. The class with the highest score is the final prediction.

[0226] E. Ablation Studies

[0227] Next, ablation studies are presented, based on subsetting of the search space, use of STE, concatenation of input and LogicLayers, simultaneous optimization of neurons and links, application of fixed thresholds, use of Gumbel-Softmax, progressive freezing, and pruning. In Table VI (FIG. 18), the average DLN accuracy and rankings based on both accuracy and the number of basic logic gate-level operations are shown. Rank is derived based on placing the corresponding column in Tables III (FIG. 12) and IV (FIG. 13). The DLN results are reproduced in the first column from those two tables. These original DLN results are derived under subsetting both gate and link search space to 8, using STE in all layers, and concatenating different ThresholdLayers to all middle LogicLayers. In the other columns, differences relative to the original setting are listed. For freezing and pruning, a 32-sample hyperparameter search was run based on the hyperparameters of the original model. Only the parameters related to freezing and pruning were searched over: starting epoch for freezing, starting and ending epochs, and convergence threshold for pruning. For other cases, hyperparameter search, training, and evaluation similar to the one performed for the original model w ere performed.

[0228] Subset Search Space: results of searching over the full space are presented, using a link subspace of size 16 for inputs of each neuron, and using both a link and gate subspace of size 4. We found that size 8 represents a local optimum.

[0229] STE: The results of when STE is not applied are shown. On average, these results are worse.

[0230] Input Concatenation: Not concatenating the ThresholdLayer with the LogicLayer and concatenating the same ThresholdLayer to every middle LogicLayer are tested. It was concluded that using different ThresholdLayers is better than using the same ThresholdLayer, and that itself is better than not concatenating.

[0231] Unified Phase: The results indicate that optimizing neuron functionalities and link connections in an alternating fashion is better in terms of both accuracy and model size than optimizing them together.

[0232] Fixed ThresholdLayer: results when the ThresholdLayers are initialized and frozen, i.e.. only LogicLayers and SumLayers are trained, are presented. This not only reduces model accuracy but also results in larger models. Thus, it is better to train the ThresholdLayers.

[0233] Gumbel-Softmax: Softmax activations were replaced with Gumbel-Softmaxes, using a Gumbel noise scale of 0.5. However, this does not improve the model.

[0234] Progressive Freezing: Based on the observation that layers closer to inputs perform more pattern extraction and layers closer to outputs perform more abstract pattern aggregation, experiments with progressively freezing layer parameters to enable better gradient flow as training progresses was performed. Results show that progressive freezing does not significantly improve model performance.

[0235] Progressive Pruning: Some neurons are not helpful in making predictions, e.g., a neuron ANDed with an always False neuron. Neurons that do not converge well may be redundant or even harmful to prediction accuracy. Hence, pruning out neurons that do not converge well (i.e., probabilities do not concentrate on one type of gate) was tested in a progressive manner. However, this does not improve model accuracy. One reason may be that a larger DLN architecture search space is needed when applying pruning. Another reason may be that a non-useful neuron is already ignored in the link search phase.

[0236] In various embodiments, the disclosed techniques may be implemented in various manners. For example, a non-transitory computer readable storage medium may contain instructions that, when executed by one or more processing units, causes the one or more processing units to (collectively) perform an embodiment of the disclosed method to train the interpretable differentiable logic network. In other examples, a non-transitory computer readable storage medium may contain instructions that, when executed by one or more processing units, causes the one or more processing units to (collectively) infer one or morevalues using an interpretable differentiable logic network trained according to an embodiment of a method as disclosed herein.

[0237] Systems may involve one or more processing units operably coupled to memory and / or a non-transitory computer-readable storage medium. As used herein, “processing unit” refers to any hardware-based or software-emulated computational component configured to perform operations required for executing neural network algorithms, including but not limited to matrix multiplications, activation functions, gradient computations, and parameter updates. This term encompasses a range of architectures such as central processing units (CPUs), graphics processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), field-programmable gate arrays (FPGAs), and application-specific integrated circuits (ASICs), as well as virtualized or cloud-based compute instances designed to accelerate machine learning workloads. The processing unit may operate individually or in parallel with other units to support distributed or parallelized training and inference processes.

[0238] Features of any of the examples or embodiments outlined above may be combined to create additional examples or embodiments without losing the intended effect. It should be understood that the description of an embodiment or example provided above is by way of example only, and various modifications could be made by one skilled in the art. Furthermore, one skilled in the art will recognize that numerous further modifications and combinations of various aspects are possible. Accordingly, the described aspects are intended to encompass all such alterations, modifications, and variations that fall within the scope of the appended claims.

Claims

CLAIMS1 . A method for training an interpretable differentiable logic network for classification tasks, the method comprising: initializing a differentiable logic network (DLN); and iteratively training the DLN by alternating, for a predetermined number of iterations, between: a first phase of learning neuron functionality parameters for the DLN while fixing neuron connections such that a link between any two neurons is either 0 or 1; and a second phase of learning neuron connection parameters for the DLN while fixing neuron functionalities such that each neuron performs only one discrete logic operation among all possible operations.

2. The method of claim 1, wherein the DLN comprises: a ThresholdLayer configured to convert continuous inputs to binary values; at least one LogicLayer configured to perform binary logic operations; and a SumLayer configured to aggregate outputs to determine classification.

3. The method of claim 2, wherein the first phase comprises: applying a Softmax function to weights associated with logic operators in the at least one LogicLayer; and updating the weights using gradient descent.

4. The method of claim 2 or 3, wherein the ThresholdLayer comprises trainable bias and scale parameters for each neuron.

5. The method of any of claims 2 to 4, further comprising concatenating outputs of the ThresholdLayer with inputs to intermediate LogicLayers.

6. The method of any one of claims 1 to 4, wherein the second phase comprises: applying a Softmax function to weights associated with neuron connections; and updating the weights using gradient descent.

7. The method of any one of claims 1 to 6, further comprising applying a Straight-Through Estimator during training to sharpen function outputs while enabling gradient flow.

8. The method of any one of claims 1 to 7, wherein training the DLN comprises searching over a subset of possible logic operators and neuron connections.

9. The method of claim 8, wherein the subset of possible logic operators and neuron connections comprises eight logic operators and eight possible input connections for each neuron.

10. The method of any one of claims 1 to 9, further comprising simplifying quantized discrete logic rules of the DLN using symbolic mathematics techniques.

11. A non-transitory computer readable storage medium, containing instructions that, when executed by one or more processing units, causes the one or more processing units to perform the method of any one of claims 1 to 10.

12. A non-transitor ' computer readable storage medium, containing instructions that, when executed by one or more processing units, causes the one or more processing units to make one or more inferences using an interpretable differentiable logic network training using a method of any one of claims 1 to 10.

13. A system for performing classification tasks, the system configured to implement the method of any one of claims 1 to 10, the system comprising: a differentiable logic network (DLN) comprising: a ThresholdLayer configured to convert continuous inputs to binary values; at least one LogicLayer configured to perform binary' logic operations; and a SumLayer configured to aggregate outputs to determine classification.

14. The system of claim 13, wherein each neuron in the at least one LogicLayer is configured to receive two binary' inputs and output a binary' value based on a learned logic operation.

15. The system of claim 13 or 14, wherein the SumLayer is configured to: compute a weighted sum of inputs from the at least one LogicLayer; anddetermine a classification based on the weighted sum.

16. The system of any one of claims 13 to 15, wherein the ThresholdLayer comprises: a plurality of neurons, each neuron associated with a continuous input feature and configured to output a binary' value based on a learned threshold.

17. The system of any one of claims 13 to 16. wherein the DLN is configured to output interpretable logic rules for classification decisions.

Images