Clause Logic Switching Reduction for Convolutional Tsetlin Machine Systems

NO20240236A1Active Publication Date: 2025-09-15UNIVERSITETET I AGDER
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
NO20240236
Authority / Receiving Office
NO · NO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2024-03-12
Publication Date
2025-09-15
Estimated Expiration
2044-03-12
Patent Text Reader

Abstract

Abstract and drawing info chevron_right An electronic circuit for a convolutional Tsetlin machine (CTM) comprising a logic element and a first feedback element. The logic element comprises an input stage and an output stage, wherein a logical expression implemented by the logic element is evaluated for a current patch during a convolution process. An output signal of the output stage is connected to the input signal of the first feedback element, and an output signal of the first feedback element is connected to the input stage.
Need to check novelty before this filing date? Find Prior Art

Description

[0002] CLAUSE LOGIC SWITCHING REDUCTION FOR CONVOLUTIONAL TSETLIN MACHINE []SYSTEMS

[0003]

[0004] FIELD OF INVENTION

[0005]

[0006]

[0001] Aspects of the present disclosure relate to clause logic switching reduction. Specifically, but not exclusively, aspects of the present disclosure are directed to clause logic switching reduction for machine learning systems. Specially, but not exclusively, aspects of the present disclosure are directed to clause logic switching reduction for convolutional Tsetlin machine systems.

[0007]

[0008] BACKGROUND

[0009]

[0010]

[0002] Machine learning models, particularly those empowered by hardware such as integrated circuits, are invaluable across numerous fields due to their ability to analyse vast datasets, recognize patterns, and make predictions or decisions with unprecedented speed and accuracy. In healthcare, these models aid in medical diagnosis, drug discovery, and personalized treatment plans. In transportation, they enhance route planning, autonomous vehicle navigation, and traffic management. Moreover, in manufacturing, they optimize production processes and predict equipment failures, while in agriculture, they optimize crop yields and monitor soil conditions. Additionally, machine learning increasingly plays vital roles in cybersecurity, natural language processing, image recognition, and more, ultimately driving innovation and efficiency across a wide spectrum of industries.

[0011]

[0012]

[0003] Machine learning systems often face significant hindrances due to the high energy and computational resource requirements. The complexity of many machine learning models, particularly deep learning architectures like convolutional neural networks and recurrent neural networks, demands extensive computational power to train and deploy effectively. This requirement poses challenges in terms of energy consumption, as training these models on large datasets can consume significant amounts of electricity, contributing to environmental concerns and energy inefficiency. Additionally, deploying these models in resource-constrained environments such as mobile devices or internet of things (loT) devices can be challenging due to limitations in computational power and battery life.Furthermore, the typical need for specialized hardware accelerators, such as graphics processing units (GPUs) or tensor processing units (TPUs), adds to the cost and complexity of implementing machine learning solutions. Consequently, optimizing machine learning systems for energy efficiency and developing hardware solutions tailored to the specific requirements of machine learning workloads are important steps in overcoming these hurdles and enabling widespread adoption of machine learning technologies.

[0013]

[0014]

[0004] In an attempt to overcome such issues, prior art discloses Convolutional Tsetlin Machine and / or Convolutional coalesced Tsetlin Machine (CTM) systems for e.g. image classification, such as: Granmo et al. “The Convolutional Tsetlin Machine”, arXiv 2019, which describes a CTM; Tunheim et al. “A Convolutional Tsetlin Machine-Based Field Programmable Gate Array Accelerator for Image Classification”, ISTM 2022, which describes a Field Programmable Gate Array (FPGA) implementation of an image classification accelerator based on the CTM; and Glimsdal and Granmo “Coalesced Multi-Output Tsetlin Machines with Clause Sharing”, arXiv 2021.

[0015]

[0016]

[0005] CTM systems can provide performance and energy benefits over traditional machine learning systems or machine learning system adapted to alter energy consumption as required, such as described in Youssef et al., “Energy Adaptive Convolution Neural Network Using Dynamic Partial Reconfiguration” MWSCAS 2020. However, prior machine learning systems for image classification or other complex tasks still face challenges for wider application in energy-restricted systems, such as battery powered devices or IoT devices, due to e.g. energy requirements associated with training and usage.

[0017]

[0018]

[0006] Tsetlin Machine (TM) systems are machine learning systems that utilise a TM, which benefits from natural logic underpinning and low-complexity. A TM foundation is based on propositional logic that leads to primarily Boolean operations. This operational concept of a TM is especially applicable to hardware, which makes it highly suitable for low-energy implementations on e.g. FPGAs and Integrated Circuits (ICs) that support online training and updating. The TM has shown competitive performance on several benchmarks in terms of accuracy, memory footprint and learning speed, and it has been tested on tabular data, images, regression, natural languages, and many other applications.

[0019]

[0007] In addition to hardware applicability and potential energy efficiency, a TM is highly interpretable, which makes it promising for rigorous applications, such as medical and environmental related applications. For example, among various applications in loT devices, image classification is central. Low-power hyperspectral sensors and cameras are usually deployed for attention detection, gesture recognition and event / occupancy detection for heating / ventilation / alarm systems. Prior art is mainly based on varieties of Convolutional Neural Networks (CNNs). For low-power applications, CNNs are gaining widespread adoption and there are numerous FPGA and ASIC implementations. Similarly, in the TM domain, a CTM is best suited for image classification, and has obtained a peak test accuracy of 99.4% on MNIST, 96.31% on Kuzushiji-MNIST and 91.5% on Fashion-MNIST.

[0020]

[0021]

[0008] Unlike the limited requirements on hardware of a TM / CTM due to its Boolean / logical operations, Deep Neural Networks (DNNs) often require complex and power-hungry digital arithmetic. Flowever, while power consumption in circuits is a known issue, with many potential ways of reducing the power consumption emerging such as TM-based ICs, there have been limited solutions directed to reducing the switching activity of an electronic circuit used within a machine learning system that do not impact either accuracy or scalability. Therefore, traditional TM and CTM implementation can still result in power and energy inefficiencies, especially when hardware implementations are scaled up for complex machine learning systems.

[0022]

[0023] SUMMARY OF INVENTION

[0024]

[0025]

[0009] According to an aspect of the present disclosure, there is provided: an electronic circuit for a convolutional Tsetlin machine or convolutional coalesced Tsetlin machine; an electronic circuit configured to be implemented within a convolutional Tsetlin machine or convolutional coalesced Tsetlin machine; and an electronic circuit configured to implement within a convolutional Tsetlin machine or convolutional coalesced Tsetlin machine.

[0026]

[0027]

[0010] According to an aspect of the present disclosure, the electronic circuit comprises logic element configured to implement a logical expression and a first feedback element. The logic element comprises an input stage and an output stage, wherein the logical expression is evaluated for a current patch of a convolution process. The output signal of the output stage is connected to the input signal of the first feedback element, and the output signal of the first feedback element is connected to the input stage.

[0028]

[0029]

[0011] According to an aspect of the present disclosure, the logical expression is evaluated for a current patch during the convolution process, wherein the logical expression is configured to operate as a filter used for performing convolutions.

[0030]

[0031]

[0012] According to a further aspect of the present disclosure, the electronic circuit further comprises a second feedback element configured such that the clause will always output 1 for the current patch if the clause has output 1 for a previous patch.

[0032]

[0033]

[0013] According to a further aspect of the present disclosure, there is provided a system comprising any electronic circuit described herein.

[0034]

[0035]

[0014] According to a further aspect of the present disclosure, there is provided a convolutional Tsetlin machine or a convolutional coalesced Tsetlin machine comprising any electronic circuit described herein.

[0036]

[0037]

[0015] According to a further aspect of the present disclosure, there is provided a method for using a convolutional Tsetlin machine or a convolutional coalesced Tsetlin machine comprising any electronic circuit described herein. The method comprises obtaining input data comprising a plurality of datapoints and applying an encoding technique to the input image. A number of patches is determined based on a predefined convolution window and a predetermined number of layers. The method further comprises calculating a number of features applied to the CTM per patch based on the plurality of datapoints, the encoding technique, the predefined convolution window, and the number of patches. A plurality of feature signals is thereby generated, as is a plurality of negated feature signals, wherein each negated feature signal comprises a negated version of each feature applied to the CTM.For each patch, one or more feature signals of the plurality of feature signals or one or more negated feature signals of the plurality of negated feature signals is input to the electronic circuit.

[0038]

[0039]

[0016] When the method described here is used for the electronic circuit, the input signal to the input stage comprises the one or more feature signals or the one or more negated feature signals; the output signal of the output stage is input to the first feedback element, resulting a switch-reduction signal; and the switch-reduction signal is connected to the input stage.

[0040]

[0041]

[0017] According to an additional aspect of the present disclosure, there is provided a computer-readable storage medium, which is transitory or non-transitory, comprising one or more program instructions which, when executed by one or more processors, cause the one or more processors to perform any of the methods or processes disclosed herein.

[0042]

[0043]

[0018] Beneficially, the electronic circuits described herein, and systems or method utilising these electronic circuits, reduces the switching activity of the clause logic during inference, thereby reducing the power consumption compared to both typical machine learning systems and a standard CTM solution. Further beneficially, the electronic circuits described herein, and systems or method utilising this circuit, are highly scalable, with power and energy efficiency further enhanced for complex machine learning systems.

[0044]

[0045]

[0019] The skilled person will understand that the above described electronic circuits, system, and method are not limited to CTM systems and may be applied to alternative machine learning systems. For example, the circuit is alternatively applied to integrated circuits (ICs), application-specific integrated circuits (ASICs), graphics processing units (GPUs), or field-programmable gate arrays (FPGAs) directed to operating e.g. artificial neural networks such as convolutional neural networks, transformer-based models such as bidirectional encoder representations from transformers, random forests and decision trees, support vector machines, or other algorithms not listed here. The electronic circuits described herein, especially but not exclusively the feedback elements described herein, and systems or method utilising these electronic circuits, are applicable to any hardware implemented machine learning system.

[0046]

[0047] BRIEF DESCRIPTION OF DRAWINGS

[0048]

[0049]

[0020] Embodiments of the invention will now be described, by way of example only, and with reference to the accompanying drawings, in which:

[0050]

[0051]

[0021] Figure 1 illustrates a system architecture diagram for an electronic circuit for implementing a CTM;

[0052]

[0022] Figure 2 illustrates a hardware implementation of an electronic circuit for implementing a CTM;

[0053]

[0054]

[0023] Figure 3 illustrates a system architecture diagram for an electronic circuit for training a CTM;

[0055]

[0056]

[0024] Figure 4 illustrates a hardware implementation of an electronic circuit for training a CTM;

[0057]

[0058]

[0025] Figures 5A-5B illustrate system architecture diagrams for a combined electronic circuit;

[0059]

[0060]

[0026] Figures 6A and 6B illustrate hardware implementations of a combined electronic circuit;

[0061]

[0062]

[0027] Figure 7 illustrates a flowchart of a method for using a CTM;

[0063]

[0064]

[0028] Figure 8 illustrates a basic state diagram for a typical Tsetlin Automaton (TA);

[0065]

[0066]

[0029] Figures 9A - 9C illustrate basic architecture diagrams for a typical TM; and

[0067]

[0030] Figure 10 shows an example computing environment for performing any of the methods described herein.

[0068]

[0069] DETAILED DESCRIPTION

[0070]

[0071]

[0031] Embodiments of the present disclosure will now be described with reference to the attached figures. It is to be noted that the following description is merely used for enabling the skilled person to understand the present disclosure, without any intention to limit the applicability of the present disclosure to other embodiments which could be readily understood and / or envisaged by the reader. In particular, whilst the present disclosure is primarily directed to clause logic switching reduction for CTM systems, the skilled person will appreciate that the circuits, methods and systems described herein are applicable to hardware enabled machine learning systems used in a variety of applications. For example, CTM is used to refer to either a Convolutional Tsetlin Machine or Convolutional coalesced Tsetlin Machine. Flowever, the embodiments herein may also apply to other hardware enabled machine learning algorithms.

[0072]

[0032] Beneficially, CTMs offer multiple improvements over traditional TMs, including: a reduced number of clauses without impacting accuracy; faster training time; and improved performance on imbalanced datasets. Incorporating coalescing operations, as with convolutional coalesced Tsetlin machines, comprises incorporating a process where outputs from multiple convolutional kernels are combined into a single feature map. The coalescing operation helps further reduce the dimensionality of the output space, making the model more efficient in terms of computation and memory requirements. Therefore, the convolutional coalesced Tsetlin machine may be regarded as a convolutional Tsetlin machine, able to capture spatial features and perform convolutional processes, but with further improved scalability and efficiency.

[0073]

[0074]

[0033] Any of the methods, systems or circuits described are configured for implementation in a CTM. A CTM algorithm is especially suited for pattern recognition in images or in time series data, where a benefit of convolutions is a significant reduction in the number of features that a TM algorithm needs to process simultaneously. As mentioned above, TMs / CTMs result in hardware accelerators offering promising solutions for energy-frugal systems.

[0075]

[0076]

[0034] Figures 1-6B below describe electronic circuits or clauses implemented by electronic circuits configured to operate as filters within a CTM. Any of the concepts of Figures 1-6B can be combined within one or more CTMs and, preferably, a CTM comprises a plurality of circuits and / or clauses described herein. Figure 7 is an example method of using a CTM implementing any of the electronic circuits or clauses described in Figures 1-6B. Figures 8 and 9A - 9C represent basic operations of a TM, which are implemented using any of the circuits or clauses described in Figures 1-6B, or according to the method illustrated in Figure 7.

[0077]

[0078]

[0035] Figure 1 illustrates a system architecture diagram for an electronic circuit. Specifically, Figure 1 shows a block diagram of an embodiment of a switching reduction electronic circuit 100 according to example embodiments of the present disclosure. Specifically, but not exclusively, Figure 1 shows a block diagram of an embodiment of a switching reduction electronic circuit 100 configured to operate within, or otherwise implement, a CTM or alternative machine learning system.

[0079]

[0080]

[0036] Switching reduction electronic circuit 100 comprises: input stage 102; output stage 104; output signal 106; first feedback element 108; first feedback signal 110; input signal 112; and intermediate signal 114. Optionally, input stage 102 and output stage 104 form a logic element configured to implement one or more logical expressions, e.g. the logic element implements a clause of the CTM as described below.

[0081]

[0082]

[0037] Use of switching reduction electronic circuit 100 results in greatly increased energy efficiency without effecting applicability or usability. In particular, a feedback element for reduced switching activity, e.g. first feedback element 108, is beneficial to both accuracy and energy efficiency. When such a feedback element, such as a D-type flip-flop, latch, or alternative storage element, is applied to a CTM, the machine learning system is capable of traditionally resource-heavy tasks such as image recognition and classification using a fraction of the power required by e.g. traditional GPU / software emulated CNN systems.

[0083]

[0084]

[0038] As an example, a CTM learns to classify images based on a training set of images, each with dimensions X x Y and comprising of Z channels, such as an RGB channel, e.g. each channel has a thermometer encoded pixel value of U bits, or a Boolean layer, e.g. a logical layer where Boolean operations are applied to inputs to produce desired outputs, such as switching reduction electronic circuit 100, switching reduction circuit 200, feedback electronic circuit 300, feedback circuit 400, combined electronic circuit 500, and / or combined circuit 600. Any appropriate encoding technique is optionally applied to the original pixels of the sample images, e.g. a Boolean encoding technique where each pixel is presented to the CTM by Boolean values (0 or 1).

[0085]

[0086]

[0039] In a CTM, a convolution window of size Wx x WY is applied, with Z channels, to generate different patches of Wx x WY x Z x U features. In addition, a clause is location-aware, with Boolean encoded (e.g. thermometer or one-hot) patch coordinates appended to the features for binary implementations. For example, thermometer or one-hot encoding is used, requiring (Y - WY ) (X - Wx ) bits.

[0087]

[0088]

[0040] The number of evaluations of the convolution window across the image is given by Eq. 1, where Bx and BY define the patches and are shown in Eq. 2 and Eq. 3 respectively, and where dx and dY are the stride values (step sizes) of the convolution window in the X and Y directions, respectively.

[0089]

[0090]

[0091] The number of features, NF, applied to the CTM per patch is given by:

[0092]

[0093]

[0094]

[0095] where (Y - WY ) (X - Wx ) represents the number of bits that encode the patch’s position.

[0096]

[0097]

[0041] In addition, the CTM also takes as input the negated versions of the features. The features and their negations are denoted as the literals e.g. literal signals 210 of Figure 2. The clauses of a CTM can be viewed as filters, where each clause is composed by 2 x NF literals. Beneficially, convolution processes performed by CTMs, i.e. both convolutional and convolutional coalesced Tsetlin machines, reduce the number of literals that have to be processed. The total model size and complexity is therefore reduced and, while the throughout (classifications per second) can be lower, the resources required significantly reduced compared to a non-convolutional solution.

[0098]

[0099]

[0042] During inference, a CTM follows the classical TM operation for pattern recognition, as detailed below, e.g. in relation to Figures 8 - 9C, which can also be implemented using the circuits disclosed herein. Flowever, for the CTM, each clause will output B values per image, i.e., one value per patch. A clause in a CTM will output 1 if it recognizes a pattern at least once in any of the B patches for a given image, i.e.,

[0100]

[0101]

[0102]

[0103] where cj is the jth clause of the CTM, and cf is the bth output of this clause obtained during the window sliding, i.e. the convolution process.

[0104]

[0105]

[0043] While learning, e.g. during training mode, a classical TM procedure is applied. Preferably, for a given clause, cj , the CTM randomly selects a single patch among those that made this clause evaluate to 1. The clause is then updated according to this patch with the classical TM feedback types of Type la, Type Ib or Type II, which are detailed further below, e.g. in relation to Figures 8 - 9C. If no patch makes cj evaluate to 1 , Type lb feedback will be applied, which is not dependent on the literal. This increases the efficiency and effectiveness of training.

[0106]

[0044] For a multi-class CTM, the Target Class during training is determined by the label, y = 1, of the current training example. A different class, i.e., the Negative Target Class, is randomly selected, which is labelled as y = 0. For two-class systems, the Negative Target Class will always be the class other than the intended / target training example.

[0107]

[0108]

[0045] TM and CTM hardware accelerators and emulators are typically implemented by either FPGAs or ASICs, e.g. using switching reduction electronic circuit 100. In most cases, the dominating part of the energy consumption is related to the switching of digital circuits. The dynamic power, P, consumed by a digital circuit with load capacitance C, operating frequency / , supply voltage Vs, and an activity factor a (transitions per clock cycle) is given by Eq. 6,

[0109]

[0110]

[0111]

[0112]

[0046] A classical approach for implementing clause logic does not incorporate a first feedback element, e.g. first feedback element 108, or a second feedback element, e.g. second feedback element 302, such as a D flip-flop, as described herein. The input to the CTM comprises o propositional variables, preferably Boolean variables, (described in more detail below e.g. in relation to Figures 8 - 9C). In addition, when the CTM comprises o propositional Boolean variables, the negated versions of the o propositional Boolean variables are generated, and a new vector L named the literals comprising 2 o Boolean variables is generated. Flere, each literal l1,· · ·, l2o is either included or excluded by the associated include signal i1,· · ·, i2o (active low) by using input stages, e.g. input stages 102, such as OR-gates. The include signals can be simultaneously available from a register. The outputs from the input stages, e.g. intermediate signals 1 14, are then fed to an output stage, e.g.output stage 104, such as a wide-input AND-gate. Optionally, the wide-input AND-gate is implemented by several smaller AND-gates connected in a tree-structure to reduce path delay.

[0113]

[0114]

[0047] The circuit of Figure 1 (and methods, systems, clauses and / or circuits described in Figures 2-7 below) reduces the α value of e.g. the logical gates implementing the clause. Power saving of implementing any of the embodiments compared to a standard CTM solution is further enhanced as the average number of patches processed per image before a clause goes high is reduced, e.g. typically half of the patches depending on the input data. In an example where half the patches are processed per image, reduction in switching activity of the clause logic is approximately 50%.

[0115]

[0116]

[0048] As a further example to illustrate the number of logical gates affected, the MNIST dataset is used as an example implementation case. For this example, each clause processes 136 features (100 from the patch window and 2 times 18 for the position), and thereby 272 literals. If implemented purely with 2-input OR-gates and 2-input AND-gates in a tree structure, 546 logic gates are required (272+136+72+36+18+9+3=546 logical gates). Using 512 clauses, the total clause logic consists of about 279.6 k gates. The number of clauses is a predefined parameter associated with the expression power of the CTM. By reducing the switching activity of this part of the accelerator with 50%, a significant overall reduction in the total power consumption of the CTM accelerator is achieved. As the reduction of switching will apply to clauses that evaluate to 1, the absolute power savings are greater for a configuration and a dataset where many clauses are activated per test example.Therefore, even greater energy savings can be made when implementing a trained CTM into a device such as a mobile or IoT device.

[0117]

[0118]

[0049] According to one embodiment, switching reduction electronic circuit 100 comprises a logic element configured to implement a logical expression, e.g. a clause, where the logical expression is evaluated for a current patch during a convolution process. Additionally, or alternatively, switching reduction electronic circuit 100 comprises a logic element configured to implement a logical expression, e.g. a clause, configured to operate as a filter for a CTM, e.g. implemented using hardware such as integrated circuits. The logic element comprises input stage 102 and output stage 104, wherein the output signal of the output stage, output signal 106, is input to first feedback element 108. The output signal of the first feedback element 108, first feedback signal 110, is then fed back to the input stage 102 alongside input signal 112.

[0119]

[0120]

[0050] The output from the input stage 102, intermediate signal 114, is the input to the output stage 104. Optionally, the switching reduction electronic circuit 100 comprises one or more intermediate stages used to manipulate or otherwise transform intermediate signal 114.

[0121]

[0122]

[0051] Beneficially, connecting first feedback signal 110 to input stage 102 avoids switching of the clause logic after the clause has evaluated to 1, reducing the power consumption of the switching reduction electronic circuit 100. Further beneficially, if the switching reduction electronic circuit 100 is being operated as a filter of a hardwareimplemented CTM, first feedback signal 110 is connected to input stage 102 such that the clause logic is kept high throughout operation as a filter of the CTM. Optionally, first feedback signal 110 is connected to the input stage 102 such that the first feedback element 108 is reset when a new signal is input to the input stage, e.g. input signal 116 (not depicted).

[0123]

[0124]

[0052] Optionally, the first feedback element 108 is a D flip-flop and inputs to the D flip-flop comprise a clock input and the output signal from the output stage. Alternatively, the first feedback element 108 is a latch and inputs to the latch comprise a reset signal and the output signal from the output stage, wherein the reset signal is a synchronous or asynchronous signal and the output signal from the output stage is an asynchronous signal, or the first feedback element is an alternative form of storage element. Whether the first feedback element 108 is synchronous or asynchronous, the output signal of the first feedback element is connected to the input stage 102 such that the first feedback element 108 is reset after a convolution process. For example, the reset signal that is applied after each convolution process.

[0125]

[0126]

[0053] Further optionally, the input stage 102 comprises one or more input logic gates, where an input signal for the one or more input logic gates comprises the output signal of the first feedback element 108 and input signal 112. Input signal 112 comprises a feature signal or a negated feature signal, and an include signal. Alternatively, or additionally, the output stage 104 comprises an output logic gate, where the output signal for the output logic gate is connected to the input signal of the first feedback element 108. In one example, an output signal for the one or more input logic gates is connected to the output stage 104, and an input signal for the output logic gate is connected to an output signal of the input stage 102.

[0127]

[0128]

[0054] Preferably, the one or more input logic gates perform OR logic and / or the output logic gate performs AND logic, as shown in relation to Figure 2 below.

[0129]

[0130]

[0055] Figure 2 illustrates a hardware implementation of an electronic circuit. Specifically, Figure 2 shows a hardware implementation diagram of switching reduction circuit 200 applied to a CTM system according to example embodiments of the present disclosure.

[0131]

[0056] A circuit utilising switching reduction circuit 200 has the potential for significant reduction in switching activity of the clause logic and thereby also reduction in dynamic power. The absolute level of reduction in power consumption will depend on the dataset, the typical included number of literals, and the number of clauses, e.g. the number of switching reduction clauses 200, employed in the CTM. According to one embodiment of the present disclosure, switching reduction circuit 200 requires a D flipflop per clause, and an input to each of a plurality of OR gates in the first stage of the clause logic (the exact implementation and connection will depend on the logic synthesis tool). However, this small amount of additional hardware is justified by the resulting power savings. The embodiment can be used during inference for all TMs utilizing convolution, including CTMs such as convolutional TMs and convolutional coalesced TMs.

[0132]

[0133]

[0057] Switching reduction circuit 200 comprises: signal A; clause output signal 202; D-input 204; D flip-flop 206; input stage 208; literal signals 210; include signals 212; intermediate signals 214; and output stage 216. Optionally, input stage 208 comprise a plurality of logic gates configured to perform OR logic, e.g. OR gates. Further optionally, the input signal for each OR gate comprises signal A, a literal signal 210, and include signal 212. For example, switching reduction circuit 200 comprises first input OR gate 208-1 and final input OR gate 208-N, where N is the number of logic gates within the input stage 208, with input signals comprising signal A, a first literal signal 210-1, a Nth literal signal 210-N, a first include signal 212-1, and a Nth include signal 212-N. Optionally, output stage 216 comprises one or more logic gates configured to perform AND logic, e.g. an AND gate.For example, the input signal to output stage 216 comprises intermediate signals 214, which are the output signals of input stage 208.

[0134]

[0135]

[0058] As noted above, literal signal 210 comprises a feature signal or a negated feature signal (a negated version of each feature applied to a CTM). For example, the number of features and negated features sums to N, the number of logic gates within the input stage 208.

[0136]

[0137]

[0059] According to one embodiment, the clause output signal 202 of switching reduction circuit 200 is connected to the D-input 204 of a D flip-flop 206. The output of the D flip-flop 206, signal A, is connected to the input stages 208 of the clause logic of switching reduction circuit 200 in a way ensures e.g.:

[0138]

[0139] i. Further switching of the clause logic is avoided after switching reduction circuit 200 has evaluated to 1, and / or

[0140]

[0141] ii. Switching reduction circuit 200 is kept high throughout the remaining part of the convolution.

[0142]

[0143] When a new input, e.g. an image sample, is processed by switching reduction circuit 200, the D flip-flop 206 is reset. Optionally, one D flip-flop 206 is used per clause, e.g. each switching reduction circuit 200 has one D flip-flop 206. For example, a CTM comprises one or more circuits comprising a plurality of switching reduction clauses 200. Alternatively, multiple switching reduction clauses result in a combined clause output signal that is input to a single D flip-flop.

[0144]

[0145]

[0060] According to an alternative embodiment, D flip-flop 206 is optional, e.g. D flipflop 206 is replaced by a synchronous alternative such as a T (toggle) flip-flop, which toggles output based on the clock signal CLK, or an asynchronous alternative, such as a Set-Reset (SR) latch, with optional additional logic to prevent a metastable state. For example, D flip-flop 206 is optionally replaced with a storage element with an output A that feeds back to input stage 208 as depicted in Figure 2.

[0146]

[0147]

[0061] For systems with a relatively large number of clauses, the clause logic will occupy a significant percentage of the digital circuitry, and reducing its switching activity enables significant energy savings. In a typical implementation, e.g. as in Figure 2, the load capacitance of the first stage of the clause logic has to be driven (by the clause and include / exclude outputs). However, if the clause logic is implemented / synthesized with a tree structure, the clause logic itself will not switch after the output of the D flip-flop goes high.

[0148]

[0149]

[0062] Figure 3 illustrates a system architecture diagram for an electronic circuit. Specifically, Figure 3 shows a block diagram of an embodiment of a feedback electronic circuit 300 according to example embodiments of the present disclosure.

[0150]

[0151]

[0063] Feedback electronic circuit 300 comprises: input stage 102; output stage 104; output signal 106; input signal 112; intermediate signal 114; second feedback element 302; second feedback signal 304; and clause output 306.

[0152]

[0064] Feedback electronic circuit 300 is configured such that second feedback signal 304 will always output 1 for a current patch if the clause output was 1 for a previous patch, as outlined in Eq. 5 above regarding use of CTMs. In other words, by feeding back the output of the second feedback element 302, second feedback signal 304 is used by second feedback element 302 to keep clause output 306 at 1 even if a new output signal 308 (not depicted) from output stage, e.g. from new input signal 116 (not depicted), is not 1. This not only allows for a CTM to output 1 if it recognizes a pattern at least once in any of the patches for a given image, but also further reduces unnecessary switching, resulting in improved energy efficiency.

[0153]

[0154]

[0065] For example, the second feedback element comprises a training logic gate, a first sequence logic gate and a second sequence logic gate. According to this configuration, the input signal of the first sequence logic gate is connected to the output signal of the training logic gate and the output signal of the output stage, and the input signal of the second sequence logic gate is connected to the output signal of the first logic sequence gate and the input signal of the first feedback element.

[0155]

[0156]

[0066] Preferably, as shown in relation to Figure 4 below, the first sequence logic gate and the second sequence logic gate perform OR logic and, optionally, the input signal to the training logic gate comprises a training signal and an empty signal, where the training signal is high during a training mode of the CTM, and low during inference mode of a CTM. The empty signal for the clause is high if no literals are included for the given clause. Optionally, the empty signal is inversely connected to an input signal for the output stage and, additionally or alternatively, the training logic gate performs AND logic.

[0157]

[0158]

[0067] Figure 4 illustrates a hardware implementation of an electronic circuit. Specifically, Figure 4 shows a hardware implementation diagram of feedback circuit 400 according to example embodiments of the present disclosure.

[0159]

[0160]

[0068] Feedback circuit 400 comprises: D flip-flop 402, include signals 404, signal EMPTY, signal TRAIN, combined signal 406, effective clause output 408, OR gates 410 (e.g. first sequence logic gate and / or second sequence logic gate of Figure 3), input stages 412, literal signals 414, intermediate signals 416, output stage 418, learning gate 420 (e.g. training logic gate of Figure 3), and learning signal 422.

[0161]

[0069] Feedback circuit 400 uses OR gates 410 and a D flip-flop 402 as a feedback element to implement Eq.5, above. The signal EMPTY is high if all include signals 404 are low, and the signal TRAIN is high during training mode. The combined signal 406 is the combinational output for the current patch, and the effective clause output 408 is obtained during the convolution.

[0162]

[0163]

[0070] According to one embodiment, input stages 412 are OR gates, e.g. gates configured to perform OR logic, with an input signal comprising include signals 404 and literal signals 414. The output signal from each input stage 412, intermediate signals 416, form the input signal to the output stage 418, which is an AND gate, e.g. one or more gates configured to perform AND logic to combine a plurality of intermediate signals 416. The input signal for the output stage 418 further comprises a negative signal EMPTY. The output signal from the output stage 418 forms an input signal to the OR gate 410-1 alongside the output signal from the learning gate 420, e.g. learning signal 422, where learning gate 420 is an AND gate configured to combine the signal EMPTY and the signal TRAIN. The output signal from the OR gate 410-1 is the combined signal 406, which forms the input signal to the OR gate 410-2.The output signal from the OR gate 410-2 is input to D flip-flop 402 to output the effective clause output 408, which is fed back to OR gate 410-2.

[0164]

[0165]

[0071] According to an alternative embodiment, D flip-flop 402 is optional, e.g. D flipflop 402 is replaced by another synchronous alternative, such as a T flip-flop, or an asynchronous alternative, such as a Set-Reset (SR) latch. For example, D flip-flop 402 is optionally replaced with a storage element with an output (effective clause output 408) that feeds back to OR gate 410-2 as depicted in Figure 4.

[0166]

[0167]

[0072] Beneficially, inclusion of a D flip-flop 402 or alternative implementations alongside the OR gates 410 not only reduces switching when applied to a single feedback circuit 400, but also further reduces switching when the number of clauses or output stages 418 is scaled up e.g. for parallel CTM implementations, and / or for CTMs comprising many inputs and consequently multiple input stages 412.

[0168]

[0169]

[0073] By feeding back the output of the OR gates 410, unwanted switching in the combinational clause logic of inputs stages 412 and output stage 418 is avoided, as shown in relation to Figures 5A-6B, below. Optionally, clock gating is applied to feedback circuit 400, where disabling clocking of D flip-flop 402 reduces power requirements of a circuit comprising feedback circuit 400. For example, clock gating provides additional power saving benefits in an embodiment where a CTM comprises multiple D flip-flops 402, e.g. where many D flip-flops 402 can be turned off simultaneously.

[0170]

[0171]

[0074] Figures 5A and 5B illustrate system architecture diagrams for an electronic circuit. Specifically, Figures 5A and 5B show block diagrams of a combined electronic circuit 500 according to example embodiments of the present disclosure.

[0172]

[0173]

[0075] Combined electronic circuit 500 comprises: input stage 102; output stage 104; output signal 106; first feedback element 108; first feedback signal 110; input signal 112; intermediate signal 114; second feedback element 302; second feedback signal 304; and clause output 306.

[0174]

[0175]

[0076] Combined electronic circuit 500 is configured to operate as a filter for a CTM such that the CTM will output 1 if it recognizes a pattern at least once in any of the patches for a given image. As described above in relation to feedback electronic circuit 300, by feeding back the output of the second feedback element 302, second feedback signal 304 is used by second feedback element 302 to keep clause output 306 at 1 (a previous patch was recognised) even if a patch for a given image is not recognised.

[0176]

[0177]

[0077] Both first feedback element 108 and second feedback element 302 reduce unnecessary switching, and hence combined electronic circuit 500 represents a highly energy efficient circuit for e.g. implementing convolutions of a CTM. Optionally, first feedback element 108 and second feedback element 302 are implemented using the same hardware, reducing power requirements and time delays in combined electronic circuit 500. For example, second feedback element 302 is first feedback element 108, as shown in Figure 5A. Alternatively, second feedback element 302 and first feedback element 108 are implemented using at least partially different hardware components, as shown in Figure 5B. Specific implementations are shown in relation to Figures 6A and 6B, below.

[0178]

[0179]

[0078] Figures 6A and 6B illustrate hardware implementations of an electronic circuit. Specifically, Figures 6A and 6B show alternative hardware implementation diagrams of combined circuit 600 according to example embodiments of the present disclosure.

[0180]

[0079] Combined circuit 600 comprises: signal A; clause output signal 202 / effective clause output 408; D-input 204; D flip-flop 206; D flip-flop 402; input stage 208; literal signals 210; include signals 212; intermediate signals 214; output stage 216; signal EMPTY; signal TRAIN; combined signal 406; OR gates 410; learning gate 420; and learning signal 422.

[0181]

[0182]

[0080] Input stage 208 comprise a plurality of logic gates configured to perform OR logic, e.g. OR gates, and the input signal for each OR gate comprises signal A, a literal signal 210, and include signal 212. For example, combined circuit 600 comprises first input OR gate 208-1 and final input OR gate 208-N, where N is the number of logic gates within the input stage 208, with input signals comprising signal A, a first literal signal 210-1, a Nth literal signal 210-N, a first include signal 212-1, and a Nth include signal 212-N. Optionally, output stage 216 comprises one or more logic gates configured to perform AND logic, e.g. an AND gate. For example, the input signal to output stage 216 comprises intermediate signals 214, which are the output signals of input stage 208.

[0183]

[0184]

[0081] Combined circuit 600 are example implementations of the OR gates 410 of feedback circuit 400 applied to the switching reduction circuit 200 of Figure 2. According to a first implementation, shown in Figure 6A, the output of D flip-flop 402, signal A, is fed back into input stages 210, such that D flip-flop 402 is D flip-flop 206. According to a second implementation, shown in Figure 6B, the output of D flip-flop 402, e.g. D-input 204, is an input to D flip-flop 206, such that D flip-flop 402 and D flipflop 206 operate independently.

[0185]

[0186]

[0082] Beneficially, the first implementation, shown in Figure 6A, utilises the benefits of both switching reduction circuit 200 and feedback circuit 400 without increasing propagation / time delays, as each clause only has one D flip-flop. This results in both a time and an energy efficient clause that can be scaled up for highly parallel CTM implementations with multiple inputs, greatly reducing the power consumption required by such a CTM.

[0187]

[0188]

[0083] Beneficially, the second implementation, shown in Figure 6B, results in a shift register. In this shift register, the output of D flip-flop 402 is connected to the input of the next flip-flop, D flip-flop 206, creating a chain of storage elements. The output of the last flip-flop in the series, signal A, is optionally fed back to the input of D flip-flop 402 to create a circular or looped shift register, allowing for continuous shifting of data, which can be used to tailor the stability of the CTM, e.g. during inference and / or training.

[0189]

[0190]

[0084] Optionally, the second implementation of Figure 6B is performed using a single D flip-flop, where the output signal of D flip-flop 402 is fed back to D flip-flop 402 and fed back to input stages 210. The effective looping of signal A results in a feedback loop or latch configuration, where D flip-flop 402 acts as a memory element. For example, when the output of the D flip-flop 402 is connected back to its input, any change in the input will propagate through the D flip-flop 402 and appear at the output after a certain delay, typically determined by the propagation delay of D flip-flop 402. This output is then fed back to the input, causing the flip-flop to continuously hold, i.e. latch, its state. Effectively, this creates a storage element where the output of the D flip-flop 402 represents the stored value, and this stored value persists until a new value is applied to the input, which is beneficial for storing and manipulating binary information of the CTM clause.

[0191]

[0192]

[0085] In summary, looping the output of a D flip-flop to its input creates a memory element, allowing the D flip-flop to store and retain its state until a new input is provided.

[0193]

[0194]

[0086] Figure 7 illustrates a flowchart of a method for using a CTM implemented using the circuits or clauses described herein. Specifically, method 700 is a method performed by components of switching reduction electronic circuit 100 of Figure 1 and / or switching reduction circuit 200 of Figure 2.

[0195]

[0196]

[0087] Method 700 comprises: step 702, obtain an input data; step 704, apply an encoding technique; step 706, determine a number of patches; step 708, calculate a number of features and generate feature signals; step 710, generate negated feature signals; step 712, input feature signals and / or negated feature signals; and optional step 714, determine if there are more patches.

[0197]

[0198]

[0088] Step 702 comprises obtaining input data comprising a plurality of datapoints. The plurality of datapoints is based on the input data. When the input data comprises an input image, the plurality of datapoints comprises a plurality of pixels. When the input data comprises time-series data, the plurality of datapoints comprises a plurality of values and / or a plurality of time-points. Examples of time-series data comprise, but are not limited to, audio signals, medical data, or health data. For example, the electronic circuit is within a wearable device, and the time-series data comprises heartrate or temperature information associated with the wearer of the wearable device.

[0199]

[0200]

[0089] Optionally, an input image is any of: an RGB image, e.g. comprising three colour channels with pixel values ranging from 0 to 255 for each channel; a grayscale image, e.g. comprising one channel representing the intensity of light with pixel values ranging from 0 to 255; a binary image e.g. with pixel values of 0 or 1; a multispectral image or hyperspectral image, e.g. comprising one or more colour channels in addition or as an alternative to RGB channels such as infrared or ultraviolet; a depth image, e.g. pixel value ranges correspond to a measured depth; a thermal image, e.g. pixel value ranges correspond to a measured temperature; a medical image, e.g. X-rays, MRIs, CT scans, or ultrasound images; a satellite image; a simulated image; a synthetic image; or another type of image not listed here where colour, intensity, or other image data can be represented numerically as one or more pixels.

[0201]

[0202]

[0090] Step 704 comprises applying an encoding technique to the input image, such as a Boolean encoding technique. For example, the Boolean encoding technique is any of: one-hot encoding; thermometer encoding; label encoding; binary encoding; integer binary encoding; dummy coding; effect coding; or another suitable technique for representing categorical data or discrete variables as binary values. Alternatively, step 704 comprises applying alternative data processing techniques not limited to representing the input data of step 702 as binary values. While binary values are efficiently implemented by the electronic circuit, alternative value ranges and discrete variables are useable e.g. based on the configuration of the logic element of the electronic circuit.

[0203]

[0204]

[0091] Step 706 comprises determining a number of patches based on a predefined convolution window and a predetermined number of layers. Each patch is a section of the input data, and using patch inputs to the electronic circuit results in reduced complexity and increased efficiency. For example, the number of patches is determined by Eq.1, above.

[0205]

[0206]

[0092] Step 708 comprises calculating a number of features applied to the CTM per patch based on the plurality of datapoints, the encoding technique, the predefined convolution window, and the number of patches, thereby generating a plurality of feature signals. For example, the number of features is calculated based on Eq. 4, above, where the plurality of feature signals comprises each feature applied to the CTM. Optionally, before the plurality of feature signals is generated, step 708 further comprises processing the Boolean encoded input image of step 704 or the input image of step 702, such as performing any of: noise removal, image enhancement, normalisation, edge detection, or other standard image processing techniques.

[0207]

[0208]

[0093] Step 710 comprises generating a plurality of negated feature signals, wherein each negated feature signal comprises a negated version of each feature applied to the CTM. For example, complement coding or complementary representation techniques are used to generate one or more negated features. In one example where edge detection is used for generation of feature signals, e.g. if an edge detection algorithm identifies the presence of an edge in a particular location and orientation, a feature signal is be generated to indicate the presence of that edge, the negated feature signal would indicate the absence of that edge. This process effectively captures both positive and negative patterns in the input image.

[0209]

[0210]

[0094] Step 712 comprises inputting one or more feature signals of the plurality of feature signals and / or one or more negated feature signals of the plurality of negated feature signals to the electronic circuit of any of Figures 1 - 6. By incorporating both feature signals and their negations, the CTM is able to capture a broader range of patterns and relationships in the input data, leading to improved performance in tasks such as image classification or pattern recognition.

[0211]

[0212]

[0095] For example, the input signal to the input stage (e.g. input stage 102) comprises the one or more feature signals and the one or more negated feature signals; the output signal of the output stage (e.g. output stage 104) is input to the first feedback element (e.g. first feedback element 108), resulting in a switch-reduction signal; and the switchreduction signal is connected to the input stage. In a parallel CTM implementation, all the literals (i.e., all the features and their negated versions) are input simultaneously / in parallel. In alternative implementations, e.g. where a clause is partially evaluated, at least one feature signal or negated feature signal is input.

[0213]

[0214]

[0096] Step 714 is optional and comprises determining whether there are more patches. If there are more patches, step 712 is repeated.

[0215]

[0216]

[0097] For example, step 712 is repeated for each patch. In one example, step 712 is repeated using a different electronic circuit for each patch, such that step 712 is repeated in parallel rather than in series to increase efficiency. In another example, step 712 is repeated and, if the electronic circuit comprises the second feedback element (e.g. second feedback element 302 of Figures 3 and 5), the electronic circuit is configured to output 1 for a current patch if the clause output 1 for a previous patch.

[0217]

[0218]

[0098] Optionally, method 700 is performed using a combination of hardware and software. Alternatively, method 700 is performed using mostly or wholly hardware, such as using integrated circuits, FPGAs, optics, or other electromagnetic or chemical processes.

[0219]

[0220]

[0099] Figure 8 illustrates a basic state diagram for a typical Tsetlin Automaton (TA). Specifically, Figure 8 shows a state diagram of an embodiment of a TA 800 for twoaction environments according to example embodiments of the present disclosure. Figures 9A-9C illustrate basic architecture diagrams for a typical TM. Specifically, Figure 9A shows a block diagram of a TM 900-a, Figure 9AB shows a block diagram of an embodiment of a two-class TM 900-b, and Figure 9AC shows a block diagram of an embodiment of a q-class TM 900-c according to example embodiments of the present disclosure.

[0221]

[0222]

[0100] Figure 8 comprises: a TA 800, action 1 A1, action 2 A2, reward 802, penalty 804, and states 806.

[0223]

[0224]

[0101] Figures 9A - 9C comprise: TA automaton 900, inputs 902, literals 904, TA team 906, TA decisions 908, conjunction operator 910, clause 912, output sum 914, TMs 920, Argmax operator 922, and output 924.

[0225]

[0226]

[0102] The input to the TM is a feature vector X = [x1, x2,· · · , xo] consisting of o propositional Boolean variables, xu e (0, 1}° , u = 1 ,· · · ,ο, e.g. inputs 902. The negation of the variables are appended to the input forming a new input vector L with in total 2 o literals: [χ1, -ιχ1, χ2, -ιΧ2, · · · , χ0> -ιχο\ , e.g. literals 904.

[0227]

[0228]

[0103] A TM consists of several teams of Tsetlin Automata (TAs), e.g. several TA teams 906 comprising a plurality of TAs 800, that operates on the literals. Each team of TAs forms a discriminative conjunctive clause, e.g. clause 912, by including or excluding literals. There are m clauses, c,·, where j = 1,· · · , m, and m is an even userspecified integer. The odd numbered clauses are defined as positive, and the even numbered ones as negative. The basic building block of the TM is the TA of two-action type, e.g. TA 800. Such a module will either include or exclude a literal from a conjunctive clause. This decision is achieved during the learning process.

[0229]

[0230]

[0104] Figure 8 shows the structure of a single TA 800 with 2 N states 806. Action 2 ( include ) is employed if the TA 800 is in one of the states from N 1 to 2 N, while the states 1 to N result in Action 1 (exclude). The TA 800 is a finite-state-machine and in hardware it can typically be implemented as a binary up- / down-counter. The output of a single clause, cj, is given by:

[0231]

[0232]

[0233]

[0234] where lk is the literal with index k, and k belongs to Ij c (1, .... , 2 o}. Ij denotes the set of indexes of all the TAs that select action “include” in cj, as shown in Figure 9A.

[0235]

[0236]

[0105] In a basic two-class TM, classification is given by:

[0237]

[0238]

[0239]

[0240] where the output sum 914, v, is defined in Eq. 9, as shown in Figure 9B.

[0241]

[0242]

[0243]

[0244] A q-class TM, as shown in Figure 9C, is constituted by several TMs 920, one for each class, 1 to q. In this case the final decision is made by an argmax operator, e.g. Argmax operator 922, that classifies the input data according to the highest vote sum, wherein the output 924 is the classification.

[0245]

[0246]

[0106] In a special version of the TM, each clause can be associated with an integer weight, w. In this case the class sum, for a two-class TM, will be v =

[0247]

[0248] In a coalesced TM with q classes, there is only one clause pool and all

[0249] clauses operate on all outputs. In this case there is one set of integer weights per output, Wij, where denotes the output (class) and

[0250] denotes the clause number. The clause integer weights can have both positive and negative polarity, and their values are obtained during the TM training. The class sum, vi, is given by The argmax operator is applied to predict the output class.

[0251]

[0252]

[0107] A TM learns online, processing one training example ( X,y ) at a time. Learning takes place through a novel finite state learning automata game that coordinates the collective of TAs and leverages resource-allocation and frequent pattern mining principles. Feedback mechanisms are employed and each TA is given either a reward, a penalty or inaction. If a reward is applied, the TA will move deeper, i.e. towards state 1 or 2 N depending on the action. With penalty the TA will move towards the centre and will eventually switch to the other action.

[0253]

[0254]

[0108] During learning there are two types of feedback: Type I and Type II.

[0255]

[0256]

[0109] Table 1 below shows the update probabilities and conditions for Type I feedback.

[0257]

[0258]

[0259]

[0260] TABLE 1 : Type I feedback conditions and update probabilities.

[0261]

[0262]

[0110] Type I feedback is given stochastically to clauses with positive polarity when y = 1 and to clauses with negative polarity when y = 0. Each clause, in turn, updates each of its TAs based on: (1) its output the action of the TA controlling the literal - Include or Exclude ; and (3) the value of the literal lk assigned to the TA. Type I feedback is governed by two rules, as can be seen from Table 1 :

[0263]

[0264] · Include is rewarded and Exclude is penalized with probability

[0265] if Cj(X ) = 1 and lk = 1. This reinforcement is strong and makes the clause remember and refine the pattern it recognizes in X.

[0266] • Include is penalized and Exclude is rewarded with probability if Cj (X) = 0 or lk = 0. This reinforcement is weak and coarsens infrequent patterns, making them frequent.

[0267]

[0268] Greater values of the hyperparameter s will stimulate a TA team to include more literals in a clause.

[0269]

[0270]

[0111] Table 2 below shows the update probabilities and conditions for Type II feedback.

[0271]

[0272]

[0273]

[0274] TABLE 2: Type II feedback conditions and update probabilities.

[0275]

[0276]

[0112] Type II feedback is given stochastically to clauses with positive polarity when y = 0 and to clauses with negative polarity when y = 1. It penalizes Exclude actions with probability 1 if Cj (X) = land lk = 0. Thus, this feedback combats false positive outputs.

[0277]

[0278]

[0113] To ensure that clauses distribute themselves across different frequent patterns, the learning procedure includes a mechanism for resource allocation. For any input X, the probability of updating a clause gradually drops to zero as the TM output sum in Eq. 9 approaches a user-configured target / hyperparameter T. A higher T increases the robustness of learning by allocating more clauses to learn each sub-pattern.

[0279]

[0280]

[0114] Eq. 10 shows the update probability, for positive clauses of the Target Class and negative clauses for the Negative Target Class. For negative clauses of the Target Class and positive clauses for the Negative Target Class, the update probability, , is given by Eq. 11.

[0281]

[0282]

[0283] Optimum settings of m, T and s is dependent on the specific ML problem.

[0284]

[0285]

[0115] Any CTM, e.g. convolutional Tsetlin machine or a convolutional coalesced Tsetlin machine, described at least in part in relation to Figures 8 - 9C above optionally comprises any of the electronic circuits and / or clauses described above in relation to Figures 1 - 6. For example, the CTM is used for classifying an image associated with a number of patches, where the CTM comprises a number of electronic circuits based on the number of patches.

[0286]

[0287]

[0116] Figure 10 shows an example computing system. Specifically, Figure 10 shows a block diagram of an embodiment of a computing system 1000 according to example embodiments of the present disclosure.

[0288]

[0289]

[0117] Computing system 1000 can be configured to perform any of the operations disclosed herein such as, for example, any of the operations discussed with reference to method 700 of Figure 7. Computing system 1000 includes one or more computing devices 1002. Computing device 1002 of computing system 1000 comprises one or more processors 1004 and memory 1006. device(s) 1002 of computing system 1000 comprise one or more processors 1004 and memory 1006. One or more processors 1004 can be any general-purpose processor(s) configured to execute a set of instructions. For example, one or more processors 1004 can be one or more generalpurpose processors, one or more field programmable gate array (FPGA), and / or one or more application specific integrated circuits (ASIC). In one embodiment, one or more processors 1004 include one processor. Alternatively, one or more processors 1004 include a plurality of processors that are operatively connected.One or more processors 1004 are communicatively coupled to memory 1006 via address bus 1008, control bus 1010, and data bus 1012. Memory 1006 can be a random-access memory (RAM), a read-only memory (ROM), a persistent storage device such as a hard drive, an erasable programmable read-only memory (EPROM), and / or the like. Computing device(s) 1002 further comprise input / output (I / O) interface 1014 communicatively coupled to address bus 1008, control bus 1010, and data bus 1012.

[0290]

[0118] Memory 1006 can store information that can be accessed by one or more processors 1004. For example, memory 1006 (e g., one or more non-transitory computer-readable storage mediums, memory devices) can include computerreadable instructions (not shown) that can be executed by one or more processors 1004. Alternatively, or additionally, memory 1006 can be a combination of information storage and the option to process data in-memory without having to read out the data first, and let the processor 1004 perform operations on the memory data. For example, in memory computing is optionally used for CTM implementation, e.g. for “weighting” of clauses, where the weights from a trained model are stored in memory and the weighting is automatically performed when a clause evaluates to 1.

[0291]

[0292]

[0119] The computer-readable instructions can be software written in any suitable programming language or can be implemented in hardware. Additionally, or alternatively, the computer-readable instructions can be executed in logically and / or virtually separate threads on one or more processors 1004. For example, memory 1006 can store instructions (not shown) that when executed by one or more processors 1004 cause one or more processors 1004 to perform operations such as any of the operations and functions for which computing system 1000 is configured, as described herein. In addition, or alternatively, memory 1006 can store data (not shown) that can be obtained, received, accessed, written, manipulated, created, and / or stored. The data can include, for instance, the data and / or information described herein in relation to Figures 1 to 6.In some implementations, computing device(s) 1002 can obtain from and / or store data in one or more memory device(s) that are remote from the computing system 1000.

[0293]

[0294]

[0120] Computing system 1000 further comprises storage unit 1016, network interface 1018, input controller 1020, and output controller 1022. Storage unit 1016, network interface 1018, input controller 1020, and output controller 1022 are communicatively coupled to central control unit or computing devices 1002 via I / O interface 1014.

[0295]

[0296]

[0121] Storage unit 1016 is a computer readable medium, preferably a non-transitory computer readable medium, comprising one or more programs, the one or more programs comprising instructions which when executed by one or more processors 1004 cause computing system 1000 to perform the method steps of the present disclosure. Alternatively, storage unit 1016 is a transitory computer readable medium.

[0297] Storage unit 1016 can be a persistent storage device such as a hard drive, a cloud storage device, or any other appropriate storage device.

[0298]

[0299]

[0122] Network interface 1018 can be a Wi-Fi module, a network interface card, a Bluetooth module, and / or any other suitable wired or wireless communication device. In an embodiment, network interface 1018 is configured to connect to a network such as a local area network (LAN), or a wide area network (WAN), the Internet, or an intranet.

[0300]

[0301]

[0123] Figure 10 illustrates one example computer system 1000 that can be used to implement the present disclosure. Other computing systems can be used as well. Computing tasks discussed herein as being performed at and / or by one or more functional unit(s) (e.g., as described in relation to Figure 1) can instead be performed remote from the respective system, or vice versa. Such configurations can be implemented without deviating from the scope of the present disclosure. The use of computer-based systems allows for a great variety of possible configurations, combinations, and divisions of tasks and functionality between and among components. Computer-implemented operations can be performed on a single component or across multiple components. Computer-implemented tasks and / or operations can be performed sequentially or in parallel. Data and instructions can be stored in a single memory device or across multiple memory devices.

[0302]

[0303]

[0124] Regarding the above disclosure, references to items in the singular should be understood to include items in the plural, and vice versa, unless explicitly stated otherwise or clear from the context. Additionally, grammatical conjunctions are intended to express any and all disjunctive and conjunctive combinations of conjoined clauses, sentences, words, and the like, unless otherwise stated or clear from the context. Thus, the term “or” should generally be understood to mean “and / or” and so forth. The use of any and all examples, or exemplary language (“e.g.,” “such as,” “including,” or the like) provided herein, is intended merely to better illuminate the embodiments and does not pose a limitation on the scope of the embodiments or the claims.

[0304]

[0305]

[0125] Methods described herein may relate to a computer storage product with a nontransitory computer-readable medium (also can be referred to as a non-transitory processor-readable medium) having instructions or computer code thereon for performing various computer-implemented operations, such that the methods are performed. The computer-readable medium (or processor-readable medium) is nontransitory in the sense that it does not include transitory propagating signals per se (e.g., a propagating electromagnetic wave carrying information on a transmission medium such as space or a cable). The media and computer code (also can be referred to as code) may be those designed and constructed for the specific purpose or purposes.Examples of non-transitory computer-readable media include, but are not limited to, magnetic storage media such as hard disks, floppy disks, and magnetic tape, optical storage media such as Compact Disc / Digital Video Discs (CD / DVDs), Compact Disc-Read Only Memories (CD-ROMs), and holographic devices; magneto-optical storage media such as optical disks; carrier wave signal processing modules; and hardware devices that are specially configured to store and execute program code, such as Application-Specific Integrated Circuits (ASICs), Programmable Logic Devices (PLDs), Read-Only Memory (ROM) and Random-Access Memory (RAM) devices. Other embodiments described herein relate to a transitory computer program product, which can include, for example, the instructions and / or computer code discussed herein.

[0306]

[0307]

[0126] Some embodiments and / or methods described herein can be performed by software (executed on hardware), hardware, or a combination thereof. Hardware modules include, for example, a general-purpose processor, a field programmable gate array (FPGA), and / or an application specific integrated circuit (ASIC). Software modules (executed on hardware) can be expressed in a variety of software languages (e.g., computer code), including C, C++, Java, Ruby, Visual Basic, Python, and / or other object-oriented, procedural, or other programming language and development tools. Examples of computer code include, but are not limited to, micro-code or microinstructions, machine instructions, such as produced by a compiler, code used to produce a web service, and files containing higher-level instructions that are executed by a computer using an interpreter. For example, embodiments can be implemented using imperative programming languages (e.g., C, Fortran, etc.), functional programming languages (Haskell, Erlang, etc.), logical programming languages (e.g., Prolog), object-oriented programming languages (e.g., Java, C++, etc.) or other suitable programming languages and / or development tools. Additional examples of computer code include, but are not limited to, control signals, encrypted code, and compressed code. Optionally, the embodiments and / or methods described herein are implemented using an operating system such as Robot Operating System (ROS).

[0308]

[0309]

[0127] The following statements encompass example embodiments of the systems and method described herein, and do not define the scope of the invention, which is instead defined in the appended claims.

[0310]

[0311] Statements of invention:

[0312]

[0313] 1. An electronic circuit for a convolutional Tsetlin machine or convolutional coalesced Tsetlin machine, both referred to as a CTM, the electronic circuit comprising:

[0314]

[0315] a logic element configured to implement a logical expression, the logic element comprising an input stage and an output stage, wherein the logical expression is evaluated for a current patch of a convolution process; and

[0316]

[0317] a first feedback element, wherein an output signal of the output stage is connected to the input signal of the first feedback element, and an output signal of the first feedback element is connected to the input stage.

[0318]

[0319] 2. The electronic circuit of statement 1, wherein the first feedback element is a D flipflop and inputs to the D flip-flop comprise a clock input and the output signal from the output stage.

[0320]

[0321] 3. The electronic circuit of statement 1, wherein the first feedback element is a latch and inputs to the latch comprise a reset signal and the output signal from the output stage, wherein the reset signal is a synchronous or asynchronous signal and the output signal from the output stage is an asynchronous signal.

[0322]

[0323] 4. The electronic circuit of any preceding statement, wherein the output signal of the first feedback element is connected to the input stage such that switching of the logic element logic is avoided after the logical expression has evaluated to 1.

[0324]

[0325] 5. The electronic circuit of any preceding statement, wherein the output signal of the first feedback element is connected to the input stage such that the logic element is kept high throughout operation as a filter.

[0326]

[0327] 6. The electronic circuit of claim 5, wherein the logic element is kept high throughout operation as the filter for the remainder of the convolution process.

[0328] 7. The electronic circuit of any preceding statement, wherein the output signal of the first feedback element is connected to the input stage such that the first feedback element is reset after the convolution process.

[0329]

[0330] 8. The electronic circuit of any preceding statement, wherein:

[0331]

[0332] the input stage comprises one or more input logic gates;

[0333]

[0334] an input signal for the one or more input logic gates comprises the output signal of the first feedback element, a feature signal or a negated feature signal, and an include signal; and

[0335]

[0336] an output signal for the one or more input logic gates is connected to the output stage.

[0337]

[0338] 9. The electronic circuit of statement 8, wherein the one or more input logic gates perform OR logic.

[0339]

[0340] 10. The electronic circuit of any preceding statement, wherein:

[0341]

[0342] the output stage comprises an output logic gate;

[0343]

[0344] an input signal for the output logic gate is connected to an output signal of the input stage; and

[0345]

[0346] the output signal for the output logic gate is connected to the input signal of the first feedback element.

[0347]

[0348] 11. The electronic circuit of statement 10, wherein the output logic gate performs AND logic.

[0349]

[0350] 12. The electronic circuit of any preceding statement, further comprising a second feedback element configured such that the logic element will always output 1 for the current patch if the logic element has output 1 for a previous patch.

[0351]

[0352] 13. The electronic circuit of statement 12, wherein:

[0353]

[0354] the second feedback element comprises a training logic gate, a first sequence logic gate and a second sequence logic gate;

[0355]

[0356] the input signal of the first sequence logic gate is connected to the output signal of the training logic gate and the output signal of the output stage; and

[0357] the input signal of the second sequence logic gate is connected to the output signal of the first logic sequence gate and the input signal of the first feedback element.

[0358]

[0359] 14. The electronic circuit of statement 13, wherein the first sequence logic gate and the second sequence logic gate perform OR logic.

[0360]

[0361] 15. The electronic circuit of statement 13 or 14, wherein:

[0362]

[0363] the input signal to the training logic gate comprises a training signal and an empty signal; and

[0364]

[0365] the training signal is high during a training mode of the CTM.

[0366]

[0367] 16. The electronic circuit of statement 15, wherein the empty signal is inversely connected to an input signal for the output stage.

[0368]

[0369] 17. The electronic circuit of statement 15 or 16, wherein the training logic gate performs AND logic.

[0370]

[0371] 18. The electronic circuit of any preceding statement, wherein the electronic circuit is an integrated circuit.

[0372]

[0373] 19. A CTM comprising the electronic circuit of any of the preceding statements 1-18.

[0374]

[0375] 20. The CTM of statement 19, wherein the CTM is used for any of: image classification, object detection, imaging analysis, medical imaging analysis, natural language processing, time series data analysis, anomaly detection, forecasting, feature extraction, or feature detection.

[0376]

[0377] 21. The CTM of statement 19 or 20, wherein:

[0378]

[0379] input data for the CTM is associated with a number of patches and

[0380]

[0381] the CTM comprises a number of electronic circuits based on the number of patches.

[0382]

[0383] 22. A method for using a convolutional Tsetlin machine or convolutional coalesced Tsetlin machine, both referred to as CTM, comprising the electronic circuit of any of statements 1-18, the method comprising:

[0384]

[0385] obtaining input data comprising a plurality of datapoints;

[0386] applying an encoding technique to the input data;

[0387]

[0388] determining a number of patches based on a predefined convolution window and a predetermined number of layers;

[0389]

[0390] calculating a number of features applied to the CTM per patch based on the plurality of datapoints, the encoding technique, the predefined convolution window, and the number of patches, thereby generating a plurality of feature signals;

[0391]

[0392] generating a plurality of negated feature signals, wherein each negated feature signal comprises a negated version of each feature applied to the CTM;

[0393] for each patch, inputting one or more feature signals of the plurality of feature signals or one or more negated feature signals of the plurality of negated feature signals to the electronic circuit, wherein:

[0394]

[0395] the input signal to the input stage comprises the one or more feature signals or the one or more negated feature signals;

[0396]

[0397] the output signal of the output stage is input to the first feedback element, resulting in a switch-reduction signal; and

[0398]

[0399] the switch-reduction signal is connected to the input stage.

[0400]

[0401] 23. The method of statement 22, wherein input data comprises an input image and a plurality of datapoints comprises a plurality of pixels.

[0402]

[0403] 24. The method of statement 22, wherein input data comprises time-series data.

[0404] 25. The method of statement 24, wherein the time-series data comprises an audio signal.

[0405]

[0406] 26. The method of statement 24 or 25, wherein the time-series data is associated with medical or health data.

[0407]

[0408] 27. The method of any of statements 22-26, wherein the encoding technique is a Boolean encoding technique.

[0409]

[0410]

[0128] Any of the above statements can be combined and the skilled person would understand such examples do not limit the potential embodiments of the present disclosure.

[0411]

[0129] The skilled person will also understand that any use of “or” throughout the statements of invention or description herein encompasses use of “or”, “and / or”, and “and”. For example, the term "or" within the discourse is construed to encompass both "and" and "and / or" owing to its inherent inclusivity. Within linguistic reasoning, "or" denotes an inclusive disjunction, allowing for the consideration of scenarios wherein either one condition holds true, the other condition holds true, or both conditions hold true concurrently. This interpretation inherently incorporates the conjunction "and", permitting the acknowledgment of scenarios wherein multiple conditions coexist. Additionally, the term "and / or" explicitly acknowledges the possibility of either condition being singularly true or both conditions being true simultaneously, thus aligning with the broader meaning of "or" within the context of this disclosure.Consequently, "or" functions as a flexible connector within the statements of invention, accommodating both exclusive and inclusive interpretations to suit the nuanced requirements of embodiments described herein.

Claims

PATENT CLAIMS1 . An electronic circuit for a Convolutional Tsetlin Machine, CTM, the electronic circuit comprising:a logic element configured to implement a logical expression, the logic element comprising an input stage and an output stage, wherein the logical expression is evaluated for a current patch of a convolution process; anda first feedback element, wherein an output signal of the output stage is connected to the input signal of the first feedback element, and an output signal of the first feedback element is connected to the input stage.

2. The electronic circuit of claim 1 , wherein the first feedback element is a D flip-flop and inputs to the D flip-flop comprise a clock input and the output signal from the output stage.

3. The electronic circuit of claim 1 , wherein the first feedback element is a latch and inputs to the latch comprise a reset signal and the output signal from the output stage,wherein the reset signal is a synchronous or asynchronous signal and the output signal from the output stage is an asynchronous signal.

4. The electronic circuit of any preceding claim, wherein the output signal of the first feedback element is connected to the input stage such that switching of the logic element logic is avoided after the logical expression has evaluated to 1 .

5. The electronic circuit of any preceding claim, wherein the output signal of the first feedback element is connected to the input stage such that the logic element is kept high throughout operation as a filter for the convolution process.

6. The electronic circuit of any preceding claim, wherein the output signal of the first feedback element is connected to the input stage such that the first feedback element is reset after the convolution process.

7. The electronic circuit of any preceding claim,wherein:the input stage comprises one or more input logic gates; an input signal for the one or more input logic gates comprises the output signal of the first feedback element, a feature signal or a negated feature signal, and an include signal; andan output signal for the one or more input logic gates is connected to the output stage.

8. The electronic circuit of any preceding claim, wherein:the output stage comprises an output logic gate;an input signal for the output logic gate is connected to an output signal of the input stage; andthe output signal for the output logic gate is connected to the input signal of the first feedback element.

9. The electronic circuit of any preceding claim, further comprising a second feedback element configured such that the logic element will always output 1 for the current patch if the logic element has output 1 for a previous patch.

10. The electronic circuit of claim 9, wherein:the second feedback element comprises a training logic gate,a first sequence logic gate and a second sequence logic gate;the input signal of the first sequence logic gate is connected to the output signal of the training logic gate and the output signal of the output stage; and the input signal of the second sequence logic gate is connected to the output signal of the first logic sequence gate and the input signal of the first feedback element.

11. The electronic circuit of claim 10, wherein:the input signal to the training logic gate comprises a training signal and an empty signal; andthe training signal is high during a training mode of the CTM.

12. The electronic circuit of any preceding claim, wherein the electronic circuit is an integrated circuit.

13. A CTM comprising the electronic circuit of any of the preceding claims 1-12.

14. A method for using a Convolutional Tsetlin Machine, CTM, comprising the electronic circuit of any of claims 1-17,the method comprising:obtaining input data comprising a plurality of datapoints;applying an encoding technique to the input data;determining a number of patches based on a predefined convolution window and a predetermined number of layers;calculating a number of features applied to the CTM per patch based on the plurality of datapoints, the encoding technique, the predefined convolution window, and the number of patches, thereby generating a plurality of feature signals;generating a plurality of negated feature signals, wherein each negated feature signal comprises a negated version of each feature applied to the CTM; for each patch, inputting one or more feature signals of the plurality of feature signals or one or more negated feature signals of the plurality of negated feature signals to the electronic circuit,wherein:the input signal to the input stage comprises the one or more feature signals or the one or more negated feature signals;the output signal of the output stage is input to the first feedback element, resulting in a switch-reduction signal; andthe switch-reduction signal is connected to the input stage.

15. The method of claim 14, wherein input data comprises:an input image and a plurality of datapoints comprises a plurality of pixels; or time-series data and a plurality of datapoints comprises at least one value at least one timepoint.,

Citation Information

Patent Citations

  • Low power hardware architecture for handling accumulation overflows in a convolution operation

    EP4290395A1

  • Accumulator hardware

    GB2617159A

  • Sparsity-based reduction of gate switching in deep neural network accelerators

    US20230325665A1