Method and apparatus for a bit-parallel inference architecture for tsetlin machines

Abstract

There is provided a method of compressing Feature input data associated with a Clause computation in a Tsetlin Machine, comprising only providing instructions for Feature data sets having an Include value associated with the Clause computation, wherein each instruction includes an indication to a Literal to be used in the computation of the Clause, a data bit (value) indicating whether a value of the Literal to be included is positive or negative, and whether the Literal is the last Literal to be processed for the Clause. There are also provided a method of inferencing in a Tsetlin Machine, a Tsetlin Machine instruction, and a processing circuit for a Tsetlin Machine.

Description

[0001] METHOD AND APPARATUS FOR A BIT-PARALLEL INFERENCE ARCHITECTURE FOR TSETLIN MACHINES

[0002] FIELD OF THE INVENTION

[0003] The present invention relates to a method of, and apparatus for, compression for a bit-parallel inference architecture for Tsetlin machines.

[0004] BACKGROUND OF THE INVENTION

[0005] There is an emerging field which aims at deploying Machine Learning (ML) algorithms to edge devices that consume micro to milliwatts of power, which is currently referred to as Tiny Machine Learning (TinyML). TinyML aims at ultra-low power / energy realization of Machine Learning algorithms on microcontrollers or dedicated chip on battery-operated platforms, to improve energy-efficiency and battery longevity. There were over 250 billion microcontrollers in use in the year 2020, and the forecasts show that the number of shipments are increasing every year. A lot of these microcontrollers already in use in loT, smartphones and other electronic devices already use some ML algorithms. For example, currently, TinyML finds applications in Google Assistant, Alexa, and applications which require continuous monitoring in industry, medicine and agriculture. Such TinyML implementations are capable of distinguishing between signals, words, images, numerical values and alphabets. Among others factors, TinyML also provides data security and privacy, because data is processed in situ, instead of being sent off to the cloud. TinyML therefore uses less to no bandwidth and is capable of working without internet.

[0006] Accordingly, there is a desire to provide improved processing architectures for use in TinyML, and other ML use cases.

[0007] BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Examples of the invention are further described hereinafter with reference to the accompanying drawings, in which:

[0009] Figure 1 shows an example of a generalised Tsetlin Machine according to an example of the disclosure; Figure 2 shows an example of a Tsetlin Automaton according to an example of the disclosure;

[0010] Figure 3 shows an example hardware arrangement and signalling for a Clause computation method during an inference routine, according to an example of the disclosure;

[0011] Figure 4 shows an example of a proposed compression scheme for the ‘includes’ in a Tsetlin Automaton, using 16-bit output instructions, according to an example of the disclosure;

[0012] Figure 5 shows an example bit-packing scheme for the binary Feature input data used by a Tsetlin Machine accelerator, according to an example of the disclosure;

[0013] Figure 6 shows an example inference routine algorithm that uses the disclosed TA compression scheme of Figure 4, and the bit-packed Feature input data of Figure 5, according to an example of the disclosure;

[0014] Figure 7 shows a high level schematic diagram of an example RISC-V based SoC architecture employing at least one Tsetlin Machine accelerator, according to an example of the disclosure;

[0015] Figure 8 shows a schematic diagram of one of the Tsetlin Machine accelerator(s) of Figure 7, according to an example of the disclosure; Figure 9 shows a schematic diagram of the Clause Accumulator and Inference Result Accumulator of Figure 8, according to an example of the disclosure;

[0016] Figure 10 shows an example logical implementation of the disclosed Tsetlin Machine accelerator apparatus.

[0017] DETAILED DESCRIPTION

[0018] The technical operation and advantages of the present disclosure shall now be provided by way of a plurality of examples that are merely illustrative of the novel and inventive features, and the disclosed examples are intended to be fully combinable in any reasonable combination (i.e. not otherwise explicitly called out as being excluded from combination in the disclosure, or where the combination defies the laws of physics). This is to say, as per usual patent practice, the disclosure is written to be inclusive.

[0019] One form of processing architecture that may be used for TinyML implementations is the Tsetlin Machine architecture, which is a novel approach to TinyML applications. Tsetlin Machines (TM) is based on a Machine Learning algorithm that relies on the principles of learning automata called Tsetlin Automata (TA), and game theory, to create logic propositions for classification of input data.

[0020] The Tsetlin Machine (TM) is an existing machine learning algorithm, as disclosed by Ole-Christoffer Granmo, in his academic paper entitled “The Tsetlin Machine -- A Game Theoretic Bandit Driven Approach to Optimal Pattern Recognition with Propositional Logic," ArXiv: 1804.01508v6, pp. 1-42, 16 April 2018, the teaching of which is herein incorporated by reference in its entirety, but especially the portions detailing how a Tsetlin Machine operates in general theory such as, but not limited to Section 3 of that paper.

[0021] A particular strength of the Tsetlin Machine approach is that it provides state-of-the-art performance when problem properties are unknown and stochastic, while the problem must be solved as quickly as possible through trial and error, which is an often seen set of circumstances in the Machine Learning area.

[0022] Unlike traditional Machine Learning (ML) approaches (e.g. deep learning and neural networks), the Tsetlin Machine approach is fundamentally based on logic rather than arithmetic (i.e. uses Boolean values and Boolean functions, such as AND, (X)OR, NOT and the like).

[0023] We refer to the textbook “An Introduction to Tsetlin Machines”, by Ole-Christoffer Granmo, published at https:z / teetij ;Yi8c jne.om, that describes the basic principles of Tsetlin Machines, which is hereby incorporated by reference.

[0024] A block diagram of a basic Tsetlin Machine algorithm is shown in Figure 1 , which is composed of a collective of Tsetlin Automata, known as a Team of Tsetlin Automata. These use propositional logic to map an arbitrary sequence of input bits to an arbitrary sequence of output bits, to thereby classify an input data set - i.e. for example pattern match the input.

[0025] To solve ML problems with the Tsetlin Machine, the input data are first encoded into terms of Boolean Literals. These Literals form part of one or more logic proposition units, called Clauses, which select different patterns of Boolean expressions. Some Clauses propose in favour of a certain class (where a class is a definable item that will be inferred during use, typically after suitable training has occurred - as will be appreciated, the present disclosure relates to a classification problem in machine learning, in which a class label is anticipated for a specific example of input data), while some Clauses oppose the definition of that class. When a majority of Clauses can provide enough confidence in their propositions, that class is selected as the outcome (i.e. resultant output data). In brief summary, to derive a proposition from each Clause, a supervised training process is used. As will be appreciated by the skilled person in the art of Tsetlin Machines, a Feature is a single Boolean variable corresponding to a particular element of a Boolean vector describing an input data item. This is to say, a Boolean vector comprises one or more Features, and a set of such Boolean vectors may form the data set used by examples of the present disclosure for Machine Learning. Furthermore, according to a more general definition: Machine Learning Features may be thought of as the independent (Boolean) variables that are in the form of rows and columns in a structured dataset that acts as input to the learning model (see uncompressed binary data in Figure 5). As such, a Feature is a measurable property of the objects that need to be analysed. By way of a very brief example, a ‘car’ may be defined as a (specific form of) vehicle that has wheels (which may be defined as a positive Feature of having wheels), and transports people (which may be defined as a positive Feature of being able to transport people), but does not fly because it does not have wings (which may be defined as a negative Feature of having wings). Similarly, using the same basic Feature definitions, an airplane can be defined as a vehicle that has wheels (defined as a positive Feature of having wheels), and transports people (defined as a positive Feature of being able to transport people), and does fly (defined as a positive Feature of having wings). Thus, in these examples, it can be seen that the Features that define the type of vehicle are: wheels (or not), transport people (or not) and has wings (or does not).

[0026] For training, each of these Features, as part of a Clause, is then included or excluded based on a finite state machine, called a Tsetlin Automaton (TA) (see Figure 2, explained in more detail below).

[0027] As shown in Figure 1 , an example of a Tsetlin Machine algorithm according to an example of the present disclosure comprises two portions, one responsible for inference (i.e. classification) 102, and the other responsible for reinforcement and feedback for learning (i.e. training) 103.

[0028] The following describes the different parts of these two portions, and their usage parameters.

[0029] As briefly described above, the input data of a Tsetlin Automata according to an example is defined as a set of Boolean (i.e. binarized) variables, called Features, which are represented by what are called Literals 105. To explain further, Features and negated Features are called Literals, as customary in Boolean algebra, where if the Features are called Xi, X2, ... XN, then the corresponding Literals are typically Xi, -■Xi, X2,-,X2, ..XN,-,XN (where -■ is the negated version of the variable, which may also be labelled using a line, called ‘bar’, above the respective value (see Figure 3, for example). Hence there are 2 Literals per Feature. In the above example of a car and airplane, Xi is ‘has wheels’, -■ Xi is ‘does not have wheels’, X2 is ‘transports people’, -> X2 is ‘does not transport people’, etc.

[0030] The Boolean Literals 105 are fed into both portions of the Tsetlin Automaton, i.e. into both the inference portion 102 and the reinforcement portion 103. The use of two arrows 104 between the Literals 105 and a conjunctive Clauses 130 (described in more detail below) signifies that any Literal can potentially be involved in multiple Clauses - so there is a fan-out from Literals to Clauses. Likewise the use of two arrows 138 between the conjunctive Clauses 130 and the Sum & Threshold function(s) 140 (described in more detail below) signifies any Clause can potentially vote for or against multiple classes - so there is a fan-out from Clauses to classes.

[0031] There is also proved a further data input, called Expected-Out 106, which is an expected class output that is fed into the Sum and threshold unit 140, which may be thought of as an initial classification decision that may be reinforced or diminished as the overall TM learns from the data inputs, etc., as discussed in more detail below.

[0032] Inference:

[0033] The main inference component is the conjunctive Clause 130, which uses propositional logic expressions for output classification. The composition of each Clause (Ci 132, C2 134, ... CM 136) is controlled by a Team of Tsetlin Automata 1 10 (comprising individual Tsetlin Automatons TAi 112, TA2 114... TAN 116), each of which has a pre-defined number of states, divided between Actions (i.e. Include or Exclude - see Figure 2). In summary, the individual Tsetlin Automata 112-116 decide whether their associated Literal should be included in the Clause or not, following a number of reinforcement steps. The Clauses are controlled by the TAs using suitable control and feedback signals propagated through the system as described in detail below.

[0034] Each inference class (i.e. classification type) has a set of Clauses associated with it, which is to say the set of Clauses defines how to decide whether a given input data point is of that class, or not. When applied to the input data, each Clause produces a yes vote or no vote on whether the input data is of that class. Half of the Clauses can vote positively, and half of the Clauses can vote negatively. The inclusion of inhibition (i.e. negative voting) in the voting system of the (Team of) Tsetlin Automaton is what enables non-linearity in the inference process. The votes, communicated by the outputs of the Clauses 138, are summed, for example in the Sum and Threshold function 140 (i.e. comprising a Sum and Threshold per class - items 142 to 146) to produce a collective result which gives an indication of confidence in the respective classification decision. This confidence in the form of feedback: Type I, Type II or None (as will be explained later) and the outputs of the Clauses are communicated, via 129 and 137, respectively, to the feedback section 120 (see details of feedback below), to influence future decisions of the Team of Tsetlin Automata 110. The output 150 of the Sum and Threshold function 140 is a set of classifications.

[0035] In a single-class inference problem, the output layer may be a simple thresholding function (e.g. set by a threshold settings value). For example, if enough (i.e. above the threshold) of the votes are positive (or zero = inaction), the input data is determined to belong to the class. The opposite is true for a negative sum (i.e. the Input Data is determined to not be part of the class). For multi-class problems, instead of a simple threshold function, an Argmax function can be used to determine the output class, where the Argmax function chooses the class with the highest confidence. In this case, the class summation becomes indicative of confidence for that particular class.

[0036] Reinforcement:

[0037] Fundamental to reinforcement are the Team of Tsetlin Automata 110, where, in the example shown, the team comprises N Tsetlin Automatons (i.e. state machines, like Figure 2), where N is derived from the number of Features (f) and the number of Clauses (M) in use, specifically where N = 2 x f x M (there are 2, because Features can be true or false, represented by Literals X and -■X). Such a Team of Tsetlin Automata are also known as Team of Tsetlin Automata with linear tactics (which means they change states in equal and discrete steps, and that makes them readily implementable in digital hardware, e.g. using up / down counters or shift registers), to emphasize the fact that they allow gradual ascent, or reinforcement, in performing a particular Action, and equally gradual descent (i.e. diminish) from the particular Action to performing another Action. The relative ‘speed’ of the ascent and decent may be set by variables, and it is common for the decent to be more rapid than the ascent, to ensure appropriate learning reinforcement levels. In practice there are a variety of different types of such learning automata which may also be applied to the present disclosure, for example as disclosed in the textbook: Narendra KS, Thathachar MAL. 1989 Learning automata: an introduction. Upper Saddle River, NJ: Prentice-Hall, Inc., which is incorporated by reference (i.e. block 530 of Figure 5 may implement any suitable learning automaton).

[0038] A state diagram of one exemplary Tsetlin Automata approach is shown in Figure 2, which shows an Exclude action (Action 1), and an Include action (Action 2), and how these may be reinforced by way of Penalty and Reward (and Inaction) functions. Half of the states within the TA favour Exclusion (i.e. states 1 to ri), while the other half favours Inclusion (i.e. states n+1 to 2n). As training proceeds with more data and rounds, these states are reinforced until each Tsetlin Automaton offers sufficiently high confidence in the decision of including or excluding the input Boolean or its complement, for example ‘far enough away’ from the mid-point n in Figure 2 (to the left for Exclude, and to the right for Include). What constitutes ‘far enough away’ may also be set using a threshold settings value. As such, the confidence level is related to 2n.

[0039] In more detail, Figure 2 shows the state diagram for a two-action Tsetlin Automaton as an example portion of an overall Tsetlin Machine implementation, having 2n states 201 , and a centre point / midstate 250, also called the decision boundary. The current state of the automaton decides which action to perform. The automaton in the figure has 2n states. Action 1 = Exclude is performed in the states with index 1 to n, while Action 2 = Include is performed in states with index n + 1 to 2n. The state transitions of the automaton govern learning. One set of state transitions is activated on reward (solid lines), and one set of state transitions is activated on penalty (dotted lines). As seen, rewards and penalties trigger specific transitions from one state to another, designed to reinforce successful actions (i.e. those eliciting rewards), and diminish, or deprecate, the unsuccessful ones. Implementation-wise, a Tsetlin Automaton simply maintains an integer (the state index), and learning is performed through increment and decrement operations, according to the transitions specified in the figure. Thus the Tsetlin Automaton is extremely simple computationally, with a very small memory footprint.

[0040] In more specific detail, according to this simple example, the Tsetlin Automaton may be given reward (solid arrows, e.g. 202, going away from the centre point / midstate 250), causing it to reinforce the current action decision (e.g., in this case either Action 1 Exclude or Action 2 Include) by moving away from the midstate. Conversely, the Tsetlin Automaton may be given a penalty (dotted arrows, e.g. 203, going towards the centre point / midstate 250), which moves the state towards the centre point decision boundary 250. In some cases, a state may be given an Inaction reward 204, which simply maintains the current state.

[0041] When applied to processing the Boolean Literals 105 in Figure 1 , i.e. through the Team of Tsetlin Automata operating on Clauses 132-136, the two actions may be Include and Exclude 118. The update of the Tsetlin Automata 110 uses reinforcement through Penalty, Reward and Inaction 128. The decision on whether to update individual Tsetlin Automata (112-116) may be controlled by a number of conditions based on: 1) the values of Literals, 2) votes from Clauses described earlier (i.e. a data input from a previous LPE), and 3) current Actions of Tsetlin Automata in terms of Include and Exclude. For further details of how algorithm reinforces these updates, we again refer to the document O.-C. Granmo, “The Tsetlin Machine - A Game Theoretic Bandit Driven Approach to Optimal Pattern Recognition with Propositional Logic,”, and the Tsetlin Machine manual that can be found at https: / / tsetlinmachine.org noted above).

[0042] The characteristics of Tsetlin Machine(s) can be enumerated as follows:

[0043] 1 . Tsetlin Machine(s) can be visualized as a single layer ML algorithm, in contrast to multi-layer deep neural networks (DNN).

[0044] 2. Tsetlin Machine(s) operates on binarized values and does not include any floating-point operations.

[0045] 3. Tsetlin Machine(s) include a feedback process that includes randomized increment or decrement of state values of TA, instead of gradient descent based backpropagation.

[0046] 4. Tsetlin Machine(s) include an inference process that involves logical OR, AND and NOT operations (or derivatives, such as XOR), along with arithmetic add or subtract, while the usage of multiplication and division can be entirely mitigated. 5. Tsetlin Machine(s) is deterministic (as in the logic works in a deterministic way overall, but for a randomisation factor, which in the specific example is provided by a pseudo random generator) and predictable because the Tsetlin Automata (that are the basic component of the Tsetlin Machine(s) architecture) profile can be analyzed to understand the information being learned (unlike the black-box nature of other ML methodologies, such as DNN).

[0047] Although the aforementioned characteristics make Tsetlin Machine(s) an ideal candidate for TinyML hardware implementations, such implementations encounter bottlenecks due large model size and the consequent inference time. This increased inference time will directly affect the energy used per inference (or classification) if not mitigated, and reducing energy usage is particularly important in the low power edge processing / TinyML space. Accordingly, examples of the present disclosure provide a novel method and apparatus that may achieve much higher compression of the model size (e.g. up to 99%), thereby making the model size used smaller (or at least similar to) the size used in state of the art Neural Network (NN) based hardware solutions. For example, there are disclosed at least the following aspects: a novel compression technique to reduce the memory footprint of a trained TA, and store it in the form of bespoke instructions, and a novel inference technique that works in this compressed domain (i.e. works with the bespoke instructions), in particular using compressed stored TA instructions to process the input data. According to some examples, the inference method is designed to exploit bit-parallelism of input data (which may be real-time or otherwise) to perform many inferences, concurrently. According to examples, there is also provided a detailed hardware implementation for operation in this compressed domain that may be implemented as an ASIC, for example.

[0048] Compression Technique and Instruction Generation

[0049] Figure 3 shows an example of (uncompressed) proposition logic 300 that results in the computation of a Clause output. As can be seen from the Figure, the proposition logic may include NOT 310, OR 320 and AND 330 Boolean operations between input Literals L, 340 and Tsetlin Automata TA, 350, where / e [0,f — 1] and f is the number of Boolean Features in the input data. As will be appreciated, particular TA, 350 can be two different values (derived from x, or -,x, and labelled here as TAo and TAo bar) and so there are 2 x f automata per Clause, meaning there are 2 x f OR gates included / excluded 360, and hence can be a 1 or 0 accordingly.

[0050] Figure 3 presents the Clause computation method that is a fundamental part of the TM inference routine. In the Figure, each Clause consists of (2xf) TAs 350 that are ORed 320 with corresponding Literals 340 and ANDed 330 in the end to obtain the Clause output 370, which in the example shown is (written out in full) Lo & 1 &...& 1 & -■ LM , which resolves down to Lo & ...& LM . Here, f is the number of Boolean Features in the input data, and the number of Literals (or TAs) are 2 * f because they include their complements (i.e. x and -■x) hence doubling the number. A notable part of the Clause computation shown is that the Clause output only depends on the Literals whose corresponding TAs are ‘includes’, i.e. have an input bit ’1 ’. Thus, storing the information associated only with the ‘includes’ should suffice for inference needs. Accordingly, examples of the present disclosure propose a method of compression which converts only this ‘includes’ information and stores it as instructions (see figures 4-5, for example), which will be used by the later disclosed hardware (see figures 7-9, for example) to perform inference.

[0051] Figure 4 shows an example of the generally proposed instruction compression / encoding scheme 400, which in this example uses a 16-bit encoding scheme that may be used for the ‘includes’ in an example TA. Of course, the general scheme may be applied using any number of bits, as would be appreciated by the skilled person. According to this example, there are shown the TAs 410 whose values, their number being 2 x f, indicate that the Literal, direct or inverted Feature, corresponding to such a TA, may be included (i.e. value = 1), or excluded (i.e. value = 0). As will be appreciated, the TA include / exclude values are listed in column 410 typically as a contiguous sequence (shown for the first four items in the sequence, by way of example only, as bit items 410i to 4104), where each bit item in column 410 indicates (i.e. acting as an index) to a Feature in the Feature data set (see Figure 5) in terms of the Literals, i.e. the positive, x, and negated, ->x, values for the given Feature. These Feature data sets are sequenced in order from a base address (usually 0), where the index pointing to a positive Literal (e.g. x) 410i for a given Feature is positioned before the index pointing to a negated Literal (e.g. ~‘x) 4102 for the same given Feature, and so each pair of indices to a given Feature (out of f Features) is listed in ascending sequence (so for the second Feature, there are the positive Literal indexed by 4103 and negated Literal indexed by 4104 and so on). This indexing is explained and utilised in more detail in Figure 5.

[0052] When the indices 410 are listed in order according to the Features 402 (i.e. in terms of their relative addresses within a set of TAs corresponding to each Clause), these, in effect, have respective offsets 420 (from 0, 421 , to ((2x f)-1 , 422), which may be indicated using a specific offset value 435. As can be seen, only those that are ‘included’, hence have a value of 1 , are actually turned into instructions 430, with each instruction comprising some initial control bits 432, 433, and an offset value 435 (where the offset value comprises both a Feature offset value 434, and a further final Literal polarity bit 436 - this is because the initial TA include / exclude data set 410 covers both positive and negative Literals of all the Features, in sequence, as discussed above - i.e. there are two Literals per Feature). Put another way, the LSB (i.e. Literal Polarity bit) can be part of the 14- bit offset, because the offset addresses a Literal, not the Feature. A synopsis / interpretation of this encoding scheme is shown in the Right Hand Side of the whole figure, 440.

[0053] According to this example, the encoded values also act as instructions to operate on the Integrated Circuit (IC) chip. In the example shown, the first two Most Significant Bits (MSB) of the instructions (Left hand side, in blue, which may be called a first control bit portion) act as the indicators of change in a Clause 433, and the Clause’s polarity 432, while computing Clause outputs during inference. Meanwhile, there is also a LSB 436 (right hand side, in red, which may be called a second control bit portion) that indicates Literal polarity. Together, these 3 different bits in the instruction may be thought of as a set of control bits.

[0054] Figure 4 therefore presents the scheme for compressing (i.e. encoding) the Feature input data used by TAs in terms of their relative addresses (offsets 434) within a set of TAs corresponding to each Clause. To form an instruction, used by the later disclosed hardware, these offsets are prefixed (or may be postfixed, in other implementations, as noted above) by two most significant bits (MSBs) that hold Clause-related information. These bits are Clause polarity (0 for positive, and 1 for negative) and Clause change flag (if this Literal is the last in the current set of included Literals for a given Clause, its flag / value is 1 , otherwise the value is 0). In effect, the control bits (i.e. the two MSBs, and the LSB, in the example of Figure 4) are providing a routing function within the instruction as well as the actual data for processing, and so an advantage of using the two MSBs is that in the usual Big-Endian architecture, these are read first and so can be acted on sooner. However, in other architectures (e.g. Little-Endian), the control bits may be from the other end, i.e. the two LSBs. Of course, any specified position may be used, according to implementation needs, and the disclosure is not specifically limited to the MSB usage described with reference to Figure 4. Moreover, since each offset is associated with one included Literal, whose polarity can be either positive (x) or negative (x-barZ-'x), the included Literal’s polarity is represented by the final bit of the offset. In the example shown in Figure 4, only three Literals are included because their corresponding TA include / exclude index values from 410 are equal to 1 , while others are 0. Their corresponding MSBs are 00, 00 and 01 . This means that only these three Literals will help calculate the Clause whose contribution to the class sum is positive, and that the third included Literal is the last Literal to be processed in this group of Literals for this current Clause, thereby indicating that the following instruction will start with the new set of Literals for the next Clause. The final bits of the three offsets in this example are 0, 0 and 1 , to indicate that the first two Literals are of positive polarity (x) while the third one is of negative polarity (-ix, or x-bar). As can be deduced from Figure 4, the offset bits 434, taken without the LSB (literal polarity) 436, forms a unique address of a Feature, which has its corresponding bit in the Input data points (see Figure 5).

[0055] Furthermore, examples of the present disclosure also propose a bit packing scheme for the inputted Features (i.e. Feature input data). Figure 5 shows an example bit-packing scheme 500 for the uncompressed binary Feature input data 501 (formed of Features, (f) 502) that may reside in a Feature Cache (see Figure 8) according to the present disclosure. Each Feature is one of the rows 503, and the binary uncompressed data 501 may be used to store W input datapoints together (i.e. can be seen as a row 503 having a length, i.e. a number of columns 504, of a predetermined word size (W) 505, where the word size 505 provides the level of concurrency possible). Put another way, the Feature cache stores D data (in f rows, one for each Feature involved), which are addressable as groups of W data points, which we call Feature data words. These uncompressed binary data may then be converted into Hexadecimal for subsequent storage and use (for example, in software implementations, when Hex representation is customary). There now follows an explanation of how the indexing functionality provided by the use of the instructions described in Figure 4 operates.

[0056] As shown on the left of Figure 5, there are a set of pointers (i.e. indexes) 510 derived from decoding the instructions of Figure 4 to provide offset values, and control bits. Because the includes / excludes 410 are in a specific sequence, which specifically index positive or negated Feature data sets, as described with reference to Figure 4 above, only those Features that are included are involved, where their relative position (offset) will determine which of the rows are used, and then the Literal polarity control bit (which is, in effect, an indication of the specific position in a pair of values for a given Feature) determines whether a given Feature row is negated (i.e. inverted) or not. So, in the example shown, because the first two values 512 are 0, this determines (shown by arrow 513) that the first row of Feature input data is excluded 514, hence this first Features data row will not be retrieved from memory during inference at all (The same goes for all pairs of values, positive or negated, of the same Feature which are both 0). Whereas, for the next two values 515, at least one is a value 1 , and this determines (shown by arrow 516) that this second Feature input data row is included 517, hence this second row is retrieved from memory during inference. A summary of how the original uncompressed input data 501 is assessed to be included in the retrieval process (or not) is indicated by the row highlighting in data set 501 ’, and this sort of assessment, by row of the Feature input data and dependent on the decoded instruction contents including offset, is carried out for all Features data rows. This selectivity of Feature input data being retrieved is shown by block 520, and results in a reduced Feature input data set 525 being retrieved as a whole.

[0057] Meanwhile, the Literal polarity bit in effect indicates whether the 1 value was the first or second bit in the pair of values, positive or negated, for the Feature. Being the first value is the positive version, so that means the Feature data stays the same during retrieval (i.e. no negation). Now looking at all the Included Feature pairs, i.e. items 512, 515, 518 and 519, it can be seen that the final one, item 519 has a 1 for the second bit. This means the negated version (of the 6thFeature row) is to be retrieved, which means the final set of values 527 in the reduced set 525 are inverted, by a selective inversion process (here indicated by arrow 530), resulting in the actual Feature input data to be used in the inference 535, including all necessary negation (or not). The negated data is shown by 537.

[0058] According to examples, the W input datapoints for each Feature may be any kind of data including image, numeral, sound or alphabet, may be bit-packed together and may be stored in real-time (e.g. stored at time of processing / classifying the input data). Notably, in most real-life scenarios, only some of the Features are actually involved in any given current class computation, (as per the indices determined by the include / exclude values of TAs, as per Figure 4), hence providing an aspect of the compression according to the present disclosure (also see Figure 4).

[0059] Thus, it can be seen that whilst the Feature input data is stored uncompressed, only the relevant Feature input data (negated as needed) is actually retrieved and acted upon during inference, and so the overall architecture operates in the compressed domain, using instructions as described with reference to Figure 4. These bit-packed W datapoints, called Feature data words, corresponding to the included Features, may then be inferred concurrently, thereby providing orders of magnitude increase in inference speed (i.e. speedups in computation time) and hence also save significant amount of energy.

[0060] Since the ‘include / exclude’ bit being equal to logical 1 , referring to a particular Feature input, is the same value for all bits in the W-bit word size, the offset 435 (e.g. 14-bit in a 16-bit binary instruction, in this example, but other bit depths may also be used in other implementations), corresponding to the index of the Feature 402, enables fetching all W bit-values of this Feature in a single access to the storage of the input data. In this way, only those W-bit words corresponding to the Features being included (i.e. their include / exclude bit values are equal to logical 1) need to be fetched from the input data storage. This provides a further compression benefit (see Figure 5), because the relative percentage of ‘include / exclude’ bits that are equal to logical 1 are considerably smaller than those whose ‘include / exclude’ bits are equal to 0, as can be observed from Figure 4 (in this case, 3 out of 2 x f potential).

[0061] Figure 6 shows an example inference routine algorithm 600 that uses the disclosed compressed TA and (compressed) Bit-packed Input data. Algorithm 1 in Figure 6 presents the pseudocode of the proposed inference algorithm. In this example algorithm, the inference of W data points is done simultaneously in each iteration, and hence the total number of iterations performed is D / W, where D is total number of data points in the testing dataset. The specific algorithm shown is based on the assumption that Word Size, W = 32 (see line 1) and 16 bits or 2 bytes are used to encode each include instruction (the structure of the instruction was explained earlier in connection with Figure 4 - see Input setup at the top, where Clause polarity bit 432 is bit 15, Clause change bit 433 is bit 14, and the rest is the offset 435 - i.e. 14 bits), which justifies the constants appearing in the pseudocode. However, generally, different values of Word size, W, can be used depending on the maximum word size of the underlying hardware, and of course total number of datapoints, D, can vary as well. The example algorithm iterates in its outer ‘for’ loop (lines 6-25) through M classes. For each class ‘i’ a set of ‘includes’ is defined by the lnc_per_class[i] variable. Within the range of addresses of the include instructions corresponding to a particular class ‘I’, in the next nested ‘for’ loop (lines 10-20), the algorithm iterates through the array of encoded ‘includes’ (called IncEnc) once for every W data points, where it computes the corresponding Clauses ‘j’ and finds their sum of W data points simultaneously. This enables parallel inference of W data points and significantly reduces the total computation time. While iterating through the array IncEnc, indexed by the Clause address ‘j in the range defined in line 10, the algorithm scrolls through each Clause, using the method of finding the value of Clause change bit (line 11) and keeps adding the Clause outputs to the class sums (lines 12-14), according to the Clause polarity bit value (line 16). The calculation of the Clause’s Boolean value is done in lines 16-19, by pulling the values of the included Literal’s Feature (addressed by the offset value in lncEnc[j]) from the input data set inp_f and performing the logical operation bit-wise described earlier in Figure 3. The Literal polarity 436 is used in lines 18-19 to determine whether the input Feature or its complement is needed to compute the Clause output. In line 20, the position of the starting value for the Clause index for the next class is incremented by lnc_per_class[i]. The algorithm keeps track of the maximum class sum and the corresponding class in lines 21-24.

[0062] According to the present disclosure, there is also provided a hardware architectural implementation for the disclosed inference method that operates in the compressed domain, so does not need decompression. An advantage of the disclosed hardware is that it is oblivious to the underlying dataset and works with all TM models trained for different datasets.

[0063] Figure 7 shows an overview of an example RISC-V based SoC architecture 700 employing at least one TM accelerator 800 according to the present disclosure. The architecture shown in Figure 7 comprises an arbitrary number of RISC-V general propose processor cores 710, an arbitrary number of Tsetlin Machine Accelerators 800, a General propose input and output interface hardware 730, and a unified memory architecture 740 shared between the RISC-V cores 710 and the TM accelerators 800, in this example operating via a shared bus 720. Note that whilst the general processor (GP) cores 710 are RISC-V in this example implementation, they may use any form of GP core architecture (which will potentially affect use of MSB / LSB, for example if based on a different, Little-Endian type).

[0064] According to the example shown, the RISC-V processor cores are responsible for, potentially amongst other things, binarising the input Features (i.e. converting (normally) digital input data values into their Boolean representation of Features 502, as per Figure 5, for example), programming the TM accelerators cores 800, and collecting and acting upon the results computed at the TM accelerator cores 800. The RISC-V processor cores 710 may store the binarised input Features and the model (discussed later, but in brief, meaning - what a certain machine has been trained to look for / do in terms of the classifications (i.e. the relationship between the Literal and Clauses -that is to say, what is included or excluded in the interactions) in a memory storage device, for example a unified memory 740. Pointers to the Features and the model can then be provided into the accelerator(s) 800 for use in its / their processing. Multiple TM Accelerators 800 may be employed, to enable parallel classification of multiple classes, as mentioned above. In any parallel processing configuration, each TM accelerator 800 may receive a different model to be used to compute the respective class score, but may still operate on the same Feature set. Alternatively, each TM accelerator 800 may operate on a different Feature set (as well as different model).

[0065] An example TM accelerator architecture 800 that implements the above described methods (including the algorithm described in Figure 6) is shown in Figure 8, and comprises a pipeline architecture featuring: a dedicated model cache 810, a model fetching controller 820, an instruction decoder 830, a dedicated Features cache 840, a Feature fetching controller 850, a Clause accumulator 860 (i.e. AND-reduction - see below), and an Inference (classification) results accumulator 870. The dedicated model cache 810, dedicated Features cache 840 and Inference (classification) results accumulator 870 may be connected to the system bus 720, to allow data transfer in and out of these units.

[0066] On ideal conditions (where, for example, ideal conditions here means that both caches are fully populated with all required data, thus no cache misses occur; it also considers that enough cycles have elapsed to fill the pipeline), the TM accelerator 800 is capable of processing 1 model instruction per-cycle (e.g. per clock cycle for synchronous hardware implementations). Each model instruction operates on a parameterised number of independent data points, typically 32 or 64, depending on the wordsize W (see above) selected during the design of any given implementation. An example of the structure of each model instruction was described in Figure 4.

[0067] The model cache 810 is akin to an instruction cache in a traditional processor organisation, because the model cache 810 stores a local copy of the model instructions (e.g. see Figure 4) from memory 740. The modelfetching controller 820 is responsible for keeping the model cache 810 full, and since the instruction flow has no branching, the model fetching controller 820 is able to predict exactly the execution flow and prefetch blocks of instructions before they are needed. The Feature cache 840 is akin to a data cache, because the Feature cache 840 holds the Features referenced by the model. The overall TM accelerator 800 may only read from memory 740, via system bus 720, which simplifies the cache(s) architecture, because there are no requirements to commit writes to the main memory.

[0068] According to examples, the model, akin to a program, may comprise a sequence of instructions, in which case each instruction may point to a Literal that may be included in a Clause computation. In effect the instruction’s offset 434, taken without its Literal polarity 436, points to exactly one of the f Features amongst all f Features 502. The instructions may also signal the completion of a Clause computation (i.e. Clause change, via the second blue control bit 433), as well as the Clause contribution, i.e. either via increment or decrement (i.e. Clause polarity, via the first blue control bit 432), to the classification result. Instructions may also command how Features are processed into Literals; if the Literal polarity bit 436 is enabled (set to logical 1) in the instruction, the Literal is formed from the negation (i.e. Boolean inversion) of the Feature. The LSB Literal polarity bit 436 is a control bit used by the logic in the Feature fetching controller 850 and Clause Accumulator 860 in Figures 8 and 9, which are explained in more detail below.

[0069] According to the specific example of the hardware shown in Figure 8, the algorithm (of Figure 6) may be implemented in the disclosed pipeline structure (of Figure 7) as follows. Every instruction (one of those 430 as per Figure 4) fetched from Model Cache 810 by Model Fetching Controller 820, passes first via Instruction Decoder 830, which separates the control bits, which comprise the two MSBs described earlier as Clause polarity bit 432 and Clause change bit 433, as well as the least significant bit described as the Literal polarity bit 436, from the offset part of the instruction 435. Then Feature Fetching Controller 850 uses the (Feature) offset part 434 of the instruction as a Feature address and fetches a bit-packed ‘included’ Feature word (e.g. line 503 as per Figure 5) from the Features Cache memory 840 (for all W-bits corresponding to the included Feature) and AND-reduces this word (bit-wise) in Clause Accumulator 860. When the Literal polarity bit 436 is enabled (i.e. logical 1 in the specific example), the Feature word is inverted prior to the AND-reduction. Upon receiving the instruction, and with the Clause change (i.e. Last Literal for the clause) bit 433 being equal to 1 , and processing the instruction via Instruction Decoder 830, Feature Fetching controller 850 and Clause Accumulator 860, the Clause value (i.e. all its W bits) is passed to the Inference Result Accumulator. At this point, if the Clause polarity bit 432 is high (i.e. logical 1 in the specific example), the class sum, accumulated in Inference Result Accumulator 870, corresponding to every active bit received from the Clause accumulator 860 is decremented; otherwise if this Clause polarity bit is low (logical 0), the class sum is incremented. The implementation of class sums in the Inference Result Accumulator 870 is carried out by means of counters 940- 949 described later. In addition, when the last included Literal (i.e. Clause change) bit is high (logical 1) the Clause accumulator 860 is cleared (i.e. set to all bits to one) for the next Clause computation. Note that due to the fact that W values of each ‘included’ Feature are processed in parallel, for each Clause, we have effectively W values accumulated in Clause Register 930 when the last included Literal has been processed. Those W values are passed to the Inference Result Accumulator 870, where their values can then be distributed to each counter 940-949, as will be shown in Figure 9, in order to increment or decrement each counter depending on the Clause polarity value.

[0070] As noted above (esp. w.r.t Figure 8), at the core of the TM accelerator 800 of the present disclosure, sits the Clause Accumulator 860 and the Inference Result Accumulator 870, both depicted in detail in Figure 9. As shown in Figure 9, the Clause Accumulator 860 comprises a Features input (which takes as an input W values of one Feature, which is shown by the use of data set 525, but rotated 90 degree clockwise, to show how each word comprises a whole horizontal row of input data 501 in Figure 5, for an included Feature), one or more (i.e. W of them) exclusive OR gates (XOR - see further explanation below) 910 operating on those included Features under control of a respective Literal parity bit, one or more (i.e. W of them) AND gates 920 operating on the W Features, and a Clause Register 930. By way of further explanation, the Features input 905 (which receives a W-bit word for each included Feature in Figure 5) would have been interpreted as ‘included’, as per Figure 4 (as a ‘1 ’ in TA include / exclude column 410), and so fetched from Features Cache 840 by Feature Fetching Controller 850, using the offset extracted from the corresponding instruction (item 430 in Figure 4). These retrieved Feature data may be inverted, as described above with reference to Figure 5, to form a set of Literals 906, which are in effect the values in item 535 of Figure 5 (also rotated clockwise to show how the Feature data words are computed concurrently).

[0071] Reverting to Figure 9, the two-input AND gates 920 (whose number is W, in correspondence to W-bit Features 905 passing through a set of XOR gates 910 whose number is also W, described below) operate in an iterative loop (iteration is carried out for a sequence of instructions corresponding to one Clause) with the Clause Register 930 to compute, iteratively, the values for each Clause (i.e. a multi-input AND function shown as 330 in Figure 3 is thus computed sequentially by using two-input AND gates 920 in this iteration process to produce the value of the Clause 370 in Figure 3). This process is shown in Figure 9 by ANDing each of the values in a row of the clockwise rotated date 535 together (exemplified by ellipse 927), to form a single value output (exemplified by ellipse 937) of end result data, i.e. the W values of the current Clause 935.

[0072] After the last included Literal (i.e. the Feature, whose whole set of Literals 906, for W data items, are derived from values from 905 having passed through an XOR gate 910 and hence either been uninverted if the Literal Polarity bit 436 is 0 or inverted if the Literal Polarity bit 436 is 1) has been accounted for in the AND gates 920, the outputs of these gates contain W values of the Clause, one for each data item. These W values of the Clause are then passed from the Clause Accumulator 860 into the Inference Result Accumulator 870, which comprises a W-bit bus 936 carrying the W values of the current Clause 935. The W values of the current Clause are distributed into individual 1 -bit signals, for use in respective counters 940-949 (where there are W counters = a counter per bit involved in W, so for example W may be 32 in the example). Put another way, for every instruction, the Clause Accumulator 860 ‘AND-reduces’ (i.e. uses AND gates) the Literal (in Wdata items) into the current Clause (again W values of the Clause, one per data item), in parallel as one action. The current Clause (W-bit) value is stored in Clause Register 930, and is being iteratively used for the AND-reduction of the Clause until the last Literal is fed into the Clause Accumulator 860. As shown in Figure 9, the ‘bespoke’ inversion of the Literal 906 (i.e. from Feature values 905), depending on its polarity (un-inverted if Literal polarity bit value is logical 0, and inverted if it is logical 1) means that the OR function (required as per 320 in Figure 3) can be conveniently implemented using a bitwise XOR applied also to the same Literal’s W data items in parallel. After this XOR-ing, this W-bit word is passed to the AND unit(s) 330. It is to be noted that the polarity of literal bit 436 can be used as a control bit to determine the value of the second input to the XOR unit 910 in Figure 9. Hence, this is a way to control whether the Boolean value of a Feature should be inverted (polarity = 1) or non-inverted (polarity = 0) before it is applied to the AND-based calculation of the Clause.

[0073] According to some examples, if the last Literal bit (i.e. Clause change bit 433) is enabled (logical 1), the iteration process for the current Clause computation is terminated, the W values of the current Clause are now ready to be transferred from the Clause Accumulator 860 to the Inference Clause Accumulator 870 via a W-bit bus 936. Taken individually these bits from the bus 936 are then used to increment or decrement, according to the Clause polarity control bit 432 value (i.e. if it is 0 then increment, and if it is 1 then decrement), each of the W counters 940-949, and this Clause polarity bit 432 is the same for all W such increment / decrement operations. It should be noted that the actual increment or decrement action depends on whether the value of the current Clause bit (one of the W bits) is equal to 0 or 1 . If the value of the current Clause bit is 0, then the increment or decrement by 0 will effectively mean that the value of the counter will remain the same. If the value of the current Clause bit is 1 , then the increment or decrement by 0 will effectively mean that the value of the counter x will be changed to x+1 for increment and to x-1 for decrement respectively. All these actions are performed by the W pairs of AND gates 872 (one of which has an inverting input to account for the Clause polarity bit 432 value being 0) in front of the counters 940-949. It should be noted that the W counters 940-949 are only enabled to perform their increment or decrement operations when the control bit Last Literal 433 is equal to 1 . This prevents the activation of the counters during the iteration process associated with the calculation of the current Clause values, until the last included Literal for the current Clause has been accounted for. Simultaneously, upon the action carried out when the Last Literal bit 433 is equal to 1 , the Clause Register 930 in the Clause Accumulator 860 is also cleared to allow the next Clause computation to occur. It is emphasized that increment or decrement of all W counters is controlled by the Clause polarity bit 432 (part of control bits - the left hand most bit in this example) received from the current instruction.

[0074] Figure 9 shows a detailed diagram of the Clause and Inference Result Accumulators. Here, W is the number of data points being processed in parallel (i.e. the Words of Figure 5). The Features are fetched from memory and the control signals (i.e. control bits - change in Clause 433, Clause polarity 432 and final Literal polarity bit 436) are fed from the instruction fetcher / decoder.

[0075] The classification results are stored in the Counters 940-949 in the Inference Result Accumulator 870. These counters 940-949 are controlled by two inputs, one being the Clause polarity bit value 432 (from control bits), which determines which operation, increment or decrement, should be performed on all W counters in parallel; the other input is the last Literal bit 433 (also from the control bits), which is used as an enable signal to the counters 940-949 to perform the increment or decrement operation every time the last Literal is processed. These counters 940-949 are accessible by the processors (e.g. RISC-V processor cores 710) via the system bus 720. The same interface may also be used to provide the Model and Features pointers and start the Clause computation. Actual implementation of the interface between the System Bus 720 and Model Cache 810, Features Cache 840 and Inference Result Accumulator 870 will depend on the type of the system architecture in use, and would be readily understood by the skilled person in the art of processor design. The fact that these communications links in Figure 8 are shown using bidirectional arrows indicates that the exchange of information via System Bus 720 with these different units is two way, e.g. involving addresses, data and read / write commands, plus control signals such as request and acknowledgement handshakes.

[0076] Examples of the present disclosure may provide significant performance improvements over known DNN based solutions - for example providing more accurate results, while also providing significant speedups (and consequential energy savings). In some examples, these speedups have been between 5-5500x, but more could be achievable.

[0077] Some examples of the present disclosure may be implemented by suitably programmed computer hardware, for example as shown in Figure 10.

[0078] Figure 10 is a block diagram 1000 illustrating components, according to some example embodiments, able to read instructions from a machine-readable or computer-readable medium (e.g., a non-transitory machine-readable storage medium) and perform any one or more of the methodologies discussed herein, or to provide hardware description, or software emulations, of any of the described apparatus to carry out the described methods. Specifically, Figure 10 shows a diagrammatic representation of hardware resources 1005 including one or more processors (or processor cores) 1010, one or more memory / storage devices 1020, and one or more communication resources 1030, each of which may be communicatively coupled via a bus 1040. The processors 1010 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP) such as a baseband processor, an application specific integrated circuit (ASIC), a cloud processing function (such as an AWS instance), another processor, or any suitable combination thereof) may include, for example, a processor 1012 and a processor 1014.

[0079] The memory / storage devices 1020 may include main memory, disk storage, or any suitable combination thereof. The memory / storage devices 1020 may include, but are not limited to any type of volatile or non-volatile memory such as dynamic random access memory (DRAM), static random-access memory (SRAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), Flash memory, solid-state storage device (SSD), magnetic storage based hard disk drive (HDD) media, etc.

[0080] The communication resources 1030 may include interconnection or network interface components or other suitable devices to communicate with one or more peripheral devices 1004 or one or more databases 1006 via a network 1008. For example, the communication resources 1030 may include wired communication components (e.g., for coupling via Ethernet, a Universal Serial Bus (USB) or the like), cellular communication components, NFC components, Bluetooth® components (e.g., Bluetooth® Low Energy), Wi-Fi® components, and other communication components.

[0081] Instructions 1050 may comprise software, a program, an application, an applet, an app, or other executable code for causing at least any of the processors 1010 to perform any one or more of the methodologies discussed herein. The instructions 1050 may reside, completely or partially, within at least one of the processors 1010 (e.g., within the processor’s cache memory), the memory / storage devices 1020, or any suitable combination thereof. Furthermore, any portion of the instructions 1050 may be transferred to the hardware resources 1005 from any combination of the peripheral devices 1004 or the databases 1006. Accordingly, the memory of processors 1010, the memory / storage devices 1020, the peripheral devices 1004, and the databases 1006 are examples of computer-readable and machine-readable media.

[0082] In some embodiments, the electronic device(s), network(s), system(s), chip(s) or component(s), or portions or implementations thereof, of Figures 10, or some other figure herein may be configured to perform one or more processes, techniques, or methods as described herein, or portions thereof.

[0083] In the foregoing, functions are described as modules or blocks, i.e. functional units that are operable to carry out the described function, algorithm, or the like. These terms may be interchangeable. Where modules, blocks, or functional units have been described, they may be formed as processing circuitry, where the circuitry may be general purpose processor circuitry configured by program code to perform specified processing functions. The circuitry may also be configured by modification to the processing hardware. Configuration of the circuitry to perform a specified functions may be entirely in hardware, entirely in software or using a combination of hardware modification and software execution. Program instructions may be used to configure logic gates of general purpose or special-purpose processor circuitry to perform a processing function.

[0084] Circuitry may be implemented, for example, as a hardware circuit comprising custom Very Large Scale Integrated, VLSI, circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. These hardware circuits may be derived from Register Transfer Level (RTL) code defining the described logic, and any other form of logic operable to carry out the described methods. Circuitry may also be implemented in programmable hardware devices such as field programmable gate arrays, FPGA, programmable array logic, programmable logic devices, A System on Chip, SoC, or the like.

[0085] Machine readable program instructions may be provided on a transitory medium such as a transmission medium or on a non-transitory medium such as a storage medium. Such machine readable instructions (computer program code) may be implemented in a high level procedural or object oriented programming language. However, the program(s) may be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language, and combined with hardware implementations. Program instructions may be executed on a single processor or on two or more processors in a distributed manner.

[0086] Examples provide a method of compressing Feature input data associated with a Clause computation in a Tsetlin Machine, comprising only providing instructions for Feature data sets having an Include value associated with the Clause computation, wherein each instruction includes an indication to a Literal to be used in the computation of the Clause, a data bit (value) indicating whether a value of the Literal to be included is positive or negative, and whether the Literal is the last Literal to be processed for the Clause.

[0087] Examples may further comprise: providing the Feature input data as a set of one or more Feature data sets, wherein each Feature data set comprises one or more Literal values for use in an inference operation in the Clause computation; and for each Feature data set having an Include value associated with the Clause computation, determining a position of the Feature data set in a contiguous sequence of the one or more Feature data sets from a base address; providing an offset value (435) indicative of the determined position; appending first portion of control bits to the offset value, to form a Tsetlin Machine instruction; wherein the first portion of control bits comprise: a Clause polarity bit; and a Clause end bit; and wherein the offset comprises a second portion of control bits, comprising a Literal polarity bit.

[0088] According to examples, a plurality of the Tsetlin Machine instructions relate to the Include values that are related to (i.e. used in) the same Clause computation.

[0089] According to examples, the method may further comprise aligning the one or more Feature data sets into a contiguous sequence of Features data sets. The contiguous sequence of the one or more Features data sets from a base address may comprise a positive and negated Literal for each Feature, wherein the positive Literal for a given Feature is positioned before the negated Literal for the same given Feature, and each given Feature is listed in ascending sequence. The first and second portion of the control bits may be positioned at one or more predetermined positions in the Tsetlin machine instruction. The Clause polarity bit and Clause end bit may be in the form of a prefix or postfix of the compressed Tsetlin machine instruction. The Literal polarity bit may be the LSB or MSB of the compressed Tsetlin machine instruction, depending on Endian type. The offset value may be a multi-bit binary value, and the base address is may be 0. According to some examples, each offset may be associated with one included Literal, whose polarity can be either positive (x) or negative (x-bar), therefore the included Literal’s polarity is represented by the final bit of the offset. According to examples, compressing the Feature input data may comprise compressing the Feature input data used by the TAs in terms of their relative addresses (i.e. offsets) within a set of TAs corresponding to each Clause.

[0090] According to some examples, the method may operate on binary format values and data, and the method may further comprise converting a binary format instruction into a hexadecimal instruction.

[0091] The variously disclosed example methods operate together to provide compression in a two-fold way. This is by virtue of a) compressing access to Literals for inference operations, by effectively only fetching values of Features from input data that are included by TAs (thereby exploiting the sparsity of includes in the TA outputs); and b) compressing the processing of multiple data items in the inference, by fetching W values of data at once and subsequently processing those W values are the same time, through calculation of Clauses having W values for each Clause, and accumulating their respective inference accumulators in parallel.

[0092] According to some examples, the Clause polarity bit indicates a contribution of the computed Clause to a classification, wherein positive comprises an increment and negative comprises a decrement (of a counter).

[0093] According to some examples, the Clause end bit indicates a last Literal to be processed in a Clause computation (also known as a Clause change bit, or a last Literal bit). The Literal polarity bit may indicate whether an associated Literal is formed from a negation of the Feature, and a value of 1 for the Literal polarity bit indicates the negation.

[0094] Examples also provide a method of inferencing in a Tsetlin Machine, comprising: receiving a plurality of Tsetlin Machine instructions, wherein each of the plurality of Tsetlin Machine instructions are indicative of at least one Feature data set in the input data having an Include value; and for each of the plurality of instructions indicative of at least one Feature data set in the input data having an Include value, the method comprises: decoding an offset value in the Tsetlin Machine instruction indicative of a relative position of the at least one Feature data set in a total number of Features operable for use in the Tsetlin Machine; decoding, in the instruction, a plurality of control bits; and processing the received input data in accordance with the offset and plurality of control bits to provide output data (i.e. one or more classifications). In some examples, the control bits comprise a Clause polarity bit, a Clause end bit (=last Literal bit) and a Literal polarity bit. The total number of Features data sets operable for use in the Tsetlin Machine = f. Each Tsetlin Machine instruction causes W values of input data (not instruction input data) to be fetched from memory for each Feature to form a Feature data word, i.e. each instruction operates on a parameterised number of independent data points, W. According to some examples, each Feature data set comprises W data bits fetched at once.

[0095] Examples of the disclosed inferencing method in a Tsetlin Machine exploits bit-parallelism of real-time input data to perform many inferences, concurrently.

[0096] There is also provided any Tsetlin apparatus comprising circuitry configured to carry out any of the disclosed methods. The input data may comprise one or more Feature data sets, wherein each Feature data set is suitably indicated (i.e. using indices and corresponding offset inside instructions) by the Include / Exclude values of TAs for a plurality of Literals potential in use during inference processing. The output data may comprise Clauses and / or Classifications.

[0097] There is also provided a Tsetlin Machine instruction indicative of an Included Literal, comprising: a plurality of control bits; and an offset value indicative of a position of an Include value for use in at least one Tsetlin Automata; wherein the control bits comprise a Clause polarity value, a change in Clause value and a Literal polarity value. The Clause polarity value, the change in Clause value and the Literal polarity value may all be Boolean values, i.e. bits. The Clause polarity value and the change in Clause value may be the Most Significant Bits in a Big-endian Architecture. The Clause polarity value and the change in Clause value may be Least Significant Bits in a Little-endian architecture. The Literal polarity value may be the Least Significant Bit in a Big-endian Architecture, or the Most Significant Bits in a Little-endian architecture.

[0098] There is also provided a processing circuit for a Tsetlin Machine, comprising: a Clause accumulator, wherein the Clause accumulator comprises: an input operable to receive a plurality of Features data words, each Features data word having a plurality of Feature data values, W; and circuitry operable to bit-wise AND- reduce the Feature data words (which may be formed into bitwise Clause values, i.e. into Clauses, W values per Clause); and an inference results accumulator, wherein the inference results accumulator receives the AND- reduced Feature data words (e.g. in the form of Clause values, W of them (935)), and comprises: a plurality of counters, wherein each counter increments or decrements under control of a respective Clause value of the AND-reduced Feature word, a Clause polarity bit, and a last Literal bit.

[0099] In examples of the processing circuit for a Tsetlin Machine, the circuitry operable to bit-wise AND- reduce the Features words comprises: a plurality of XOR function blocks, one per respective data value, each under control of a Literal polarity bit; a plurality of two-input AND gates, each having as input a one of the W data values in the Features data word; and a clause register operable to feed back into the plurality of AND gates under control of a last Literal bit. The inference results accumulator may further comprise a plurality, W, of sets of two AND gates, one set per counter, operable to configure the counter to increment or decrement.

[0100] In examples of the processing circuit for a Tsetlin Machine, the circuitry of the clause accumulator and inference result accumulator may be operable to ensure that if the last Literal bit is enabled, an iteration process for the current Clause computation is terminated, and W values of the current Clause are transferrable from the Clause Accumulator to the counters of the Inference Clause Accumulator via a W-bit bus.

[0101] Examples of the processing circuit for a Tsetlin Machine may further comprise a model fetching controller, operable to retrieve compressed Tsetlin Machine instructions from a model cache, wherein the Tsetlin Machine instructions each comprise values indicative of an Included Literal, an offset value and a plurality of control bits.

[0102] Examples of the processing circuit for a Tsetlin Machine may further comprise an instruction decoder, operable to extract the plurality of control bits and the offset value from the Literal value in the compressed Tsetlin Machine instruction. Examples of the processing circuit for a Tsetlin Machine may further comprise a Feature fetching controller, operable to fetch a bit-packed Included Features data word based upon the extracted offset value. The Feature data word may be a W-bit data word.

[0103] Examples of the processing circuit for a Tsetlin Machine may further comprise a model cache for storing the compressed TM instructions, and a Features Cache, for storing the Features data words, and a System bus for coupling the different parts of the processing circuitry.

[0104] Examples of the processing circuit for a Tsetlin Machine operate in the compressed domain, using instructions as formed by any of above described methods. Examples of the processing circuit for a Tsetlin Machine operates in an inference according to any of the above described methods.

[0105] According to examples, the instructions are operable to index each included Literal by means of an offset contained in, and decoded from, a respective instruction (and so the instructions as a whole set gives all Literals involved, by way of using multiple offsets). The Feature data sets are the data that are processed via the proposed inference architecture, and the offsets decoded from the instructions point (or indicate, or address) to those Features in the uncompressed Features data set 501 (in binary form)- i.e. to their W-bit rows, which correspond only to the included Literal positions in the column of include / exclude bits 410. This is by virtue of the encoding of those positions exactly in the offsets of the instructions, as per Figures 4 and 5, and their associated descriptions.

[0106] There is also provided a computer readable medium, comprising instructions, which, when executed by one or more processors, cause the one or more processors to carry out any of the described methods. There is also provided a computer readable medium, comprising instructions, which, when executed by one or more processors, cause the one or more processors to define construction of a plurality of logic gates or components, to realise the apparatus of described circuitry.

[0107] Examples provides a novel compression technique to reduce the memory footprint of trained TA and store it in the form of instructions, and a novel inference method that works in the compressed domain and uses stored TA instructions to process the input data. The inference disclosed method(s) is designed to exploit bitparallelism of real-time input data to perform many inferences, concurrently. Note there are compressed instructions, as shown in Figure 4, where any given Literal may or may not be included, operating on actual input data for classification, as per Figure 5. As will be appreciated, this mean the values shown in Figure 4 (especially 410), act as an index into the data that is actually being processed during inference. This is to say, there is complex routing in place regarding which data that is being used at any given point, and how that data is being manipulated by the overall circuits implementing the disclosed methods and apparatus.

[0108] While preferred embodiments of the present invention have been shown and described herein, it will be obvious to those skilled in the art that such embodiments are provided by way of example only. Numerous variations, changes, and substitutions will now occur to those skilled in the art without departing from the scope of the disclosure. It should be understood that various alternatives to the embodiments of the invention described herein may be employed in any combination in practicing the disclosure. It is intended that the following claims define the scope of the invention and that methods and structures within the scope of these claims and their equivalents be covered thereby.

Claims

Claims:1 . A method of compressing Feature input data associated with a Clause computation in a Tsetlin Machine, comprising only providing instructions for Feature data sets having an Include value associated with the Clause computation, wherein each instruction includes an indication to a Literal to be used in the computation of the Clause, a data bit indicating whether a value of the Literal to be included is positive or negative, and whether the Literal is the last Literal to be processed for the Clause.

2. The method of claim 1 , further comprising: providing the Feature input data as a set of one or more Feature data sets, wherein each Feature data set comprises one or more Literal values for use in an inference operation in the Clause computation; and for each Feature data set having an Include value associated with the Clause computation, determining a position of the Feature data set in a contiguous sequence of the one or more Feature data sets from a base address; providing an offset value (435) indicative of the determined position; appending first portion of control bits to the offset value, to form a Tsetlin Machine instruction; wherein the first portion of control bits comprise: a Clause polarity bit; and a Clause end bit; and wherein the offset comprises a second portion of control bits, comprising a Literal polarity bit.

3. The method of claim 1 or 2, wherein a plurality of the Tsetlin Machine instructions relating to the Include values are related to the same Clause computation.

4. The method of claim 2 or 3, wherein the first and second portion of the control bits are positioned at one or more predetermined positions in the Tsetlin machine instruction.

5. The method of claim 4, wherein the Clause polarity bit and Clause end bit are a prefix or postfix of the Tsetlin machine instruction.

6. The method of any preceding claim, wherein the Literal polarity bit is the LSB or MSB of the Tsetlin machine instruction.

7. The method of any preceding claim, wherein: the Clause polarity bit indicates a contribution of the computed Clause to a classification, optionally wherein a positive Clause polarity bit comprises an increment and a negative Clause polarity bit comprises a decrement; and / or wherein the Clause end bit indicates a last Literal to be processed in a Clause computation; and / or wherein the Literal polarity bit indicates whether an associated Literal is formed from a negation of the Feature, optionally wherein a value of 1 for the Literal polarity bit indicates the negation.

8. A method of inferencing in a Tsetlin Machine, comprising:receiving a plurality of Tsetlin Machine instructions, wherein each of the plurality of Tsetlin Machine instructions are indicative of at least one Feature data set in the input data having an Include value; and for each of the plurality of instructions indicative of at least one Feature data set in the input data having an Include value, the method comprises: decoding an offset value in the Tsetlin Machine instruction indicative of a relative position of the at least one Feature data set in a total number of Features operable for use in the Tsetlin Machine; decoding, from the instruction, a plurality of control bits; and processing the received input data in accordance with the offset value and plurality of control bits to provide output data.

9. The method of claim 8, wherein the control bits comprise a Clause polarity bit, a Clause end bit and a Literal polarity bit.

10. The method of claim 8 or 9, wherein the total number of Feature data sets operable for use in the Tsetlin Machine = f.11 . The method of any of claims 8 to 10, wherein each Tsetlin Machine instruction causes W values of input data to be fetched from memory for each Feature.

12. The method of any of claims 1 to 11 , wherein each Feature data set comprises W data bits fetched at once.

13. A Tsetlin Machine instruction indicative of an Included Literal, comprising: a plurality of control bits; and an offset value indicative of a position of an Include value for use in at least one Tsetlin Automata; wherein the control bits comprise a Clause polarity value, a change in Clause value and a Literal polarity value.

14. The instruction of claim 13, wherein the Clause polarity value and the change in Clause value are the Most Significant Bits in a Big-endian Architecture, optionally, wherein the Clause polarity value and the change in Clause value are Least Significant Bits in a Little-endian architecture.

15. A processing circuit for a Tsetlin Machine, comprising: a Clause accumulator, wherein the Clause accumulator comprises: an input operable to receive a plurality of Feature data words, each Feature data word having a plurality of Feature data values, W; and circuitry operable to bit-wise AND-reduce the Feature data words; and an inference results accumulator, wherein the inference results accumulator receives the AND-reduced Feature data words, and comprises: a plurality of counters, wherein each counter increments or decrements under control of a respective current Clause value of the AND-reduced Feature data word, a Clause polarity bit, and a last Literal bit.

16. The processing circuit of claim 15, wherein the circuitry operable to bit-wise AND-reduce the Feature data words comprises: a plurality of XOR function blocks, one per respective data value, each under control of a Literal polarity bit; a plurality of two-input AND gates, each having as input a one of the Wdata values in a respective Feature data word; and a Clause register operable to feed back into the plurality of AND gates under control of the last Literal bit.

17. The processing circuit of claim 15 or 16, wherein the inference results accumulator further comprises a plurality, W, of sets of two AND gates, one set per counter, operable to configure the counter to increment or decrement.

18. The processing circuit of any of claims 15 to 17, wherein the circuitry of the clause accumulator and inference result accumulator are operable to ensure that if the last Literal bit is enabled, an iteration process for the current Clause computation is terminated, and W values of the current Clause are transferrable from the Clause Accumulator to the counters of the Inference Clause Accumulator via a W-bit bus.

19. The processing circuit of any of claims 15 to 18, further comprising a model fetching controller, operable to retrieve Tsetlin Machine instructions from a model cache, wherein the Tsetlin Machine instructions each comprise values indicative of an Included Literal, an offset value and a plurality of control bits.

20. The processing circuit of any of claims 15 to 19, further comprising an instruction decoder, operable to extract the plurality of control bits and the offset value from the Literal value in the Tsetlin Machine instructions.21 . The processing circuit of any of claims 15 to 20, further comprising a Feature fetching controller, operable to fetch a bit-packed Included Features data word based upon the extracted offset value.

22. The processing circuit of any of claims 15 to 21 , wherein the Feature data word is a W-bit data word.

23. The processing circuit of any of claims 15 to 22, wherein the processing circuit operates in the compressed domain, using instructions as formed by any of methods 1 to 7.

24. A computer readable medium, comprising instructions, which, when executed by one or more processors, cause the one or more processors to carry out the method of any of claims 1 to 14.

25. A computer readable medium, comprising instructions, which, when executed by one or more processors, cause the one or more processors to define construction of a plurality of logic gates or components, to realise the apparatus of any of claims 15 to 23.

Images