Low-resource compute blocks for trained neural networks

Abstract

A computation block for performing an inference task by applying a plurality of low-resource computation operations to a binary input feature tensor to generate an integer feature tensor equivalent to the output of a multiplication-accumulation operation performed on a ternary weight tensor and the binary input feature tensor; and performing a comparison operation on the generated integer feature tensor with a comparison threshold to generate a binary output feature tensor. The plurality of low-resource computation operations are applied to the binary input feature tensor using a first weight tensor and a second weight tensor, each of which includes n binary elements and collectively represents the corresponding n elements of the ternary weight tensor.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. patent application Ser. No. 16 / 900,658, filed on Jun. 12, 2020, entitled “LOW RESOURCE COMPUTATIONAL BLOCK FOR A TRAINED NEURAL NETWORK,” which is incorporated herein by reference as if reproduced in its entirety. Technical Field

[0003] The present invention generally relates to artificial neural networks. More specifically, the present application relates to a computational block with quantized inputs and parameters for a layer of a trained neural network. Background Art

[0004] An artificial neural network (NN) is a computational system that mimics the workings of a biological brain. NNs consist of multiple layers (e.g., computational blocks), each layer comprising multiple computational units (called neurons), with connections between units in different layers. Each computational unit (e.g., each neuron) in the NN transforms data using a series of calculations. These calculations involve multiplying an initial value by some weight, adding the result to the other values fed into the same unit, adjusting the resulting value using the unit's bias, and then normalizing the output using an activation function. The bias is a factor that adjusts the value of the unit after all connections have been processed, and the activation function ensures that the value passed to subsequent units is within a tunable expected range. This series of calculations is repeated until the NN's final output layer generates a score or prediction related to a specific reasoning task. NNs can learn to perform reasoning tasks such as object detection, image classification, clustering, speech recognition, or pattern recognition. NNs typically do not require programming with any task-specific rules. Instead, NNs typically perform supervised learning tasks, building information from a dataset for which the correct answer is pre-determined. The NN then learns by tuning itself to find the correct answer on its own, thereby improving the accuracy of its predictions.

[0005] NNs are becoming larger (i.e., deeper) and more complex. This inevitably increases the number and size of layers in the NN, to the point where the cost of implementing the NN in software or hardware can be high. NNs increasingly rely on the use of specially designed, computationally powerful hardware devices to perform the operations of each layer of the NN (hereinafter generally referred to as NN operations), which include one or more processing units, accelerators (e.g., accelerators designed to perform certain operations of the NN), and supporting memory. In some examples, the dedicated processing unit, accelerator, and supporting memory are packaged in a single integrated circuit. The computationally powerful hardware devices required to perform the NN operations of deep NNs increase economic costs, as well as ancillary costs in terms of physical space and heating and cooling requirements.

[0006] Deep NNs are typically full-precision NNs built using full-precision layers composed of full-precision computational units. Full-precision layers perform NN operations on values that each have more than 8 bits, such as matrix multiplication, addition, batch normalization, and multiply-accumulate (MAC) operations (for example, individual elements in a feature tensor such as an input feature vector or feature map are each real values represented using 8 bits or more, and network layer parameters such as weights included in a weight tensor are also real values represented using 8 bits or more). NN operations performed in the context of full-precision layers are called high-bit NN operations. Specifically, each element output of a computational unit in a NN layer (for example, the i-th layer of a NN) is the weighted sum of all feature elements input to the computational unit, which requires each full-precision layer to perform a large number of multiply-accumulate (MAC) operations. Therefore, the high-bit NN operations performed by the full-precision NN layer are computationally intensive. This imposes limitations on the use of full-precision NNs in computationally constrained hardware devices.

[0007] Consequently, there is growing interest in NN compression techniques that can reduce the number of NN operations required to perform a particular inference task, thereby enabling the NN to be deployed in computationally constrained hardware devices, which, for example, can utilize less powerful processing units, less powerful (or no) accelerators, less memory, and / or less power than would be required to deploy an uncompressed NN. NN compression techniques can, for example, be applied to cost-effective, computationally constrained hardware devices that can be implemented to solve real-world problems in applications such as robotics, autonomous driving, drones, and the Internet of Things (IoT). Neural network quantization is a NN compression technique used to address the challenge of compressing trained NNs so that NN operations can be executed on resource-constrained hardware devices. NN quantization can, among other things, be used to replace high-bit MAC operations performed at NN layers with bitwise operations. Since the smallest bit-width MAC operator supported by general-purpose processing units is an 8-bit MAC operator, conventional quantization techniques reduce full-precision matrix multiplication operations to 8-bit matrix multiplication operations, which are performed by 8-bit MAC operators. Any NN quantization technique that reduces full-precision matrix multiplication operations to matrix multiplication operations with a bit width less than 8 bits (i.e., a number of bits less than 8 bits) must be performed by computational units that use bitwise operators or specific hardware devices. NN layers that use bitwise operations can be used as low-bit layers, where operations are performed using elements represented as 1-bit or 2-bit values.

[0008] Low-level neural network quantization techniques can generally be divided into two different categories: (i) weight quantization techniques that quantize real-valued weight tensors received by the NN layer but use real-valued input feature map tensors in the NN operations of the NN layer; (ii) weight and feature map quantization techniques that quantize real-valued weight tensors and input feature map tensors.

[0009] A limitation of low-bit quantization technology is that when a compressed trained NN is deployed for inference (i.e., making predictions on new data), the compressed trained NN suffers from deficiencies in one or more aspects of computational cost and memory requirements. For example, the compressed trained NN still requires a large number of multiply-accumulate (MAC) operations, which can be computationally and memory intensive.

[0010] Therefore, a low-bit computation block is needed that can be used to implement a layer of a NN and that can accurately generate binary feature map tensors for subsequent computation blocks of the NN (e.g., subsequent layers of the NN) after training, with lower computational cost and memory requirements than existing low-bit NN schemes. Summary of the Invention

[0011] The present invention provides a method and system for efficiently performing inference tasks on resource-constrained hardware devices. In a trained NN, an inference phase computation block uses low-resource computation operations to replace a large number of resource-intensive multiply-accumulate (MAC) operations. These low-resource computation operations may include basic logical and mathematical operations, such as AND operations, POPCOUNT operations, subtraction operations, and comparison operations. The methods and systems disclosed herein can use less computational resources, such as processing power and memory, to perform inference tasks and generate feature vectors in some applications. In an exemplary embodiment, the inference phase computation block enables a trained NN including one or more inference phase computation blocks to be deployed on a resource-constrained hardware device, thereby enabling it to use less processor computation, less memory, and / or less power than existing solutions.

[0012] According to a first exemplary aspect, a computer-implemented method is provided, comprising: applying a plurality of low-resource computing operations to generate an integer feature tensor, the integer feature tensor being equivalent to the output of a multiplication-accumulation operation performed on a ternary weight tensor and the binary input feature tensor; and performing a comparison operation on the generated integer feature tensor with a comparison threshold to generate a binary output feature tensor.

[0013] In at least some applications, low-resource computational operations are used to provide an output equivalent to a set of multiply-accumulate operations, enabling the method to be performed without using computationally and memory-intensive multiply-accumulate (MAC) operations. This can enable the method to be performed using computationally constrained hardware devices that have one or more of limited processing power, limited memory, or limited power, including, for example, edge devices. The method can, for example, enable neural networks (NNs) to be deployed in computationally constrained hardware devices that can, for example, use less powerful processing units, less powerful (or no) accelerators, less memory, and / or less power than previously required.

[0014] According to some examples of the above aspects, the generated integer feature tensor includes a set of m integer elements, each integer element has a value, and the value is equivalent to the dot product of the set of n ternary elements of the ternary weight tensor and the set of n binary elements of the binary input feature tensor.

[0015] According to an example of one or more of the above aspects, for each integer element of the integer feature tensor: applying the multiple low-resource computing operations using the first weight tensor and the second weight tensor, the first weight tensor and the second weight tensor each including n binary elements and jointly representing the corresponding n elements of the ternary weight tensor, applying the multiple low-resource computing operations includes, for each integer element of the integer feature tensor: applying a first bitwise logical AND operation between the n binary elements of the first weight tensor and the n binary elements of the binary input feature tensor; applying a second bitwise logical AND operation between the n binary elements of the second weight tensor and the n binary elements of the binary input feature tensor; generating the integer element based on the number of bits of a predetermined value included in the output of the first bitwise logical AND operation and the output of the second bitwise logical AND operation.

[0016] According to an example of one or more of the above aspects, applying the multiple low-resource computing operations includes, for each integer element of the integer feature tensor: applying a first POPCOUNT operation to count the number of bits of the predetermined value included in the output of the first bitwise logical AND operation; applying a second POPCOUNT operation to count the number of bits of the predetermined value included in the output of the second bitwise logical AND operation; wherein, generating the integer element includes applying a subtraction operation between the output of the first POPCOUNT operation and the output of the second POPCOUNT operation to output the integer element.

[0017] According to one or more examples of the above aspects, the n elements of the ternary weight tensor are decomposed into n binary element pairs to generate the n binary elements of the first weight tensor and the n binary elements of the second weight tensor.

[0018] According to one or more examples of the above aspects, applying the multiple low-resource computing operations and performing the comparison operation are performed in a layer of a second neural network trained to perform an inference task, and the method also includes training the first neural network to perform the inference task by learning network parameters of the first neural network including the ternary weight tensor.

[0019] According to an example of one or more of the above aspects, training the first neural network includes: using the network parameters to infer an output tensor for each of a plurality of input feature tensors of a training data set; determining a cost of the network parameters based on the inference; updating the network parameters using gradient descent and backpropagation, wherein the inference, the determination, and the update are repeated until the cost of the network parameters reaches an optimization threshold.

[0020] According to one or more examples of the above aspects, the layer of the first neural network corresponds to the layer of the second neural network, and the layer of the first neural network is used to: apply a multiplication-accumulation operation using the ternary weight tensor and the binary input feature tensor in the training data set to generate a training integer feature tensor; apply a batch normalization operation, an activation operation, and a binarization operation to the generated training integer feature tensor to generate a training binary output feature tensor, the batch normalization is based on a batch normalization parameter set, wherein the learned network parameters include the batch normalization parameters, and the comparison threshold is a tensor of m integer thresholds calculated based on the learned batch normalization parameters.

[0021] According to one or more examples of the above aspects, the batch normalization parameters include a scaling factor tensor and a bias tensor, and training the first neural network includes initializing one or more of the network parameters to maintain a variance of values within the network parameters within a target variance range.

[0022] According to an example of one or more of the above aspects, during training of the first neural network, the ternary weight tensor is generated by quantizing the values of the corresponding real-valued weight tensor using a non-differentiable ternary operation, the binarization operation applies a non-differentiable binarization operation, and during backpropagation, the non-differentiable ternary operation and the non-differentiable binarization operation are each approximated by a corresponding differentiable operation.

[0023] According to a second exemplary aspect, a processing unit is provided, which is configured to execute the method of one or more of the above aspects.

[0024] According to another exemplary aspect, a processing device is provided, the processing device including a memory operably connected to the processing device and storing executable instructions that, when executed by the processing device, cause a processing unit to implement a computational block of a trained neural network for performing an inference task. The computational block is configured to: apply a plurality of low-resource computational operations to generate an integer feature tensor, the integer feature tensor being equivalent to an output of a multiplication-accumulation operation performed on a ternary weight tensor and the binary input feature tensor; and perform a comparison operation on the generated integer feature tensor with a comparison threshold to generate a binary output feature tensor.

[0025] According to another exemplary aspect, a computer-readable medium is provided, the computer-readable medium storing instructions for causing a processing unit to implement a computational block of a trained neural network for performing an inference task, the computational block being configured to: apply a plurality of low-resource computational operations to generate an integer feature tensor, the integer feature tensor being equivalent to an output of a multiplication-accumulation operation performed on a ternary weight tensor and the binary input feature tensor; and perform a comparison operation on the generated integer feature tensor with a comparison threshold to generate a binary output feature tensor.

[0026] According to another exemplary aspect, a computer program is provided, the computer program storing instructions for causing a processing unit to implement a computational block of a trained neural network for performing an inference task, the computational block being configured to: apply a plurality of low-resource computational operations to generate an integer feature tensor, the integer feature tensor being equivalent to an output of a multiplication-accumulation operation performed on a ternary weight tensor and the binary input feature tensor; and perform a comparison operation on the generated integer feature tensor with a comparison threshold to generate a binary output feature tensor. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Reference will now be made, by way of example, to the accompanying drawings which show exemplary embodiments of the present application, in which:

[0028] Figure 1 is a computational graph representation of a training phase computational block of a layer of a NN that can be used to implement the NN model while training the NN model, provided by an exemplary embodiment;

[0029] Figure 2 Shown Figure 1 Further representation of the training phase computational blocks, and examples of ternary weight tensors provided by exemplary embodiments;

[0030] Figure 3 is a computational graph representation of computational blocks of the inference phase of the training layer of a NN provided by an exemplary embodiment;

[0031] Figure 4 It is a truth table representation of bitwise operations;

[0032] Figure 5 Two 1-bit weight tensors representing a ternary weight tensor are shown;

[0033] Figure 6 It is used to implement Figure 2 A block diagram of the low-resource computing operations of the inference phase computing block;

[0034] Figure 7 The exemplary embodiment provides a method for using Figure 3 An exemplary method for performing reasoning tasks using low-resource computing operations of B;

[0035] Figure 8 graphically represents a differentiable function with three different parameters provided by an exemplary embodiment;

[0036] Figure 9 is available for execution including Figure 1 or Figure 3 A block diagram of an exemplary processing system for machine-readable instructions of a NN of one or more computational blocks is shown.

[0037] Like reference numerals are used in different drawings to denote like components. DETAILED DESCRIPTION

[0038] The present invention relates to a neural network (NN) comprising one or more layers implemented as inference-phase computational blocks that use a set of low-resource computational operations to replace resource-intensive computations used in corresponding training-phase computational blocks. As used herein, an inference computational block that uses "low-resource computational operations" is a computational block that performs a nonlinear inference function on an input feature tensor based on a set of learning parameters without applying multiplication and accumulation (MAC) operations. In an exemplary embodiment, the low-resource computational operations include basic logical and mathematical computational operations, such as AND operations, POPCOUNT operations, subtraction operations, and comparison operations, and exclude computationally intensive operations, such as matrix multiplication operations that rely on MAC operations. As used herein, a "tensor" refers to an array of elements, such as a vector (e.g., a one-dimensional array, such as a row array or column array including multiple scalar feature elements) or a matrix or map (e.g., a multidimensional array, each dimension including multiple scalar feature elements). A "feature tensor" refers to an array of elements that represents features of an object being processed. For example, the object being processed can be an instance of image data, audio data, digital data, or other forms of structured or unstructured data. A "weight tensor" is an array of scalar weight elements, such as a weight vector (e.g., a one-dimensional array, such as a row array or column array including multiple scalar weight elements) or a weight matrix (e.g., a multi-dimensional array, each dimension including multiple scalar weight elements). In exemplary embodiments, a "low-order element" refers to a scalar element represented using 1 or 2 bits. In some exemplary embodiments, a "low-order feature element" is represented by a 1-bit value and a "low-order weight element" is represented by a 2-bit value. In at least some NN implementations, an inference phase computation block that applies low-resource computation operations may help reduce computational costs by simplifying computational operations and / or reducing memory storage requirements compared to a full-process scheme or a bit-by-bit scheme that relies on multiplication operations, accumulation operations, and activation operations. In at least some applications, reducing computational costs can improve performance latency and power efficiency of hardware devices.

[0039] In an exemplary embodiment, a trained NN including one or more layers implemented as corresponding inference-stage computational blocks is deployed for execution on a hardware device to perform a specific inference task. For example, the inference task may be to classify an object as belonging to one of a set of possible candidate categories. The disclosed inference-stage computational blocks perform low-resource computational operations to process an input quantized feature tensor (i.e., each feature element of the input tensor is represented as a corresponding 1-bit value). The inference-stage computational blocks process the input quantized feature tensor based on trained weight parameters defined in a 2-bit width quantized weight tensor (i.e., each weight element of the weight tensor is represented as a 2-bit value). In at least some embodiments, a trained NN including one or more of the disclosed inference-stage computational blocks can be deployed for execution by a computationally constrained hardware device (e.g., a device with one or more of limited processing power, limited memory, or limited power supply). In at least some applications, a NN including the inference-stage computational blocks described herein can perform inference tasks in a manner that accurately approximates the performance of a full-precision NN, thereby alleviating traditional issues such as the extensive use of MAC operations and / or the loss of accuracy that can occur when existing bit-wise NN structures are used to provide discrete approximations of full-precision NNs.

[0040] In an exemplary embodiment, the NN layer including the inference phase computing blocks described in the present invention can be implemented using low-resource computing operations (e.g., 1-bit AND operations, 1-bit POPCOUNT operations) (the low-resource computing operations can be supported using low-bit memory storage) and other low-resource computing operations (e.g., basic subtraction operations and comparison (CMP) operations). This approach enables a trained NN model (the trained NN model includes one or more layers, each layer including a corresponding inference computing block) to be loaded into the cache memory of a processing unit of a hardware device, thereby allowing a processing unit with limited computing power (e.g., a microcontroller used in a resource-constrained device (e.g., IoT devices, mobile devices, and other edge devices)) to perform high-speed operations.

[0041] In some exemplary embodiments, when training a NN to learn parameters of one or more training phase computational blocks, feature tensor and weight tensor quantization functions are used to generate corresponding low-order features and weight elements. During backpropagation, the quantization function is approximated by a differentiable function (e.g., a gradient estimate). Using differentiable gradient estimates can reduce gradient noise in backpropagation during NN training. In at least some exemplary embodiments, when training a NN to learn parameters, a scaling factor for a batch normalization (BN) operation of the NN is initialized to improve the accuracy of the parameters of the trained deployed NN.

[0042] Computational blocks for training purposes

[0043] Figure 1 is a computational graph representation of a training phase computation block 100 that can be used to implement a layer of a NN during the training phase, as provided by an exemplary embodiment. In the embodiment described herein, the training phase computation block 100 represents the i-th fully connected layer of the NN. During the forward pass, the input to the training phase computation block 100 is the quantized feature tensor X i bin In an exemplary embodiment, the quantized feature tensor may be represented as a feature vector of n 1-bit elements (eg, each element having a binary value of “0” or “1”), such that In the present invention, uppercase letters (e.g., "X", "W", "Z") are used to represent tensors (e.g., multi-element vectors or arrays) and lowercase letters (e.g., "x", "w", "z") are used to represent individual scalar elements that make up a tensor. The training phase computation block 100 is used to perform a set of low-order computation operations to process the quantized feature tensor. And output the corresponding quantized feature tensor In an exemplary embodiment, the output quantized feature tensor can be represented as a feature vector of m 1-bit elements (e.g., each element has a binary value of "0" or "1") such that As will be explained in more detail below, the low-order computational operations performed by the training phase computation block 100 include matrix multiplication operations (MatMul bit )206, batch normalization (BN) operation 208, activation function (e.g., PreLU) 210 and binarization operation 212.

[0044] In some cases, the layer i training phase computation block 100 is the first computation block in the NN, for example, where i=1, and the training phase computation block 100 implements the first hidden layer immediately following the input layer of the NN. In this case, an initial quantization operation, i.e., a binarization operation 202, is required that quantizes the features included in the real-valued input feature tensor. to output the binary input feature tensor The binarization operation 202 converts the real-valued input feature tensor X into i Each real-valued element x in is quantized to one of two possible values (1, 0).

[0045] In some embodiments, the binarization operation 202 may be mathematically represented by the following equation (1):

[0046]

[0047] Where, as mentioned above, X irepresents a real-valued input feature tensor (e.g., a real-valued n-element feature vector), Represents the binarized version of the tensor (e.g., the corresponding n-element feature vector of a 1-bit binary value).

[0048] The quantized feature tensor Provided for matrix multiplication operation (MatMul bit )206. Matrix multiplication operation (MatMul bit )206 in the ternary weight tensor With the input quantized feature tensor Perform a bitwise multiply-accumulate operation between to generate a tensor Z of m integer-valued elements i int . Ternary weight tensor is an m×n set of 2-bit width weight parameters learned during the training of the bitwise computation block 100. More specifically, the ternary weight tensor From the real-valued weight tensor W i The real-valued weight tensor is derived from the weight elements w learned during training in the training phase computation block 100. i As mentioned above, n is the binary input tensor In the present description, m is the binary output tensor generated by the bitwise computation block 100. size.

[0049] In this regard, the ternary operation 204 converts the real-valued weight tensor Quantized into ternary weight tensors Specifically, the ternary operation 204 converts each real-valued weight element w a，b Quantized into the corresponding ternary weight element w ter(a，b) , the ternary weight element is one of three possible values (1, 0, –1) (where 1≤a≤m, 1≤b≤n).

[0050] In some embodiments, the ternary weight tensor generated by the ternary operation 204 is It is expressed mathematically by the following equation (2):

[0051]

[0052] Among them, as mentioned above, W i represents a real-valued weight tensor, Represents a ternary weight tensor.

[0053] As explained in more detail below, in some examples, during back-propagation, the ternary operation 204 is approximated using a differentiable function (eg, a gradient estimate that approximates the non-differentiable discrete quantization function represented by equation (2)).

[0054] The matrix multiplication operation 206 performs a bitwise matrix multiply-accumulate operation, as shown in equations (3A) and (3B):

[0055]

[0056]

[0057]

[0058]

[0059] As mentioned above, is an integer-valued feature tensor In the case of a fully connected layer, the feature vector Each integer element of is a ternary weight tensor The set of n weights and quantized feature tensors for the corresponding rows The dot product of n elements.

[0060] In an exemplary embodiment, the integer feature tensor is provided to a BN operation 208, which may be implemented according to known batch normalization techniques. The BN operation 208 is configured by trainable BN parameters, namely a trainable bias tensor β and a trainable scaling factor tensor γ. As is known in the art, batch normalization can improve the training of a NN when used instead of a non-batch bias vector addition operation. The BN operation 208 can be mathematically represented by equation (4):

[0061]

[0062] Among them, μ B is the mini-batch mean of X (X is an integer-valued vector for BN operation groups or mini-batches, and), σ B is the mini-batch variance of X, γ is a trainable scaling factor tensor, β is a trainable bias tensor, and ∈ is a very small number to avoid division by zero errors. In an exemplary embodiment, each of the m outputs of the bitwise computation block will have a corresponding scaling factor and bias, so that γ and β are each tensors (e.g., vectors) of m elements.

[0063] exist Figure 1 In the example, the output of the BN operation 208 is a real-valued feature tensor This real-valued feature tensor is provided to the activation function 210. The combined operation of the matrix multiplication operation 206, the BN operation 208, and the activation function 210 can be mathematically represented by equation (5), where σ represents the activation function 210:

[0064]

[0065] Output tensor X i+1 is a real-valued output vector,

[0066] In one exemplary embodiment, the activation function σ210 is implemented using a parametric rectified linear unit (PReLU). Examples of suitable parametric ReLUs are described in Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” CoRR, abs / 1502.01852, 2015. In another exemplary embodiment, the activation function σ210 is implemented using a leaky ReLU. Examples of suitable leaky ReLUs are described in Andrew L. Maas, Awni Y. Hannun, and Andrew Y. Ng, “Rectifier nonlinearities improve neural network acoustic models,” in ICML Workshop on Deep Learning for Audio, Speech, and Language Processing, 2013. In an exemplary embodiment, when the bitwise NN block 100 is in the training phase, using Leaky ReLU or parametric ReLU as the activation function σ 210 may help improve the accuracy of back-propagation.

[0067] As mentioned above, the output tensor X i+1 is a real-valued output vector. To provide a binary output, a binarization operation 212 is applied to convert the real-valued output vector Quantized into the corresponding tensor of 1-bit binary elements, that is, the output feature tensor The binarization operation 212 may be the same as the binarization operation 202 described above.

[0068] Output feature tensor can be used as the input feature tensor for a subsequent NN layer (i.e., layer i+1) implemented using another computational block 100.

[0069] refer to Figure 2, the training phase computation block 100 can be considered as a set of “m” neurons or computational units 250(1) to 250(m), which together form the i-th fully connected layer of the NN. Figure 2 As shown, the matrix multiplication operation 206 includes m dot product operations 252(1) to 252(m). In this regard, each computation unit 250(a) (where a is an integer from 1 to m) computes a corresponding integer feature element The corresponding integer feature element is composed of a ternary weight tensor The "m" lines of the quantized input feature tensor define the corresponding n-element weight vector Therefore, in the fully connected layer, n weight elements are added at each of the dot product operations 252(1) to 252(m). to Different sets of are applied to n 1-bit binary input feature elements to As an example, if Figure 2 As shown, the integer characteristic element is a m-row by n-column ternary weight tensor The first row 216(1) is the quantized feature tensor with n elements The dot product of . Integer feature elements is a ternary weight tensor The last row 216(m) is the quantized feature tensor with n elements The integer feature tensor consists of m integer elements. to m integer elements to Each of them performs batch normalization, activation and binarization operations to obtain the output binary vector binary feature elements Each of is generated by a corresponding computation unit 250(a).

[0070] The forward propagation of each computation unit 250(a) of the computation block 100 during the training phase can generally be represented by the function shown in Equation (6A):

[0071]

[0072] Where "a" is an integer between 1 and m, is composed of a ternary weight tensor The ternary weight vector consisting of the n weight elements in the a-th row of Represents the elements of the scaling factor tensor γ and bias tensor β corresponding to computational unit 250(a).

[0073] like Figure 1 and Figure 2 As shown in FIG, when the bitwise calculation block 100 performs forward propagation, the output binary feature tensor of the calculation block 100 in the training phase is It can be expressed mathematically by the following equation (6B):

[0074]

[0075]

[0076] Among them, BatchNorm is the BN operation 208, σ is the activation function 210, and Binarize is the binarization operation 202.

[0077] In an exemplary embodiment, the training phase computation block 100 is trained using supervised training to learn the nonlinear function Specifically, it includes the weight tensor W i The weight element w in i , trainable scaling factor tensor γ i The scaling factor elements in and the trainable bias tensor β i The parameter set for the bias element in is learned through an iterative training process that measures the NN performance on a training dataset with respect to the parameter set, determines the cost of the parameters, updates the parameters using backpropagation, and repeats until an optimal parameter set is achieved. Further training details are described below.

[0078] After the computation block 100 is trained in the training phase, the learned parameter tensor W i , γ i , β i Can be used with any computational block having the same structure as the training phase computational block 100 to implement nonlinear functions

[0079] Computational blocks for inference purposes

[0080] Despite Figure 1 The training phase computing block 100 shown can eliminate some high-order computations during NN training, but the structure of the training phase computing block 100 still relies on a large number of MAC operations (e.g., matrix multiplication operations), as well as high-order BN operations and activation functions, which can be computationally and memory-intensive. Therefore, when deployed on computationally constrained devices such as edge devices, the training phase computing block 100 may not be suitable for continuous reasoning operations on new data.

[0081] Thus, exemplary embodiments of the present invention describe methods that can be used to implement nonlinear functions based on parameters learned by computation block 100 during the training phase. In an exemplary embodiment, the alternative computational block structure is used to replace the computational operations of the computational block 100 during the training phase with a plurality of low-resource computational operations. For example, after training (i.e., when backpropagation is no longer required), for inference purposes, the matrix multiplication operation 206 may be alternatively represented as the input quantized feature tensor with a ternary weight tensor Specifically, multiple low-resource computation operations can be applied to generate integer feature tensors. This integer feature tensor is equivalent to the matrix multiplication operation 206 on the ternary weight tensor and the binary input feature tensor The output of the multiply-accumulate operation performed.

[0082] in this regard, Figure 3 An exemplary embodiment of an inference phase computation block 200 is shown that can be used with parameters learned during training of the training phase computation block 100. The inference phase computation block 200 replaces the matrix multiplication operation 206 with a set of low-resource computation operations 256 described below. In addition, the inference phase computation block 200 replaces the BN operation 208, the activation operation 210, and the binarization operation 212 with a simple Boolean comparison operation 258 described in more detail below.

[0083] In the training phase calculation block 100, the input quantized feature map tensor is and a ternary weight tensor Provided to the matrix multiplication operation 206, which applies a multiply-accumulate operation to generate an intermediate output tensor This is mathematically expressed by equations (3A) and (3B) as described above.

[0084] In the case of computing block 100 during the training phase, given division Figure 4 Shows w i ter(a，b) Element-wise truth table 400 of the scalar product of (where 1≤a≤m, 1≤b≤n). To use the low-resource compute operation 256 of the inference phase compute block 200, the ternary weight tensor is mapped using a mapping function 260. The learning values of are each decomposed into corresponding binary value pairs. In this regard, Figure 5 The operation of the mapping function 260 is shown, where each weight element w i ter(a，b) is decomposed into two 1-bit elements, denoted as w i -1 and w i+1 For example, w i ter =-1 can be mapped to w i -1 =1 and w i +1 =0;w i ter =0 can be mapped to w i -1 =0 and w i +1 =0;w i ter =1 can be mapped to w i -1 =0 and w i +1 =1. 1-bit positive weight element w i +1 can be collectively represented as a 1-bit positive weight tensor 1 negative weight element w i -1 can be collectively represented as a 1-bit negative weight tensor Therefore, equations (3A) and (3B) can be equated to a 1-bit positive weight tensor With the input quantized feature tensor Inference phase operations between , and 1-bit negative weight tensor With the input quantized feature tensor The inference phase operations between.

[0085] Therefore, regarding the training phase, the ternary weight tensor learned by block 100 is calculated as Decomposed into positive weight tensors by mapping function 260 operation and negative weight tensors They are stored as trained weight parameters for the inference phase computation block 200. The low resource computation operation 256 operates on a 1-bit positive weight tensor. With binary input feature map tensor Performs a set of bitwise logical and basic mathematical operations between 1-bit negative weight tensors With binary input feature map tensor Performs a set of logical operations between them.

[0086] Will refer to Figure 6 The low resource computing operation 256 is explained in more detail. The low resource computing operation 256 includes two AND operations 302 (1), 301 (2) (generally referred to as AND operations 302), two POPCOUNT operations 304 (1), 304 (2) (generally referred to as POPCOUNT operations 304) and a subtraction operation 306. Specifically, a 1-bit positive weight tensor is used. and 1-bit negative weight tensor To represent the ternary weight tensor exist and The matrix multiplication (Matmulbit) operation 206 performed between can be replaced by a plurality of low-resource computing operations (AND operation 302, POPCOUNT operation 304, subtraction operation 306) represented in the following equation (7).

[0087]

[0088] Where: (i) Each AND() corresponds to Figure 6 (ii) each POPCOUNT() corresponds to a corresponding bitwise popcount operation 304(1), 304(2), wherein the popcount operation 304(1) is for The number of 1s in the , popcount operation 304 (2) Count the number of 1s in the is a positive weight tensor, is a negative weight tensor; (iii) is an output tensor of integers (eg, an output vector of m integer elements) resulting from the subtraction operation 306 .

[0089] As shown in Equation (7), the generated integer feature tensor Contains m integer elements A set where each integer element Has a value equivalent to the ternary weight tensor n ternary elements of The set of binary input feature tensors n binary elements of The dot product of the set of . For each integer element A plurality of low-resource computational operations 256 are applied using a first weight tensor and a second weight tensor, each of which includes n binary elements, i.e., positive weight tensors n binary elements of (e.g., row a ) and negative weight tensors n binary elements of (e.g., row a ), and together represent the ternary weight tensor The corresponding n elements of (for example, row a ). Applying multiple low-resource computation operations 256 includes, for each integer element (i) In the positive weight tensor n binary elements of With binary input feature tensor n binary elements of Applying a first bitwise logical AND operation (e.g., AND operation 302()) between the negative weight tensor n binary elements of With binary input feature tensor n binary elements of (iii) generating an integer element based on the number of bits (e.g., the number of "1"s) of the predetermined value included in the output of the first AND operation and the output of the second AND operation. Specifically, if Figure 6 As shown in Equation (7), the integer feature tensor Each integer element of The calculation of includes: applying a first POPCOUNT operation 304(1) to count the number of bits (e.g., the number of "1's") of a predetermined value included in the output of the first AND operation 302(1); applying a second POPCOUNT operation 304(2) to count the number of bits of the predetermined value included in the output of the second AND operation 302(2). Then, an integer element is generated by applying a subtraction operation 306 between the output of the first POPCOUNT operation 304(2) and the output of the second POPCOUNT operation 304(2). To get the integer elements In an exemplary embodiment, subtraction operation 306 performs a simple integer subtraction. Integer subtraction may use less computing power and memory than floating point subtraction. In some examples, subtraction operation 306 is implemented by using an addition operation to add the output of POPCOUNT operation 304() to a changed sign version of the output of POPCOUNT operation 304(2). and negative weight tensors Perform low-resource computation operations 256 on all m rows of to obtain an integer feature tensor

[0090] For illustration purposes, the binary matrix multiplication operation 206 will be provided in the context of the ath neuron or computational unit of the training phase computational block 100 and the inference phase computational block 200 (e.g., ) and low resource computing operation 256. In the example shown, n=4, ternary weight vector Among them, the ternary weight vector Corresponding to the ternary weight tensor Thus, the training phase computes the integer feature element of the ath neuron of block 100 (e.g., computation unit 250(a)) is the dot product

[0091] In the case of low resource computing operation 256, Breaks down to: and The result is (0, 0, 0, 0), The result is (0, 0, 1, 1). The result is 0, The result of is 2. Therefore, the intermediate integer output of the a-th neuron of the inference phase computation block 200 (eg, the computation unit corresponding to unit 250(a)) is

[0092] Thus, the example above illustrates that a binary matrix multiplication operation 206 and a low-resource compute operation 256 will generate the same output, an integer feature tensor

[0093] In the inference phase computation block 200, the integer feature tensor is a vector of m integers, which is then converted into a binary output feature vector by comparison operation 308 In an exemplary embodiment, the comparison operation 308 may be implemented, for example, using a CMP(x, y) instruction that returns a Boolean result of whether the integer x is less than the integer y. For example, the comparison operation 308 may be used to convert the tensor Integer value of Each of the quantized features is converted into an output tensor The corresponding binary value element of As shown in equation (8):

[0094]

[0095] As described above, in the inference phase computation block 200, the comparison operation 308 replaces the BN operation 208, the activation operation 210, and the binarization operation 212 of the training phase computation block 100. As will now be explained in the following paragraphs, the threshold value is calculated by the threshold calculation function 310 based on the trainable scaling factor γ learned during the training of the training phase computation block 100. i and a trainable bias β i The parameter value is the comparison threshold value calculated by the calculation block 200 in the inference phase. The value of .

[0096] Reference again Figure 1In the training phase, the output tensor of the training phase calculation block 100 is calculated by implementing the BN operation 208, the activation function 210 and the binarization operation 212. As shown below:

[0097]

[0098] Whether the output of the activation function σ210 is positive or negative will be determined based on BatchNorm(Z i ) (e.g., the output of BN operation 208) is positive or negative. Therefore, by incorporating the BatchNorm equation (4) discussed above into equation (9), equation (9) can be transformed into the following equation (10):

[0099]

[0100] where, as mentioned above, the scaling factor tensor γ i and the bias tensor β i is the learning parameter of the BN operation 208. The parameter μ of the BN operation 208 B (Mini-batch mean of X, where X is the intermediate vector provided to BN operation 208 groups or mini-batches) and the parameter σ B (the mini-batch variance of X) is also known from the training of the bitwise computation block 100. As mentioned above, the value of ε is predefined as a small constant to avoid division by zero errors during forward propagation.

[0101] For the sake of The threshold value of Z is solved. i The value of can be expressed by the following equation (11):

[0102]

[0103] Therefore, the equation Defines the binary output feature map tensor used to generate the i-th computational block of the NN The threshold tensor (e.g., threshold i ). As shown in equation (11), threshold i The value of may be based on the parameters μ of the BN operation 208 learned during the training of the bitwise computation operation 100 B , β i , σ B , γ i As mentioned above, the value of ε is predefined as a small constant to avoid division by zero errors during forward propagation.

[0104] The scaling factor tensor γ for the BN operation 208 iFor all cases where the elements of are greater than zero, the output tensor of block 100 is calculated during the training phase as represented by equation (11) It can be described by the following equation (13):

[0105]

[0106] Among them, the scaling factor tensor γ i >0.

[0107] In an exemplary embodiment, to ensure that the scaling factor tensor γ i is greater than zero, in the scaling factor vector γ i The trained scaling factor elements in are less than zero (e.g., ), the scaling factor element is used before being used to calculate the parameters for the inference calculation model 200. and a ternary weight tensor The corresponding row to Multiply by -1.

[0108] It should be noted that in equation (8), the vector The elements of are integers. In this regard, refer to Figure 6 , considering the first scaling factor The output of the popcount operation 304() and the second scaling factor applied to the negative weight tensor Applied to the case of the output of the negative weight tensor popcount operation 304 (2), the output of the subtraction operation 306 can be expressed by equation (14):

[0109]

[0110]

[0111] In equations (7) and (8), is a tensor of m integers, and in equation (14), Zi is a tensor of m real numbers. However, when and When both are equal to 1, the tensor Z in equation (14) i The tensor equal to equation (7) Such that the output of the subtraction operation 306 is the integer-valued tensor of equation (7) In the case of low resource computing operation 256, and is considered equal to 1 because The intermediate output is always an integer Therefore, in an exemplary embodiment, threshold iThe value of (calculated as described below) can be rounded to an integer so that Using integer values for thresholds may reduce the computing power and memory required to perform comparison operation 308 relative to using real-valued thresholds.

[0112] In some embodiments, the parameter μ is uniquely trained for each of the m computational units forming the training phase computational block 100. B , β i 、 γ i One or more of the threshold value calculation function 310 is calculated by the inference phase calculation block 200. is an m-dimensional vector of single integer thresholds applied to the m neurons or computational units of the inference phase computational block 200, respectively.

[0113] Training and deployment

[0114] Figure 7 is a flow chart of an exemplary method 700 for training a NN including the training phase computation block 100 and deploying the trained NN including the inference phase computation block 200 .

[0115] like Figure 7 As shown, the method 700 includes initializing the network parameters (e.g., initial weight tensor W) of the computing block 100 for the training phase. i , scaling factor tensor γ i and the bias tensor β i ) pre-training action (action 702). In an exemplary embodiment, the network parameters are initialized to control the initial weight tensor W i , scaling factor tensor γ i and the bias tensor β i The variance of the output binary tensor of block 100 will be calculated during the training phase. The variance of and the binary input tensor The variance of is kept within the target variance range. In some embodiments where the NN includes a computation block 100 with a BN operation 208, the scaling factor tensor γ of the BN operation 208 is initialized to the following equation (15):

[0116]

[0117] Where n is the input tensor The number of feature elements in (also equal to the number of features included in the initial weight tensor W i Thus, the initial elements of the scaling factor tensor are scaled based on the number of element-wise multiplications that occur at each corresponding neuron (e.g., computational unit) of the bitwise computation block 100.

[0118] In some exemplary embodiments using the ReLU activation function to implement the activation function 210, the initial weight tensor W i All weight elements of are initialized with random values uniformly distributed between 1.5 and +1.5, so the corresponding ternary weight tensor W i ter will have an equal number of -1, 0, and +1 values.

[0119] Ternary weight tensor W i ter The weight elements and / or scaling factor tensors γ in i Initialization of the scaling factor elements of can reduce the time required to train the NN and can improve the performance of the trained NN during the post-training inference phase.

[0120] After the network parameters have been initialized, the training phase computation block 100 and the NN forming its layers are trained using supervised training to learn the nonlinear function (Action 704). Specifically, a set of network parameters is learned through an iterative training process, wherein the network parameters include a weight tensor W i The weight element w and the scaling factor tensor γ in i The trainable scaling factor elements and bias tensor β i The iterative training process includes: using a network parameter set to infer an output tensor for each of a plurality of input feature tensors of a training dataset; determining a cost of the network parameters based on the inference performed by the NN; updating the network parameters of the corresponding layer (including the parameters of the bitwise computation block 100) using gradient descent and backpropagation; repeating the inference, cost determination, and parameter update until the cost of the network parameters reaches an optimization threshold (e.g., until an optimal network parameter set that minimizes the cost is achieved).

[0121] As described above, the feature tensor and weight tensor quantization operations (e.g., the binarization operation 212 and the ternary operation 204) are mathematically represented by equations (1) and (2), respectively. Equations (1) and (2) are not differentiable functions. This may pose a challenge to the backpropagation during training of the bitwise calculation block 100. Therefore, in some embodiments, a differentiable function (i.e., a gradient estimation function) is used to approximate the discrete functions of the binarization operation 212 and the ternary operation 204 during backpropagation. In some embodiments, the differentiable function used to represent the discrete function of the feature tensor and weight tensor quantization operations can be approximated by the Gaussian function δ represented by the following equation (16) ε (x):

[0122]

[0123] Where ε is the controlling differentiable Gaussian function δε (x) is a hyperparameter of the shape. The differentiable Gaussian function δ is represented by Equation (16) ε (x) is called the zero-centered Gaussian distribution density function.

[0124] Figure 8 shows the zero-centered Gaussian distribution density function δ ε (x) (where c = 1, 0.5, 0.25) of different shapes. Figure 8 As shown, three zero-centered Gaussian distribution density functions δ with different shapes ε Each of (x) is symmetric about the y-axis.

[0125] Equation (16) and Figure 8 An example of a differentiable function is shown that can be used during backpropagation to represent the discrete functions of the binarization operation 212 and the ternary operation 204. In other examples, different differentiable functions can be used to represent the discrete functions of the feature tensor and weight tensor quantization operations. Using differentiable functions during backpropagation can mitigate gradient noise introduced during backpropagation.

[0126] After the training of the convolution block 100 in the training phase is completed, the learned parameters are converted into a suitable format for the convolution block 200 in the inference phase (action 706). Specifically, as described above, the ternary weight tensor learned by the computation block 100 in the training phase is converted into a suitable format for the inference phase convolution block 200 (action 706). Decomposed into positive weight tensors by mapping function 260 operation and negative weight tensors They are stored as trained weight parameters for the inference phase computation block 200. In addition, based on the parameters μ of the BN operation 208 learned during the training of the training phase computation operation 100 B , β i , σ B , γ i , calculate the threshold tensor according to equation (11) Threshold Tensor Stored as trained threshold parameters of the inference phase computation block 200 .

[0127] Positive weight tensor Negative weight tensor and threshold tensor It may be deployed along with software code for implementing a NN including the inference phase computation block 200 onto one or more hardware devices for inference operations (act 708 ).

[0128] As described above, the inference phase computation block 200 does not use matrix multiplication operations that require a multiply-accumulate function to determine the tensor dot product, but instead relies on low-resource computation operations 256, including AND operations, POPCOUNT operations, and subtraction operations. In addition, the inference phase computation block 200 relies on simple Boolean comparison operations 258 to replace the BN operation 208, the activation operation 210, and the binarization operation 212.

[0129] The low-resource computational operations 256 of the inference-phase computational block 200 can, in some applications, enable hardware devices to perform a given inference task at a lower computational cost and higher efficiency than previous solutions. A NN comprising one or more layers implemented by the inference-phase computational block 200 can be suitable for deployment on computationally constrained hardware devices (e.g., hardware devices with one or more of limited processing power, limited memory, or limited power supply). Using a trained NN comprising layers based on low-resource computational operations 256 (e.g., AND operations, POPCOUNT operations, subtraction operations, comparison operations) can enable the trained NN to run on a processing system at a lower computational cost, making it more suitable for execution by a general-purpose CPU or microcontroller. Other benefits may include reduced hardware costs, improved device battery life, and enabling the trained NN to be deployed on a wider range of hardware devices.

[0130] Although described above in the context of an inference phase computation block 200 implementing a fully connected NN layer, the low-resource computation operations 256 and comparison operations 258 may also be applied to inference phase computation blocks that implement non-fully connected layers of a NN (e.g., a convolutional layer of a convolutional NN). As described above, in the case of a fully connected layer, each individual computation unit of the computation block receives the same set of input feature elements, but applies a corresponding set of weight elements. In the case of a convolutional layer, each individual computation unit of the computation block receives a different, possibly overlapping set of input feature elements (produced by convolving the input feature quantities with filters), but applies the same set of weight elements (e.g., filters). In the case of the convolution computation units of the convolution computation block, the dot product is determined based on matrices of similar size (e.g., sets) of weights and feature elements rather than vectors as described above with respect to the fully connected layer; nonetheless, the same low-resource computation operations described above may also be applied to provide integer feature element outputs corresponding to the dot product of the two matrices.

[0131] In an exemplary embodiment, a NN including an inference stage block using the low-resource computation operations 256 and comparison operations 258 described above can be used for many different inference tasks, including, for example, extracting features from video images for facial or object recognition applications.

[0132] In some embodiments, a deep NN having an input layer, an output layer, and one or more hidden layers between the input layer and the output layer may include one or more training phase computation blocks 100 during a training phase. When the NN is deployed for inference purposes, one or more training phase computation blocks 100 are replaced by corresponding inference phase computation blocks 200. The training parameters learned using the training phase computation blocks 100 are converted into a format suitable for low-resource computation operations 256 of the inference phase computation blocks 200. The NN may be software-implemented using machine-readable instructions executed by a processing unit such as a tensor processing unit or a neural processing unit. Alternatively, the NN may be implemented using software including machine-readable instructions executed by a dedicated hardware device, such as a compact, energy-efficient AI chip including a small number of logic gates (e.g., a microprocessor specifically designed to perform NN operation tasks faster, using less power than a traditional microprocessor). In an exemplary embodiment, the NN is trained using a processing unit that is more powerful than the processing system of the trained NN that is ultimately deployed for inference operations.

[0133] Figure 9 is a block diagram of an exemplary inference-stage hardware device including a processing unit 900 that may be used for training purposes to execute machine-executable instructions of a NN including one or more training-stage computational blocks 100, or to execute machine-executable instructions of a trained NN including one or more inference-stage computational blocks 200 during post-training inference. Other processing unit configurations suitable for implementing the embodiments described herein may be used and may include components different from those discussed below. Although Figure 9 A single instance of each component is shown, but multiple instances of each component may be present in processing unit 900 .

[0134] The processing unit 900 may include one or more processing devices 902, such as a processor, a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a dedicated logic circuit, or a combination thereof. In an exemplary embodiment, the processing unit 900 for training purposes may include an accelerator 906 connected to the processing device 902. The processing unit 900 may also include one or more input / output (I / O) interfaces 904, which may support connection to one or more suitable input devices 914 and / or output devices 916. The processing unit 900 may include one or more network interfaces 906 for wired or wireless communication with a network (e.g., an intranet, the internet, a P2P network, a WAN, and / or a LAN) or other nodes. The network interface 906 may include a wired link (e.g., an Ethernet cable) and / or a wireless link (e.g., one or more antennas) for intranet communication and / or extranet communication.

[0135] The processing unit 900 may also include one or more storage units 908, which may include a mass storage unit, such as a solid-state drive, a hard disk drive, a magnetic disk drive, and / or an optical disk drive. The processing unit 900 may include one or more memories 910, which may include volatile or non-volatile memory (e.g., flash memory, random access memory (RAM), and / or read-only memory (ROM)). The one or more non-transitory memories 910 may store instructions executed by the one or more processing devices 902, for example, to perform the examples described herein. The one or more memories 910 may include other software instructions, such as software instructions for implementing an operating system and other applications / functionality. In some examples, the memory 910 may include software instructions for execution by the processing device 902 to implement and train a neural network including the bitwise computation block 100 of the present invention. In some examples, the memory 910 may include software instructions and data (e.g., weights and threshold parameters) for execution by the processing device 902 to implement a trained neural network including the inference phase computation block 200 of the present invention.

[0136] In some examples, one or more data sets and / or modules may be provided by an external memory (e.g., an external drive in wired or wireless communication with the processing unit 900) or by a transient or non-transitory computer-readable medium. Examples of non-transitory computer-readable media include RAM, ROM, erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, CD-ROM, or other portable memory.

[0137] A bus 912 may be present to provide communications between components of the processing unit 900, including one or more processing devices 902, one or more I / O interfaces 904, one or more network interfaces 906, one or more storage units 908, and / or one or more memories 910. The bus 912 may be any suitable bus architecture, including, for example, a memory bus, a peripheral bus, or a video bus.

[0138] Although the present invention describes methods and processes by steps performed in a certain order, one or more steps in the methods and processes may be omitted or changed as appropriate. In appropriate cases, one or more steps may be performed in an order other than the order described.

[0139] Although the present invention has been described at least in part in terms of methods, it will be understood by those skilled in the art that the present invention is also directed to various components for performing at least some aspects and features of the methods, whether through hardware components, software, or any combination thereof. Accordingly, the technical solutions of the present invention may be embodied in the form of software products. Suitable software products may be stored in pre-recorded storage devices or other similar non-volatile or non-transitory computer-readable media, including DVDs, CD-ROMs, USB flash drives, removable hard drives, or other storage media. The software product includes instructions tangibly stored thereon that enable a processing device (e.g., a personal computer, a server, or a network device) to perform examples of the methods disclosed herein.

[0140] The present invention may be embodied in other specific forms without departing from the subject matter of the claims. The exemplary embodiments described are intended in all respects to be illustrative only and not restrictive. Selected features from one or more of the above-described embodiments may be combined to create alternative embodiments not explicitly described, and it is understood that features suitable for such combinations are within the scope of the present invention.

[0141] All values and subranges within the disclosed ranges are also disclosed. Furthermore, although the systems, devices, and processes disclosed and illustrated herein may include a specific number of elements / components, the systems, devices, and assemblies may be modified to include more or fewer of such elements / components. For example, although any disclosed element / component may be referenced as a single quantity, the embodiments disclosed herein may be modified to include a plurality of such elements / components. The subject matter described herein is intended to cover and encompass all appropriate technical variations.

[0142] The contents of all publications identified in this application are incorporated herein by reference.

Claims

1. A computer-implemented method, characterized in that include: Applying a plurality of low-resource computational operations to a binary input feature tensor to generate an integer feature tensor, the integer feature tensor being equivalent to an output of a multiplication-accumulation operation performed on a ternary weight tensor and the binary input feature tensor; wherein applying the plurality of low-resource computational operations to the binary input feature tensor to generate the integer feature tensor comprises: applying the plurality of low-resource computational operations to the binary input feature tensor using a first weight tensor and a second weight tensor to generate the integer feature tensor, the first weight tensor and the second weight tensor each comprising n binary elements and collectively representing corresponding n elements of the ternary weight tensor; The generated integer feature tensor is compared with a comparison threshold to generate a binary output feature tensor.

2. The method according to claim 1, wherein: The generated integer feature tensor includes a set of m integer elements, each integer element has a value, and the value is equivalent to the dot product of the set of n ternary elements of the ternary weight tensor and the set of n binary elements of the binary input feature tensor.

3. The method according to claim 2, characterized in that For each integer element of the integer feature tensor: Applying the plurality of low-resource computational operations comprises, for each integer element of the integer feature tensor: applying a first bitwise logical AND operation between the n binary elements of the first weight tensor and the n binary elements of the binary input feature tensor; applying a second bitwise logical AND operation between the n binary elements of the second weight tensor and the n binary elements of the binary input feature tensor; The integer element is generated based on the number of bits of a predetermined value included in the output of the first bitwise logical AND operation and the output of the second bitwise logical AND operation.

4. The method according to claim 3, characterized in that Applying the plurality of low-resource computational operations comprises, for each integer element of the integer feature tensor: applying a first POPCOUNT operation to count the number of bits of the predetermined value included in the output of the first bitwise logical AND operation; applying a second POPCOUNT operation to count the number of bits of the predetermined value included in the output of the second bitwise logical AND operation; Wherein generating the integer element comprises applying a subtraction operation between an output of the first POPCOUNT operation and an output of the second POPCOUNT operation to output the integer element.

5. The method according to claim 3 or 4, characterized in that The n elements of the ternary weight tensor are decomposed into n binary element pairs to generate the n binary elements of the first weight tensor and the n binary elements of the second weight tensor.

6. The method according to claim 5, characterized in that Applying the plurality of low-resource computing operations and performing the comparison operation are performed in a layer of a second neural network trained to perform inference tasks, The method also includes training the first neural network to perform the inference task by learning network parameters of the first neural network including the ternary weight tensor.

7. The method according to claim 6, characterized in that Training the first neural network includes: inferring an output tensor for each of a plurality of input feature tensors of a training dataset using the network parameters; determining a cost of the network parameter based on the inference; Update the network parameters using gradient descent and backpropagation, The inferring, determining, and updating are repeated until the cost of the network parameter reaches an optimization threshold.

8. The method according to claim 7, characterized in that The layers of the first neural network correspond to the layers of the second neural network, and the layers of the first neural network are used to: Applying a multiply-accumulate operation using the ternary weight tensor and binary input feature tensors in the training dataset to generate a training integer feature tensor; applying a batch normalization operation, an activation operation, and a binarization operation to the generated training integer feature tensor to generate a training binary output feature tensor, wherein the batch normalization is based on a batch normalization parameter set, The learned network parameters include the batch normalization parameters, and the comparison threshold is a tensor of m integer thresholds calculated based on the learned batch normalization parameters.

9. The method according to claim 8, characterized in that The batch normalization parameters include a scaling factor tensor and a bias tensor, and training the first neural network includes initializing one or more of the network parameters to maintain a variance of values within the network parameters within a target variance range.

10. The method according to claim 8 or 9, characterized in that During training of the first neural network, the ternary weight tensor is generated by quantizing values of a corresponding real-valued weight tensor using a non-differentiable ternary operation, the binarization operation applies a non-differentiable binarization operation, and during backpropagation, the non-differentiable ternary operation and the non-differentiable binarization operation are each approximated by a corresponding differentiable operation.

11. A processing system, characterized in that include: processing equipment; a memory operatively connected to the processing device and storing executable instructions that, when executed by the processing device, cause a processing unit to implement computational blocks of a trained neural network for performing an inference task, the computational blocks being configured to: Applying a plurality of low-resource computational operations to a binary input feature tensor to generate an integer feature tensor, the integer feature tensor being equivalent to an output of a multiplication-accumulation operation performed on a ternary weight tensor and the binary input feature tensor; wherein applying the plurality of low-resource computational operations to the binary input feature tensor to generate the integer feature tensor comprises: applying the plurality of low-resource computational operations to the binary input feature tensor using a first weight tensor and a second weight tensor to generate the integer feature tensor, the first weight tensor and the second weight tensor each comprising n binary elements and collectively representing corresponding n elements of the ternary weight tensor; The generated integer feature tensor is compared with a comparison threshold to generate a binary output feature tensor.

12. The processing system according to claim 11, characterized in that: The generated integer feature tensor includes a set of m integer elements, each integer element has a value, and the value is equivalent to the dot product of the set of n ternary elements of the ternary weight tensor and the set of n binary elements of the binary input feature tensor.

13. The processing system according to claim 12, characterized in that For each integer element of the integer feature tensor, the computation block is used to: Applying the plurality of low-resource computational operations to each integer element of the integer feature tensor in the following manner: applying a first bitwise logical AND operation between the n binary elements of the first weight tensor and the n binary elements of the binary input feature tensor; applying a second bitwise logical AND operation between the n binary elements of the second weight tensor and the n binary elements of the binary input feature tensor; The integer element is generated based on the number of bits of a predetermined value included in the output of the first bitwise logical AND operation and the output of the second bitwise logical AND operation.

14. The processing system according to claim 13, characterized in that The computation block is configured to apply the plurality of low-resource computation operations to each integer element of the integer feature tensor in the following manner: applying a first POPCOUNT operation to count the number of bits of the predetermined value included in the output of the first bitwise logical AND operation; applying a second POPCOUNT operation to count the number of bits of the predetermined value included in the output of the second bitwise logical AND operation; Wherein generating the integer element comprises applying a subtraction operation between an output of the first POPCOUNT operation and an output of the second POPCOUNT operation to output the integer element.

15. The processing system according to claim 13, wherein: The n binary elements of the first weight tensor and the n binary elements of the second weight tensor are stored in the memory and correspond to decomposing the corresponding n elements of the ternary weight tensor into n binary element pairs to generate the n binary elements of the first weight tensor and the n binary elements of the second weight tensor.

16. The processing system according to claim 15, characterized in that The ternary weight tensor is part of a set of network parameters learned by training a neural network to perform the inference task.

17. The processing system according to claim 15 or 16, characterized in that The training neural network is trained in the following way: inferring an output tensor for each of a plurality of input feature tensors of a training dataset using the network parameters; determining a cost of the network parameter based on the inference; Update the network parameters using gradient descent and backpropagation, The inferring, determining, and updating are repeated until the cost of the network parameter reaches an optimization threshold.

18. The processing system according to claim 17, wherein: The computational blocks of the training neural network correspond to the computational blocks of the trained neural network, the computational blocks of the training neural network being used to: Applying a multiply-accumulate operation using the ternary weight tensor and binary input feature tensors in the training dataset to generate a training integer feature tensor; applying a batch normalization operation, an activation operation, and a binarization operation to the generated training integer feature tensor to generate a training binary output feature tensor, wherein the batch normalization is based on a batch normalization parameter set, The learned network parameters include the batch normalization parameters, and the comparison threshold is a tensor of m integer thresholds calculated based on the learned batch normalization parameters.

19. The processing system according to any one of claims 16 to 18, characterized in that In combination with another processing unit, the another processing unit is used to train the training neural network to perform the inference task.

20. A computer-readable medium, characterized in that Instructions are stored for causing a processing unit to implement a computational block of a trained neural network for performing an inference task, the computational block being configured to perform the method of any one of claims 1 to 10.

21. A computer-readable product, characterized in that Instructions are stored for causing a processing unit to implement a computational block of a trained neural network for performing an inference task, the computational block being configured to perform the method of any one of claims 1 to 10.

Images