Methods and systems for generating binary weights of neural networks

Abstract

A processor for a neural network, comprising: a weight probability analysis module configured to generate, based on a set of input weights for one or more layers of the neural network, data representing at least a probability that each input weight in the set of input weights is associated with a binary value; a scheme generation module configured to generate, for at least one selected grouping of the weights, data representing at least one or more potential binary weight matrices based on the probabilities determined for the selected weights; a scheme selection module configured to generate at least data representing a matrix-specific probability value for each potential binary weight matrix, generate data representing a number of data bits used to represent each potential binary weight matrix according to a predetermined encoding method, and select a potential binary weight matrix based on the matrix-specific probability value and the number of data bits; and a weight generation module configured to generate data representing one or more binary weights according to the selected potential binary weight matrix.

Description

Technical Field

[0001] The present invention relates to an improved method and system for generating binary weights for neural networks, wherein, specifically, the binary weights are generated based on probability factors and their compression properties. Background Technology

[0002] Currently, different neural network architectures have been developed for various applications. For neural networks using the same architecture, those with more layers and more parameters generally achieve higher accuracy in performing neural network tasks. For example, a VCG-16-based convolutional neural network (CNN) (with 16 layers and 138 million parameters) typically achieves higher accuracy than an AlexNet-based CNN (with 8 layers and 60 million parameters), which in turn typically achieves higher accuracy than a LeNet-5-based CNN (with 5 layers and 60,000 parameters). The same principle applies to more modern architectures such as ResNet and DenseNet.

[0003] The problem with neural networks, especially convolutional neural networks, is that the operations they perform typically consume significant amounts of hardware resources, hindering their application in resource-constrained environments (e.g., small, battery-powered devices). For example, multiply-accumulate (MAC) operations performed on floating-point weights in convolutional layers can require substantial data processing and memory resources. Furthermore, significant memory resources are also needed to store the weights in each convolutional or fully connected layer of the neural network. However, the amount of such hardware resources available for implementing neural networks is often physically or practically limited, depending on the implementation environment.

[0004] Several methods have been proposed previously to help reduce the hardware resources required for neural networks. These methods include, for example, pruning connections in the network based on weight size, and quantizing weights from raw floating-point values ​​(e.g., 10 to 64 bits in length) to fixed-point values ​​of predetermined bit lengths (e.g., 16 bits, 8 bits, 4 bits, 2 bits, or 1 bit). Another approach is to develop binary neural network (BNN) architectures that use binary weights to process binary data inputs. Operations in BNNs are generally easier to implement in hardware, where, for example, simpler XNOR logical operations can be performed on the relevant binary weights without performing MAC operations.

[0005] Several methods have been proposed to reduce the resource requirements of BNNs. For example, the flipping frequency of each weight can be determined, where weights with flipping frequencies above a predetermined threshold can be pruned (because their impact on the BNN's output is limited). In another example, the weight matrix of a binary convolutional neural network (BCNN) can be pruned and then compressed. Starting from one end of the weight matrix, non-zero bits in the binary weight matrix are progressively pruned (or changed) to zero bits, stopping when the change in the binary weight matrix causes a significant decrease in recognition accuracy. Then, consecutive sequences of weights with the same value in the weight matrix can be compressed (e.g., mapped to predefined values ​​representing consecutive zero-bit sequences of different lengths). In yet another example, the sensitivity of each weight in the BNN can be estimated, and each weight can then be divided into sensitive and insensitive weights based on a threshold. This threshold is determined based on the error caused by variations in the values ​​of insensitive binary weights stored in unreliable memory operating at near / below a threshold voltage in the BNN, and this threshold is adjusted to achieve an optimal set of insensitive weights.

[0006] In the methods described above, a binary weight matrix is ​​generated and then processed separately to achieve compression. This type of matrix is ​​generated without considering the compression properties of the resulting matrix. Furthermore, in these methods, the weight matrix can only contain one of two binary values. These methods do not consider the possibility of subtle variations in a given binary weight matrix (when considering the probabilities associated with each weight), some of which may be more favorable for compression than others.

[0007] The present invention aims to solve one or more of the problems mentioned above. Specifically, representative embodiments of the present invention aim to provide an improved method for generating a binary weight matrix with ideal compression properties by considering the probability of the input weights. Summary of the Invention

[0008] In one aspect, the present invention provides a processor for a neural network, the processor comprising:

[0009] A weight probability analysis module is configured to generate data representing at least the probability that each of the input weights in the set of input weights is associated with a binary value, based on a set of input weights of one or more layers of a neural network.

[0010] A scheme generation module is configured to generate data representing at least one or more potential binary weight matrices based on probabilities determined for at least one selected group of the weights.

[0011] A scheme selection module is configured to at least: generate data representing matrix-specific probability values ​​for each of the latent binary weight matrices; generate data representing the number of data bits used to represent each of the latent binary weight matrices according to a predetermined encoding method; and perform selection on the latent binary weight matrices based on the matrix-specific probability values ​​and the number of data bits; and

[0012] A weight generation module is configured to generate data representing one or more binary weights based on the selected potential binary weight matrix.

[0013] On the other hand, the present invention provides a method for binary quantization of weights in a neural network, the method comprising:

[0014] A set of input weights based on one or more layers of a neural network generates data that at least represents the probability that each of the input weights in the set of input weights is associated with a binary value;

[0015] For at least one selected group of the weights, data representing at least one or more potential binary weight matrices is generated based on probabilities determined for the selected weights.

[0016] Generate data representing matrix-specific probability values ​​for each of the latent binary weight matrices; generate data representing the number of data bits used to represent each of the latent binary weight matrices according to a predetermined encoding method; and perform selection on the latent binary weight matrices based on the matrix-specific probability values ​​and the number of data bits; and

[0017] Data representing one or more binary weights is generated based on the selected potential binary weight matrix. Attached Figure Description

[0018] This document describes representative embodiments of the invention by way of example only and with reference to the accompanying drawings, wherein the same reference numerals denote the same features, in which:

[0019] Figure 1 The diagram shown is an exemplary internal structure of a data processing system for implementing the method of the present invention.

[0020] Figure 2 The diagram shown is an exemplary flowchart of the training process for generating binary weights using the method of the present invention.

[0021] Figure 3 The diagram shown is an exemplary flowchart of key steps in a binarization method according to a representative embodiment of the present invention;

[0022] Figure 4The diagram shown is an exemplary flowchart of key steps in an alternative binarization method according to a representative embodiment of the present invention;

[0023] Figure 5 The diagram shown is an exemplary flowchart of a key step in the binary matrix selection process according to a representative embodiment of the present invention.

[0024] Figure 6 The diagram shown is an exemplary flowchart of key steps in an alternative binary matrix selection step according to a representative embodiment of the present invention;

[0025] Figure 7 The diagram shown is an exemplary linear relationship for generating normalized weights according to a representative embodiment of the present invention;

[0026] Figure 8 The diagram shown is an example of a predetermined relationship for determining the local weight probabilities of different input weights;

[0027] Figure 9 The diagram shown is an example of selected groups that can form weights according to a representative embodiment of the present invention;

[0028] Figure 10 The diagram illustrates a method for generating a potential binary weight matrix according to a representative embodiment of the present invention.

[0029] Figure 11 The diagram illustrates a method for selecting a potential binary weight matrix according to a representative embodiment of the present invention.

[0030] Figure 12 The illustration shows a representative embodiment of the present invention of a method for determining a bit group for encoding. Detailed Implementation

[0031] Figure 1 The diagram shown illustrates an exemplary internal structure of a data processing system 100, which includes a processor 102, a memory 104, one or more input interfaces 106 and one or more output interfaces 108, and a data communication bus 110. The bus 110 includes one or more conductors to enable communication between computer components (e.g., sending or exchanging data).

[0032] Processor 102 may include one or more conventional microprocessors, microcontrollers, or similar or equivalent data / signal processing units (e.g., application-specific integrated circuits (ASICs) or field-programmable gate arrays (FGPAs)) configured to interpret and execute instructions (including instructions in the form of code or signals) provided to processor 102. Memory 104 may include any conventional random access memory (RAM) device, any conventional read-only memory (ROM) device, or any other type of volatile or non-volatile data storage device that can store information and instructions for execution by processor 102. Memory 104 may also include storage devices for persistent storage of electronic data. Figure 1 (not shown in the image) (e.g., a hard disk drive), the storage device includes, for example, a data recording medium based on magnetism, light, or circuitry, and any associated circuitry and physical components for reading and writing data to / from the recording medium.

[0033] Input interface 106 may include one or more conventional data communication devices for receiving: (i) input data representing the weights of a neural network; (ii) configuration data representing one or more parameters for the operation of system 100; and / or (iii) instruction data representing instruction codes for the operation of system 100 (e.g., stored in memory 104). Output interface 108 may include one or more conventional data communication devices for providing output data to external systems or users, including, for example, binary weight data representing one or more binary weights generated by system 100.

[0034] In a representative embodiment of the invention, system 100 is implemented in an integrated circuit chip such as an ASIC or FPGA. Alternatively, system 100 can be implemented on a conventional desktop computer, server, or cloud-based computer system.

[0035] In a representative embodiment of the invention, the processor 102 of system 100 is configured by instructions stored in memory 104 to provide a weight probability analysis module 102a, a scheme generation module 102b, a scheme selection module 102c, and a weight generation module 102d, each of which is configured to execute the steps of binarization method 204 to generate binary weights for the neural network. The representative embodiments of the invention described herein can be applied to any neural network, including convolutional neural networks. Binarization method 204 can be implemented as follows: Figure 2 The training process shown in 200 can be performed as part of the training process, or it can be performed as a separate and independent process.

[0036] A simple neural network can consist of one or more fully connected (or hidden) layers. Each layer consists of multiple neurons, each with learnable weights and learnable biases. Each neuron receives an input value from the input layer and transforms that input value based on the weights and biases associated with that neuron to generate an output value, which is then fed as input to one or more neurons in the next layer. This process continues until the last layer generates the final output of the neural network. In this context, the weight matrix of a layer refers to the array of weight values ​​associated with each neuron in that layer.

[0037] Convolutional neural networks are typically used to extract features from input provided by an input layer. In the context of image processing, the input layer can provide input in the form of an image. The input image may include a single channel (e.g., where the elements of the input correspond to black or white image pixels). In another example, the input image may be a color image comprising multiple channels, where the elements of each channel correspond to the intensity or degree of different color components of the image (e.g., red, green, and blue channels).

[0038] Different filters can be used to detect different features of an input image (e.g., horizontal edges, vertical edges, or diagonal edges). These features are extracted by convolution using filters, where the weights of the filters are determined and adjusted through training. A kernel refers to a two-dimensional array (or weight matrix) of weights. Each kernel is unique and is used to detect or extract different features of the input image. A filter refers to a set of kernels (e.g., stacked together to form a three-dimensional array). For example, to detect specific features in an input image comprising multiple channels, a different kernel is provided for each channel to detect relevant features in each channel. In this case, each kernel will have the same size (e.g., a 3×3 matrix), where a filter refers to a set of kernels used to detect the same relevant features in different channels of the image. For an input image with only one channel, only one kernel is used to detect relevant features; therefore, the filter is the same as the kernel. In the context of convolutional neural networks, this invention can be applied to kernels in a single-channel input layer or to kernels in one or more filters in a multi-channel input layer.

[0039] Convolution is the process of generating an output value based on element-wise multiplication (or dot product) of the weights in the kernel with the corresponding matrices of the input values ​​from the input channels associated with that kernel. For example, a 3×3 kernel is first associated with a 3×3 matrix of input values ​​from the input channels (e.g., starting from a corner of the two-dimensional input matrix of the channel's input values). A dot product is performed between the weights in the kernel and the input value matrix, and the resulting values ​​are summed to generate a single output value for the initial input weight matrix. Then, the kernel "slides over" to different positions in the input matrix (e.g., sliding one element to the left if the stride parameter is set to 1) so that the kernel is associated with different 3×3 input value matrices. A convolution operation is then performed on the new 3×3 input value matrix to generate the corresponding output value. This process is repeated until the kernel has "slid over" the entire input matrix.

[0040] Figure 2 The diagram shown is an exemplary flowchart of a training process 200 according to a representative embodiment of the present invention, wherein the binarization method of the present invention is used to generate binary weights for a neural network. Figure 2 In the exemplary embodiment shown, the training process 200 begins at step 202, where processor 102 obtains the initial weights of the neural network. The initial weights may be randomly determined floating-point weight values ​​of one or more layers (i.e., dense or fully connected layers) of the neural network (e.g., a convolutional neural network). Alternatively, the initial weights may be floating-point values ​​or binary weight values ​​previously determined for the neural network. Such predetermined weight values ​​may be provided to processor 102 via input interface 106 (e.g., from a separate device communicating with system 100), or alternatively, processor 102 may retrieve any such predetermined weight values ​​stored in memory 104.

[0041] At step 204, processor 102 (and its associated modules 102a to 102d) is configured to execute a binarization method for generating binary weights for a neural network. Reference will be made below. Figures 3 to 6 This will be described in more detail. The binary weights generated at step 204 can be stored in memory 104 for use by processor 102 in subsequent steps of training process 200.

[0042] At step 206, processor 102 is configured to provide the neural network with the binary weights generated at step 204. This neural network may be implemented by processor 102 of system 100, or alternatively by a processor on a separate device or machine communicating with processor 102 or system 100. The neural network is configured (e.g., configured by processor 102) to perform a training task on a set of training data to determine a training outcome. For example, the training data may include multiple training images related to different topics, and the neural network (when configured with the binary weights generated at step 204) will generate a training outcome indicating what the neural network ultimately determines as what topic in each training image.

[0043] At step 208, processor 102 evaluates the training results generated at step 206 to assess the accuracy of the data model represented by the binary weights generated at step 204. If step 208 determines that the model's accuracy is acceptable (e.g., the training results are sufficiently close to the expected results or within an acceptable range of the expected results), the training process 200 proceeds to step 210, where it is determined that the model is usable. In this case, the binary weights corresponding to the data model can be stored in memory 104 and can also be provided via output interface 108 to a separate device for configuring the neural network. The training process 200 ends after step 210. However, if step 208 determines that the model's accuracy is unacceptable (e.g., the training results are not sufficiently close to the expected results or are not within an acceptable range of the expected results), the training process 200 proceeds to step 212.

[0044] At step 212, processor 102 is configured to use a suitable cost function to generate the cost or error associated with the training result generated at step 206. At step 214, the cost or error determined in step 212 is used to determine new weight values ​​(e.g., floating-point weights) to be provided as input to the binarization method at step 204. For example, step 214 may involve generating new floating-point weight values ​​based on the cost or error determined at step 212 and the input weights previously provided to the binarization method at step 204 (e.g., the input weights last obtained at step 202 or the weights last determined by step 214). Alternatively, step 214 may involve modifying the input weights previously provided to the binarization method at step 204 according to the values ​​determined based on the cost or error determined at step 212. The weights determined by step 214 are stored in memory 104. The training process 200 then proceeds to step 204, in which the binarization method performs the above steps using the weights determined by step 214 as input weights.

[0045] Although steps 202 and 206 through 214 may be executed by the processor 102 of system 100 according to one representative embodiment of the invention, in other representative embodiments of the invention, such steps may be executed on one or more computing devices, processors, chipsets, etc., that are separate from (but communicate with and work with) system 100.

[0046] Figure 3 The diagram illustrates an exemplary flowchart of key steps in a binarization method 204 according to a representative embodiment of the present invention. At step 204a, the weight probability analysis module 102a of processor 102 is configured to generate data representing at least a “local” weight probability associated with each input weight and a binary value, based on a set of input weights (e.g., one or more layers) from a neural network. In this context, a binary value refers to one of two potential values ​​(e.g., 1 and -1, or 1 and 0). The input weights can be derived from… Figure 2 The weights received in step 202 or 214 of the training process 200 may be weights retrieved from memory 104 or weights provided to processor 102 via input interface 106. Figure 3 In the illustrated embodiment, step 204a may include steps 204a-1 and 204a-2.

[0047] At step 204a-1, the weight probability analysis module 102a is configured to generate a set of normalized weights based on the input weights. This involves transforming the input weights into corresponding weight values ​​within a predetermined weight range and using those corresponding weight values ​​as input weights. For example, the maximum absolute value of the floating-point input weights of each layer after training (e.g., weights in each BNN layer) is typically not equal to 1, or may not necessarily fall within the range of expected values ​​(e.g., between 1 and -1). To help estimate the probability of each input weight more accurately and efficiently, step 204a-1 is configured to linearly scale each input weight based on the relationship between the value (x) of the specific input weight to be normalized and the maximum absolute value of all input weights in the layer to which the input weight (x) belongs. This relationship is represented by Equation 1, where x represents the value of the specific input weight to be normalized, X represents the values ​​of all input weights in the layer to which the input weight (x) belongs, and x' represents the normalized weight value generated by Equation 1.

[0048]

[0049] The normalized weights generated according to Equation 1 will fall within the range of +1 and -1. Those skilled in the art will understand that the weights can also be normalized within different ranges of binary values ​​(e.g., between 1 and 0). Figure 7The diagram shown is an exemplary function graph represented by Equation 1, in which the original input weight values ​​ranging from +1.5 to -1.5 are scaled to normalized weight values ​​ranging from +1 to -1.

[0050] At step 204a-2, the weight probability analysis module 102a is configured to generate data, based on a first set of normalized input weights (e.g., one or more layers) of a neural network, to generate "local" weight probabilities that at least represent the association of each input weight in a second set of selected or specific input weights with a binary value. In this context, the second set of selected or specific input weights may include:

[0051] i) Input weights selected from at least one layer of the neural network;

[0052] ii) Some input weights selected from all layers of the neural network;

[0053] iii) All input weights of at least one layer of the neural network; or

[0054] iv) All input weights of all layers of the neural network.

[0055] The "local" weighted probability can be determined based on the following:

[0056] i) The predetermined relationship between different potential values ​​of input weights and their corresponding probabilities;

[0057] ii) one or more previously determined probabilities of the weights corresponding to the input weights (e.g., based on summation, average, count, frequency, or other values ​​determined based on one or more previously determined probabilities of the weights corresponding to the input weights, wherein such values ​​may be determined based on one or more previous training events performed for or by the neural network); and / or

[0058] iii) One or more previously determined weight values ​​corresponding to the input weights (e.g., based on summation, average, count, frequency, or other values ​​determined based on one or more previously determined weight values ​​corresponding to the input weights according to previous training events, wherein such values ​​may be determined based on one or more previous training events performed for or by the neural network).

[0059] Figure 8The diagram illustrates an exemplary predetermined relationship for determining the local weight probabilities of different input weights. A first function 800 can be used to determine the probability (on the vertical axis) of an input weight (on the horizontal axis) associated with a binary value +1. Function 800 can be based on a sigmoid function, a hyperbolic tangent function, an arctangent function, or any other suitable function that defines the association between different input weight values ​​and their corresponding probabilities. Function 800 is characterized by an inflection point where an input weight of 0 corresponds to a probability of 0.5, and the slope at the inflection point is approximately 1.6. A second function 802 can be used to determine the probability of an input weight associated with a binary value -1. Figure 8 In the example shown, function 802 is the inverse function of function 800.

[0060] In a representative embodiment of the invention, a predetermined relationship for determining the local weight probabilities of different input weights is defined based on equations 2a, 2b, and 3 below, wherein equation 2a represents the probability (p) of an input weight (x') being associated with a binary value +1. +1 Equation 3 represents the probability (p) that the input weight (x') is associated with the binary value -1. -1 Equation 2b represents the probability (p) that the input weight (x') is associated with the binary value +1. +1 Alternative methods.

[0061]

[0062]

[0063] p -1 =1-p +1 Equation 3

[0064] refer to Figure 3 At step 204b, for a selected group (or encoded group) of input weights, the scheme generation module 102b of processor 102 is configured to generate data representing at least one or more potential binary weight matrices (or schemes) based on probabilities determined for the selected input weights. Figure 3 In the illustrated embodiment, step 204b may include steps 204b-1 and 204b-2.

[0065] At step 204b-1, the scheme generation module 102b of processor 102 is configured to define at least one selected group (or encoding group) of input weights. The weight matrix of each layer typically has multiple dimensions. Convolutional layers typically have at least four dimensions, including, for example, input channels, output channels, kernel rows, and kernel columns. Fully connected layers typically have at least two dimensions, including, for example, input size and output size. Step 204b-1 involves dividing the weight matrix of each layer into smaller selected groups (or encoding groups) of weights. Selected groups of weights can be formed along any one or more dimensions of the layer. One or more selected groups of weights for a layer can be formed in this way. After generating multiple selected groups of weights, each selected group is processed according to steps 204b-2, 204c, and 204d below, and the selection at step 204e is performed based on the output generated for all (or at least some) of the selected groups in steps 204c and 204d. For example, as... Figure 9 As shown, a 3×3 kernel 900 can be viewed as a selected group (or encoded group) of weights, where the kernel has 3 rows and 3 columns. Figure 9 In this context, w0 to w8 represent the positions of different weights in the 3×3 kernels 900, 908, 910, and 912. Alternatively, the selected grouping of weights can be formed based on one or more dimensions of the layer according to other predetermined selection criteria, including, for example:

[0066] i) One or more weights from selected row 902 of kernel 900 of the convolutional layer;

[0067] ii) One or more weights from selected column 904 of kernel 900 of the convolutional layer;

[0068] iii) One or more weights 906 from different kernels 908, 910, 912 associated with the same channel or filter of the convolutional layer (e.g., where the weights 906 are selected from the same corresponding positions in each kernel 908, 910, 912—such as including the first bit of the first row in each kernel 908, 910, 912, then the second bit of the first row in each kernel 908, 910, 912, then the third bit of the first row in each kernel 908, 910, 912, then the first bit of the second row in each kernel 908, 910, 912, etc.);

[0069] iv) One or more weights 914 from different corresponding kernels of different filters 916, 918, 920 associated with the same convolutional layer (e.g., where weights 914 are selected from the same corresponding position in the kernel of each filter 916, 918, 920 for a particular channel (e.g., the red channel) - such as including the first bit of the first row in the kernel of the filter 916, 918, 920 for the red channel, then the second bit of the first row in the kernel of the filter 916, 918, 920 for the red channel, then the third bit of the first row in the kernel of the filter 916, 918, 920 for the red channel, then the first bit of the second row in the kernel of the filter 916, 918, 920 for the red channel, etc.);

[0070] v) One or more input weights of a fully connected layer;

[0071] vi) One or more output weights of the fully connected layer;

[0072] vii) A combination of one or more of the above; and / or

[0073] viii) A combination of one or more of the above with one or more other predetermined selection criteria.

[0074] As can be understood from the examples above, a predefined selection criterion defines the basis for how the weights of one or more kernels (of one or more filters) from a layer are selected and arranged in a certain order. Any basis can be used as a predefined selection criterion, as long as it does not involve random selection and each weight in the layer is selected only once (to avoid selecting the same weight again).

[0075] At step 204b-2, the scheme generation module 102b of processor 102 is configured to generate data representing at least one or more potential binary weight matrices (or schemes) based on the probabilities determined for selected groups of input weights generated in step 204b-1.

[0076] In each selected group (or encoded group) of weights, each floating-point weight has a first “local” weight probability associated with a first binary value (e.g., +1), and may also have a second “local” weight probability associated with a second binary value (e.g., -1). One or more latent binary weight matrices are generated based on one or both of the first and second “local” weight probabilities associated with each weight.

[0077] For example, according to a representative embodiment of the invention, processor 102 generates all potential binary weight matrices that can be formed based on different combinations of first "local" weight probabilities and second "local" weight probabilities associated with each weight. For example, in the case where the selected grouping of weights is a 3×3 kernel, a total of 2 9 (i.e., 512) distinct 3×3 latent binary weight matrices. Further processing is performed on each generated latent binary weight matrix according to steps 204c and 204d described below.

[0078] In another representative embodiment of the invention, the processor 102 is configured to generate one or more potential binary weight matrices based on at least one of a first “local” probability and a second “local” probability for each weight, and a predetermined probability threshold. For example, an initial binary weight matrix is ​​first generated from the binary weights corresponding to the larger of the first and second “local” probabilities determined for each weight. Alternatively, the initial binary weight matrix can be generated by comparing one of the first and second “local” probabilities for each weight with a predetermined selection threshold representing a predetermined probability value (e.g., if the first probability is equal to or higher than the predetermined selection threshold, the weight is set to a binary value associated with the first “local” probability; if the first probability is lower than the predetermined selection threshold, the weight is set to another binary value).

[0079] The processor 102 can then compare the “local” probability of each weight in the initial weight matrix with a predetermined evaluation threshold representing a predetermined probability value. For weights whose “local” probability is equal to or greater than the evaluation threshold, the binary weights remain unchanged. However, for weights whose “local” probability is less than the evaluation threshold, these weights do not have a sufficiently strong probability of being associated with their current binary value; therefore, further analysis is required to consider the possibility that the weight is associated with its current binary value (e.g., +1) or another binary value (e.g., -1).

[0080] Processor 102 can then generate all potential binary weight matrices that can be formed from the initial weight matrix described above, based on different combinations of first and second "local" weight probabilities associated with weights whose "local" probabilities are below an evaluation threshold. Processor 102 generates multiple potential binary weight matrices based on the number of selected input weights determined to be associated with the first or second binary value, wherein each binary weight matrix includes different combinations of weight values. For example, depending on the number (n) of weights in the initial binary weight matrix whose "local" probabilities are below an evaluation threshold, processor 102 will generate 2... n Potential binary weight matrices with different combinations.

[0081] It can be done as follows Figure 10The example shown provides a better understanding of the method used to generate the potential binary weight matrix. Processor 102 first receives the input floating-point weights of one layer (e.g., corresponding to...). Figure 2 In step 202 or 214), these input floating-point weights are shown in this example as input weight matrix 1002. The input weights are normalized to values ​​within a predetermined range from +1 to -1 (e.g., corresponding to...). Figure 3 Step 204a-1 in the above steps is used to generate the normalized weight matrix 1004. The "local" weight probability (e.g., corresponding to one or both of the potential binary values) associated with each normalized input weight is determined. Figure 3 Step 204a-2 in [the original text]. In this example, matrix 1006 represents the "local" weight probability associated with the normalized input weights and the binary value +1, and matrix 1008 represents the "local" weight probability associated with the normalized input weights and the binary value -1. In this example, the selected group (or encoded group) of weights is a 3×3 weight matrix (e.g., corresponding to [the original text]). Figure 3 Step 204b-1 in the table. Then, an initial binary matrix 1010 is generated based on the "local" weighted probabilities (e.g., corresponding to...). Figure 3 (Step 204b-2 in the original text). This can be achieved in different ways. One approach is to consider the weights in either matrix 1006 or 1008, where, in the case of matrix 1006, if the probability of that weight in matrix 1006 is equal to or greater than a predetermined selection threshold (e.g., 50%), the binary value of the weight in the initial weight matrix is ​​determined to be +1, otherwise the binary value of the weight is determined to be -1. Matrix 1008 can be used in a similar manner to determine which weights in the initial weight matrix should have a value of -1. Alternatively, the generation of the binary values ​​of the weights in the initial binary matrix 1010 depends on which matrix (1006 or 1008) has a greater probability for that weight. For example, if the probability value corresponding to a certain weight in matrix 1006 is higher than that in matrix 1008, then the binary value of that weight is +1. The processor 102 then generates one or more potential binary weight matrices (e.g., corresponding to the binary weights and the "local" weight probabilities of each weight in the initial weight matrix 1010) based on the binary weights and the initial weight matrix 1010. Figure 3 Step 204b-2 in the process. For example, when comparing the “local” weight probability of each weight in the initial weight matrix 1010 with an evaluation threshold (e.g., 55%), processor 102 determines that only one weight has a “local” weight probability below the evaluation threshold (shown in gray in matrix 1012). Then, processor 102 generates two potential binary weight matrices 1014 and 1016 (i.e., 2...). nThere are several combinations (n=1 in this example) to represent different combinations of binary weights from the initial weight matrix, where weights with a "local" probability below the evaluation threshold can take values ​​of +1 (in matrix 1014) or -1 (in matrix 1016).

[0082] refer to Figure 3 The processor 102 is configured to provide a scheme selection module 102c, which performs... Figure 3 Steps 204c, 204d, and 204e are used to select one of the potential binary weight matrices generated in the previous steps.

[0083] At step 204c, the scheme selection module 102c is configured to generate data representing at least a matrix-specific probability value for each potential binary weight matrix. According to one embodiment, this involves generating a value based on the product of all “local” weight probabilities for each potential binary weight matrix. Figure 11 In the example shown, matrices 1102, 1104, 1106, and 1108 represent the binary weights in four different potential binary weight matrices, and matrices 1102', 1104', 1106', and 1108' represent the weight probabilities corresponding to each binary weight in matrices 1102, 1104, 1106, and 1108. The matrix-specific probability values ​​of matrices 1102', 1104', 1106', and 1108' are 0.025038, 0.02507, 0.020060, and 0.020052, respectively.

[0084] In step 204d, the scheme selection module 102c generates data representing at least a series of data bits, which are used to represent each potential binary weight matrix according to a predetermined encoding method. First, the binarized weights in each potential binary weight matrix are encoded as strings of 1s and 0s. For example, in... Figure 11 In the process, the binary weights (i.e., 1, -1, -1, 1, 1, -1, -1, 1, 1) obtained row by row in matrix 1102 are encoded into a bit string (i.e., 1, 0, 0, 1, 1, 0, 0, 1, 1), where bit value 1 represents a binary weight 1, and bit value 0 represents another binary weight (e.g., -1).

[0085] Then, the bit string is divided into several bit groups for encoding. (Reference) Figure 12 In the example, the bits (b0 to b8) representing the weights from a 3×3 potential binary weight matrix (e.g., 1102, 1104, 1106, and 1108) can be grouped into groups of 3 consecutive bits (1202, 1204, and 1206), or into two groups of 4 consecutive bits (1208 and 1210) and a remaining bit (1212). Figure 12The grouping shown is merely an example, and any form of grouping can be used (according to any predefined grouping criteria).

[0086] According to a representative embodiment of the present invention, generalized run-length encoding (RLE) is used to encode the bit string representing binary weights. However, other encoding methods, such as generalized Huffman coding, can be used alternatively. The result of generalized run-length encoding consists of two main parts: the first part represents the encoding arrangement of the same symbol or group of symbols, where each symbol consists of one or more bits, and the second part represents the symbol information corresponding to the same symbol or group of symbols. The generalized Huffman coding method defines its corresponding encoding length according to the occurrence probability of each symbol or group of symbols.

[0087] The following text is for reference only. Figure 11 , Figure 12 Table 1 describes an example of RLE encoding for a bit string used to represent a 3×3 binary weight matrix, which has been divided into 3 groups of 3 consecutive bits (e.g., as shown in Table 1). Figure 12 (As shown).

[0088] i) If the bit patterns in each row 1202, 1204, and 1206 are the same, these rows can be encoded using the encoding pattern indicator “00” along with the bit string of one of the rows (see the Data0 field in Table 1 below). In this case, encoding requires 5 bits.

[0089] ii) If the bit patterns in rows 1202 and 1204 are the same, but the bit patterns in row 1206 are different, then these rows can be encoded using the encoding pattern indicator “01” along with the bit strings of rows 1202 or 1204 (see the Data0 field in Table 1) and the bit string of row 1206 (see the Data1 field in Table 1). In this case, encoding requires 8 bits.

[0090] iii) If the bit patterns in rows 1202 and 1206 are the same, but the bit pattern in row 1204 is different, then these rows can be encoded using the encoding pattern indicator “01” along with the bit string of row 1202 or 1206 (see the Data0 field in Table 1) and the bit string of 1204 (see the Data1 field in Table 1). In this case, encoding requires 8 bits.

[0091] iv) If the bit patterns in rows 1204 and 1206 are the same, but the bit patterns in row 1202 are different, then these rows can be encoded using the encoding pattern indicator “11” along with the bit string of each row (see the Data0, Data1, and Data2 fields in Table 1). In this case, encoding requires 11 bits.

[0092] v) If the bit patterns in each of rows 1202, 1204, and 1206 are different, these rows can be encoded using the encoding pattern indicator “11” along with the bit string of each row (see the Data0, Data1, and Data2 fields in Table 1). In this case, encoding requires 11 bits.

[0093]

[0094] Table 1

[0095] As those skilled in the art will understand, in the above examples, encoding mode indicators 01 and 10 can be assigned to the bit string under any two of the three conditions (ii) to (iv) shown above, while encoding mode indicator 11 can be assigned to the remaining condition. Furthermore, the encoding mode indicators used to encode the bit string under the above five conditions can be defined in any way (e.g., by predetermined settings or parameters), not limited to the examples above. Each row in Table 1 can be associated with a unique encoding mode indicator different from those shown in Table 1. For example, the encoding mode indicators used to encode the bit string under the above five conditions can be defined as 11, 10, 01, 00, 00 (so that the encoding mode indicators for rows 1 to 4 of Table 1 would be 11, 10, 01, 00, respectively). Note that assigning a different (e.g., 2-bit) encoding mode indicator to each row of Table 1 does not change the basis for encoding the data types described in each row of Table 1 (or in the conditions (i) to (v) above). Furthermore, the bit string representing the binary weights can be segmented in any way, such as segmenting by column, and is not limited to... Figure 12 The segmentation method shown is (i.e., segmented by line).

[0096] At step 204e, the scheme selection module 102c performs selection on the potential binary weight matrix based on the corresponding matrix-specific probability value of the potential binary weight matrix (determined at step 204c) and the number of data bits used for encoding (determined at step 204d). Figure 5 and Figure 6 As shown, there are two methods to achieve this goal.

[0097] like Figure 5As shown, according to a representative embodiment of the present invention, the scheme selection module 102c (at step 204e-1) first selects one or more potential binary weight matrices based on the matrix-specific probability values ​​of the potential binary weight matrices (e.g., the potential binary weight matrix with the highest matrix-specific probability value, or the potential binary weight matrix with a matrix-specific probability value higher than a predetermined probability threshold). At step 204e-2, the scheme selection module 102c selects one from the potential binary weight matrices selected at step 204e-1 based on the number of bits required to encode each matrix (e.g., the minimum number of bits required for encoding).

[0098] like Figure 6 As shown, according to another representative embodiment of the present invention, the scheme selection module 102c (at step 204e-1') first selects one or more potential binary weight matrices based on the number of bits required to encode each matrix (e.g., the minimum number of bits required, or the number of bits required is below a predetermined bit threshold). At step 204e-2', the scheme selection module 102c selects one from the potential binary weight matrices selected at step 204e-1' based on the matrix-specific probability value of each matrix (e.g., having the highest matrix-specific probability value).

[0099] refer to Figure 3 At step 204e, processor 102 provides weight generation module 102d to generate data for one or more binary weights based on the potential binary weight matrix selected in step 204e.

[0100] Figure 4 The diagram shown is an exemplary flowchart of key steps in another binarization method 204' according to a representative embodiment of the present invention. Except for steps 204a-1', 204a-2', and 204a-3', the steps in binarization method 204' are similar to... Figure 3The method 204 shown is essentially the same (where the same reference numerals indicate the same steps). At step 204a-1', a binarized weight matrix is ​​generated based on the sign of each floating-point input weight (e.g., assigning a binary weight +1 to input weights equal to or greater than 0, and assigning a binary weight -1 to input weights less than 0). At step 204a-2', the binarized weight matrix is ​​stored in memory 104, and steps 204a-1' and 204a-2' are repeated for a predetermined number of iterations (N), wherein, for example, different sets of input weights are generated based on different training data used in each iteration. At step 204a-3', "local" weight probabilities are generated for each input weight based on the stored binarized weight matrix. For example, in step 204a-3', the "local" probability of a weight can be determined based on the count (n) of binary weights whose weight is assigned +1 in N iterations (in this case, the probability of a weight being associated with binary +1 can be generated based on the relation n / N). Similarly, a “local” probability can be generated that weight is associated with a binary value of -1 (e.g., based on the relation 1-n / N).

[0101] In this application, unless otherwise specified, the terms “comprising”, “comprise”, and their grammatical variations are intended to indicate open or inclusive language such that they include not only the listed elements but also additional, non-explicitly listed elements.

[0102] While the invention has been described above with reference to exemplary embodiments, those skilled in the art will understand that various changes and equivalent substitutions of its elements can be made without departing from the spirit and scope of the invention. Furthermore, modifications can be made to adapt the invention to specific situations and materials without departing from its essential scope. Therefore, the invention is not limited to the specific examples disclosed in this specification, but covers all embodiments falling within the scope of the appended claims.

Claims

1. A processor for a neural network, the processor comprising: A weight probability analysis module is configured to generate data representing the probability of each input weight in the set of input weights being associated with a binary value, based on a set of input weights from one or more layers of a neural network. The input weights are determined based on errors associated with training results and randomly determined floating-point weight values ​​from one or more layers of the neural network obtained by a processor. The neural network is further configured to perform a training task on a set of training data to determine training results, the training data including multiple training images related to different topics. The multiplication and addition operations performed on the floating-point weights in the convolutional layers of the neural network require significant data processing and memory resources, as well as substantial memory resources to store the weights in each convolutional or fully connected layer of the neural network. A scheme generation module is configured to generate data representing at least one or more potential binary weight matrices based on probabilities determined for at least one selected group of the weights. A scheme selection module is configured to at least: generate data representing matrix-specific probability values ​​for each of the potential binary weight matrices; generate data representing the number of data bits for each of the potential binary weight matrices according to a predetermined encoding method; and select a potential binary weight matrix based on the matrix-specific probability values ​​and the number of data bits; and A weight generation module is configured to generate data representing one or more binary weights based on the selected potential binary weight matrix, and store the data in memory for use by the processor in performing training tasks.

2. The processor according to claim 1, wherein, The set of input weights includes: Some input weights selected from at least one of the input weights of said layer; Some input weights are selected from the input weights of all the layers; All input weights of at least one of the layers; or All input weights of all the layers.

3. The processor according to claim 1, configured to convert the input weights into corresponding weight values ​​within a predetermined weight range, and to use the corresponding weight values ​​as input weights.

4. The processor of claim 1, configured to generate the data representing the probability of each of the input weights based on: The predetermined relationship between the different potential values ​​of the input weights and their corresponding probabilities; One or more previously determined probabilities of the weights corresponding to the input weights; or One or more previously determined weight values ​​corresponding to the input weights.

5. The processor according to claim 4, wherein, The previously determined probabilities and / or the previously determined weight values ​​of the weights are determined based on training events performed by the neural network.

6. The processor of claim 1, configured to select the weight group based on a predetermined selection criterion.

7. The processor according to claim 6, wherein, The predetermined selection criteria include at least one of the following: One or more weights from a selected row of the kernel of the convolutional layer; One or more weights from selected columns of the kernel of the convolutional layer; One or more weights from different kernels associated with the same channel of the convolutional layer; One or more weights from different kernels of different channels associated with the same filter in the convolutional layer; One or more input weights of a fully connected layer; as well as One or more output weights of a fully connected layer.

8. The processor of claim 1, configured to determine a binary weight value for each of the input weights based on a comparison of the data representing the probability of the input weights with a predetermined probability threshold.

9. The processor according to claim 8, wherein, Based on the comparison, the input weights are determined as follows: Associated with the first binary value; Associated with the second binary value; or It is associated with either the first binary value or the second binary value.

10. The processor of claim 9, configured to generate a plurality of the potential binary weight matrices based on a plurality of input weights determined to be associated with the first binary value or the second binary value, wherein, Each of the potential binary weight matrices includes different combinations of the first binary value or the second binary value associated with the input weights.

11. The processor of claim 1, configured to generate the matrix-specific probability of each of the latent binary weight matrices based on the probability of the input weights in each of the latent binary weight matrices.

12. The processor according to claim 1, wherein, The encoding method is at least one of generalized run-length encoding and generalized Huffman encoding.

13. The processor according to claim 1, configured as follows: Based on data representing matrix-specific probability values ​​for each of the potential binary weight matrices, one or more of the potential binary weight matrices are selected; and According to a predetermined encoding method, a potential binary weight matrix is ​​selected from the selected potential binary weight matrices based on the data representing the number of data bits of each potential binary weight matrix.

14. The processor of claim 13, configured to: Select one or more of the potential binary weight matrices whose corresponding probability values ​​are higher than a specific value; and select one potential binary weight matrix from the selected potential binary weight matrices whose corresponding number of data bits is the lowest.

15. The processor according to claim 1, configured as follows: According to a predetermined encoding method, one or more latent binary weight matrices are selected from the latent binary weight matrices based on the number of data bits representing each latent binary weight matrix; and Based on the data representing matrix-specific probability values ​​for each of the potential binary weight matrices, a potential binary weight matrix is ​​selected from the selected potential binary weight matrices.

16. The processor of claim 15, configured to: Select one or more of the potential binary weight matrices whose corresponding number of data bits is lower than a specific value; and select one potential binary weight matrix from the selected potential binary weight matrices whose corresponding matrix has the highest specific probability value.

17. A binarization method for weights in a neural network, the method comprising: A set of input weights from one or more layers of a neural network generates data representing at least the probability that each of the input weights in the set is associated with a binary value; wherein the input weights are determined based on errors associated with training results and floating-point weight values ​​randomly determined by a processor for one or more layers of the neural network, the neural network being configured to perform a training task on a set of training data to determine training results, the training data including multiple training images related to different topics, the multiplication and addition operations performed on the floating-point weights in the convolutional layers of the neural network require significant data processing and memory resources, as well as significant memory resources to store the weights in each convolutional or fully connected layer of the neural network; For at least one selected group of the weights, data representing at least one or more potential binary weight matrices is generated based on probabilities determined for the selected weights. Generate data representing matrix-specific probability values ​​for each of the latent binary weight matrices; generate data representing the number of data bits for each latent binary weight matrix according to a predetermined encoding method; and select the latent binary weight matrix based on the matrix-specific probability values ​​and the number of data bits; and Data representing one or more binary weights is generated based on the selected potential binary weight matrix, and the data is stored in memory for use by the processor in performing training tasks.

18. The method according to claim 17, wherein, The set of input weights includes: Some input weights selected from at least one of the input weights of said layer; Some input weights are selected from the input weights of all the layers; All input weights of at least one of the layers; or All input weights of all the layers.

19. The method of claim 17, further comprising converting the input weights into corresponding weight values ​​within a predetermined weight range, and using the corresponding weight values ​​as input weights.

20. The method of claim 17, further comprising generating the data representing the probability of each of the input weights based on: The predetermined relationship between the different potential values ​​of the input weights and their corresponding probabilities; One or more previously determined probabilities of the weights corresponding to the input weights; or One or more previously determined weight values ​​corresponding to the input weights.

21. The method according to claim 20, wherein, The previously determined probabilities and / or the previously determined weight values ​​of the weights are determined based on training events performed by the neural network.

22. The method of claim 17, further comprising selecting the weight grouping based on a predetermined selection criterion.

23. The method according to claim 22, wherein, The predetermined selection criteria include at least one of the following: One or more weights from a selected row of the kernel of the convolutional layer; One or more weights from selected columns of the kernel of the convolutional layer; One or more weights from different kernels associated with the same channel of the convolutional layer; One or more weights from different kernels of different channels associated with the same filter in the convolutional layer; One or more input weights of a fully connected layer; as well as One or more output weights of a fully connected layer.

24. The method of claim 17, further comprising determining a binary weight value for each of the input weights based on a comparison of the data representing the probability of the input weights with a predetermined probability threshold.

25. The method according to claim 24, wherein, Based on the comparison, the input weights are determined as follows: Associated with the first binary value; Associated with the second binary value; or It is associated with either the first binary value or the second binary value.

26. The method of claim 25, further comprising generating a plurality of the potential binary weight matrices based on a plurality of the input weights already determined to be associated with the first binary value or the second binary value, wherein, Each of the potential binary weight matrices includes different combinations of weight values.

27. The method of claim 17, further comprising generating the matrix-specific probability of each of the latent binary weight matrices based on the probability of the input weights in each of the latent binary weight matrices.

28. The method according to claim 17, wherein, The encoding method is at least one of generalized run-length encoding and generalized Huffman encoding.

29. The method of claim 17, comprising: Based on data representing matrix-specific probability values ​​for each of the potential binary weight matrices, one or more of the potential binary weight matrices are selected; and According to a predetermined encoding method, a potential binary weight matrix is ​​selected from the selected potential binary weight matrices based on the data representing the number of data bits of each potential binary weight matrix.

30. The method of claim 29, comprising: Select one or more of the potential binary weight matrices, the corresponding matrix having a specific probability value higher than a specific value; and Choose one potential binary weight matrix from the selected potential binary weight matrices, the one with the lowest number of data bits.

31. The method of claim 17, comprising: Based on a predetermined encoding method and the number of data bits representing each of the potential binary weight matrices, one or more of the potential binary weight matrices are selected; and Based on the data representing matrix-specific probability values ​​for each of the potential binary weight matrices, a potential binary weight matrix is ​​selected from the selected potential binary weight matrices.

32. The method of claim 31, comprising: Select one or more of the potential binary weight matrices, the number of data bits corresponding to which is lower than a specific value; Furthermore, from the selected potential binary weight matrices, one potential binary weight matrix is ​​chosen, whose corresponding matrix has the highest specific probability value.

Images