Method, electronic device, and computer-readable medium for neural network computation

Abstract

The present disclosure provides a method for neural network calculation, comprising: dividing a weight matrix into multiple groups, and storing non-zero weights in each group in a corresponding memory slice respectively; wherein the weight matrix is a connection weight matrix of neurons; each group comprises multiple consecutive columns of the weight matrix; the memory slice corresponds to a group one by one, and stores the value and position of the weight; and calculating output data according to the input data corresponding to the weight matrix and the weight stored in the memory slice. The present disclosure also provides an electronic device and a computer readable medium.

Description

Technical Field

[0001] This disclosure relates to the field of neural network technology, and in particular to a method for neural network computation, an electronic device, and a computer-readable medium. Background Technology

[0002] In the use of neural networks, the calculation of the weight matrix accounts for a large amount of computation. For example, in spiking neural networks based on the LIF (Leaky Integrity-and-Fire) model, the calculation of the synaptic integral of the weight matrix is ​​the main load.

[0003] Therefore, the calculation of the weight matrix requires a large amount of memory space, access bandwidth, and computing resources, resulting in a large computational overhead. Summary of the Invention

[0004] This disclosure provides a method for neural network computation, an electronic device, and a computer-readable medium.

[0005] In a first aspect, embodiments of this disclosure provide a method for neural network computation, comprising:

[0006] The weight matrix is ​​divided into multiple groups, and the non-zero weights in each group are stored in the corresponding memory slices. The weight matrix is ​​the connection weight matrix of the neuron. Each group includes multiple consecutive columns of the weight matrix. The memory slices correspond one-to-one with the groups and store the values ​​and positions of the weights.

[0007] The output data is calculated based on the input data corresponding to the weight matrix and the weights stored in the memory slices.

[0008] In some embodiments, at most one weight per row of each group is stored in a memory slice; the calculation of output data based on the input data corresponding to the weight matrix and the weights stored in the memory slice includes:

[0009] Repeat the following multiplication operation: synchronously input the current weight values ​​of each memory chip into the corresponding multiplier, and synchronously input the corresponding data from the input data into each multiplier;

[0010] In each multiplication operation, the multiplication results output by each multiplier are shifted according to their respective weights. The multiplication results that are in the same position after shifting are then accumulated to calculate the output data.

[0011] In some embodiments, the neural network is a spiking neural network, and the weight matrix is ​​a synaptic connection weight matrix.

[0012] In some embodiments, before dividing the weight matrix into multiple groups, the method further includes:

[0013] Swap the positions of some columns in the weight matrix.

[0014] In some embodiments, each group has one and only one weight stored in a memory slice for each row, and all weights stored in the memory slice are non-zero weights.

[0015] In some embodiments, storing the non-zero weights in each group in their respective memory slices includes:

[0016] In response to a row in a set where all weights are zero, store one of the zero-weight elements in the corresponding memory slice.

[0017] In some embodiments, any group has the same number of columns.

[0018] In some embodiments, the position of the weight is the offset of the weight within the row of its group.

[0019] In some embodiments, shifting the multiplication result output by each multiplier in each multiplication operation according to the position of the corresponding weight includes: inputting the multiplication result output by each multiplier in each multiplication operation to the corresponding shifter, and inputting the position of each corresponding weight to the corresponding shifter; the shifter shifts the multiplication result according to the corresponding position;

[0020] The step of accumulating the multiplication results that are in the same position after shifting includes: inputting the output of each shifter to the corresponding accumulator, and the accumulator accumulating the multiplication results that are in the same position.

[0021] In a second aspect, embodiments of this disclosure provide an electronic device, including:

[0022] The processing unit includes multiple memory chips;

[0023] A storage unit on which computer programs are stored;

[0024] When the computer program is executed by the processing unit, it implements any of the above-described methods for neural network computation.

[0025] Thirdly, embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any of the above-described methods for neural network computation.

[0026] In this embodiment of the disclosure, only the non-zero weights of the weight matrix need to be stored in memory, which is equivalent to "compressed encoding" of the weight matrix, thereby reducing the memory space and access bandwidth required for calculation and reducing computational overhead.

[0027] Furthermore, in this embodiment of the present disclosure, only the non-zero weights of the weight matrix can be calculated, thereby reducing the amount of computation required, further reducing computational overhead, and improving computational efficiency.

[0028] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0029] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the embodiments of the present disclosure to explain the disclosure and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed description of exemplary embodiments with reference to the accompanying drawings, in which:

[0030] Figure 1 A flowchart illustrating a method for neural network computation provided in this disclosure embodiment;

[0031] Figure 2 A flowchart illustrating another method for neural network computation provided in this disclosure embodiment;

[0032] Figure 3 A schematic diagram of the grouping process in a neural network computation method provided in this embodiment of the disclosure;

[0033] Figure 4 A schematic diagram illustrating the storage format of weights in a neural network calculation method provided in this embodiment of the present disclosure;

[0034] Figure 5 A block diagram of an electronic device provided in an embodiment of this disclosure;

[0035] Figure 6 A schematic diagram of the architecture of a processing unit in an electronic device provided in this disclosure embodiment;

[0036] Figure 7 This is a block diagram illustrating the composition of a computer-readable medium provided in an embodiment of the present disclosure. Detailed Implementation

[0037] To enable those skilled in the art to better understand the technical solutions of this disclosure, exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments of this disclosure to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0038] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0039] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0040] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Words such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0041] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0042] Firstly, referring to Figures 1 to 4 , Figure 6 This disclosure provides a method for neural network computation.

[0043] The method of this disclosure embodiment is used for computation in a neural network.

[0044] Among them, neural networks can realize any pre-trained function, such as image recognition, action recognition, speech recognition, and text recognition.

[0045] In this embodiment, the method of present disclosure is used to calculate the product of the input data input to a neuron in a neural network and the corresponding connection weight matrix. That is, in a neural network, neurons in different layers are interconnected, and the connections between different neurons have different weight matrices, so the input data input to a neuron needs to be multiplied by the weight matrix of that neuron.

[0046] In some embodiments, the neural network is a spiking neural network, and the weight matrix is ​​a synaptic connection weight matrix.

[0047] As one embodiment of this disclosure, the applicable neural network may be a spiking neural network (SNN), wherein the weight matrix is ​​the weight matrix of the synaptic connections between corresponding neurons.

[0048] For example, the above spiking neural network can be based on the LIF (Leaky Integrity-and-Fire) model.

[0049] In the LIF model, the pulse current input signal (Xi) of the pre-neuron i is connected to the current neuron through the weight matrix (Wi). When the membrane voltage (Vt) of the current neuron accumulates (integrates) due to the charging of all the input current signals of the pre-neurons and reaches a predetermined threshold (Vth), the current neuron will emit a pulse current signal and reset the membrane voltage (Vt) to the resting voltage (Vreset).

[0050] Of course, the scope of application of the embodiments disclosed herein is not limited thereto. They can also be used in other modes of spiking neural networks, or in the calculation of weight matrices in other types of neural networks (such as convolutional neural networks).

[0051] The embodiments of this disclosure are suitable for processing sparse weight matrices, such as weight matrices with a sparsity (i.e., the proportion of zero elements) of at least 50%, at least 70%, at least 90%, or at least 95%.

[0052] Reference Figure 1 The method in this disclosure includes:

[0053] S101. Divide the weight matrix into multiple groups and store the non-zero weights in each group in their respective memory slices.

[0054] The weight matrix is ​​the connection weight matrix of the neurons; each group includes multiple consecutive columns of the weight matrix; each memory slice corresponds to a group and stores the weight values ​​and positions.

[0055] S102. Calculate the output data based on the input data corresponding to the weight matrix and the weights stored in the memory slice.

[0056] Reference Figure 3 In this embodiment of the disclosure, before calculating the weight matrix, the weight matrix is ​​first vertically grouped, so that each group includes multiple consecutive columns of the weight matrix; then, all non-zero weights in each group are stored in a corresponding memory slice, and the memory slice stores the value (the "numerical value" of the weight) and position (the "relative position" or "offset" of the weight in the row, group, or weight matrix); subsequently, the corresponding calculation can be performed based on the weights stored in the memory slice.

[0057] Obviously, the result of zero-weight calculation is also zero, which is meaningless. Therefore, in this embodiment, only the non-zero weights of the weight matrix need to be stored in memory, which is equivalent to "compressed encoding" of the weight matrix, thereby reducing the storage and access bandwidth required for calculation and reducing computational overhead.

[0058] In some embodiments, at most one weight per group per row is stored in a memory slice.

[0059] The output data (S102) calculated based on the input data corresponding to the weight matrix and the weights stored in the memory slice includes:

[0060] S1021. Repeat the following multiplication operation: synchronously input the current weight values ​​of each memory chip to the corresponding multiplier, and synchronously input the corresponding data from the input data to each multiplier.

[0061] S1022. Shift the multiplication results output by each multiplier in each multiplication operation according to their respective weights, and accumulate the multiplication results that are in the same position after shifting to calculate the output data.

[0062] As one embodiment of this disclosure, reference is made to... Figure 3 After grouping, each row in each group has at most one non-zero weight (represented by black squares in the diagram, and colorless squares represent zero weights), so each row in each group has at most one weight stored in the memory slice.

[0063] Of course, to achieve the above grouping, the weight matrix should usually be a sparse weight matrix.

[0064] Reference Figure 2 When the above grouping is satisfied, the weight values ​​can be obtained synchronously from the memory slices and input into the corresponding multipliers respectively. Since there is at most one weight per row in each memory slice, the weights output by each memory slice must correspond to the same row. Therefore, the data corresponding to the row in the input data can be input into each multiplier for multiplication calculation at the same time.

[0065] By repeatedly performing the above multiplication operation, the weights of each row can be processed in sequence, so that each multiplier outputs the calculated value (multiplication result) of the weights of each row in sequence.

[0066] Reference Figure 6 Each multiplier outputs a multiplication result that only represents the numerical value of the corresponding calculation result, without specifying the position of that result. Therefore, it is necessary to shift the multiplication result to the corresponding position (the position of the corresponding weight in the row of the group) according to the position of the corresponding weight, and then sum all the multiplication results at the same position (equivalent to summing the calculation results of each row). After concatenation and processing, the final result of the weight matrix calculation (output data) can be obtained.

[0067] In the above calculation process, it is equivalent to processing the data of different groups in different "channels", and each channel is only used to process a part of the data of the weight matrix. Therefore, the computational overhead can be further reduced and the computational efficiency can be improved.

[0068] Moreover, in the above calculation process, only the "non-zero weights" were actually calculated, while the "zero weights" were not calculated, which greatly reduced the total amount of calculation and could further reduce the computational cost.

[0069] In some embodiments, refer to Figure 2 Before dividing the weight matrix into multiple groups, the following steps are also included:

[0070] S100, Swap the positions of some columns in the weight matrix.

[0071] The above scheme requires that each row of each group has at most one non-zero weight, but the weight matrix in the neural network may not always meet this requirement. Therefore, for weight matrices that do not meet the requirement, the positions of some columns can be swapped to change the group in which some weights belong, making it easier for the swapped weight matrix to achieve the above grouping.

[0072] Of course, when the column positions are swapped, the corresponding data in the weight calculation results should also be swapped before they can be actually output as the final output data.

[0073] Of course, in order to obtain a weight matrix (sparse weight matrix) that can perform the above grouping, restrictions can be imposed on the weight matrix during the training process of the neural network.

[0074] For example, the sparsity of the weight matrix can be constrained during training.

[0075] For example, the sparsity of the weight matrix can also be made to meet the requirements during training by using mask constraints.

[0076] For example, during training, for each row in a predetermined group, only the largest weight can be retained, while the other weights can be set to zero.

[0077] In some embodiments, each group has one and only one weight stored in a memory slice for each row, and all weights stored in the memory slice are non-zero weights.

[0078] As one embodiment of this disclosure, reference is made to... Figure 3 After grouping, the weight matrix can have one and only one non-zero weight in any row of any group.

[0079] In some embodiments, storing the non-zero weights in each group in their respective memory slices includes: in response to the case that all weights in a row of a group are zero weights, storing one of the zero weights in the corresponding memory slice.

[0080] When all weights in a row of a group are zero, it may result in no corresponding data being stored in the memory slice for that row, and no data available for subsequent multiplication operations. This can lead to inconsistent processing progress across different memory slices (channels), causing a mismatch between the input data and weights entering the multiplier, resulting in errors. Therefore, in this case, a zero weight (which can be random or at a specific location) can be stored in the corresponding memory slice for each row, so that the zero weight can be used for subsequent calculations (of course, the calculation result will be zero).

[0081] Of course, when all weights in a certain row of a certain group are zero, it is also feasible to achieve synchronous processing by not storing zero weights in the memory slice, but by having the corresponding memory slice "pause" output when processing the corresponding row.

[0082] In some embodiments, any group has the same number of columns.

[0083] As one embodiment of this disclosure, the number of columns in all groups can be the same. For example, see [reference needed]. Figure 3 There are 4 columns in each group.

[0084] Of course, the specific number of columns in each group is not limited to this, and can be determined according to the weight matrix. For example, the number of columns in each group can be at least 2, or at least 4, or at least 10, etc.; or the number of columns in each group can not exceed 20, or not exceed 10, or not exceed 4, etc.

[0085] In some embodiments, the position of a weight is the offset of the weight within the row of its group.

[0086] Reference Figure 4 As one embodiment of this disclosure, the position of a weight can be the offset (or relative position) of the weight in the corresponding row of the corresponding group, such as the weight being at position 0, position 1, position 2, position 3, etc. in that row.

[0087] Thus, when the number of columns in a group is N, the position of the weight only needs to occupy log2N bits, which can further save memory space.

[0088] In some embodiments, shifting the multiplication result output by each multiplier in each multiplication operation according to the position of the corresponding weight includes: inputting the multiplication result output by each multiplier in each multiplication operation to the corresponding shifter, and inputting the position of each corresponding weight to the corresponding shifter; the shifter shifts the multiplication result according to the corresponding position;

[0089] Accumulating the multiplication results that are in the same position after shifting involves: inputting the output of each shifter to the corresponding accumulator, and the accumulator sums the multiplication results that are in the same position.

[0090] As one embodiment of this disclosure, each channel further includes a shifter and an accumulator, so that after the output of the multiplier and the position of the weight are both input to the shifter, the shifter can perform the shifting, and the output of the shifter then enters the corresponding accumulator for accumulation.

[0091] In some embodiments, the value and location of each weight may be stored at a memory address; thus, it can be referenced. Figure 6 The weights in the memory slices are obtained through a splitter. The splitter separates the weight values ​​and positions, and inputs the values ​​to the corresponding multipliers and the positions to the corresponding shifters.

[0092] Secondly, referring to Figure 5 , Figure 6 This disclosure provides an electronic device that can implement the above-described neural network calculation method.

[0093] Reference Figure 5 The electronic device provided in this disclosure includes:

[0094] The processing unit comprises multiple memory chips;

[0095] The storage unit stores a computer program; when the computer program is executed by the processing unit, it implements any of the above-mentioned methods for neural network computation.

[0096] The above neural network calculation method can be implemented in the processing unit, therefore the processing unit should include the above memory chips.

[0097] The specific hardware forms of electronic devices, processing units, and storage units are diverse.

[0098] For example, a processing unit can be a single physical processor device or processing core device; or, a processing unit can be composed of multiple processor devices or processing core devices, such as one processor device or processing core device corresponding to each channel.

[0099] For example, the storage unit can be one or more independent memory devices, or it can be the memory device of the processing unit, as long as it stores a computer program that can be executed by the processing unit to implement any of the above-mentioned neural network calculation methods.

[0100] For example, the above electronic device can be a processor or processing core device, or it can be a many-core system consisting of multiple processing cores and on-chip networks.

[0101] In some embodiments, the architecture of the processing unit may refer to Figure 6 It also includes an address generator, input buffer, output buffer, and the aforementioned multipliers, shifters, accumulators, splitters, etc.; among which:

[0102] The address generator is used to determine the memory address to be processed, or in other words, to determine the data to be processed.

[0103] The input buffer is used to input the data corresponding to the current row of the input data to each multiplier according to the control of the address generator.

[0104] Each memory chip, under the control of the address generator, inputs its stored weight corresponding to the current row into the splitter.

[0105] The splitter is used to input the values ​​of each input weight to each multiplier, and to input the position of each weight to the shifter.

[0106] Each multiplier performs a multiplication calculation on the input weights and the input data, and inputs the multiplication result to the shifter.

[0107] Each shifter is used to shift the multiplication result to the corresponding position based on the position of the splitter input and the multiplication result of the multiplier input, and then input it to the accumulator.

[0108] Each accumulator is used to accumulate the input data. Specifically, it accumulates the data in the same position in the input data (the multiplication result) and outputs the result to the output buffer.

[0109] The output buffer is used to concatenate the final outputs of each accumulator to obtain the result of the weight matrix calculation (output data).

[0110] Among them, the memory chips, multipliers, shifters, and accumulators mentioned above can all be in one-to-one correspondence, that is, each channel includes one memory chip, one multiplier, one shifter, and one accumulator.

[0111] The above splitter can be a single unit, or it can be one-to-one with each channel.

[0112] However, there may only be one address generator, input buffer, and output buffer.

[0113] For example, regarding the reference Figure 3 The weight matrix is ​​divided into 4 groups, each with 4 columns. Assuming that each weight value requires 16 bits of memory space, the resource consumption is as follows when calculating it using related technologies (storing and calculating all weights) and the method of this embodiment:

[0114] Table 1. Comparison of Computational Resource Consumption

[0115]

[0116] As can be seen, when using the embodiments of this disclosure, the number of devices required for computation, memory space, access bandwidth, etc. are significantly reduced. Therefore, the embodiments of this disclosure can significantly reduce computational overhead and improve computational efficiency.

[0117] Thirdly, referring to Figure 7 This disclosure provides a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any of the above-described neural network computation methods.

[0118] It will be understood by those skilled in the art that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0119] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. A method for neural network computation, comprising: dividing a weight matrix into a plurality of groups, and storing non-zero weights in each group in a corresponding memory slice respectively, wherein the weight matrix is a connection weight matrix of neurons, each group comprises a plurality of consecutive columns of the weight matrix, and each memory slice stores values and positions of weights; computing output data according to input data corresponding to the weight matrix and the weights stored in the memory slices; in a case that all weights in any row of a group are zero weights, storing one zero weight in the any row in a corresponding memory slice, or not storing zero weights in the corresponding memory slice and indicating the memory slice to pause output in a computation process of the any row; wherein at most one weight in each row of each group is stored in a memory slice, and the computing output data according to input data corresponding to the weight matrix and the weights stored in the memory slices comprises: repeating the following multiplication operation: synchronously inputting values of the current weights in each memory slice to corresponding multipliers respectively, and synchronously inputting corresponding data in the input data to each multiplier; shifting multiplication results output by each multiplier in each multiplication operation according to positions of corresponding weights, and accumulating multiplication results located at the same position after shifting to compute output data. 2.The method of claim 1, wherein the neural network is a spiking neural network, and the weight matrix is a synaptic connection weight matrix.

3. The method of claim 1, wherein, Before the dividing the weight matrix into a plurality of groups, the method further comprises: interchanging positions of part of columns in the weight matrix. 4.The method of claim 1, wherein at most one weight in each row of each group is stored in a memory slice, and all weights stored in the memory slices are non-zero weights.

5. The method of claim 1, wherein, The storing non-zero weights in each group in a corresponding memory slice respectively comprises: in response to a case that all weights in a row of a group are zero weights, storing one zero weight in the corresponding memory slice. 6.The method of claim 1, wherein a number of columns of any group is the same. 7.The method of claim 1, wherein the position of the weight is an offset of the weight in the row of the group where the weight is located. 8.The method of claim 1, wherein the shifting multiplication results output by each multiplier in each multiplication operation according to positions of corresponding weights comprises: synchronously inputting multiplication results output by each multiplier in each multiplication operation to corresponding shifters respectively, and inputting positions of corresponding weights to corresponding shifters respectively; and the shifters shift the multiplication results according to the corresponding positions; the accumulating multiplication results located at the same position after shifting comprises: inputting outputs of the shifters to corresponding accumulators respectively, and the accumulators accumulate multiplication results located at the same position. 9.An electronic device, comprising: a processing unit comprising a plurality of memory slices; a storage unit having a computer program stored thereon; the computer program, when executed by the processing unit, implements the method for neural network computation of any one of claims 1 to 8.

10. A computer readable medium having stored thereon a computer program, wherein, The computer program, which when executed by a processor implements the method of neural network computation of any one of claims 1 to 8.

Images