Sparse data processing method and device of neural network processor

Abstract

The present disclosure provides a sparse data processing method and device of a neural network processor, the neural network processor comprising a basic computing unit, the method comprising: obtaining a plurality of groups of weight sub-vectors, wherein the weight sub-vectors are obtained by sparse processing of a to-be-computed weight vector based on an information unit supported by the basic computing unit; determining a to-be-computed feature vector corresponding to the to-be-computed weight vector; controlling the basic computing unit to perform vector inner product operation on each group of weight sub-vectors and the to-be-computed feature vector to obtain vector operation results; and performing shift operation on part of the group vector operation results, adding the vector operation results obtained by the shift operation to the remaining group vector operation results, and taking the addition result as a sparse data processing result, wherein the part of the group vector operation results and the remaining group vector operation results together constitute the plurality of group vector operation results. Through the present disclosure, the distribution of weights can be fully utilized, the data processing accuracy of the neural network processor can be effectively guaranteed, the hardware unit cost and the execution power consumption can be effectively reduced, and the weight storage space can be effectively reduced.

Description

Technical Field

[0001] This disclosure relates to the field of deep learning technology, and in particular to a method and apparatus for sparse data processing of a neural network processor. Background Technology

[0002] The core structure of deep learning neural network computation is matrix multiplication. Matrix computation generally consists of vector inner product calculations. The two inputs to matrix multiplication are features (or some integer operation on the features) and weights. On the inference side, quantization techniques are generally used to perform inference through integer calculations, such as INT8 and INT4.

[0003] In related technologies, the computation of deep learning neural networks generally exhibits sparse properties, with the weights of deep learning neural networks often following a normal distribution. For normally distributed data, the range of values ​​varies considerably. For a set of weights, the value range is more often distributed within the INT2 / INT4 range, and less often within the INT6 / INT8 range.

[0004] In this approach, existing neural network processors rely solely on INT8 inference, resulting in unnecessary computations on data belonging to the INT2 / INT4 / INT6 value range, consuming more area and power, and requiring more storage space for weights. Summary of the Invention

[0005] This disclosure aims to at least partially address one of the technical problems in the related art.

[0006] Therefore, the purpose of this disclosure is to propose a sparse data processing method, apparatus, computing unit array, neural network processor, electronic device, non-transient computer-readable storage medium storing computer instructions, and computer program product for a neural network processor, which can fully utilize the distribution of weights, effectively ensure the data processing accuracy of the neural network processor, effectively reduce hardware unit costs and execution power consumption, and effectively reduce weight storage space.

[0007] The first aspect of this disclosure discloses a method for sparsifying data processing in a neural network processor, the neural network processor comprising: a basic computing unit; the method comprising: acquiring multiple sets of weight sub-vectors, wherein the weight sub-vectors are obtained by sparsifying the weight vectors to be computed based on the information units supported by the basic computing unit; determining the feature vectors to be computed corresponding to the weight vectors to be computed; controlling the basic computing unit to perform a vector inner product operation on each set of weight sub-vectors and the feature vectors to be computed to obtain a vector operation result; performing a shift operation on a portion of the vector operation results, and adding the vector operation result obtained from the shift operation to the remaining vector operation results, and using the result obtained from the addition operation as the sparsified data processing result, wherein the partial vector operation results and the remaining vector operation results together constitute multiple sets of vector operation results.

[0008] The second aspect of this disclosure provides a sparse data processing apparatus for a neural network processor, the neural network processor comprising: a basic computing unit; the apparatus comprising: an acquisition module for acquiring multiple sets of weight sub-vectors, wherein the weight sub-vectors are obtained by sparse processing of the weight vectors to be computed based on the information units supported by the basic computing unit; a determination module for determining the feature vectors to be computed corresponding to the weight vectors to be computed; a control module for controlling the basic computing unit to perform a vector inner product operation on each set of weight sub-vectors and the feature vectors to be computed to obtain a vector operation result; and a calculation module for performing a shift operation on a portion of the vector operation results, adding the shifted vector operation result to the remaining vector operation results, and using the summed result as the sparse data processing result, wherein the partial vector operation results and the remaining vector operation results together constitute multiple sets of vector operation results.

[0009] The computing unit array proposed in the third aspect embodiment of this disclosure includes: a plurality of basic computing units, wherein different basic computing units support different information units, and the information units supported by the basic computing units are used to sparsify the weight vectors to be calculated to obtain multiple sets of weight sub-vectors, wherein the weight vectors to be calculated have corresponding feature vectors to be calculated; wherein the basic computing units are used to perform vector inner product operation on each set of weight sub-vectors and the feature vectors to be calculated to obtain vector operation results, and to perform shift operation on a portion of the vector operation results, and to add the vector operation results obtained by the shift operation to the remaining vector operation results, and to use the result of the addition operation as the sparsified data processing result, wherein the partial set of vector operation results and the remaining set of vector operation results together constitute multiple sets of vector operation results.

[0010] The neural network processor proposed in the fourth aspect of this disclosure includes: a processing unit and a computing unit array. The computing unit array includes: multiple basic computing units, each supporting a different information unit. The processing unit is configured to acquire multiple sets of weight sub-vectors, wherein the weight sub-vectors are obtained by sparsifying the weight vectors to be computed based on the information units supported by the basic computing units; determine the feature vectors to be computed corresponding to the weight vectors to be computed; control the basic computing units to perform a vector inner product operation on each set of weight sub-vectors and the feature vectors to be computed to obtain a vector operation result; perform a shift operation on a portion of the vector operation results; add the shifted vector operation result to the remaining vector operation results; and use the sum as the sparsified data processing result. The partial vector operation results and the remaining vector operation results together constitute multiple sets of vector operation results.

[0011] An electronic device according to a fifth aspect embodiment of this disclosure includes: at least one neural network processor; and a memory communicatively connected to the at least one neural network processor; wherein the memory stores instructions executable by the at least one neural network processor, the instructions being executed by the at least one neural network processor to enable the at least one neural network processor to perform a sparse data processing method for a neural network processor as proposed in a first aspect embodiment of this disclosure.

[0012] A sixth aspect of this disclosure provides a non-transitory computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the sparse data processing method of a neural network processor as proposed in the first aspect of this disclosure.

[0013] A seventh aspect of this disclosure provides a computer program product in which, when instructions in the computer program product are executed by a processor, the sparse data processing method of a neural network processor as described in a first aspect of this disclosure is performed.

[0014] Additional aspects and advantages of this disclosure will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this disclosure. Attached Figure Description

[0015] The above and / or additional aspects and advantages of this disclosure will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, in which:

[0016] Figure 1 This is a schematic flowchart of a sparse data processing method for a neural network processor according to an embodiment of the present disclosure;

[0017] Figure 2 This is a flowchart illustrating a sparse data processing method for a neural network processor according to another embodiment of this disclosure;

[0018] Figure 3a This is an application diagram of one embodiment of the present disclosure;

[0019] Figure 3b This is another application illustration in the embodiments of this disclosure;

[0020] Figure 4a This is another application illustration in the embodiments of this disclosure;

[0021] Figure 4b This is another application illustration in the embodiments of this disclosure;

[0022] Figure 5a This is another application illustration in the embodiments of this disclosure;

[0023] Figure 5b This is another application illustration in the embodiments of this disclosure;

[0024] Figure 6 This is a schematic diagram of the architecture of multiple basic computing units in an embodiment of this disclosure;

[0025] Figure 7 This is a schematic diagram of the structure of a sparse data processing device for a neural network processor according to an embodiment of the present disclosure;

[0026] Figure 8 This is a schematic diagram of the structure of a computing unit array proposed in an embodiment of this disclosure;

[0027] Figure 9 This is a schematic diagram of the structure of a neural network processor proposed in an embodiment of this disclosure;

[0028] Figure 10 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present disclosure is shown. Detailed Implementation

[0029] Embodiments of this disclosure are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are used only to explain this disclosure, and should not be construed as limiting this disclosure. Rather, embodiments of this disclosure include all variations, modifications, and equivalents falling within the spirit and scope of the appended claims.

[0030] Figure 1 This is a schematic flowchart of a sparse data processing method for a neural network processor proposed in an embodiment of this disclosure.

[0031] This embodiment illustrates the example of a sparse data processing method for a neural network processor being configured in a sparse data processing device for a neural network processor. In this embodiment, the sparse data processing method for a neural network processor can be configured in a sparse data processing device for a neural network processor. The sparse data processing device for a neural network processor can be located in a server or in an electronic device. This embodiment does not limit the scope of the invention.

[0032] This embodiment uses the example of a sparse data processing method for a neural network processor being configured in an electronic device. The electronic device includes hardware devices with various operating systems, such as smartphones, tablets, personal digital assistants, and e-readers.

[0033] It should be noted that the execution entity of the embodiments disclosed herein may be, in hardware, a central processing unit (CPU) in a server or electronic device, and in software, a related background service in a server or electronic device, without limitation.

[0034] The neural network processing unit (NPU) in this embodiment includes a basic computing unit, which can be understood as a basic computing unit in a deep learning neural network. This basic computing unit can be used to process input data, such as a set of feature vectors and a set of weight vectors. The basic computing unit can then perform online inference based on the input set of feature vectors and the corresponding set of weight vectors.

[0035] like Figure 1 As shown, the sparse data processing method of this neural network processor includes:

[0036] S101: Obtain multiple sets of weight sub-vectors, where the weight sub-vectors are obtained by sparsifying the weight vector to be computed based on the information units supported by the basic computing unit.

[0037] The information unit (i.e., bit) supported by the basic computing unit can be, for example, 4 bits x 16 bits, 8 bits x 8 bits, or 4 bits x 8 bits, and there are no restrictions on this.

[0038] The weight vector to be calculated can be represented by W, and the corresponding feature vector to be calculated can be represented by F. The basic computing unit can then perform inference based on the weight vector W and the feature vector F to obtain the inference result, which is the sparse data processing result, and can be represented by Y.

[0039] The weight sub-vector refers to the sparse partitioning of the weight vector W to be computed. When performing the sparse partitioning, the information units supported by the basic computing unit can be used to perform the sparse partitioning of the weight vector W to be computed, and there are no restrictions on this.

[0040] S102: Determine the feature vector to be calculated corresponding to the weight vector to be calculated.

[0041] In some embodiments of this disclosure, the weight sub-vector includes multiple weight sub-elements. The weight sub-elements are obtained by sparsifying the weight elements in the weight vector to be computed based on the information units supported by the basic computing unit, thereby enabling efficient sparsification computation of the weight vector to be computed and facilitating full utilization of the weight distribution.

[0042] After obtaining multiple sets of weight sub-vectors by sparsifying the weight vector to be calculated based on the information units supported by the basic computing unit, the feature vector to be calculated corresponding to the weight vector to be calculated can be determined, that is, the feature vector to be calculated corresponding to the weight vector W to be calculated can be determined.

[0043] S103: Control the basic calculation unit to perform a vector inner product operation on each set of weight sub-vectors and the feature vector to be calculated, so as to obtain the vector operation result.

[0044] After obtaining multiple sets of weight sub-vectors by sparsifying the weight vector to be calculated based on the information units supported by the basic computing unit, and determining the feature vector to be calculated corresponding to the weight vector to be calculated, the basic computing unit can be controlled to perform a vector inner product operation on each set of weight sub-vectors and the feature vector to be calculated to obtain the vector operation result.

[0045] For example, the basic computing unit can be controlled to select feature elements from the feature vector to be calculated based on the weight sub-vector. For example, several feature elements can be selected as feature sub-elements (the selected multiple feature sub-elements can form a feature sub-vector). Then, the vector inner product operation is performed based on the weight sub-vector and the feature sub-vector to obtain the vector operation result. There are no restrictions on this.

[0046] S104: Perform a shift operation on the results of partial group vector operations, add the result of the shift operation to the results of the remaining group vector operations, and use the result of the addition as the result of sparse data processing. The results of partial group vector operations and the results of the remaining group vector operations together constitute multiple groups of vector operation results.

[0047] Then, a shift operation can be performed on the results of the vector operations on some groups, and the vector operation results obtained from the shift operation can be added to the results of the vector operations on the remaining groups. The result of the addition can be used as the result of sparse data processing. The results of the vector operations on some groups and the results of the vector operations on the remaining groups together constitute multiple sets of vector operation results.

[0048] For example, in the offline computation stage: the weight vector W to be computed is preprocessed. Assuming that the weight vector W to be computed includes weight elements Wi, which are 8 bits, it is split into two groups of 4-bit weight elements. Then, several weight elements can be selected from each group of 4-bit weight elements (the selected weight elements can be called weight sub-elements, and weight sub-elements belonging to the same group can form a weight sub-vector). Then, based on whether each 4-bit weight element is 0, a sparsified index is constructed according to the judgment result. That is, in the offline computation stage, the weight vector to be computed is sparsified based on the information units supported by the basic computing unit to obtain the weight sub-vector. Then, in the online inference phase: based on the sparse index, sparse basic computational units are selected layer by layer to perform integer vector inner product operations. The information unit of the basic computational unit can be 4 bits x 8 bits. The result of the operation can be called the vector operation result. Then, a partial group of vector operation results are first shifted, and the vector operation result obtained from the shift operation is added to the remaining group of vector operation results. The result of the addition is used as the sparse data processing result. Similarly, it can also be applied to basic computational units with other information units. For example, assuming that the weight element Wi is 16 bits, it can be split into 4 groups of 4-bit weight elements, and sparse data processing can be performed using a 4-bit x 16-bit basic computational unit.

[0049] In some embodiments of this disclosure, there are multiple basic computing units, and different basic computing units support different information units. This enables the basic computing units to effectively adapt to the weight distribution of the weight vector to be computed, and effectively supports the hardware module design of the neural network processor for mixed-precision sparse computing, so that the hardware module supports sparse computing with variable precision.

[0050] In this embodiment, multiple sets of weight sub-vectors are obtained. These weight sub-vectors are obtained by sparsifying the weight vector to be calculated based on the information units supported by the basic computing unit. The feature vector to be calculated corresponding to the weight vector to be calculated is determined. The basic computing unit is controlled to perform a vector inner product operation on each set of weight sub-vectors and the feature vector to be calculated to obtain the vector operation result. A shift operation is performed on some of the vector operation results, and the vector operation result obtained by the shift operation is added to the vector operation results of the remaining sets of vectors. The result of the addition is used as the sparsified data processing result. The vector operation results of some sets of vectors and the remaining sets of vectors together constitute multiple sets of vector operation results. This can make full use of the weight distribution, effectively ensure the data processing accuracy of the neural network processor, effectively reduce the hardware unit cost and execution power consumption, and effectively reduce the weight storage space.

[0051] Figure 2 This is a flowchart illustrating a sparse data processing method for a neural network processor according to another embodiment of this disclosure.

[0052] like Figure 2 As shown, the sparse data processing method of this neural network processor includes:

[0053] S201: Based on the information units supported by the basic computing unit, determine multiple weight elements of first bits and / or multiple weight elements of second bits from the weight vector to be calculated, wherein the first bits are lower than the second bits.

[0054] In this embodiment, the weight vector W to be calculated is 8 bits. The weight element of the first bit can be, for example, the weight element of the lower four bits of the weight vector W to be calculated, and the weight element of the second bit can be, for example, the weight element of the higher four bits of the weight vector W to be calculated. That is, the first bit is lower than the second bit.

[0055] S202: Select a first number of weight elements from multiple weight elements of the first bit as the first weight sub-element, wherein each first weight sub-element has a corresponding first index.

[0056] In order to effectively achieve sparsification of the weight vector to be computed based on the information units supported by the basic computing unit, in this embodiment of the disclosure, several weight elements can be selected from the weight elements of the first bit, and the selected weight elements can be called the first weight sub-elements.

[0057] For example, sparsification calculation is performed on the lower four bits (e.g., 4 bits) of the weight vector W to be calculated. Six numbers, 0, 1, 3, 4, 6, and 7, are selected from the lower four bits (e.g., 4 bits) of the weight element as the first weight sub-element, and the first index W[3:0] of the first weight sub-element is formed. This first index can be used to index the position of the first weight sub-element in the weight vector W to be calculated.

[0058] S203: Select a second number of weight elements from multiple weight elements of the second bit as second weight sub-elements, wherein each second weight sub-element has a corresponding second index.

[0059] For example, sparsification calculation is performed on the high four bits (e.g., 4 bits) of the weight vector W to be calculated. Two numbers, 1 and 3, are selected from the high four bits (e.g., 4 bits) of the weight element as the second weight sub-element, and the second index W[7:4] of the second weight sub-element is formed. This second index can be used to index the position of the second weight sub-element in the weight vector W to be calculated.

[0060] S204: Combine the first weighted sub-element and / or the second weighted sub-element as multiple weighted sub-elements.

[0061] S205: Determine the feature vector to be calculated corresponding to the weight vector to be calculated.

[0062] S206: The control unit selects the first feature element corresponding to the first weighted sub-element from the feature vector to be calculated according to the first index.

[0063] The first index can be used to index the position of the first weight sub-element in the weight vector W to be calculated. The feature sub-element selected from the feature vector to be calculated based on the first index, which has the same position as the first weight sub-element, can be called the first feature sub-element.

[0064] S207: The control unit selects the second feature element corresponding to the second weight element from the first feature element according to the second index.

[0065] The second index can be used to index the position of the second weight sub-element in the weight vector W to be calculated. The feature sub-element selected from the first feature sub-element based on the second index, which has the same position as the second weight sub-element, can be called the second feature sub-element.

[0066] S208: Control the basic calculation unit to perform a vector inner product operation on the first weight sub-element and its corresponding first feature sub-element to obtain the first vector inner product result.

[0067] S209: Control the basic calculation unit to perform a vector inner product operation on the second weight sub-element and its corresponding second feature sub-element to obtain the second vector inner product result.

[0068] S210: Take the result of the first vector inner product and / or the result of the second vector inner product as the result of the vector operation.

[0069] S211: Perform a shift operation on the results of a partial group of vector operations, add the result of the shift operation to the result of the remaining group of vector operations, and use the result of the addition as the result of sparse data processing. The results of the partial group of vector operations and the results of the remaining group of vector operations together constitute multiple sets of vector operation results.

[0070] The example uses the result of a partial vector operation, which is the inner product of the second vector mentioned above.

[0071] A specific calculation example for the above steps can be illustrated as follows:

[0072] like Figure 3a and Figure 3b As shown, Figure 3a This is an application diagram of one embodiment of the present disclosure. Figure 3b This is another application illustration in the embodiments of this disclosure. Figure 3a The offline calculation process is described. Figure 3b The online computation process is described, where W represents the weight vector to be computed and F represents the feature vector to be computed. For example, both W and F are 8 bits. W can be split into 4 bits for sparsification.

[0073] During offline calculations:

[0074] Perform sparsification calculation on the lower 4 bits of W (wherein, when performing sparsification calculation on the lower bits and the higher bits separately, the division of the lower bits and the higher bits can be achieved by judging the value range. In order to maintain precision, the division of the lower bits and the higher bits can be fine-tuned), and select 6 numbers (the first weighted sub-elements) of 0, 1, 3, 4, 6, and 7.

[0075] Sparsification calculation is performed on the high 4 bits of W to select 2 numbers (second weighted sub-elements).

[0076] Store the selected 4-bit number and its index (first index and / or second index).

[0077] During online calculations:

[0078] For the 8-bit feature vector F to be calculated, select data by selecting 6 numbers (first feature sub-elements) according to the index of the lower 4 bits of the feature vector W to be calculated, and record them as F_stage1. Perform a 4-bit x 8-bit inner product operation with the corresponding 6 numbers of W (0, 1, 3, 4, 6, 7) to obtain the result A.

[0079] For F_stage1, select two numbers (second feature elements) based on the index of the high 4 bits of W (second index), and perform a 4-bit x 8-bit inner product calculation with the corresponding W to obtain the result B.

[0080] Calculate A + (B << 4) to obtain the inner product calculation result (shift B and then add it to A), and then feed the calculation result into the accumulator for accumulation.

[0081] The above scheme can be called 8:6:2, which means that for 8-bit vector data (feature vector or weight vector to be calculated), 6 numbers are selected for the low bits and 2 numbers are selected for the high bits.

[0082] like Figure 4a and Figure 4b As shown, Figure 4a This is another application illustration in the embodiments of this disclosure. Figure 4b This is another application illustration in the embodiments of this disclosure. Figure 4a The offline calculation process is described. Figure 4b The online computation process is described, where W represents the weight vector to be calculated and F represents the feature vector to be calculated. An example is given where both W and F are 8 bits; W can be split into 4-bit segments for sparsity processing. A special case is shown where the second stage selects all data, called 8:4:4, where the same number of data bits are selected for both the low and high bits.

[0083] like Figure 5a and Figure 5b As shown, Figure 5a This is another application illustration in the embodiments of this disclosure. Figure 5b This is another application illustration in the embodiments of this disclosure. Figure 5a The offline calculation process is described. Figure 5b The online computation process is described, where W represents the weight vector to be calculated and F represents the feature vector to be calculated. An example is given where both W and F are 8 bits; W can be split into 4-bit segments for sparsity processing. A special case is shown where no data is selected in the second stage, referred to as 8:8:0, where 8 numbers are selected from the low bits and none from the high bits.

[0084] In some embodiments of this disclosure, there are multiple basic computing units, and different basic computing units support different information units. This enables the basic computing units to effectively adapt to the weight distribution of the weight vector to be computed, and effectively supports the hardware module design of the neural network processor for mixed-precision sparse computing, so that the hardware module supports sparse computing with variable precision.

[0085] In some embodiments of this disclosure, the multiple basic computing units may respectively support the above-mentioned Figures 3a-5b The application scenarios shown will support the above. Figures 3a-5b The application scenarios shown are used in combination to achieve sparse hybrid computing, which can realize a wider range of inner product calculations.

[0086] like Figure 6 As shown, Figure 6 This is a schematic diagram of the architecture of multiple basic computing units in the embodiments of this disclosure. Each sparse inner product module can be regarded as a basic computing unit. Different basic computing units implement sparse computation on W (W0, W1, W2, W3) and corresponding F (F0, F1, F2, F3) with different weight distributions, based on the sparse data processing method of the neural network processor provided in the above embodiments.

[0087] In this embodiment, multiple sets of weight sub-vectors are obtained. These weight sub-vectors are obtained by sparsifying the weight vector to be calculated based on the information units supported by the basic computing unit. The feature vector to be calculated corresponding to the weight vector to be calculated is determined. The basic computing unit is controlled to perform a vector inner product operation on each set of weight sub-vectors and the feature vector to be calculated to obtain the vector operation result. A shift operation is performed on some of the vector operation results, and the vector operation result obtained by the shift operation is added to the vector operation results of the remaining sets of vectors. The result of the addition is used as the sparsified data processing result. The vector operation results of some sets of vectors and the remaining sets of vectors together constitute multiple sets of vector operation results. This can make full use of the weight distribution, effectively ensure the data processing accuracy of the neural network processor, effectively reduce the hardware unit cost and execution power consumption, and effectively reduce the weight storage space.

[0088] Figure 7 This is a schematic diagram of the structure of a sparse data processing device for a neural network processor according to an embodiment of the present disclosure.

[0089] The processor includes: basic computing units.

[0090] like Figure 7 As shown, the sparse data processing device 70 of the neural network processor includes:

[0091] The acquisition module 701 is used to acquire multiple sets of weight sub-vectors, wherein the weight sub-vectors are obtained by sparsifying the weight vector to be calculated based on the information units supported by the basic computing unit.

[0092] The determination module 702 is used to determine the feature sub-vectors corresponding to each set of weight sub-vectors.

[0093] The control module 703 is used to control the basic calculation unit to perform vector inner product operation on each group of weight sub-vectors and corresponding feature sub-vectors to obtain the vector operation result.

[0094] The calculation module 704 is used to perform shift operations on the results of partial group vector operations, add the result of the shift operation to the result of the remaining group vector operations, and use the result of the addition as the result of sparse data processing. The results of partial group vector operations and the results of the remaining group vector operations together constitute multiple groups of vector operation results.

[0095] It should be noted that the foregoing explanation of the sparse data processing method for neural network processors also applies to the sparse data processing device for neural network processors in this embodiment, and will not be repeated here.

[0096] In this embodiment, multiple sets of weight sub-vectors are obtained. These weight sub-vectors are obtained by sparsifying the weight vector to be calculated based on the information units supported by the basic computing unit. The feature vector to be calculated corresponding to the weight vector to be calculated is determined. The basic computing unit is controlled to perform a vector inner product operation on each set of weight sub-vectors and the feature vector to be calculated to obtain the vector operation result. A shift operation is performed on some of the vector operation results, and the vector operation result obtained by the shift operation is added to the vector operation results of the remaining sets of vectors. The result of the addition is used as the sparsified data processing result. The vector operation results of some sets of vectors and the remaining sets of vectors together constitute multiple sets of vector operation results. This can make full use of the weight distribution, effectively ensure the data processing accuracy of the neural network processor, effectively reduce the hardware unit cost and execution power consumption, and effectively reduce the weight storage space.

[0097] Figure 8 This is a schematic diagram of the structure of a computing unit array proposed in an embodiment of this disclosure.

[0098] like Figure 8 As shown, the computing unit array 80 includes: multiple basic computing units 801, each supporting different information units. The information units supported by the basic computing units 801 are used for sparsification of the weight vector to be computed to obtain multiple sets of weight sub-vectors. The weight vector to be computed has a corresponding feature vector to be computed; wherein,

[0099] The basic computing unit 801 is used to perform a vector inner product operation on each group of weight sub-vectors and the feature vector to be calculated to obtain the vector operation result, and to perform a shift operation on the vector operation result of a portion of the groups, and to add the vector operation result obtained by the shift operation to the vector operation result of the remaining groups, and to use the result of the addition calculation as the result of sparse data processing. The vector operation result of a portion of the groups and the vector operation result of the remaining groups together constitute multiple groups of vector operation results.

[0100] Figure 9 This is a schematic diagram of the structure of a neural network processor proposed in an embodiment of this disclosure.

[0101] like Figure 9 As shown, the neural network processor 90 includes a processing unit 901 and a computing unit array 902. The computing unit array 902 includes multiple basic computing units 9021, each supporting a different information unit.

[0102] The processing unit 901 is used to obtain multiple sets of weight sub-vectors, wherein the weight sub-vectors are obtained by sparsifying the weight vector to be calculated based on the information units supported by the basic computing unit 9021. The processing unit 901 determines the feature vector to be calculated corresponding to the weight vector to be calculated, and controls the basic computing unit 9021 to perform a vector inner product operation on each set of weight sub-vectors and the feature vector to be calculated to obtain the vector operation result. The processing unit 901 also performs a shift operation on some of the vector operation results, adds the vector operation result obtained by the shift operation to the vector operation results of the remaining sets of vectors, and uses the result of the addition operation as the sparsified data processing result. The vector operation results of some sets of vectors and the vector operation results of the remaining sets of vectors together constitute multiple sets of vector operation results.

[0103] Figure 10 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present disclosure is shown. Figure 10 The electronic device 12 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0104] like Figure 10 As shown, the electronic device 12 is represented in the form of a general-purpose computing device. The components of the electronic device 12 may include, but are not limited to: one or more neural network processors 16, system memory 28, and bus 18 connecting different system components (including system memory 28 and neural network processors 16).

[0105] Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0106] Electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0107] Memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM) 30 and / or cache memory 32. Electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (… Figure 10 Not shown; usually referred to as a "hard drive".

[0108] although Figure 10 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disc drive for reading and writing to a removable non-volatile optical disc (e.g., a compact disc read-only memory (CD-ROM), a digital video disc read-only memory (DVD-ROM), or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this disclosure.

[0109] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of this disclosure.

[0110] Electronic device 12 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable human interaction with electronic device 12, and / or with any device that enables electronic device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 22. Furthermore, electronic device 12 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 20. As shown, network adapter 20 communicates with other modules of electronic device 12 via bus 18. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0111] The neural network processor 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the sparse data processing method of the neural network processor mentioned in the foregoing embodiments.

[0112] To implement the above embodiments, this disclosure also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the sparse data processing method of the neural network processor as proposed in the foregoing embodiments of this disclosure.

[0113] To implement the above embodiments, this disclosure also proposes a computer program product that, when the instruction processor in the computer program product is executed, performs the sparse data processing method of the neural network processor as proposed in the foregoing embodiments of this disclosure.

[0114] It should be noted that in the description of this disclosure, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this disclosure, unless otherwise stated, "a plurality of" means two or more.

[0115] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this disclosure pertain.

[0116] It should be understood that various parts of this disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0117] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0118] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0119] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0120] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0121] Although embodiments of the present disclosure have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present disclosure. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present disclosure.

Claims

1. A sparse data processing method for a neural network processor, characterized in that, The neural network processor includes: a basic computing unit, and the method includes: Multiple sets of weight sub-vectors are obtained, wherein the weight sub-vectors are obtained by sparsifying the weight vector to be computed based on the information units supported by the basic computing unit; Determine the feature vector to be calculated corresponding to the weight vector to be calculated; The basic computing unit is controlled to perform a vector inner product operation on each group of weight sub-vectors and the feature vector to be calculated, so as to obtain the vector operation result; and A shift operation is performed on the results of a partial group of vector operations, and the vector operation result obtained from the shift operation is added to the results of the vector operations of the remaining groups of vector operations. The result of the addition is used as the result of sparsification data processing. The results of the partial group of vector operations and the results of the remaining group of vector operations together constitute multiple groups of vector operation results. The weight sub-vector includes: multiple weight sub-elements, which are obtained by sparsifying the weight elements in the weight vector to be computed based on the information units supported by the basic computing unit; The process of obtaining multiple sets of weighted sub-vectors includes: Based on the information units supported by the basic computing unit, multiple weight elements of first bits and / or multiple weight elements of second bits are determined from the weight vector to be calculated, wherein the first bits are lower than the second bits; A first number of weight elements are selected as first weight sub-elements from the weight elements of the plurality of first bits, wherein each first weight sub-element has a corresponding first index; and / or A second number of weight elements are selected as second weight sub-elements from the plurality of weight elements of the second bit, wherein each second weight sub-element has a corresponding second index; and The first weighted sub-element and / or the second weighted sub-element are used together as the plurality of weighted sub-elements.

2. The method as described in claim 1, characterized in that, The control of the basic computing unit to perform a vector inner product operation on each group of weight sub-vectors and the feature vector to be calculated to obtain the vector operation result includes: The basic computing unit is controlled to select a first feature sub-element corresponding to the first weight sub-element from the feature vector to be calculated based on the first index; and / or The basic computing unit is controlled to select a second feature sub-element corresponding to the second weight sub-element from the first feature sub-element according to the second index; The basic computing unit is controlled to perform a vector inner product operation on the first weight sub-element and its corresponding first feature sub-element to obtain a first vector inner product result; and / or The basic computing unit is controlled to perform a vector inner product operation on the second weight sub-element and its corresponding second feature sub-element to obtain the second vector inner product result. The result of the first vector inner product and / or the result of the second vector inner product are used as the result of the vector operation.

3. The method according to any one of claims 1-2, characterized in that, There are multiple basic computing units, and different basic computing units support different information units.

4. A sparse data processing device for a neural network processor, characterized in that, The neural network processor includes: a basic computing unit; the device includes: The acquisition module is used to acquire multiple sets of weight sub-vectors, wherein the weight sub-vectors are obtained by sparsification of the weight vector to be calculated based on the information units supported by the basic computing unit, and the weight sub-vectors include: multiple weight sub-elements, wherein the weight sub-elements are obtained by sparsification of the weight elements in the weight vector to be calculated based on the information units supported by the basic computing unit. A determination module is used to determine the feature vector to be calculated corresponding to the weight vector to be calculated; The control module is used to control the basic computing unit to perform a vector inner product operation on each group of weight sub-vectors and the feature vector to be calculated, so as to obtain the vector operation result; and The calculation module is used to perform shift operations on the results of partial group vector operations, add the vector operation results obtained from the shift operations to the results of the remaining group vector operations, and use the result of the addition as the result of sparse data processing. The results of partial group vector operations and the results of the remaining group vector operations together constitute multiple groups of vector operation results. The acquisition module is used to determine, based on the information units supported by the basic computing unit, multiple weight elements of first bits and / or multiple weight elements of second bits from the weight vector to be calculated, wherein the first bits are lower than the second bits. A first number of weight elements are selected as first weight sub-elements from the weight elements of the plurality of first bits, wherein each first weight sub-element has a corresponding first index; and / or A second number of weight elements are selected as second weight sub-elements from the plurality of weight elements of the second bit, wherein each second weight sub-element has a corresponding second index; and The first weighted sub-element and / or the second weighted sub-element are used together as the plurality of weighted sub-elements.

5. A computing unit array, characterized in that, The computing unit array includes: multiple basic computing units, each supporting different information units. The information units supported by the basic computing units are used to sparsify the weight vector to be computed to obtain multiple sets of weight sub-vectors. The weight vector to be computed has a corresponding feature vector to be computed. The weight sub-vector includes: multiple weight sub-elements. The weight sub-elements are obtained by sparsifying the weight elements in the weight vector to be computed based on the information units supported by the basic computing units. The resulting set of multiple weight sub-vectors includes: Based on the information units supported by the basic computing unit, multiple weight elements of first bits and / or multiple weight elements of second bits are determined from the weight vector to be calculated, wherein the first bits are lower than the second bits; A first number of weight elements are selected as first weight sub-elements from the weight elements of the plurality of first bits, wherein each first weight sub-element has a corresponding first index; and / or A second number of weight elements are selected as second weight sub-elements from the plurality of weight elements of the second bit, wherein each second weight sub-element has a corresponding second index; and The first weighted sub-element and / or the second weighted sub-element are used together as the plurality of weighted sub-elements; wherein... The basic computing unit is used to perform a vector inner product operation on each group of weight sub-vectors and the feature vector to be calculated to obtain a vector operation result, perform a shift operation on a portion of the vector operation results, add the vector operation result obtained by the shift operation to the vector operation results of the remaining groups, and use the result of the addition operation as the sparsity data processing result. The partial group vector operation results and the remaining group vector operation results together constitute multiple groups of vector operation results.

6. A neural network processor, characterized in that, The neural network processor includes: a processing unit and a computing unit array; the computing unit array includes: multiple basic computing units, each supporting different information units; wherein... The processing unit is configured to acquire multiple sets of weight sub-vectors, wherein the weight sub-vectors are obtained by sparsifying the weight vector to be calculated based on the information units supported by the basic computing unit; determine the feature vector to be calculated corresponding to the weight vector to be calculated; control the basic computing unit to perform a vector inner product operation on each set of weight sub-vectors and the feature vector to be calculated to obtain a vector operation result; perform a shift operation on a portion of the vector operation results; add the vector operation result obtained by the shift operation to the vector operation results of the remaining sets of vectors; and use the result of the addition operation as the sparsified data processing result. The partial set of vector operation results and the remaining set of vector operation results together constitute multiple sets of vector operation results. The weight sub-vector includes multiple weight sub-elements, which are obtained by sparsifying the weight elements in the weight vector to be calculated based on the information units supported by the basic computing unit. The process of obtaining multiple sets of weighted sub-vectors includes: Based on the information units supported by the basic computing unit, multiple weight elements of first bits and / or multiple weight elements of second bits are determined from the weight vector to be calculated, wherein the first bits are lower than the second bits; A first number of weight elements are selected as first weight sub-elements from the weight elements of the plurality of first bits, wherein each first weight sub-element has a corresponding first index; and / or A second number of weight elements are selected as second weight sub-elements from the plurality of weight elements of the second bit, wherein each second weight sub-element has a corresponding second index; and The first weighted sub-element and / or the second weighted sub-element are used together as the plurality of weighted sub-elements.

7. An electronic device, characterized in that, include: At least one neural network processor; as well as A memory communicatively connected to the at least one neural network processor; wherein, The memory stores instructions that can be executed by the at least one neural network processor to enable the at least one neural network processor to perform the method of any one of claims 1-3.

8. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, in, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-3.

9. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-3.

Images