Apparatus, board, method and readable storage medium for conducting sparse training

Abstract

The present disclosure relates to apparatuses, boards, methods, and readable storage media for sparsifying training of neural network layers, where a computing apparatus of the present disclosure is included in an integrated circuit apparatus that includes a general-purpose interconnect interface and other processing apparatuses. The computing apparatus interacts with the other processing apparatuses to collectively complete a user-specified computing operation. The integrated circuit apparatus can also include a storage apparatus connected with the computing apparatus and the other processing apparatuses, respectively, for data storage of the computing apparatus and the other processing apparatuses.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of neural networks in general. More specifically, the present disclosure relates to an apparatus, a board, a method and a readable storage medium for training a neural network layer in a sparse manner. BACKGROUND

[0002] In recent years, with the rapid development of deep learning, the algorithm performance in a series of fields such as computer vision and natural language processing has made a leap. However, deep learning algorithm is a kind of computing-intensive and storage-intensive tool. With the increasing complexity of information processing tasks, the real-time performance and accuracy of the algorithm are constantly increasing. The neural network is often designed to be deeper and deeper, so that the computational complexity and storage space requirement are increasing, which makes it difficult for existing artificial intelligence technology based on deep learning to be directly applied to mobile phones, satellites or embedded devices with limited hardware resources.

[0003] Therefore, the compression, acceleration and optimization of deep neural network model become particularly important. A large number of studies try to reduce the computational and storage requirements of neural networks without affecting the model accuracy, which is of great significance for the engineering application of deep learning technology in embedded and mobile terminals. Sparsity is one of the methods for model lightening.

[0004] Network parameter sparsity is to reduce the redundant components in a large network by appropriate methods to reduce the demand for computational complexity and storage space. Although the existing fine-grained parameter sparsity method model performs well, it is not friendly to hardware memory, that is, the on-chip and off-chip input / output overhead is large and the performance is low. On the other hand, the structured sparsity method based on channel and convolution kernel improves the hardware performance, but the model accuracy is greatly lost. Finally, the existing sparse algorithm is mostly in the form of offline fine-tuning, that is, the pre-trained model is sparsified and then fine-tuned. The offline fine-tuning method has many limitations and cannot have more observable performance benefits on model training.

[0005] Therefore, a scheme for using the parameter tensor updated online by sparsity for inference is urgently needed. SUMMARY

[0006] In order to at least partially solve the technical problems mentioned in the background, the scheme of the present disclosure provides an apparatus, a board, a method and a readable storage medium for training a neural network layer in a sparse manner.

[0007] In one aspect, the disclosure discloses a method for sparse training of a neural network layer, comprising: presetting a certain number of mask tensors; sequentially masking a plurality of parameters of the neural network layer based on each mask tensor to obtain a masked parameter tensor; calculating the product sum of the training data of the neural network layer based on each masked parameter tensor to obtain a parameter evaluation value; and selecting the mask tensor that produces the largest parameter evaluation value among all parameter evaluation values as a parameter mask tensor. Wherein, the parameter mask tensor is used to screen the processing area of the feature map input to the neural network layer.

[0008] In another aspect, the disclosure discloses a computer readable storage medium having stored thereon a computer program code for sparse training of a neural network layer, which, when executed by a processing device, performs the aforementioned method.

[0009] In another aspect, the disclosure discloses an integrated circuit device for sparse training of a neural network model, comprising a processing device and a computing device. The processing device comprises a storage module, a masking module, an operation module and a selection module. The storage module is used to store a certain number of mask tensors; the masking module is used to sequentially mask a plurality of parameters of the neural network layer based on each mask tensor to obtain a masked parameter tensor; the operation module is used to calculate the product sum of the training data of the neural network layer based on each masked parameter tensor to obtain a parameter evaluation value; and the selection module is used to select the mask tensor that produces the largest parameter evaluation value among all parameter evaluation values as a parameter mask tensor. The computing device is used to screen the processing area of the feature map input to the neural network layer using the parameter mask tensor.

[0010] In another aspect, the disclosure discloses a board card comprising the integrated circuit device according to the foregoing.

[0011] In the model training phase, the disclosure simultaneously masks a plurality of parameters of the neural network layer based on the mask tensor, selects an appropriate parameter mask tensor as the basis for pruning during inference, and has the technical effects of reducing input / output overhead and improving accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0012] The above and other objects, features and advantages of the exemplary embodiments of the disclosure will be readily understood through reading the detailed description of the exemplary embodiments of the disclosure below in conjunction with the accompanying drawings. In the drawings, several embodiments of the disclosure are shown by way of example and not limitation, and the same or corresponding reference numbers indicate the same or corresponding parts:

[0013] Figure 1 is a structural diagram of a board card showing an embodiment of the disclosure;

[0014] Figure 2 is a structural diagram of an integrated circuit device showing an embodiment of the disclosure;

[0015] Figure 3 is a schematic diagram showing the internal structure of a computing device according to an embodiment of the present disclosure;

[0016] Figure 4 is a schematic diagram showing the internal structure of a processor core according to an embodiment of the present disclosure;

[0017] Figure 5 is a schematic diagram showing when one processor core wants to write data to another processor core of the same cluster;

[0018] Figure 6 is a flowchart showing the process of sparse training of a convolutional layer according to an embodiment of the present disclosure;

[0019] Figure 7 is a diagram showing three exemplary 4 mask matrices;

[0020] Figure 8 is a diagram showing an exemplary masking process;

[0021] Figure 9 is a diagram showing an exemplary product-sum calculation process;

[0022] Figure 10 is a block diagram showing a processing device according to an embodiment of the present disclosure; and

[0023] Figure 11 is a diagram showing the compression of a mask matrix according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present disclosure.

[0025] It should be understood that the terms “first”, “second”, “third”, and “fourth” in the claims, specification, and drawings of the present disclosure are used to distinguish different objects, rather than to describe a specific order. The terms “include” and “contain” used in the specification and claims of the present disclosure indicate the presence of the described features, whole, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, whole, steps, operations, elements, components, and / or sets thereof.

[0026] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure. As used in this disclosure and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this disclosure and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0027] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection."

[0028] The specific embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0029] Figure 1 A schematic diagram of the structure of a board 10 according to an embodiment of this disclosure is shown. Figure 1 As shown, board 10 includes chip 101, which is a system-on-chip (SoC) integrating one or more combined processing units. These combined processing units are artificial intelligence computing units used to support various deep learning and machine learning algorithms, meeting the intelligent processing needs of complex scenarios in fields such as computer vision, speech, natural language processing, and data mining. In particular, deep learning technology is widely used in cloud intelligence. A significant characteristic of cloud intelligence applications is the large volume of input data, placing high demands on the platform's storage and computing capabilities. Board 10 in this embodiment is suitable for cloud intelligence applications, possessing massive off-chip storage, on-chip storage, and powerful computing capabilities.

[0030] Chip 101 is connected to external device 103 via external interface device 102. External device 103 may be, for example, a server, computer, camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. Data to be processed can be transmitted from external device 103 to chip 101 via external interface device 102. The calculation results from chip 101 can be transmitted back to external device 103 via external interface device 102. Depending on the application scenario, external interface device 102 may have different interface forms, such as a PCIe interface.

[0031] The board card 10 further comprises a storage device 104 for storing data, which comprises one or more storage units 105. The storage device 104 is connected and transmits data with the control device 106 and the chip 101 through a bus. The control device 106 in the board card 10 is configured to regulate the state of the chip 101. To this end, in one application scenario, the control device 106 can comprise a micro controller unit (MCU).

[0032] Figure 2 is a structural diagram of the combined processing apparatus in the chip 101 of this embodiment. As shown in Figure 2 The combined processing apparatus 20 comprises a computing apparatus 201, an interface apparatus 202, a processing apparatus 203 and a DRAM 204.

[0033] The computing apparatus 201 is configured to perform user-specified operations, mainly implemented as a single-core or multi-core intelligent processor to perform deep learning or machine learning calculation, which can interact with the processing apparatus 203 through the interface apparatus 202 to jointly complete the user-specified operations.

[0034] The interface apparatus 202 is used to transmit data and control instructions between the computing apparatus 201 and the processing apparatus 203. For example, the computing apparatus 201 can obtain input data from the processing apparatus 203 via the interface apparatus 202 and write into the storage device on the computing apparatus 201. Further, the computing apparatus 201 can obtain control instructions from the processing apparatus 203 via the interface apparatus 202 and write into the control cache on the computing apparatus 201. Alternatively or optionally, the interface apparatus 202 can also read data in the storage device of the computing apparatus 201 and transmit to the processing apparatus 203.

[0035] The processing device 203, as a general processing device, performs basic control including but not limited to data carrying, starting and / or stopping of the computing device 201, etc. Depending on the implementation, the processing device 203 can be one or more types of processors, such as a central processing unit (CPU), a graphics processing unit (GPU), or other general purpose and / or special purpose processors, including but not limited to a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic component, a discrete hardware component, etc., and the number thereof can be determined according to actual needs. As mentioned above, only in terms of the computing device 201 of the present disclosure, it can be considered as having a single-core structure or a homogeneous multi-core structure. However, when the computing device 201 and the processing device 203 are considered together, they are considered to form a heterogeneous multi-core structure.

[0036] The DRAM 204 is used to store data to be processed, is a DDR memory, and usually has a size of 16G or more, and is used to save data of the computing device 201 and / or the processing device 203.

[0037] Figure 3 The internal structure of the computing device 201 is shown. The computing device 201 is used to process input data such as computer vision, speech, natural language, data mining, etc. The computing device 201 in the figure adopts a multi-core hierarchical structure design. The computing device 201 is a system on a chip, which includes at least one cluster, and each cluster includes a plurality of processor cores. In other words, the computing device 201 is a system on a chip The cluster is composed of a hierarchy of processor cores.

[0038] From the perspective of the hierarchy of the system on a chip, as shown in Figure 3 , the computing device 201 includes an external storage controller 301, a peripheral communication module 302, an on-chip interconnection module 303, a synchronization module 304, and a plurality of clusters 305.

[0039] The external storage controller 301 can be multiple, and two are exemplarily shown in the figure, which is used to access external storage devices in response to access requests issued by the processor cores, such as Figure 2The DRAM 204 in the chip allows the computing device 201 to read data from external sources or write data to external sources. The peripheral communication module 302 receives control signals from the processing device 203 via the interface device 202, initiating the computing device 201 to execute tasks. The on-chip interconnect module 303 connects the external storage controller 301, the peripheral communication module 302, and multiple clusters 305 to transmit data and control signals between the modules. The synchronization module 304 is a global barrier controller (GBC) used to coordinate the working progress of each cluster and ensure information synchronization. The multiple clusters 305 are the computing core of the computing device 201. Four are shown exemplary in the figure; however, with hardware development, the computing device 201 of this disclosure can also include 8, 16, 64, or even more clusters 305. The clusters 305 are used to efficiently execute deep learning algorithms.

[0040] From the perspective of cluster hierarchy, such as Figure 3 As shown, each cluster 305 includes multiple processor cores (IPU cores) 306 and one memory core (MEM core) 307.

[0041] Four processor cores 306 are shown in the figure as an example, but this disclosure does not limit the number of processor cores 306. Its internal architecture is as follows: Figure 4 As shown. Each processor core 306 includes three main modules: a control module 41, an arithmetic module 42, and a storage module 43.

[0042] The control module 41 coordinates and controls the operation of the computation module 42 and the storage module 43 to complete the deep learning task. It includes an instruction fetch unit (IFU) 411 and an instruction decode unit (IDU) 412. The instruction fetch unit 411 fetches instructions from the processing device 203, and the instruction decode unit 412 decodes the fetched instructions and sends the decoding result as control information to the computation module 42 and the storage module 43.

[0043] The computation module 42 includes a vector operation unit 421 and a matrix operation unit 422. The vector operation unit 421 is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformations; the matrix operation unit 422 is responsible for the core computations of deep learning algorithms, namely matrix multiplication and convolution.

[0044] Storage module 43 is used to store or move related data, including neuron RAM (NRAM) 431, weight RAM (WRAM) 432, input / output direct memory access (IODMA) 433, and move direct memory access (MVDMA) 434. NRAM 431 is used to store feature maps for computation by processor core 306 and intermediate results after computation; WRAM 432 is used to store the weights of the deep learning network; IODMA 433 controls the memory access of NRAM 431 / WRAM 432 and DRAM 204 through broadcast bus 309; MVDMA 434 controls the memory access of NRAM 431 / WRAM 432 and SRAM 308.

[0045] Back Figure 3 The storage core 307 is primarily used for storage and communication, namely storing shared data or intermediate results among processor cores 306, and performing communication between cluster 305 and DRAM 204, communication between clusters 305, and communication between processor cores 306. In other embodiments, the storage core 307 has scalar operation capabilities and is used to perform scalar operations.

[0046] The storage core 307 includes an SRAM 308, a broadcast bus 309, a cluster direct memory access (CDMA) module 310, and a global direct memory access (GDMA) module 311. The SRAM 308 acts as a high-performance data relay station. Data multiplexed between different processor cores 306 within the same cluster 305 does not need to be obtained from the DRAM 204 by each processor core 306. Instead, it is relayed between processor cores 306 via the SRAM 308. The storage core 307 only needs to quickly distribute the multiplexed data from the SRAM 308 to multiple processor cores 306 to improve inter-core communication efficiency and greatly reduce on-chip and off-chip input / output access.

[0047] Broadcast bus 309, CDMA 310, and GDMA 311 are used to perform communication between processor cores 306, communication between clusters 305, and data transfer between cluster 305 and DRAM 204, respectively. These will be explained below.

[0048] The broadcast bus 309 is used to complete high-speed communication between the processor cores 306 within the cluster 305. In this embodiment, the broadcast bus 309 supports inter-core communication methods including unicast, multicast, and broadcast. Unicast refers to point-to-point (e.g., data transmission from one processor core to another) data transmission. Multicast is a communication method that transmits a piece of data from SRAM 308 to several specific processor cores 306. Broadcast is a communication method that transmits a piece of data from SRAM 308 to all processor cores 306, and is a special case of multicast.

[0049] CDMA 310 is used to control SRAM 308 access between different clusters 305 within the same computing device 201. Figure 5 This diagram illustrates the operation of CDMA310 when one processor core attempts to write data to another processor core in a different cluster. In this application scenario, the same computing device comprises multiple clusters. For simplicity, only clusters 0 and 1 are shown in the diagram. Both clusters 0 and 1 contain multiple processor cores; similarly, for ease of explanation, only processor core 0 is shown in cluster 0, and only processor core 1 is shown in cluster 1. Processor core 0 intends to write data to processor core 1.

[0050] First, processor core 0 sends a unicast write request to write data into its local SRAM 0. CDMA 0 acts as the master and CDMA 1 acts as the slave. The master pushes the write request to the slave, that is, the master sends the write address AW and the write data W to transmit the data to SRAM 1 of cluster 1. Then the slave sends a write response B as a reply. Finally, processor core 1 of cluster 1 sends a unicast read request to read the data from SRAM 1.

[0051] Back Figure 3The GDMA 311, in conjunction with the external memory controller 301, controls memory access from the SRAM 308 to the DRAM 204 of the cluster 305, or reads data from the DRAM 204 into the SRAM 308. As previously described, communication between the DRAM 204 and the NRAM 431 or WRAM 432 can be achieved through two channels. The first channel is a direct connection between the DRAM 204 and the NRAM 431 or WRAM 432 via the IODAM 433; the second channel involves first transmitting data between the DRAM 204 and SRAM 308 via the GDMA 311, and then transmitting data between the SRAM 308 and the NRAM 431 or WRAM 432 via the MVDMA 434. Although the second channel appears to require more components and has a longer data flow, in some embodiments, the bandwidth of the second channel is actually much greater than that of the first channel. Therefore, communication between the DRAM 204 and the NRAM 431 or WRAM 432 may be more efficient via the second channel. The embodiments of this disclosure can select the data transmission channel according to their own hardware conditions.

[0052] In other embodiments, the functions of GDMA 311 and IODMA 433 can be integrated into the same component. For ease of description, this disclosure treats GDMA 311 and IODMA 433 as different components. For those skilled in the art, any component that implements functions and achieves technical effects similar to this disclosure falls within the scope of protection of this disclosure. Furthermore, the functions of GDMA 311, IODMA 433, CDMA 310, and MVDMA 434 can also be implemented by the same component.

[0053] This embodiment, based on the aforementioned hardware structure, implements a method for sparse training of convolutional layers. Unlike existing technologies, this embodiment utilizes a multi-dimensional mask tensor to mask parameters simultaneously across multiple dimensions during model training. Each dimension of the mask tensor has m elements, where n elements are 1s and mn elements are 0s. An element of 1 indicates no masking, meaning the corresponding parameter still participates in training, while an element of 0 indicates masking, meaning the corresponding parameter is masked and does not participate in training. Here, m is greater than n, and m and n are positive integers. In other application scenarios, an element of 1 can represent masking, and an element of 0 can represent no masking. The following explanation uses the example of an element of 1 representing no masking and an element of 0 representing masking. The method is as follows... Figure 6 As shown.

[0054] In step 601, a specific number of mask tensors are preset. For ease of explanation, the mask tensors in this embodiment are exemplarily set as two-dimensional mask matrices; more specifically, they are for the input channels (c) of the convolutional kernel of the convolutional layer.in ) and output channel (c out The mask is applied, where m is 4 and n is 2, meaning the mask matrix is ​​4. 4, where each row and each column has 2 elements that are 1 and 2 elements that are 0. Figure 7 Three such 4s are shown as an example. 4. Masking matrix, taking masking matrix 701 as an example: The elements of the first row C1 are [1 0 0 1], the elements of the second row C2 are [0 1 1 0], the elements of the third row C3 are [1 0 0 1], and the elements of the fourth row C4 are [0 1 1 0]. Each row has two elements of 1 and two elements of 0. The elements of the first column R1 are [1 01 0], the elements of the second column R2 are [0 1 0 1], the elements of the third column R3 are [0 1 0 1], and the elements of the fourth column R4 are [1 01 0]. Each column has two elements of 1 and two elements of 0. This 4... There are 90 possible 4-mask matrices. Therefore, in this step, we pre-determine 90 matrices where each row and each column has two elements of 1 and two elements of 0. 4. Mask matrix.

[0055] In step 602, it is determined whether the masking is based on all mask tensors, that is, whether the convolution kernel has masked all the specific number of mask tensors. If not, proceed to step 603.

[0056] In step 603, multiple parameters of the convolutional layer are masked based on each mask tensor to obtain the masked parameter tensor. Using the aforementioned 4... Taking the 4-mask matrix as an example, in this step, the input and output channels of the convolution kernel are masked based on any one of the 90 mask matrices that has not yet been masked.

[0057] Figure 8 An exemplary masking process is shown, assuming the convolutional layer has 4 input and 4 output channels. The channel matrix 801 of 4 has elements a. 11 To a 44 In this step, using Figure 7 The mask matrix 702 is used for masking. If the corresponding element in the mask matrix 702 is 1, the element in the channel matrix 801 is retained; if the corresponding element in the mask matrix 702 is 0, the element in the channel matrix 801 is considered masked and its value is 0. Using a in the channel matrix 801... 11For example, the corresponding element in the mask matrix 702 is 0, so the corresponding element in the masked parameter matrix 802 is masked and its value is 0. The masked parameter matrix 802 is obtained in this way. Since half of the elements in the channel matrix 801 are masked, it means that about half of the computation is saved. Therefore, by setting the number of elements with 0 in the mask tensor, the amount of computation saved can be determined.

[0058] In step 604, the training data of the convolutional layer are multiplied and calculated based on each masked parameter tensor to obtain parameter evaluation values. The original parameter tensor is replaced by the masked parameter tensor, and multiplied and calculated with the training data to obtain parameter evaluation values. The purpose of obtaining these evaluation values ​​is to calculate the amount of information obtainable after masking with the mask matrix. If the parameter evaluation value is high, it indicates that the information content is not significantly lost due to masking; this mask tensor reduces computational load while retaining most of the information, making it a high-quality mask tensor. Conversely, if the parameter evaluation value is low, it indicates that too much information is lost after masking, and this mask tensor is not a high-quality mask tensor.

[0059] Figure 9 An exemplary product sum calculation process is shown. Assuming training data matrix 901 is one of the training data sets, it should originally be calculated with channel matrix 801. Instead, it is now multiplied and summed with the masked parameter matrix 802 to obtain the amount of information after masking. There are several ways to perform such a product sum calculation. For example, the absolute values ​​of the corresponding elements of training data matrix 901 and masked parameter matrix 802 can be multiplied and then added together to obtain the parameter evaluation value S1, i.e.:

[0060]

[0061] For example, the absolute values ​​of the corresponding elements of the training data matrix 901 and the masked parameter matrix 802 are multiplied and then added together to obtain the parameter evaluation value S2, that is:

[0062]

[0063] The parameter evaluation value S1 or S2 reflects the amount of information retained after masking; a higher parameter evaluation value indicates more information retained. In one application scenario, either the parameter evaluation value S1 or S2 can be calculated, while in another application scenario, both S1 and S2 can be used simultaneously. This disclosure does not impose any restrictions.

[0064] After executing step 604, the process returns to step 602 to check if all mask tensors have been masked and their parameter evaluation values ​​have been obtained. If there are still mask tensors that have not been masked, steps 603 and 604 are executed until all mask tensors have been masked and their parameter evaluation values ​​have been obtained. In the previous example, this means all 90 4's... Step 605 will only be executed after all four mask matrices have been masked and 90 parameter evaluation values ​​have been obtained.

[0065] In step 605, the mask tensor with the largest parameter evaluation value is selected as the parameter mask tensor. There are many ways to select the largest parameter evaluation value. For example, all parameter evaluation values ​​can be sorted by numerical value to obtain the largest value, or a simple two-input comparator can be used to compare the values, keeping the larger one and comparing it with the next parameter evaluation value. After comparing all 90 parameter evaluation values, the remaining one is the largest. If multiple mask tensors have the same largest parameter evaluation value, one is randomly selected.

[0066] The mask tensor with the largest parameter evaluation value is the mask tensor that retains the most information. In this embodiment, this mask tensor is used as the parameter mask tensor. During inference, this parameter mask tensor is used for pruning to select the processing regions of the feature maps input to the convolutional layer for calculation.

[0067] The foregoing Figure 6 The process is as follows: Figure 2 The processing device 203 performs the implementation; more specifically, the processing device 203, such as Figure 10 As shown, it includes a control module 1001, a storage module 1002, a mask module 1003, a calculation module 1004, and a selection module 1005.

[0068] The control module 1001 is used to control and coordinate the operation of each module.

[0069] Storage module 1001 stores a specific number of mask tensors, typically a lookup table (LUT). A lookup table is essentially a random access memory (RAM), an internal memory that directly exchanges data with mask module 1002. RAM can be read and written at any time, is fast, and is usually used as a temporary data storage medium. A specific number of mask tensors (e.g., the aforementioned 90 4x4) 4. After the mask matrix is ​​written into RAM in advance, whenever the control module 1001 sends a signal to the storage module 1002, it is equivalent to inputting an address to look up the table, find the mask tensor corresponding to the address, and then output it to the mask module 1003.

[0070] Since some elements in the mask tensor are 0, this embodiment can further discard these 0 elements and not store them, thereby removing unnecessary zero-occupying elements, thus compressing the data size of the mask tensor and saving storage space. This embodiment encodes all elements in the mask tensor, also using 4... Taking a 4-mask matrix as an example, this embodiment encodes each element as follows: Figure 11 As shown in the mask matrix 1101, the storage module 1002 only needs to record the element codes with a value of 1. Taking the mask matrix 702 as an example, the 16-element mask matrix 702 only needs to record 8 element codes with a value of 1. These element codes are stored in a lookup table in the form of a sequence 1102. In other words, the storage module 1002 only stores 90 8s. One sequence, not 90 groups of 4. A 4-mask matrix greatly reduces the required storage space.

[0071] The control module 1001 will determine whether all mask tensors have been masked. If not, the control module 1001 will control the storage module 1002 to output the next mask tensor to the mask module 1003.

[0072] The masking module 1003 masks multiple parameters of the convolutional layer based on each mask tensor to obtain the masked parameter tensor, i.e., as shown below. Figure 8 The mask matrix 702 is used to mask the channel matrix 801 to generate the masked parameter matrix 802.

[0073] More specifically, the masking module 1003 uses the element codes in the compressed sequence as indices to read the corresponding elements in the parameter tensor and writes the values ​​of these non-zero elements into a register (not shown). Taking sequence 1102 as an example, when executing... Figure 8 When the mask is shown, the mask module 1003 will mask the elements [m] in the sequence 1102. 31 m 41 m 12 m 42 m 13 m 23 m 24 m 34 ] Corresponding to the element [a] in channel matrix 801 31 a 41 a 12 a 42 a 13 a 23 a 24 a 34 Keep the values ​​of the remaining elements and change the values ​​of the other elements to 0 to obtain the masked parameter matrix 802.

[0074] The computation module 1004 performs multiplication and calculation on the training data of the convolutional layer based on the masked parameter tensor to obtain parameter evaluation values. As mentioned earlier, the purpose of obtaining parameter evaluation values ​​is to know the amount of information obtained after masking the parameter tensor. If the parameter evaluation value is high, it means that not much information is lost due to masking; if the parameter evaluation value is low, it means that too much information is lost after masking.

[0075] This embodiment does not limit the way the operation module 1004 performs product and calculation. For example, it can multiply the corresponding elements and then add the absolute values, or multiply the corresponding elements by taking the absolute values ​​and then add them, or perform other product and calculation methods that can reflect the amount of information obtained. The higher the value of the parameter evaluation, the more information is obtained.

[0076] The control module 1001 continuously determines whether all mask tensors have been masked and parameter evaluation values ​​have been obtained. If there are still mask tensors that have not been masked, it continues to control the storage module 1002, the masking module 1003 and the calculation module 1004 to mask specific mask tensors and obtain parameter evaluation values.

[0077] After all mask tensors have been masked and their parameter evaluation values ​​obtained, the control module 1001 controls the selection module 1005 to generate the mask tensor with the largest parameter evaluation value as the parameter mask tensor. There are many ways to generate the mask tensor with the largest parameter evaluation value. The selection module 1005 can sort all parameter evaluation values ​​by numerical value to obtain the largest parameter evaluation value, or it can simply use a two-input comparator to compare them, keeping the larger one and comparing it with the next parameter evaluation value. After comparing all 90 parameter evaluation values, the remaining one is the largest parameter evaluation value. If multiple mask tensors have the same largest parameter evaluation value, the selection module 1005 randomly selects one.

[0078] The mask tensor with the largest parameter evaluation value means that the most information is retained. The computing device 201 uses this mask tensor as the parameter mask tensor. When processing the input feature map in scenarios such as computer vision, speech, natural language processing, and data mining, it uses this parameter mask tensor to perform pruning, remove some parameters, and then select the processing area of ​​the input feature map input to the convolutional layer for calculation, thereby achieving the purpose of sparsity.

[0079] Although this embodiment uses the input and output channels as examples, this disclosure is not limited thereto. For example, this disclosure can use the number of pixels (H) in the vertical direction and the number of pixels (W) in the horizontal direction of the input feature map as parameters for masking. In this application scenario, since two parameters are involved, the mask tensor is a two-dimensional mask matrix. As another example, this disclosure can use the number of pixels (H) in the vertical direction, the number of pixels (W) in the horizontal direction of the input feature map, and the number of batches (N) contained in the input feature map as parameters for masking. In this application scenario, three parameters are involved, so the mask tensor is three-dimensional.

[0080] Although this embodiment uses a convolutional layer as an example for illustration, this disclosure is not limited thereto. Any neural network layer that requires parameters to participate in calculations can utilize the techniques disclosed in this disclosure to achieve the sparsity effect, such as fully connected layers and batch normalized layers.

[0081] Although the mask tensor in this embodiment has m elements in each dimension, this disclosure is not limited thereto. Those skilled in the art can appropriately adjust the number of elements in each dimension according to multiple parameters of the neural network layer. This disclosure does not require that the number of elements in each dimension be the same. Furthermore, n does not necessarily have to remain consistent in the same training session. Taking a mask vector with m=4 as an example, this disclosure can dynamically adjust the mask tensor to a 2-out-of-4 selection based on the actual application scenario. If the absolute values ​​of the four weights in a certain interval are particularly large, the interval can be dynamically adjusted to a 3-out-of-4 or a 4-out-of-4 selection. If the absolute values ​​of the four weights in a certain interval are particularly small, the interval can be dynamically adjusted to a 1-out-of-4 selection.

[0082] This embodiment can perform masking from the very beginning of the model training phase, or train the weights to a certain extent based on existing training methods, and then use the method disclosed in this embodiment to train the parameter masking.

[0083] In this embodiment, during the model training phase, multiple parameters of the convolutional layer are simultaneously masked using multiple mask tensors. The mask tensor with the largest parameter evaluation value is selected as the parameter mask tensor, serving as the basis for pruning during inference. Because of the masking, not all data needs to be loaded onto the chip for computation, reducing input / output overhead. Furthermore, this embodiment incorporates the masking factor during training, so the trained weights, combined with the masked parameter matrix, contribute to improved accuracy. Finally, online training not only trains the parameters but also updates the mask matrix, rather than performing sparsity fine-tuning based on the trained weights as in existing technologies. This embodiment can generate more significant performance gains.

[0084] Another embodiment of this disclosure is a computer-readable storage medium storing computer program code for sparse training of neural network layers. When the computer program code is run by a processor, it executes the methods described in the preceding embodiments. In some implementation scenarios, the integrated units described above can be implemented as software program modules. If implemented as software program modules and sold or used as independent products, the integrated units can be stored in a computer-readable memory. Based on this, when the solution of this disclosure is embodied in the form of a software product (e.g., a computer-readable storage medium), the software product can be stored in a memory, which may include several instructions to cause a computer device (e.g., a personal computer, a server, or a network device) to execute some or all of the steps of the methods described in the embodiments of this disclosure. The aforementioned memory may include, but is not limited to, various media capable of storing program code, such as USB flash drives, flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0085] In the model training phase, this disclosure uses a mask tensor to simultaneously mask multiple parameters of a neural network layer and selects an appropriate parameter mask tensor as the basis for pruning during inference, which has the technical effect of reducing input / output overhead and improving accuracy.

[0086] Depending on the application scenario, the electronic devices or apparatus disclosed herein may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablet computers, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, dashcams, navigators, sensors, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, autonomous driving terminals, vehicles, home appliances, and / or medical devices. The vehicles include airplanes, ships, and / or vehicles; the home appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical devices include MRI scanners, ultrasound scanners, and / or electrocardiographs. The electronic devices or apparatus disclosed herein can also be applied in fields such as the Internet, IoT, data centers, energy, transportation, public management, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and healthcare. Furthermore, the electronic devices or apparatus disclosed herein can also be used in cloud, edge, and terminal applications related to artificial intelligence, big data, and / or cloud computing. In one or more embodiments, the high-computing-power electronic devices or apparatuses according to the present disclosure can be applied to cloud devices (e.g., cloud servers), while the low-power electronic devices or apparatuses can be applied to terminal devices and / or edge devices (e.g., smartphones or cameras). In one or more embodiments, the hardware information of the cloud devices and the hardware information of the terminal devices and / or edge devices are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud devices to simulate the hardware resources of the terminal devices and / or edge devices based on the hardware information of the terminal devices and / or edge devices, so as to complete the unified management, scheduling and collaborative work of the end-to-cloud or cloud-edge-end integration.

[0087] It should be noted that, for the sake of brevity, this disclosure describes some methods and their embodiments as a series of actions and combinations thereof. However, those skilled in the art will understand that the solutions of this disclosure are not limited by the order of the described actions. Therefore, based on the disclosure or teachings of this disclosure, those skilled in the art will understand that some steps can be performed in a different order or simultaneously. Furthermore, those skilled in the art will understand that the embodiments described in this disclosure can be considered as optional embodiments, that is, the actions or modules involved are not necessarily essential for the implementation of one or more solutions of this disclosure. In addition, depending on the solution, the description of some embodiments in this disclosure also has different emphases. In view of this, those skilled in the art will understand that parts not described in detail in a certain embodiment of this disclosure can also be referred to the relevant descriptions of other embodiments.

[0088] In terms of specific implementation, based on the disclosure and teachings of this disclosure, those skilled in the art will understand that the several embodiments disclosed herein can also be implemented in other ways not disclosed herein. For example, regarding the various units in the electronic device or apparatus embodiments described above, this document has divided them based on logical functions, but in actual implementation, there may be other ways of division. As another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. Regarding the connection relationship between different units or components, the connection discussed above in conjunction with the accompanying drawings can be a direct or indirect coupling between units or components. In some scenarios, the aforementioned direct or indirect coupling involves a communication connection utilizing an interface, wherein the communication interface can support electrical, optical, acoustic, magnetic, or other forms of signal transmission.

[0089] In this disclosure, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed across multiple network units. Furthermore, depending on actual needs, some or all of the units can be selected to achieve the purpose of the solution described in the embodiments of this disclosure. Additionally, in some scenarios, multiple units in the embodiments of this disclosure may be integrated into one unit or each unit may exist physically independently.

[0090] In other implementation scenarios, the integrated units described above can also be implemented in hardware, i.e., as specific hardware circuits, which may include digital circuits and / or analog circuits. The physical implementation of the circuit's hardware structure may include, but is not limited to, physical devices, which may include, but are not limited to, transistors or memristors. Therefore, the various devices described herein (e.g., computing devices or other processing devices) can be implemented using appropriate hardware processors, such as central processing units, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage units or storage devices can be any suitable storage medium (including magnetic storage media or magneto-optical storage media), such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), ROM, and RAM.

[0091] The foregoing can be better understood in accordance with the following terms:

[0092] Clause A1. A method for sparse training of a neural network layer, comprising: presetting a specific number of mask tensors; sequentially masking multiple parameters of the neural network layer based on each mask tensor to obtain masked parameter tensors; multiplying and calculating the training data of the neural network layer based on each masked parameter tensor to obtain parameter evaluation values; and selecting the mask tensor that generates the largest parameter evaluation value as the parameter mask tensor; wherein the parameter mask tensor is used to filter the processing regions of the feature maps input to the neural network layer.

[0093] Clause A2, the method described in Clause A1, wherein each dimension of the mask tensor has m elements, of which n elements are 1, mn elements are 0, m is greater than n, and m and n are positive integers.

[0094] Clause A3, the method described in Clause A2, where m is 4.

[0095] Clause A4, the method described in accordance with Clause A3, where n is 2.

[0096] Clause A5, the method according to Clause A1, wherein the product and calculation steps include multiplying the corresponding elements and then adding the absolute values.

[0097] Clause A6. The method according to Clause A1, wherein the product and calculation steps include multiplying the absolute values ​​of the corresponding elements and then adding them together.

[0098] Clause A7, the method according to Clause A1, wherein the plurality of parameters are input channel parameters and output channel parameters.

[0099] Clause A8. The method according to Clause A1, wherein the neural network layer is one of a convolutional layer, a fully connected layer, and a batch normalization layer.

[0100] Clause A9. A computer-readable storage medium having stored thereon computer program code for sparse training of neural network layers, wherein when the computer program code is run by a processing device, it performs the method described in any one of Clauses A1 to A8.

[0101] Clause A10. An integrated circuit apparatus for sparse training of a neural network model, comprising: a processing device, including: a storage module for storing a specific number of mask tensors; a masking module for sequentially masking multiple parameters of the neural network layer based on each mask tensor to obtain masked parameter tensors; a computation module for multiplying and calculating training data of the neural network layer based on each masked parameter tensor to obtain parameter evaluation values; and a selection module for selecting the mask tensor that produces the largest parameter evaluation value as the parameter mask tensor; and a computing device for using the parameter mask tensor to filter processing regions of feature maps input to the neural network layer.

[0102] Clause A11. An integrated circuit device according to Clause A10, wherein the mask tensor has m elements in each dimension, where n elements are 1, mn elements are 0, m is greater than n, and m and n are positive integers.

[0103] Clause A12, an integrated circuit device as described in Clause A11, wherein m is 4.

[0104] Clause A13, an integrated circuit device as described in Clause A12, wherein n is 2.

[0105] Clause A14. An integrated circuit device according to Clause A10, wherein the arithmetic module multiplies the corresponding elements of the training data and then adds the absolute values.

[0106] Clause A15, the integrated circuit device according to Clause A10, wherein the arithmetic module multiplies the corresponding elements of the training data by their absolute values ​​and then adds them together.

[0107] Clause A16, the integrated circuit device according to Clause A10, wherein the plurality of parameters are input channel parameters and output channel parameters.

[0108] Clause A17. An integrated circuit device according to Clause A10, wherein the neural network layer is one of a convolutional layer, a fully connected layer, and a batch normalization layer.

[0109] Clause A18, a board including an integrated circuit device as described in any one of Clauses A10 to A17.

[0110] The embodiments of this disclosure have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The description of the above embodiments is only for the purpose of helping to understand the methods and core ideas of this disclosure. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure.

Claims

1. A method for sparse training of neural network layers, comprising: Preset a specific number of mask tensors; Multiple parameters of the neural network layer are sequentially masked based on each mask tensor to obtain the masked parameter tensor. The training data of the neural network layer are multiplied and calculated based on the parameter tensor after each mask to obtain parameter evaluation values. This multiplication and calculation to obtain parameter evaluation values ​​includes: multiplying the corresponding elements of the training data matrix and the masked parameter matrix, taking the absolute values, and then adding them to obtain the parameter evaluation values; or multiplying the corresponding elements of the training data matrix and the masked parameter matrix by their absolute values ​​and then adding them to obtain the parameter evaluation values; and Select the mask tensor that produces the largest value among all parameter evaluations as the parameter mask tensor; Specifically, the parameter mask tensor is used to filter the processing regions of the feature maps input to the neural network layers. When processing the input feature maps of computer vision and speech, the parameter mask tensor is used to prune and remove some parameters in order to filter the processing regions of the input feature maps input to the convolutional layers of the neural network layers for calculation, thereby achieving the purpose of sparsity.

2. The method according to claim 1, wherein each dimension of the mask tensor has m elements, of which n elements are 1, mn elements are 0, m is greater than n, and m and n are positive integers.

3. The method according to claim 2, wherein m is 4.

4. The method according to claim 3, wherein n is 2.

5. The method according to claim 1, wherein the plurality of parameters are input channel parameters and output channel parameters.

6. The method according to claim 1, wherein the neural network layer is one of a convolutional layer, a fully connected layer, and a batch normalization layer.

7. A computer-readable storage medium having stored thereon computer program code for sparse training of neural network layers, wherein when the computer program code is run by a processing device, it performs the method according to any one of claims 1 to 6.

8. An integrated circuit device for sparse training of neural network layers, comprising: Processing apparatus, comprising: The storage module stores a specific number of mask tensors; The masking module is used to sequentially mask multiple parameters of the neural network layer based on each masking tensor to obtain the masked parameter tensor. The computation module is used to perform product and calculation on the training data of the neural network layer based on each masked parameter tensor to obtain parameter evaluation values. The product and calculation to obtain parameter evaluation values ​​includes: multiplying the corresponding elements of the training data matrix and the masked parameter matrix, taking the absolute values ​​and adding them together to obtain the parameter evaluation values; or multiplying the corresponding elements of the training data matrix and the masked parameter matrix, taking the absolute values ​​and adding them together to obtain the parameter evaluation values; and The selection module is used to select the mask tensor that produces the largest value among all parameter evaluations as the parameter mask tensor; A computing device is used to filter the processing regions of the feature maps input to the neural network layer using the parameter mask tensor. When processing the input feature maps of computer vision and speech, the parameter mask tensor is used to prune and remove some parameters in order to filter the processing regions of the input feature maps input to the convolutional layers in the neural network layer for calculation, thereby achieving the purpose of sparsity.

9. The integrated circuit device according to claim 8, wherein each dimension of the mask tensor has m elements, of which n elements are 1, mn elements are 0, m is greater than n, and m and n are positive integers.

10. The integrated circuit device according to claim 9, wherein m is 4.

11. The integrated circuit device according to claim 10, wherein n is 2.

12. The integrated circuit device according to claim 8, wherein the plurality of parameters are input channel parameters and output channel parameters.

13. The integrated circuit device according to claim 8, wherein the neural network layer is one of a convolutional layer, a fully connected layer, and a batch normalization layer.

14. A board comprising an integrated circuit device according to any one of claims 8 to 13.

Images