Apparatus, method, and article of manufacture for neural network sparsification

Abstract

The present disclosure relates to methods, devices, boards and readable storage media for sparse training of neural network models. The data processing device of the present disclosure can be implemented as a computing device included in a combined processing device, which can further include an interface device and other processing devices. The computing device interacts with the other processing devices to jointly complete a user-specified computing operation. The combined processing device can further include a storage device connected to the computing device and the other processing devices, respectively, for storing data of the computing device and the other processing devices. Embodiments of the present disclosure provide solutions related to sparse training of neural network models, which can improve the computing capacity of the neural network models and improve the processing efficiency of the machine.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to the field of processors. More particularly, the present disclosure relates to a method, device, chip, board and readable storage medium for sparse training of a neural network model by a data processing apparatus. BACKGROUND

[0002] In recent years, with the rapid development of deep learning, the algorithm performance in a series of fields such as computer vision and natural language processing has made a leap. However, deep learning algorithm is a kind of computing-intensive and storage-intensive tool. With the increasing complexity of information processing tasks, the real-time performance and accuracy of the algorithm are increasingly required. The neural network is often designed to be deeper and deeper, so that the computational load and storage space requirement are larger and larger, which makes it difficult for existing artificial intelligence technology based on deep learning to be directly applied to mobile phones, satellites or embedded devices with limited hardware resources.

[0003] Therefore, the compression, acceleration and optimization of the deep neural network model become particularly important. A large number of studies try to reduce the computational and storage requirements of the neural network without affecting the model accuracy, which is of great significance for the engineering application of deep learning technology in embedded and mobile terminals. Sparsity is one of the methods for model lightening.

[0004] Network parameter sparsity is to reduce the redundant components in a large network by appropriate methods to reduce the computational load and storage space requirement of the network. Although the existing fine-grained parameter sparsity method model performs well, it is not friendly to hardware memory, that is, the on-chip and off-chip input / output overhead is large and the performance is low. On the other hand, although the structured sparsity method based on channel and convolution kernel improves the hardware performance, the model accuracy is greatly lost. Finally, the existing sparse algorithm is mostly in the form of offline fine-tuning, that is, the pre-trained model is sparsified and then fine-tuned. The offline fine-tuning method has many limitations and cannot have more observable performance benefits on model training.

[0005] Therefore, there is a need for a scheme capable of sparse training of a neural network model. SUMMARY

[0006] In order to at least partially solve one or more technical problems mentioned in the background, the scheme of the present disclosure provides a device, board, method and readable storage medium for sparse training of a neural network model.

[0007] In a first aspect, the disclosure provides a method of training a neural network model by a data processing apparatus, comprising: in a forward propagation, performing sparsification on at least neural network parameters based on a mask tensor to calculate a value of a loss function; in a backward propagation, calculating neuron gradients and neural network parameter gradients based on the loss function; and updating the neural network parameters based on the neural network parameter gradients.

[0008] In a second aspect, the disclosure provides a computer-readable storage medium having stored thereon computer program code to train a neural network model, which when executed by a processing apparatus, performs the method of any of the preceding first aspect.

[0009] In a third aspect, the disclosure provides a data processing apparatus, comprising a control circuit, a storage circuit and an operation circuit, wherein: the control circuit is configured to control the storage circuit and the operation circuit to perform sparsification training on a neural network model; the storage circuit is configured to store information, the information comprising at least neural network parameters and a mask tensor; and the operation circuit is configured to perform the following operations under the control of the control circuit: in a forward propagation, performing sparsification on at least neural network parameters based on a mask tensor to calculate a value of a loss function; in a backward propagation, calculating neuron gradients and neural network parameter gradients based on the loss function; and updating the neural network parameters based on the neural network parameter gradients.

[0010] In a fourth aspect, the disclosure provides a chip comprising the data processing circuit of any of the preceding third aspect.

[0011] In a fifth aspect, the disclosure provides a board card comprising the chip of any of the preceding fourth aspect.

[0012] By the data processing apparatus, the method of training a neural network model using the data processing apparatus and the related products as provided above, the embodiments of the disclosure provide a sparsification scheme in the training process of a neural network. The sparsification scheme can support sparsification in the forward propagation process of training, such as input channel dimension sparsification, or simultaneous sparsification of input channel dimension and output channel dimension. In some embodiments, when the forward propagation performs simultaneous sparsification of input channel dimension and output channel dimension, the backward propagation can also support simultaneous sparsification of input channel dimension and output channel dimension, thereby further optimizing the performance. The sparsification scheme of the disclosure can be executed in multiple stages of training, and different training stages can adopt different structured sparse data flow structures for related operations to obtain optimized operation and IO performance. BRIEF DESCRIPTION OF DRAWINGS

[0013] The above and other objects, features and advantages of the present disclosure exemplary embodiments will be more apparent from the following detailed description read in conjunction with the accompanying drawings, in which several embodiments of the present disclosure are shown by way of example, and like or corresponding reference numerals are used to refer to like or corresponding elements throughout. In the drawings:

[0014] Figure 1 is a structural diagram of a board card of an embodiment of the present disclosure;

[0015] Figure 2 is a structural diagram of an integrated circuit device of an embodiment of the present disclosure;

[0016] Figure 3 is a schematic diagram of the internal structure of a single-core computing device of an embodiment of the present disclosure;

[0017] Figure 4 is a schematic diagram of the internal structure of a multi-core computing device of an embodiment of the present disclosure;

[0018] Figure 5 is a schematic diagram of the internal structure of a processor core of an embodiment of the present disclosure;

[0019] Figure 6 shows an exemplary structural block diagram of a data processing device according to an embodiment of the present disclosure;

[0020] Figure 7 shows a method performed in one iteration according to an embodiment of the present disclosure;

[0021] Figure 8A shows a masking process of one exemplary one-dimensional mask tensor according to an embodiment of the present disclosure;

[0022] Figure 8B shows a masking process of one exemplary two-dimensional mask tensor according to an embodiment of the present disclosure;

[0023] Figure 9 is a schematic diagram showing an exemplary mask vector update;

[0024] Figure 10 is a schematic diagram showing an exemplary product-sum calculation process;

[0025] Figure 11 is a flowchart showing a sparse training method of another embodiment of the present disclosure;

[0026] Figure 12 is a flowchart showing a sparse training method of another embodiment of the present disclosure in which the mask enters a fixed phase; and

[0027] Figure 13 is a schematic diagram showing several embodiments of the present disclosure when training a neural network model to be sparse. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present disclosure will be described clearly and completely in combination with the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0029] It should be understood that the terms "first", "second", "third", and "fourth" and the like used in the claims, the specification and the drawings of the present disclosure are used to distinguish different objects, and are not used to describe a particular order. The terms "include" and "contain" used in the specification and claims of the present disclosure indicate the presence of the described features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or sets thereof.

[0030] It should also be understood that the terms used in the specification of the present disclosure are only for the purpose of describing specific embodiments, and are not intended to limit the present disclosure. As used in the specification and claims of the present disclosure, the singular forms "a", "an" and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should be further understood that the term "and / or" used in the specification and claims of the present disclosure refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0031] As used in the specification and claims of the present disclosure, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting" depending on the context.

[0032] The specific embodiments of the present disclosure will be described in detail below in combination with the accompanying drawings.

[0033] Figure 1 A structural schematic diagram of a board card 10 according to an embodiment of the present disclosure is shown. As shown in the figure, the board card 10 includes a board card body 100, a plurality of first connectors 110, a plurality of second connectors 120, a plurality of third connectors 130, and a plurality of fourth connectors 140. Figure 1As shown, the board card 10 includes a chip 101, which is a system on chip (SoC) integrated with one or more combined processing devices, which is an artificial intelligence operation unit to support various deep learning and machine learning algorithms to meet the intelligent processing needs in complex scenarios in the fields of computer vision, speech, natural language processing, data mining, etc. In particular, deep learning technology is widely used in cloud intelligent fields. A significant feature of cloud intelligent applications is the large amount of input data, which has high requirements for the storage capacity and computing capacity of the platform. The board card 10 of this embodiment is suitable for cloud intelligent applications and has a large off-chip storage, on-chip storage and strong computing capacity.

[0034] The chip 101 is connected with an external device 103 through an external interface device 102. The external device 103 is, for example, a server, a computer, a camera, a display, a mouse, a keyboard, a network card or a wifi interface, etc. The data to be processed can be transmitted from the external device 103 to the chip 101 through the external interface device 102. The computing result of the chip 101 can be transmitted back to the external device 103 through the external interface device 102. According to different application scenarios, the external interface device 102 can have different interface forms, such as a PCIe interface, etc.

[0035] The board card 10 further includes a storage device 104 for storing data, which includes one or more storage units 105. The storage device 104 is connected and transmits data with the control device 106 and the chip 101 through a bus. The control device 106 in the board card 10 is configured to regulate the state of the chip 101. For this purpose, in one application scenario, the control device 106 can include a micro controller unit (MCU).

[0036] Figure 2 is a structural diagram of the combined processing device in the chip 101 of this embodiment. As shown in Figure 2 The combined processing device 20 includes a computing device 201, an interface device 202, a processing device 203 and a DRAM 204.

[0037] The computing device 201 is configured to perform user-specified operations, mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations, which can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0038] The interface device 202 is used to transmit data and control instructions between the computing device 201 and the processing device 203. For example, the computing device 201 can obtain input data from the processing device 203 via the interface device 202 and write into the storage device on the computing device 201. Further, the computing device 201 can obtain control instructions from the processing device 203 via the interface device 202 and write into the control buffer on the computing device 201. Alternatively or additionally, the interface device 202 can also read data from the storage device of the computing device 201 and transmit to the processing device 203.

[0039] The processing device 203 is a general-purpose processing device, which performs basic controls including but not limited to data transfer, start and / or stop of the computing device 201, etc. Depending on the implementation, the processing device 203 can be one or more types of processors, such as a central processing unit (CPU), a graphics processing unit (GPU), or other general-purpose and / or special-purpose processors, including but not limited to a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc., and the number thereof can be determined according to actual needs. As mentioned above, only in terms of the computing device 201 of the present disclosure, it can be considered as having a single-core structure or a homogeneous multi-core structure. However, when the computing device 201 and the processing device 203 are considered together, they are considered to form a heterogeneous multi-core structure.

[0040] The DRAM 204 is used to store data to be processed, which is a DDR memory, usually having a size of 16G or more, for saving data of the computing device 201 and / or the processing device 203.

[0041] Figure 3 An internal structure diagram of the computing device 201 as a single-core is shown. The single-core computing device 301 is used to process input data of computer vision, voice, natural language, data mining, etc., and the single-core computing device 301 includes three modules: a control module 31, a computation module 32, and a storage module 33.

[0042] The control module 31 is configured to coordinate and control the operation of the operation module 32 and the storage module 33 to complete the task of deep learning, which includes an instruction fetch unit (IFU) 311 and an instruction decode unit (IDU) 312. The instruction fetch unit 311 is configured to fetch instructions from the processing device 203, and the instruction decode unit 312 decodes the fetched instructions and sends the decoding results as control information to the operation module 32 and the storage module 33.

[0043] The operation module 32 includes a vector operation unit 321 and a matrix operation unit 322. The vector operation unit 321 is configured to perform vector operations, which can support complex operations such as vector multiplication, addition, and nonlinear transformation; and the matrix operation unit 322 is responsible for the core calculation of the deep learning algorithm, i.e., matrix multiplication and convolution.

[0044] The storage module 33 is configured to store or transfer related data, which includes a neuron RAM (NRAM) 331, a weight RAM (WRAM) 332, and a direct memory access (DMA) 333. The NRAM 331 is configured to store input neurons, output neurons, and intermediate results after calculation; the WRAM 332 is configured to store the convolution kernel of the deep learning network, i.e., the weight; and the DMA 333 is connected to the DRAM 204 through the bus 34 and is responsible for data transfer between the single-core computing device 301 and the DRAM 204.

[0045] Figure 4 An internal structure schematic diagram of the multi-core computing device 41 is shown. The multi-core computing device 41 adopts a hierarchical structure design. The multi-core computing device 41 is a system on chip, which includes at least one cluster, and each cluster includes a plurality of processor cores. In other words, the multi-core computing device 41 is composed of a system on chip-cluster-processor core hierarchy.

[0046] As shown in the system on chip hierarchy, Figure 4 the multi-core computing device 41 includes an external storage controller 401, a peripheral communication module 402, an on-chip interconnection module 403, a synchronization module 404, and a plurality of clusters 405.

[0047] The external storage controller 401 can be multiple, and two are exemplarily shown in the figure. The external storage controller 401 is configured to respond to an access request issued by the processor core to access an external storage device, such as a DRAM 204. Figure 2The DRAM 204 in the chip allows data to be read from or written to external devices. The peripheral communication module 402 receives control signals from the processing device 203 via the interface device 202, initiating the computing device 201 to execute tasks. The on-chip interconnect module 403 connects the external storage controller 401, the peripheral communication module 402, and multiple clusters 405 to transmit data and control signals between modules. The synchronization module 404 is a global barrier controller (GBC) used to coordinate the working progress of each cluster and ensure information synchronization. The multiple clusters 405 are the computing cores of the multi-core computing device 41. Four are shown exemplary in the figure; however, with hardware development, the multi-core computing device 41 of this disclosure can also include 8, 16, 64, or even more clusters 405. The clusters 405 are used to efficiently execute deep learning algorithms.

[0048] From the perspective of cluster hierarchy, such as Figure 4 As shown, each cluster 405 includes multiple processor cores (IPU cores) 406 and one memory core (MEM core) 407.

[0049] Four processor cores 406 are shown in the figure as an example, but this disclosure does not limit the number of processor cores 406. Its internal architecture is as follows: Figure 5 As shown. Each processor core 406 is similar to Figure 3 The single-core computing device 301 also includes three main modules: a control module 51, an arithmetic module 52, and a storage module 53. The functions and structures of the control module 51, arithmetic module 52, and storage module 53 are largely the same as those of the control module 31, arithmetic module 32, and storage module 33, and will not be described again. It should be noted that the storage module 53 includes an input / output direct memory access (IODMA) module 533 and a move direct memory access (MVDMA) module 534. The IODMA 533 controls the memory access of NRAM 531 / WRAM 532 and DRAM 204 via the broadcast bus 409; the MVDMA 534 controls the memory access of NRAM 531 / WRAM 532 and SRAM 408.

[0050] Back Figure 4The storage core 407 is mainly used to store and communicate, i.e., store shared data or intermediate results among the processor cores 406, perform communication between the execution cluster 405 and the DRAM 204, perform communication among the clusters 405, perform communication among the processor cores 406, etc. In other embodiments, the storage core 407 has the capability of scalar operations to perform scalar operations.

[0051] The storage core 407 includes an SRAM 408, a broadcast bus 409, a cluster direct memory access (CDMA) 410, and a global direct memory access (GDMA) 411. The SRAM 408 serves as a high-performance data relay station. Data reused among different processor cores 406 within the same cluster 405 does not need to be obtained by the processor cores 406 from the DRAM 204 individually, but is relayed among the processor cores 406 through the SRAM 408. The storage core 407 only needs to quickly distribute the reused data from the SRAM 408 to the multiple processor cores 406, so as to improve the efficiency of inter-core communication and greatly reduce on-chip and off-chip input / output access.

[0052] The broadcast bus 409, the CDMA 410, and the GDMA 411 are respectively used to perform communication among the processor cores 406, communication among the clusters 405, and data transmission between the clusters 405 and the DRAM 204. The following will be described respectively.

[0053] The broadcast bus 409 is used to complete high-speed communication among the processor cores 406 within the cluster 405. The broadcast bus 409 of the embodiment supports inter-core communication modes including unicast, multicast, and broadcast. Unicast refers to point-to-point (e.g., single processor core to single processor core) data transmission, multicast refers to a communication mode of transmitting a piece of data from the SRAM 408 to specific processor cores 406, and broadcast refers to a communication mode of transmitting a piece of data from the SRAM 408 to all processor cores 406, which is a special case of multicast.

[0054] The CDMA 410 is used to control access to the SRAM 408 among different clusters 405 within the same computing device 201.

[0055] The GDMA 411 cooperates with the external memory controller 401 to control the access of the SRAM 408 to the DRAM 204 or the reading of data from the DRAM 204 to the SRAM 408. As mentioned above, the communication between the DRAM 204 and the NRAM 431 or the WRAM 432 can be achieved through two channels. The first channel is to directly contact the DRAM 204 and the NRAM 431 or the WRAM 432 through the IODMA 433; the second channel is to first transfer data between the DRAM 204 and the SRAM 408 through the GDMA 411, and then transfer data between the SRAM 408 and the NRAM 431 or the WRAM 432 through the MVDMA 534. Although the second channel seems to need more elements to participate and the data flow is longer, in fact, in some embodiments, the bandwidth of the second channel is much larger than that of the first channel, so the communication between the DRAM 204 and the NRAM 431 or the WRAM 432 through the second channel can be more efficient. The embodiments of the present disclosure can select the data transmission channel according to the hardware conditions.

[0056] In other embodiments, the functions of the GDMA 411 and the functions of the IODMA 533 can be integrated in the same component. For the convenience of description, the GDMA 411 and the IODMA 533 are regarded as different components in the present disclosure, and for those skilled in the art, as long as the functions achieved and the technical effects achieved are similar to the present disclosure, they belong to the protection scope of the present disclosure. Further, the functions of the GDMA 411, the functions of the IODMA 533, the functions of the CDMA 410, and the functions of the MVDMA 534 can also be achieved by the same component.

[0057] The training of the neural network is to adjust the parameters of each layer by inputting the training samples, so that the result calculated by the neural network is as close to the true result as possible. The neural network training includes forward propagation and back propagation. The forward propagation is based on the existing model, and the input training samples are calculated through each layer of the neural network to gradually extract the input feature map into abstract features. After forward propagation, an output value called predicted value is obtained. The back propagation is to calculate the loss function according to the predicted value and the true value obtained by forward propagation, and to update the parameters by using the gradient descent method and calculating the partial derivative of the loss function to each parameter. In the chain rule, the derivative of the error value corresponding to the last layer weight of the neural network is calculated first. These derivatives are called gradients, and then these gradients are used to calculate the gradients of the second last layer in the neural network. Repeat this process until the gradient corresponding to each weight in the neural network is obtained. Finally, each weight in the neural network is subtracted by the corresponding gradient, so as to update the weight once to reduce the error value. The updated parameters are used for training, and the process is repeated several times, so that the calculation result of the forward propagation meets the expectation.

[0058] In the process of training the neural network, after each forward propagation of the signal and the corresponding back propagation of the error in the neural network, the weight in the neural network is updated once by using the gradient, which is called an iteration. In order to obtain a neural network with expected accuracy, a very large sample data set is needed in the training process. In this case, it is impossible to input the sample data set into the computer at one time. Therefore, in order to solve this problem, the sample data set needs to be divided into multiple blocks, each block is transmitted to the computer, and each block of data set is forward processed to update the weight of the neural network once. When a complete sample data set is processed once by the neural network and the corresponding weight update is returned, this process is called an epoch. In practice, it is not enough to transmit a complete data set in the neural network, and the complete data set needs to be transmitted multiple times in the same neural network, that is, multiple epochs are needed, and finally the neural network with expected accuracy is obtained.

[0059] This embodiment is based on the hardware environment described above, and provides a scheme for sparse training of a neural network model. In more detail, in each iteration including the forward propagation and the backward propagation process, the neural network parameters are processed for sparse in at least the forward propagation. The sparse processing can be one-dimensional sparse (e.g., input channel dimension), or multi-dimensional sparse, such as two-dimensional sparse (e.g., input channel dimension and output channel dimension are simultaneously sparse). In some embodiments, when the forward propagation performs input channel dimension and output channel dimension simultaneously sparse, the input channel dimension and output channel dimension simultaneously sparse can also be supported in the backward propagation, thereby further optimizing the performance. The sparse scheme of the present disclosure can be performed in multiple stages of training, and different training stages can adopt different structured sparse data flow structures for related operations to obtain optimized operation and IO performance.

[0060] Figure 6 An exemplary structural block diagram of a data processing apparatus according to an embodiment of the present disclosure is shown.

[0061] The data processing apparatus 600 may, for example, be implemented in the computing apparatus 201 of Figure 2 As shown, the data processing apparatus 600 can include a control circuit 610, a storage circuit 620, and an operation circuit 630.

[0062] The function of the control circuit 610 can be similar to that of the control module 314 of Figure 3 which may, for example, include an instruction fetch unit to fetch instructions from the processing apparatus 203, for example, and an instruction decode unit to decode the fetched instructions and send the decoded results as control information to the operation circuit 630 and the storage circuit 620. Figure 2

[0063] In one embodiment, the control circuit 610 can be configured to control the storage circuit 620 and the operation circuit 630 to perform sparse training of a neural network model.

[0064] The storage circuit 620 can be configured to store information, which can at least include neural network parameters. In embodiments of the present disclosure, the storage circuit 620 can also store mask tensors. In this embodiment, the storage circuit may, for example, be the WRAM 332, the NRAM 331 of Figure 3

[0065] The operation circuit 630 can be configured to perform sparse training of a neural network model under the control of the control circuit 610 to perform the method of sparse training as shown in Figure 7

[0066] Figure 7 ​​​A method performed in one iteration is shown according to embodiments of the present disclosure.

[0067] In step 710, in forward propagation, at least the neural network parameters are sparsified based on the mask tensor to calculate the value of the loss function.

[0068] In embodiments of the present disclosure, there can be multiple cases of the mask tensor.

[0069] In some embodiments, the mask tensor is a one-dimensional tensor which sparsifies one specified dimension of data. For example, the mask tensor sparsifies the input channel dimension of the neural network parameters.

[0070] In some embodiments, the sparsification can be a structured sparsification, for example, according to a sparsity rule, n data elements are selected from every m data elements of the to-be-sparse dimension of the input data as valid data elements, where m>n. In one implementation, m=4, n=2. In other implementations, when m=4, n can also take other values, such as 1 or 3.

[0071] At this time, the mask tensor can be a one-dimensional vector which can be divided into multiple intervals of length m, each interval has n elements of 1 representing the preserved data positions, and m-n elements of 0 representing the masked data positions.

[0072] In forward propagation, neurons (e.g., training data) and neural network parameters (e.g., weights) perform operations (e.g., convolution), and the neurons can be sparsified by the mask tensor in the same way, so that the corresponding operations are performed based on the sparsified results.

[0073] Figure 8A A mask process of an exemplary one-dimensional mask tensor according to embodiments of the present disclosure is shown. Figure 8A Taking the convolution layer operation of the convolutional neural network as an example, the sparse-based convolution operation in forward propagation is shown.

[0074] As shown in the figure, the to-be-sparse dimension is the input channel dimension. The exemplary mask tensor is a vector of length 16, which is divided into 4 intervals of length 4, each interval has 2 elements of 1, as shown by the black squares in the figure. The input channel dimension of the weights is correspondingly segmented, each segment corresponds to an interval of the mask tensor, and the two interact with each other (e.g., by the multipliers in the operation circuit 630 to multiply the corresponding bits) to obtain the masked weights. The input channel dimension of the neurons is similarly sparsified using the same mask tensor. The sparsified weights and the sparsified neurons perform operations, such as multiply-accumulate operations.

[0075] In other embodiments, the mask tensor is a two-dimensional tensor that sparsifies two specified dimensions of the data simultaneously. For example, the mask tensor sparsifies both the input channel dimension and the output channel dimension of the neural network parameters.

[0076] In some embodiments, the sparsification can be a structured sparsification, e.g., according to a sparsification rule, n data elements are selected from every m data elements of the to-be-sparified dimension of the input data as valid data elements, where m > n. In one implementation, m = 4, n = 2. In other implementations, when m = 4, n can also take other values, e.g., 1 or 3.

[0077] At this time, the mask tensor can be a two-dimensional matrix, which can be divided into multiple m x m blocks, and any row in each block has n elements of 1 and m-n elements of 0, and any column in each block has n elements of 1 and m-n elements of 0, where “1” represents a preserved data position and “0” represents a masked data position. In some embodiments, assuming that m is 4 and n is 2, such a 4 x 4 mask matrix has a total of 90, which can be pre-stored in the DRAM 204.

[0078] Figure 8B An exemplary mask process is shown, assuming that the input channel and the output channel of the convolution layer are a 4 x 4 channel matrix 801, and the elements are a 11 to a 44 The channel matrix 801 is the neural network parameter. An exemplary mask matrix 802 of the aforementioned 90 4 x 4 mask matrices is also shown in the figure, which is used to mask and sparsify the channel matrix 801. Specifically, if the corresponding element in the mask matrix 802 is 1, the operation circuit 630 preserves the element in the channel matrix 801, and if the corresponding element in the mask matrix 802 is 0, the operation circuit 630 masks the element in the channel matrix 801, and the value is 0. Taking a 11 , for example, the corresponding element in the mask matrix 802 is 0, so the corresponding element of the masked parameter matrix 803 is masked and the value is 0. In this way, all element values of the masked parameter matrix 803 are obtained. Since half of the elements in the channel matrix 801 are masked, about half of the calculation amount is saved.

[0079] For each training sample, the operation circuit 630 calculates based on the mask tensor in the forward propagation to obtain the value of the loss function, which corresponds to the output error of the neural network.

[0080] Returning to Figure 7In step 720, in back propagation, neuron gradients and neural network parameter gradients are calculated based on the loss function. In embodiments of the present disclosure, based on the mask tensor used in forward propagation, in back propagation, the sparsification process can be selectively applied or not applied.

[0081] In some embodiments, in back propagation, neuron gradients and neural network parameter gradients can be calculated based on the non-sparisified neural network parameters, regardless of the mask tensor used in forward propagation; and the neural network parameters are updated based on the neural network parameter gradients.

[0082] Depending on the information stored in the storage circuit, in some implementations, the non-sparisified neural network parameters can be the neural network parameters before sparsification, or can be obtained by de-sparsification of the already sparisified neural network parameters. The de-sparsification can include recovering the already sparisified neural network parameters to the corresponding positions before sparsification according to the indication of the mask tensor, and filling predetermined information (e.g. 0) at the remaining positions to recover the shape before sparsification.

[0083] In other embodiments, when the mask tensor used in forward propagation is a two-dimensional tensor, in back propagation, sparsification can also be applied, that is, the neural network parameter gradients and neuron gradients are calculated based on the sparisified neural network parameters, and then the neural network parameters are updated based on the neuron gradients.

[0084] In back propagation of training, the calculation of neuron gradients and weight gradients is involved, as shown below:

[0085]

[0086]

[0087] wherein top_diff and bottom_diff are neuron gradients respectively, W is the weight of the current iteration, and △W is the weight gradient calculated in the current iteration, is the calculation in back propagation, similar to convolution operation. With respect to the direction of back propagation, the bottom_diff of the previous layer is the top_diff of the current layer, and the bottom_diff of the current layer is the top_diff of the next layer, thus the error can be transferred back layer by layer.

[0088] In the calculation of formula (1), the layout of the weight W is different from that in the forward propagation process, and thus the direction of accumulation in the operation is also different. In the forward propagation, the weight is used in the order of (Co, Kh, Kw, Ci) dimensions or dimension shape, where Ci represents the input channel dimension, Co represents the output channel dimension, Kh is the convolution kernel height dimension, and Kw is the convolution kernel width dimension. In the convolution operation of the forward propagation, the operation result is accumulated in the Ci direction. In the backward propagation, the weight is used in the order of (Ci, Kh, Kw, Co) dimensions or dimension shape. In the operation of the backward propagation, the operation result is accumulated in the Co direction. Therefore, in order to maintain the mathematical consistency of the operation gradient in the backward propagation, the Ci and Co directions need to be sparsified at the same time.

[0089] When the sparsification processing is performed in the backward propagation, a reverse mask tensor can be used to mask the neural network parameters to obtain the sparsified neural network parameters.

[0090] The reverse mask tensor can be consistent with the mask tensor used in the forward propagation. However, due to the different layouts of the weights in the backward propagation mentioned above, the accumulation direction in the operation is also different, and thus the mask tensor in the forward propagation cannot be directly used. In some implementations, the mask tensor used in the forward propagation (or referred to as a forward mask tensor) can be used after dimension conversion. Various existing dimension conversion methods (for example, dimension transposition, data deformation) can be used to convert the mask tensor into the layout required in the backward propagation, and used as a reverse mask tensor. In other implementations, the mask tensor generation process adopted in the forward propagation process can also be repeated once in the backward propagation process to generate a reverse mask tensor. However, the mask calculation in the Ci direction is implemented in the forward propagation process, and the mask calculation in the Co direction is implemented in the backward propagation process.

[0091] Continue Figure 7 In step 730, the neural network parameters are updated based on the neural network parameter gradients.

[0092] The sparsification training of the embodiments of the present disclosure can include several training stages, such as a no-mask stage, a mask adjustment stage, and a mask fixing stage. The processing of each stage will be described in detail below in combination with the drawings.

[0093] Based on different stages of the sparsification training, the update of the neural network parameters can also be different.

[0094] In some embodiments, updating the neural network parameters can be updating the unsparsified neural network parameters. For example, in the mask adjustment phase, the unsparsified neural network parameters are updated in each iteration. Further, in the mask adjustment phase, an updated mask tensor can be generated based on the updated unsparsified neural network parameters in every K (K > 1) iterations, whereby the mask tensor can be optimized in the training process to improve performance.

[0095] In other embodiments, updating the neural network parameters can be updating the sparsified neural network parameters. For example, in the mask fixing phase, since the mask tensor is fixed, the sparsification pattern of the neural network parameters is fixed, i.e., the valid data elements in the neural network parameters are fixed, thus the updating of the neural network parameters can only update the valid data elements, i.e., updating the sparsified neural network parameters. In one implementation, updating the neural network parameters can include sparsifying neuron gradients using the mask tensor, and updating the sparsified neural network parameters based on the sparsified neuron gradients.

[0096] The fixed mask tensor in the mask fixing phase can be the mask tensor determined in the last training phase (e.g., the mask adjustment phase). Depending on the different forms of the mask tensor, there can be different ways to generate or update the mask tensor.

[0097] When the mask tensor is a one-dimensional tensor, i.e., a mask vector, the mask vector can only mask a single parameter. The mask tensor can be generated based on the unsparsified neural network parameters. For example, n largest absolute values of every m data elements in a specified dimension of the neural network parameters are selected as valid data elements, where m > n, and the mask tensor is generated based on the positions of the n valid data elements in the m data elements. In some implementations, the specified dimension can be the input channel dimension (Ci). Specifically, this embodiment divides the parameters into multiple intervals in units of a certain parameter quantity m, and the parameters in each interval are sorted according to their absolute values. Then, in the mask tensor, the elements at positions relative to the first n parameters with larger absolute values in each interval are set to 1, and the elements at positions relative to the m-n parameters with smaller absolute values in each interval are set to 0, because the parameters with larger absolute values in the mask adjustment phase carry more obvious features and are more worthy of being retained for further calculation. There are many ways to screen the parameters with larger absolute values in the mask adjustment phase, and the present disclosure is not limited in this regard.

[0098] Figure 9 An exemplary mask vector updating diagram is shown in FIG. 9 to illustrate the foregoing updating of the mask vector. As shown in FIG. 9, a parameter vector 901 is shown, which has 64 parameters, b 01 to b 64. In this step, the element values of the mask vector are updated to retain the mask adjustment parameters with larger absolute values and mask the mask adjustment parameters with smaller absolute values. The updated mask adjustment parameters are divided into multiple intervals in units of every 4 mask adjustment parameters (i.e., m is 4), as shown in the figure, b 01 to b 04 is the first interval 902, b 05 to b 08 is the second interval 903, b 61 to b 64 is the sixteenth interval 917, and then the mask adjustment parameters in each interval are sorted according to their absolute values. Assuming that the absolute values of the parameters in the first interval 902 are b 02 > b 01 > b 04 > b 03 , the absolute values of the parameters in the second interval 903 are b 07 > b 05 > b 06 > b 08 , and the absolute values of the parameters in the sixteenth interval 917 are b 64 > b 63 > b 61 > b 62 , the mask adjustment parameters in each interval are sorted according to the absolute values of the mask adjustment parameters. Then, in the mask vector, the elements at positions relative to the first 2 (i.e., n is 2) mask adjustment parameters with larger absolute values in each interval are set to 1, and the elements at positions relative to the last 2 (i.e., m-n is 2) mask adjustment parameters with smaller absolute values in each interval are set to 0. Taking the first interval 902 as an example, the elements corresponding to b 02 and b 01 in the mask vector are set to 1, and the elements corresponding to b 04 and b 03 are set to 0. Each interval is adjusted in this way, and finally the updated mask vector 918 is completed. The updated mask vector 918 retains the mask adjustment parameters with larger absolute values after updating and masks the mask adjustment parameters with smaller absolute values after updating. In summary, every 4 mask adjustment parameters are taken as an interval, and the element values of the mask vector are updated in a 4-to-2 manner in each interval.

[0099] This embodiment is to sort the mask adjustment parameters in each interval completely to identify the n mask adjustment parameters with larger absolute values and the m-n mask adjustment parameters with smaller absolute values. However, the present disclosure does not necessarily need to sort completely, but only needs to identify the n mask adjustment parameters with larger absolute values and the m-n mask adjustment parameters with smaller absolute values. The sizes of the n mask adjustment parameters with larger absolute values and the sizes of the m-n mask adjustment parameters with smaller absolute values are not necessary information. Taking the first interval 902 as an example, the present disclosure only needs to judge b01 with b 02 are two with larger absolute values, while b 03 with b 04 are two with smaller absolute values, while b 01 with b 02 are two with larger absolute values, while b 03 with b 03 are two with smaller absolute values, while b

[0100] If the mask tensor is multi-dimensional (e.g., two-dimensional), the training data can be multiplied with each masked parameter tensor to obtain a parameter evaluation value. The purpose of obtaining the parameter evaluation value is to calculate the amount of information retained by the mask tensor. If the parameter evaluation value is high, it indicates that the amount of information has not been lost too much after the mask, and the mask tensor has reduced the amount of computation on the premise of retaining most of the information, and is a high-quality mask tensor; on the contrary, if the parameter evaluation value is low, it indicates that the amount of information has been lost too much after the mask, and the mask tensor is not a high-quality mask tensor.

[0101] Specifically, for a two-dimensional mask tensor, the following can be used to determine it: a specific number of two-dimensional mask tensors are preset, and then one of the preset two-dimensional mask tensors is selected as the mask tensor to be used. Each dimension of the two-dimensional mask tensor includes m elements, of which n elements are 1 and m-n elements are 0, and m>n. As mentioned earlier, under the condition of m=4 and n=2, there are a total of 90 such 4x4 mask matrices, so one of the 90 mask matrices is selected as the mask tensor.

[0102] Selecting one of the specific number (e.g., 90) of two-dimensional mask tensors can include: respectively masking the specified two dimensions of the neural network parameters based on each of the preset two-dimensional mask tensors to obtain a masked parameter tensor; performing a product-sum operation on the training data of the neural network layer based on each of the masked parameter tensors to obtain a parameter evaluation value; and selecting the two-dimensional mask tensor that produces the largest parameter evaluation value among all parameter evaluation values as the selected mask tensor. In some implementations, the specified two dimensions can be the input channel dimension and the output channel dimension. The product-sum operation can also be regarded as a convolution operation, but it does not accumulate in the input channel dimension, but only in the depth direction, so it can also be called a depth direction convolution operation, where the depth direction is the Kw x Kh dimension.

[0103] Figure 10An exemplary product-sum calculation process is shown. Assume that the training data matrix 1001 is one of the training data in the training set, which should be calculated with the channel matrix 801 of FIG. 8, but now is calculated with the product-sum of the masked parameter matrix 803 to identify the amount of information retained after masking. There are various ways to perform such product-sum calculation, for example, multiplying the corresponding elements of the training data matrix 1001 and the masked parameter matrix 803, taking the absolute value, and adding them up to obtain the parameter evaluation value S1, i.e.:

[0104] S1 = |d 31 ·a 31 | + |d 41 ·a 41 | + |d 12 ·a 12 | + |d 42 ·a 42 | + |d 13 ·a 13 | + |d 23 ·a 23 | + |d 24 ·a 24 | + |d 34 ·a 34 |

[0105] For another example, multiplying the corresponding elements of the training data matrix 1001 and the masked parameter matrix 803, taking the absolute value, and adding them up to obtain the parameter evaluation value S2, i.e.:

[0106] S2 = |d 31 | · |a 31 | + |d 41 | · |a 41 | + |d 12 | · |a 12 | + |d 42 | · |a 42 | + |d 13 | · |a 13 | + |d 23 | · |a 23 | + |d 24 | · |a 24 | + |d 34 | · |a 34 |

[0107] The parameter evaluation value reflects the result of the absolute value calculation, and the parameter evaluation value S1 or S2 shows the amount of information retained after masking. The higher the value of the parameter evaluation value, the more information is retained. In one application scenario, either the parameter evaluation value S1 or S2 can be selected, while in another application scenario, both the parameter evaluation value S1 and S2 can be used, and the present disclosure does not limit the selection.

[0108] All the mask tensors are masked and get parameter evaluation values. In the example above, it means all the 90 4x4 mask matrices are masked and get 90 parameter evaluation values. The mask tensor with the largest parameter evaluation value is selected as the updated mask tensor, i.e. the parameter mask tensor. There are many ways to select the mask tensor with the largest parameter evaluation value, for example, all the parameter evaluation values can be sorted by their values to get the largest parameter evaluation value, or a two-input comparator can be used to compare, leaving the larger one and comparing it with the next parameter evaluation value, and after comparing all the 90 parameter evaluation values, the one left is the largest parameter evaluation value. If there are multiple mask tensors with the same largest parameter evaluation value, one of them can be selected based on certain rules or hardware characteristics, for example, the one sorted first, the one sorted last, the one left first, the one left last, or randomly selected.

[0109] The mask tensor with the largest parameter evaluation value is the mask tensor that retains the most information, and this embodiment takes it as the parameter mask tensor.

[0110] In this embodiment, the mask tensor can be updated in each iteration or in each generation of training. If the neural network parameters are updated after each training sample is trained in the training process, the mask tensor is preferably updated in each iteration; if the neural network parameters are updated in each iteration, the parameter mask tensor is preferably updated at the end of each generation of training.

[0111] Those skilled in the art can understand that although the generation of the mask tensor is described above based on the update process, the mask tensor can be generated in a similar way when it is first generated, but the neural network parameters based on which the mask tensor is generated will be different. Depending on the stages included in the training process, when the mask tensor is first generated, the neural network parameters based on which the mask tensor is generated can be randomly initialized parameters or determined neural network parameters after the unmasked stage of training.

[0112] As mentioned above, the sparse training of the embodiments of the disclosure can include several training stages, such as the unmasked stage, the mask adjustment stage and the mask fixed stage. The processing of each stage will be described in detail below in conjunction with the drawings.

[0113] Figure 11 An exemplary flowchart including the unmasked stage and the mask adjustment stage is shown. In the unmasked stage, the processing device 203 only trains the neural network parameters, that is, does not perform mask sparsification on the neural network parameters, and after the unmasked stage ends and enters the mask adjustment stage, the parameters are trained and the mask tensor is updated.

[0114] As mentioned above, the sparse training of the embodiments of the disclosure can include several training stages, such as the unmasked stage, the mask adjustment stage and the mask fixed stage. The processing of each stage will be described in detail below in conjunction with the drawings. Figure 11As shown, in step 1101, the control circuit 610 first sets to enter the unmasked phase. In the unmasked phase, this embodiment does not mask the neural network parameters, and all elements of the parameters participate in training. At the beginning of training, the parameter values can be randomly generated. For the convenience of identification, the parameters participating in training in the unmasked phase are referred to as unmasked parameters.

[0115] In step 1102, the operation circuit 630 calculates the value of the loss function based on the unmasked parameters in the forward propagation. In this step, the operation circuit 630 calculates the loss function in the manner of the prior art, that is, the input training sample is calculated through each layer of the neural network in the forward propagation to extract the feature map as abstract features, and the loss function is calculated based on the forward propagation result and the true value.

[0116] In step 1103, the operation circuit 630 calculates the partial derivative of the loss function with respect to the unmasked parameters in the backward propagation. The operation circuit 630 calculates the partial derivative of the loss function with respect to each unmasked parameter by the chain rule in the gradient descent method.

[0117] In step 1104, the operation circuit 630 updates the unmasked parameters based on the partial derivative and takes the updated unmasked parameters as the initial value of the mask adjustment parameters. First, the operation circuit 630 updates the unmasked parameters of the entire neural network according to the influence of the unmasked parameters on the error and then multiplies the step size. In this embodiment, the operation circuit 630 can also update the unmasked parameters based on the partial derivative in each training sample or each iteration.

[0118] This embodiment can repeat steps 1102, 1103 and 1104 in a certain number of generations of training to update the unmasked parameters multiple times. After the last update, the updated unmasked parameters will be taken as the initial value of the mask adjustment parameters in the next phase.

[0119] In step 1105, the control circuit 610 sets to enter the mask adjustment phase, that is, to start masking part of the parameters by using the mask tensor. In the training, the prior art only trains all parameters (such as weights, biases, etc.), and usually does not mask the parameters. This embodiment masks the parameters, which aims to reduce the participation of parameters in the training phase to avoid overfitting and reduce the amount of calculation, and at the same time, the mask tensor is also updated with the update of the parameters in the training process to obtain a more ideal mask tensor. At the beginning of entering the mask adjustment phase, as described above, the initial value of the mask adjustment parameters is the unmasked parameters updated at the end of the unmasked phase, and the mask tensor can obtain the initial value of the mask adjustment parameters based on the unmasked parameters updated at the end of the unmasked phase, and the obtaining method is the same as the generation method of the mask tensor described above, which will not be described again.

[0120] In step 1106, the mask adjustment parameters are masked based on the mask tensor in the forward propagation to calculate the value of the loss function. In step 1107, the partial derivative of the loss function with respect to the mask adjustment parameters is calculated in the backward propagation. In step 1108, the mask adjustment parameters are updated based on the partial derivative. In step 1109, the mask tensor is updated based on the updated mask adjustment parameters. These steps can refer to the foregoing description of the hardware environment, which will not be repeated here. Figure 7

[0121] This embodiment does not limit the number of generations of training in the unmasked phase and the mask adjustment phase, which can be arranged by the person skilled in the art according to the specific situation, and the number of generations of training in the unmasked phase and the mask adjustment phase does not necessarily have to be the same.

[0122] Another embodiment of the present disclosure is also based on the foregoing hardware environment, and provides a scheme for training a neural network model in a sparse manner. Unlike the foregoing embodiment, the training is divided into three phases: an unmasked phase, a mask adjustment phase, and a mask fixed phase. In the unmasked phase, the processing device 203 only trains the parameters without masking the parameters, in the mask adjustment phase, the processing device 203 takes the updated unmasked parameters as the initial value, and simultaneously trains the parameters and the mask tensor, and in the mask fixed phase, the processing device 203 takes the updated mask adjustment parameters and the updated mask tensor in the mask adjustment phase as the initial value, and continues to train the parameters without changing or updating the mask tensor.

[0123] The processes performed in the unmasked phase and the mask adjustment phase of this embodiment are shown in FIG. 12B, and will not be repeated here. After entering the mask fixed phase, the process is shown in FIG. 12C. Figure 11 Figure 12

[0124] In step 1201, the control circuit 610 sets to enter the mask fixed phase. In the mask fixed phase, the control circuit 610 takes the mask adjustment parameters updated in the mask adjustment phase as the initial value of the parameters (hereinafter referred to as mask fixed parameters) in this phase. This embodiment updates the mask tensor in the mask adjustment phase, so the mask tensor will not be updated in this phase, but the mask fixed parameters are masked based on the mask tensor updated in the mask adjustment phase, and the mask fixed parameters are continued to be trained.

[0125] This embodiment repeats the following steps in at least one generation of training.

[0126] In step 1202, the operation circuit 630 masks the mask fixed parameters based on the mask tensor updated in the mask adjustment phase in the forward propagation to calculate the value of the loss function.

[0127] ​​​In step 1203, the operation circuit 630 calculates the partial derivative of the loss function with respect to the fixed parameter of the mask in the back propagation.

[0128] In step 1204, the update module 64 updates the fixed parameter of the mask based on the partial derivative.

[0129] The above steps can refer to the description of the foregoing embodiments, which will not be described here again. Figure 7

[0130] This embodiment is divided into three stages during training. In the unmasked stage, there is no mask tensor to mask the parameters, and only the parameters are trained to accelerate the convergence of the parameters. In the mask adjustment stage, since the initial value of the parameters is no longer randomly generated, but the unmasked parameters that have been trained, it helps to quickly obtain an ideal mask tensor. After the mask tensor is updated, enter the mask fixed stage, and continue to train the parameters using the updated mask tensor. The finally trained parameters will better match the mask tensor.

[0131] In summary, those skilled in the art understand that the present disclosure can have several embodiments when training a neural network model to be sparse, as shown in Figure 13

[0132] Embodiment 1301 only has a mask adjustment stage, the initial value of the parameter W0 is randomly generated, the initial value of the mask tensor M0 is determined based on the initial value of the parameter W0, and the parameter is trained while the mask matrix is updated to obtain the trained parameter Wf and the updated mask tensor Mf.

[0133] Embodiment 1302 only has an unmasked stage and a mask adjustment stage. In the unmasked stage, only the parameters are trained, the initial value of the parameter W0 is randomly generated, and the updated parameter W1 is obtained after training. In the mask adjustment stage, the parameter is trained while the mask matrix is updated. The initial value of the parameter in this stage is the updated parameter W1, and the initial value of the mask tensor M0 is obtained using the updated parameter W1. Finally, the trained parameter Wf and the updated mask tensor Mf are obtained.

[0134] Embodiment 1303 only has a mask adjustment stage and a mask fixed stage. In the mask adjustment stage, the initial value of the parameter W0 is randomly generated, the initial value of the mask tensor M0 is determined based on the initial value of the parameter W0, and the parameter is trained while the mask matrix is updated to obtain the updated parameter W1 and the updated mask tensor Mf. In the mask fixed stage, the parameter is continuously trained using the updated mask tensor Mf to mask the parameter. The initial value of the parameter in this stage is the updated parameter W1, and the trained parameter Wf is finally obtained.

[0135] ​​The embodiment 1304 has a mask-free phase, a mask adjustment phase and a mask fixed phase. In the mask-free phase, only the parameters are trained, the initial value of the parameters W0 is randomly generated, and the updated parameters W1 are obtained after training. In the mask adjustment phase, the parameters are trained and the mask matrix is updated, the initial value of the parameters in this phase is the updated parameters W1, and the initial value of the mask tensor M0 is obtained by using the updated parameters W1, and finally the updated parameters W2 and the updated mask tensor Mf are obtained. In the mask fixed phase, the parameters are continuously trained by using the updated mask tensor Mf to mask the parameters, the initial value of the parameters in this phase is the updated parameters W2, and finally the trained parameters Wf are obtained.

[0136] The embodiment 1305 has a mask-free phase, a mask adjustment phase and a mask fixed phase, and there are other training phases (indicated by dashed lines) between the mask-free phase and the mask adjustment phase and between the mask adjustment phase and the mask fixed phase. In the mask-free phase, only the parameters are trained, the initial value of the parameters W0 is randomly generated, and the updated parameters W1 are obtained after training. Then, any training phase disclosed or not disclosed in the present disclosure can be followed to train the parameters or update the mask matrix, and if the phase is a mask fixed phase, the initial value of the parameters in this phase is the updated parameters W1, and the initial value of the mask tensor M0 is obtained by using the updated parameters W1 to obtain the updated parameters W2.

[0137] Then, the mask adjustment phase is entered, the parameters are trained and the mask matrix is updated, the initial value of the parameters in this phase is the updated parameters W2, and the initial value of the mask tensor is still the mask tensor M0, to obtain the updated parameters W3 and the updated mask tensor M1. Then, any phase disclosed or not disclosed in the present disclosure can be followed to train the parameters or update the mask matrix, and if the phase is a parameter fixed phase, the parameters are not trained, only the mask tensor is trained, the initial value of the parameters in this phase is the updated parameters W3, and the initial value of the mask tensor is the updated mask tensor M1, to obtain the updated mask tensor Mf.

[0138] Finally, in the mask fixed phase, the parameters are continuously trained by using the updated mask tensor Mf to mask the parameters, the initial value of the parameters in this phase is the updated parameters W3, and finally the trained parameters Wf are obtained.

[0139] Figure 13 The various embodiments shown are only examples, and those skilled in the art can extend other embodiments without creative efforts after referring to the present disclosure, and these embodiments all belong to the scope disclosed in the present disclosure.

[0140] The present disclosure does not limit the number of times of one generation training in each phase of the various embodiments, and those skilled in the art can arrange according to specific conditions, and the number of times of one generation training in each phase does not necessarily have to be the same.

[0141] The foregoing embodiments do not necessarily have to perform all the pre-set specific number of generations of training. The control circuit 610 can further determine whether the percentage of all element values of the parameter mask tensor that do not change in the last two generations of training reaches a threshold. If so, it means that the training result has basically converged, and more training will have limited improvement in accuracy, so the mask adjustment phase is ended and the training is completed. Such a threshold is generally set to be more than 70%, that is, the percentage of all element values of the parameter mask tensor that do not change is more than 70% to stop training. The present disclosure does not limit the threshold, which can be 80%, 90%, 100% or any other percentage.

[0142] In embodiments of the present disclosure, in order to save the overhead brought by the sparsification and desparsification processes, different sparsification data flow structures can be used in different stages of training to obtain optimal operation and IO performance.

[0143] In some embodiments, in the mask adjustment phase, the mask tensor can be updated based on the updated neural network parameters, and the result of the update process can include the sparsification result of the neural network parameters (e.g., sparse weights) and the mask tensor. The mask tensor can be used for the sparsification of the training data. Then, subsequent operations can be performed based on the sparsified neural network parameters and the sparsified training data. In the backpropagation process of the mask adjustment phase, the neuron gradient and the neural network parameter gradient can be calculated based on the current non-sparse neural network parameters, and the non-sparse neural network parameters are updated accordingly. Alternatively, in the backpropagation of the mask adjustment phase, the neural network parameters can be sparsified based on the mask tensor used in the forward propagation, and the neuron gradient and the neural network parameter gradient can be calculated based on the sparsified neural network parameters, and the non-sparse neural network parameters are updated accordingly. The sparsification in the backpropagation process is described above and will not be repeated here.

[0144] In other embodiments, in the mask fixing phase, the mask tensor is fixed and does not need to be updated in real time. Therefore, the fixed mask tensor can be stored in the storage circuit for subsequent use. The fixed mask tensor can include the forward mask tensor used in the forward propagation and the backward mask tensor used in the backward propagation. The neural network parameters can have different storage schemes.

[0145] In one implementation, the storage circuit can store the neural network parameters that are not sparsified. In this case, in the forward propagation, the neural network parameters need to be sparsified by using the stored mask tensors. In the backward propagation, the neural network parameters that are not sparsified are directly used to calculate the neuron gradient (e.g., formula (1) above), and the neural network parameters that are not sparsified are updated and stored in the storage circuit again. Alternatively, in the backward propagation, the neural network parameters that are not sparsified can be sparsified by using the backward mask tensors stored in the storage circuit, and then the neuron gradient is calculated based on this and the neural network parameters that are not sparsified are updated accordingly.

[0146] In another implementation, the storage circuit can store the neural network parameters that are sparsified. In this case, in the forward propagation, the neural network parameters that are sparsified can be directly used for forward operation without further sparsification. In the backward propagation, the neural network parameters that are sparsified need to be updated, and therefore the neural network parameter gradient can be sparsified by using the mask tensors stored in the storage circuit, and then the neural network parameters that are sparsified are updated. In the neuron gradient calculation in the backward propagation, sparsification or non-sparification can be selected. When non-sparification is used, the neural network parameters that are sparsified need to be de-sparsified, and then the neuron gradient is calculated based on the neural network parameters after de-sparsification. When sparsification is used, the neural network parameters after de-sparsification can be sparsified again by using the backward mask tensors stored in the storage circuit, and then the neuron gradient is calculated based on this.

[0147] Another embodiment of the present disclosure is a computer-readable storage medium having stored thereon computer program codes for sparsifying training of a neural network model, which, when executed by a processor, performs the method of any of the above embodiments. In some implementation scenarios, the above integrated units can be implemented in the form of software program modules. If implemented in the form of software program modules and sold or used as independent products, the integrated units can be stored in a computer-readable memory. Based on this, when the scheme of the present disclosure is embodied in the form of a software product (e.g., a computer-readable storage medium), the software product can be stored in a memory, which can include a plurality of instructions for causing a computer device (e.g., a personal computer, a server, or a network device, etc.) to perform some or all of the steps of the method described in the embodiments of the present disclosure. The above-mentioned memory can include, but is not limited to, a U disk, a flash disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0148] After the foregoing embodiments are trained, the computing device 201 uses the updated parameter mask tensor to mask the trained parameters when performing inference, so as to control the processing area of the feature map input to the neural network model. On the one hand, the expected accuracy can be achieved, and on the other hand, the computational complexity in the inference process can be reduced, thereby achieving the purpose of sparsification.

[0149] According to different application scenarios, the electronic device or apparatus of the present disclosure can include a server, a cloud server, a server cluster, a data processing apparatus, a robot, a computer, a printer, a scanner, a tablet computer, a smart terminal, a PC device, an Internet of Things terminal, a mobile terminal, a mobile phone, a vehicle recorder, a navigator, a sensor, a camera, a camera, a camera, a projector, a watch, a headset, a mobile storage, a wearable device, a visual terminal, an autonomous driving terminal, a vehicle, a household appliance, and / or a medical device. The vehicle includes an airplane, a ship and / or a vehicle; the household appliance includes a television, an air conditioner, a microwave oven, a refrigerator, an electric rice cooker, a humidifier, a washing machine, an electric lamp, a gas stove, an exhaust hood; the medical device includes a nuclear magnetic resonance instrument, a B-ultrasound instrument and / or an electrocardiograph. The electronic device or apparatus of the present disclosure can also be applied to the fields of Internet, Internet of Things, data center, energy, transportation, public management, manufacturing, education, power grid, telecommunications, finance, retail, construction site, medical treatment, etc. Further, the electronic device or apparatus of the present disclosure can also be used in cloud, edge, terminal and other application scenarios related to artificial intelligence, big data and / or cloud computing. In one or more embodiments, the electronic device or apparatus with high computing power according to the present disclosure can be applied to a cloud device (such as a cloud server), and the electronic device or apparatus with low power consumption can be applied to a terminal device and / or an edge device (such as a smart phone or a camera). In one or more embodiments, the hardware information of the cloud device and the hardware information of the terminal device and / or the edge device are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud device according to the hardware information of the terminal device and / or the edge device to simulate the hardware resources of the terminal device and / or the edge device, so as to complete the unified management, scheduling and cooperative work of end-cloud integration or cloud-edge integration.

[0150] It should be noted that, for the purpose of clarity, the disclosure describes some methods and embodiments thereof as a series of acts and / or combinations thereof, but those skilled in the art will understand that the present disclosure is not limited to the order of the acts described. Those skilled in the art will understand and appreciate that some steps of the methods can be decided to be executed in other orders or at the same time with other steps. Further, those skilled in the art will understand and appreciate that some of the embodiments described in the disclosure can be considered optional, i.e., the acts or modules involved therein are not necessarily essential for the implementation of one or more of the aspects of the present disclosure. In addition, the disclosure describes some embodiments with different focuses according to different aspects. In view of this, those skilled in the art will understand that the parts not described in detail in some embodiments of the disclosure can also be seen from the relevant description of other embodiments.

[0151] In terms of implementation, based on the disclosure and teachings of the present disclosure, those skilled in the art can understand that some embodiments disclosed in the present disclosure can also be implemented in other ways not disclosed herein. For example, as for each unit in the electronic device or apparatus embodiments described above, the units are split based on the logical functions considered herein, and there can be other splitting manners when actually implemented. For another example, a plurality of units or components can be combined or integrated into another system, or some features or functions of the units or components can be selectively disabled. As for the connection relationship between different units or components, the connection discussed above in conjunction with the drawings can be a direct or indirect coupling between the units or components. In some scenarios, the aforementioned direct or indirect coupling involves a communication connection using an interface, where the communication interface can support electrical, optical, acoustic, magnetic or other forms of signal transmission.

[0152] In the present disclosure, the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units. The aforementioned components or units can be located in the same place or distributed on a plurality of network units. In addition, according to actual needs, some or all of the units can be selected to achieve the purpose of the aspects described in the embodiments of the present disclosure. In addition, in some scenarios, a plurality of units in the embodiments of the present disclosure can be integrated into one unit or each unit physically exists separately.

[0153] In some other implementation scenarios, the above-mentioned integrated units can also be implemented in the form of hardware, i.e., specific hardware circuits, which can include digital circuits and / or analog circuits, etc. The physical implementation of the hardware structure of the circuit can include but is not limited to physical devices, and the physical devices can include but are not limited to transistors or memristors, etc. In view of this, various apparatuses (e.g., computing apparatuses or other processing apparatuses) described herein can be implemented by appropriate hardware processors, such as central processing units, GPUs, FPGAs, DSPs, ASICs, etc. Further, the aforementioned storage units or storage devices can be any appropriate storage medium (including magnetic storage media or magneto-optical storage media, etc.), which can be, for example, resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high bandwidth memory (HBM), hybrid memory cube (HMC), ROM, and RAM, etc.

[0154] The foregoing can be better understood in light of the following clauses:

[0155] Clause 1. A method performed by a data processing apparatus for sparsifying training of a neural network model, comprising:

[0156] in the forward propagation, sparsifying at least the neural network parameters based on a mask tensor to compute a value of a loss function;

[0157] in the backward propagation, computing neuron gradients and the neural network parameter gradients based on the loss function; and

[0158] updating the neural network parameters based on the neural network parameter gradients.

[0159] Clause 2. The method of clause 1, wherein the method further comprises:

[0160] in the backward propagation, computing the neuron gradients and the neural network parameter gradients based on non-sparisified neural network parameters; and

[0161] updating the neural network parameters based on the neural network parameter gradients.

[0162] Clause 3. The method of clause 2, wherein the method further comprises:

[0163] de-sparsifying the sparsely processed neural network parameters to obtain the unsparsely processed neural network parameters.

[0164] Clause 4. The method of clause 1, wherein the method further comprises:

[0165] in backpropagation, computing the neuron gradients and the neural network parameter gradients based on the sparsely processed neural network parameters; and

[0166] updating the neural network parameters based on the neuron gradients.

[0167] Clause 5. The method of clause 4, the method further comprising:

[0168] in backpropagation, sparsely processing the neural network parameters based on the reverse mask tensor to obtain the sparsely processed neural network parameters.

[0169] Clause 6. The method of any of clauses 1-3, wherein the mask tensor is a one-dimensional tensor.

[0170] Clause 7. The method of clause 6, wherein the one-dimensional tensor sparsely processes an input channel dimension of a neural network parameter.

[0171] Clause 8. The method of any of clauses 1-5, wherein the mask tensor is a two- dimensional tensor.

[0172] Clause 9. The method of clause 8, wherein the two-dimensional tensor sparsely processes an input channel dimension and an output channel dimension of a neural network parameter.

[0173] Clause 10. The method of clause 5, wherein when the mask tensor is a two- dimensional tensor, the reverse mask tensor is generated by performing a dimension conversion on the mask tensor.

[0174] Clause 11. The method of clause 1, wherein updating the neural network parameters comprises updating unsparsely processed neural network parameters.

[0175] Clause 12. The method of clause 11, further comprising:

[0176] generating the mask tensor based on the updated unsparsely processed neural network parameters.

[0177] Clause 13. The method of clause 12, wherein when the mask tensor is a one- dimensional tensor, the method generates the mask tensor as follows:

[0178] selecting, from each m data elements of a specified dimension of the neural network parameter, n data elements with larger absolute values as valid data elements, where m > n; and

[0179] determining the mask tensor based on positions of the n valid data elements in the m data elements.

[0180] Clause 14, the method according to Clause 12, wherein when the mask tensor is a two-dimensional tensor, the method generates the mask tensor as follows:

[0181] pre-setting a certain number of two-dimensional mask tensors, each dimension of the two-dimensional mask tensors including m elements, where n elements are 1 and m-n elements are 0, and m > n;

[0182] masking, based on each pre-set two-dimensional mask tensor, respectively, two specified dimensions of the neural network parameter to obtain a masked parameter tensor;

[0183] performing a product-sum operation on training data of the neural network based on each masked parameter tensor to obtain a parameter evaluation value; and

[0184] selecting a two-dimensional mask tensor that produces the largest parameter evaluation value among all parameter evaluation values as the mask tensor.

[0185] Clause 15, the method according to any one of Clauses 1-14, wherein the method is performed in multiple iterations in a mask adjustment phase of the sparsification training.

[0186] Clause 16, the method according to Clause 15, wherein the mask adjustment phase further comprises:

[0187] determining whether a percentage of all element values of the mask tensor that have not changed in consecutive multiple iterations of training reaches a threshold value; and

[0188] if so, ending the mask adjustment phase.

[0189] Clause 17, the method according to Clause 16, wherein the threshold value is one of 80%, 90%, and 100%.

[0190] Clause 18, the method according to any one of Clauses 1-10, wherein the method is performed in multiple iterations in a mask fixing phase of the sparsification training, and the mask tensor is fixed as the mask tensor finally determined in a previous phase.

[0191] Clause 19, the method according to Clause 18, wherein updating the neural network parameter comprises updating a neural network parameter that has been sparsified.

[0192] Clause 20, the method of clause 19, wherein updating the neural network parameters further comprises:

[0193] sparsifying the neuron gradient with the mask tensor; and

[0194] updating the sparsified neural network parameters based on the sparsified neuron gradient.

[0195] Clause 21, the method of any one of clauses 18-20, wherein during the mask fixing phase, storing the fixed mask tensor and the sparsified neural network parameters.

[0196] Clause 22, the method of any one of clauses 18-20, wherein during the mask fixing phase, storing the fixed mask tensor and the unsparsified neural network parameters.

[0197] Clause 23, a computer readable storage medium having stored thereon computer program code to sparsify training of a neural network model, which when executed by a processing apparatus, performs the method of any one of clauses 1 to 22.

[0198] Clause 24, a data processing apparatus comprising a control circuit, a storage circuit and an operation circuit, wherein:

[0199] the control circuit is configured to control the storage circuit and the operation circuit to perform sparsification training of a neural network model;

[0200] the storage circuit is configured to store information, the information comprising at least neural network parameters and a mask tensor; and

[0201] the operation circuit is configured to perform the following operations under the control of the control circuit:

[0202] in forward propagation, sparsifying at least the neural network parameters based on the mask tensor to compute a value of a loss function;

[0203] in backward propagation, computing neuron gradients and neural network parameter gradients based on the loss function; and

[0204] updating the neural network parameters based on the neural network parameter gradients.

[0205] Clause 25, the apparatus of clause 24, wherein the operation circuit is further configured to:

[0206] in backward propagation, computing the neuron gradients and the neural network parameter gradients based on the unsparsified neural network parameters; and

[0207] updating the neural network parameters based on the neuron gradients.

[0208] Clause 26, the apparatus of clause 25, wherein the operation circuitry is further configured to:

[0209] de-sparsifying the sparsely processed neural network parameters to obtain the unsparsely processed neural network parameters.

[0210] Clause 27, the apparatus of clause 24, wherein the operation circuitry is further configured to:

[0211] in backpropagation, computing the neuron gradients and the neural network parameters gradients based on the sparsely processed neural network parameters; and

[0212] updating the neural network parameters based on the neuron gradients.

[0213] Clause 28, the apparatus of clause 27, wherein the operation circuitry is further configured to:

[0214] in backpropagation, sparsely processing the neural network parameters based on the reverse mask tensor to obtain the sparsely processed neural network parameters.

[0215] Clause 29, the apparatus of any of clauses 24-26, wherein the mask tensor is a one-dimensional tensor.

[0216] Clause 30, the apparatus of clause 29, wherein the one-dimensional tensor sparsely processes an input channel dimension of a neural network parameter.

[0217] Clause 31, the apparatus of any of clauses 24-28, wherein the mask tensor is a two-dimensional tensor.

[0218] Clause 32, the apparatus of clause 31, wherein the two-dimensional tensor sparsely processes an input channel dimension and an output channel dimension of a neural network parameter.

[0219] Clause 33, the apparatus of clause 28, wherein when the mask tensor is a two-dimensional tensor, the reverse mask tensor is generated by the operation circuitry performing a dimension conversion on the mask tensor.

[0220] Clause 34, the apparatus of clause 24, wherein the operation circuitry is further configured to:

[0221] updating the unsparsely processed neural network parameters.

[0222] Clause 35, the apparatus of clause 34, wherein the operation circuitry is further configured to:

[0223] generate the mask tensor based on updated non-sparse processing of the neural network parameters.

[0224] Clause 36, the apparatus according to clause 35, wherein when the mask tensor is a one-dimensional tensor, the operation circuit is configured to generate the mask tensor as follows:

[0225] select n data elements with larger absolute values from every m data elements of a specified dimension of the neural network parameters, where m > n; and

[0226] determine the mask tensor based on positions of the n effective data elements in the m data elements.

[0227] Clause 37, the apparatus according to clause 35, wherein when the mask tensor is a two-dimensional tensor, the operation circuit is configured to generate the mask tensor as follows:

[0228] preset a certain number of two-dimensional mask tensors, each dimension of the two-dimensional mask tensors including m elements, where n elements are 1 and m-n elements are 0, and m > n;

[0229] mask the specified two dimensions of the neural network parameters based on each preset two-dimensional mask tensor respectively to obtain a masked parameter tensor;

[0230] perform product-sum operations on the training data of the neural network based on each masked parameter tensor to obtain a parameter evaluation value; and

[0231] select the two-dimensional mask tensor that produces the largest parameter evaluation value among all parameter evaluation values as the mask tensor.

[0232] Clause 38, the apparatus according to any one of clauses 24-37, wherein the operation circuit is configured to perform the operations in multiple iterations in a mask adjustment phase of the sparse training.

[0233] Clause 39, the apparatus according to clause 38, wherein the operation circuit is further configured to, in the mask adjustment phase,

[0234] determine whether a percentage of all element values of the mask tensor that do not change in consecutive multiple iterations of training reaches a threshold value; and

[0235] if so, end the mask adjustment phase.

[0236] Clause 40, the apparatus according to clause 39, wherein the threshold value is one of 80%, 90%, and 100%.

[0237] Clause 41, The apparatus according to any one of clauses 24-33, wherein the operation circuitry is configured to perform the operations in multiple iterations in a mask fixing phase of the sparsification training, and the mask tensor is fixed to the mask tensor finalized in a previous phase.

[0238] Clause 42, The apparatus according to clause 41, wherein the operation circuitry is further configured to:

[0239] update the sparsified neural network parameters.

[0240] Clause 43, The apparatus according to clause 42, wherein the operation circuitry is further configured to:

[0241] sparsify the neuron gradient with the mask tensor; and

[0242] update the sparsified neural network parameters based on the sparsified neuron gradient.

[0243] Clause 44, The apparatus according to any one of clauses 41-43, wherein during the mask fixing phase, the storage circuitry is configured to store the fixed mask tensor and the sparsified neural network parameters.

[0244] Clause 45, The apparatus according to any one of clauses 41-43, wherein during the mask fixing phase, the storage circuitry is configured to store the fixed mask tensor and the unsparsified neural network parameters.

[0245] Clause 46, A chip comprising the data processing apparatus according to any one of clauses 24-45.

[0246] Clause 47, A board card comprising the chip according to clause 46.

[0247] The above has been described in detail for the embodiments of the present disclosure, and the principles and implementation manners of the present disclosure are described by applying specific examples. The above description of the embodiments is only to help understand the method of the present disclosure and its core idea; at the same time, for those skilled in the art, according to the idea of the present disclosure, the specific implementation manner and application range will be changed, and the above description should not be understood as a limitation of the present disclosure.

Claims

1. A method for sparse training of a neural network model applied to computer vision and speech scenarios, executed by a data processing device, wherein computer vision and speech are used as input data respectively, the method comprising: During forward propagation, the neural network parameters are sparsified based on the mask tensor to compute the value of the loss function; In backpropagation, the gradients of the neurons and the neural network parameters are calculated based on the loss function; and The neural network parameters are updated based on the gradient of the neural network parameters; Furthermore, in backpropagation, the gradients of the neurons and the neural network parameters are calculated based on the sparsed neural network parameters; and The neural network parameters are updated based on the gradients of the neurons. In backpropagation, the neural network parameters are sparsified based on the inverse mask tensor to obtain the sparsified neural network parameters. Updating the neural network parameters includes updating the unsparsed neural network parameters and generating the mask tensor based on the updated unsparsed neural network parameters; When the mask tensor is a two-dimensional tensor, the method generates the mask tensor as follows: A predetermined number of two-dimensional mask tensors are used, each dimension of which includes j elements, where k elements are 1, jk elements are 0, and j>k; Based on each preset two-dimensional mask tensor, the parameters of the neural network are masked in two specified dimensions to obtain the masked parameter tensor. The product and calculation to obtain the parameter evaluation value includes: multiplying the corresponding elements of the training data matrix and the masked parameter matrix and then adding the absolute values ​​to obtain the parameter evaluation value; or multiplying the corresponding elements of the training data matrix and the masked parameter matrix and then adding the absolute values ​​to obtain the parameter evaluation value. Based on the parameter tensor after each mask, the training data of the neural network are multiplied and summed to obtain parameter evaluation values; and The two-dimensional mask tensor that produces the largest value among all parameter evaluations is selected as the mask tensor.

2. The method according to claim 1, wherein the method further comprises: In backpropagation, the gradients of the neurons and the gradients of the neural network parameters are calculated based on the unsparsed neural network parameters. as well as The neural network parameters are updated based on the gradient of the neural network parameters.

3. The method according to claim 2, wherein the method further comprises: The sparsed neural network parameters are de-sparsed to obtain the unsparsed neural network parameters.

4. The method according to claim 1, wherein the mask tensor is a one-dimensional tensor.

5. The method according to claim 4, wherein the one-dimensional tensor sparsifies the input channel dimension of the neural network parameters.

6. The method according to claim 1, wherein the two-dimensional tensor sparsifies the input channel dimension and output channel dimension of the neural network parameters.

7. The method according to claim 1, wherein when the mask tensor is a two-dimensional tensor, the inverse mask tensor is generated by performing a dimensional transformation on the mask tensor.

8. The method of claim 1, wherein when the mask tensor is a one-dimensional tensor, the method generates the mask tensor as follows: From every m data elements in a specified dimension of the neural network parameters, select n data elements with larger absolute values ​​as valid data elements, where m > n; and The mask tensor is determined based on the positions of the n valid data elements among the m data elements.

9. The method according to any one of claims 1-8, wherein the method is performed in multiple iterations during the mask adjustment phase of sparse training.

10. The method of claim 9, wherein the mask adjustment stage further comprises: Determine whether the percentage of all element values ​​of the mask tensor that remain unchanged during multiple consecutive training iterations reaches a threshold. as well as If so, the mask adjustment phase ends.

11. The method of claim 10, wherein the threshold is one of 80%, 90%, and 100%.

12. The method according to any one of claims 1-8, wherein the method is performed in multiple iterations during the mask fixing phase of sparse training, and the mask tensor is fixed to the mask tensor finally determined in the previous phase.

13. The method of claim 12, wherein updating the neural network parameters includes updating the sparsed neural network parameters.

14. The method of claim 13, wherein updating the neural network parameters further comprises: The gradient of the neuron is sparsified using the mask tensor. as well as The parameters of the sparsed neural network are updated based on the sparsed neuron gradients.

15. The method of claim 14, wherein during the mask fixing phase, the fixed mask tensor and the sparsed neural network parameters are stored.

16. The method of claim 14, wherein during the mask fixing phase, the fixed mask tensor and the unsparsed neural network parameters are stored.

17. A computer-readable storage medium having stored thereon computer program code for sparse training of a neural network model, wherein when the computer program code is run by a processing device, it performs the method according to any one of claims 1 to 16.

18. A data processing apparatus, comprising a control circuit, a storage circuit, and an arithmetic circuit, wherein: The control circuit is configured to control the storage circuit and the arithmetic circuit to perform sparsity training on a neural network model applied to computer vision and speech scenarios, wherein computer vision and speech are respectively used as input data. The storage circuitry is configured to store information, which includes at least neural network parameters and a mask tensor; and The arithmetic circuit is configured to perform the following operations under the control of the control circuit: During forward propagation, the neural network parameters are sparsified based on the mask tensor to compute the value of the loss function; In backpropagation, the gradients of neurons and neural network parameters are calculated based on the loss function; and The neural network parameters are updated based on the gradient of the neural network parameters; The operational circuitry is further configured for: In backpropagation, the gradients of the neurons and the gradients of the neural network parameters are calculated based on the sparsified neural network parameters; and The neural network parameters are updated based on the gradients of the neurons. The operational circuitry is further configured for: In backpropagation, the neural network parameters are sparsified based on the inverse mask tensor to obtain the sparsified neural network parameters. The operational circuitry is further configured for: Update the parameters of the unsparsed neural network; The mask tensor is generated based on the updated, unsparsed neural network parameters; When the mask tensor is a two-dimensional tensor, the computation circuit is configured to generate the mask tensor as follows: A predetermined number of two-dimensional mask tensors are used, each dimension of which includes j elements, where k elements are 1, jk elements are 0, and j>k; Based on each preset two-dimensional mask tensor, the parameters of the neural network are masked in two specified dimensions to obtain the masked parameter tensor. The product and calculation to obtain the parameter evaluation value includes: multiplying the corresponding elements of the training data matrix and the masked parameter matrix and then adding the absolute values ​​to obtain the parameter evaluation value; or multiplying the corresponding elements of the training data matrix and the masked parameter matrix and then adding the absolute values ​​to obtain the parameter evaluation value. Based on the parameter tensor after each mask, the training data of the neural network are multiplied and summed to obtain parameter evaluation values; and The two-dimensional mask tensor that produces the largest value among all parameter evaluations is selected as the mask tensor.

19. The apparatus of claim 18, wherein the operational circuitry is further configured to: In backpropagation, the gradients of the neurons and the gradients of the neural network parameters are calculated based on the unsparsed neural network parameters; and The neural network parameters are updated based on the gradient of the neural network parameters.

20. The apparatus of claim 19, wherein the operational circuitry is further configured to: The sparsed neural network parameters are de-sparsed to obtain the unsparsed neural network parameters.

21. The apparatus of claim 20, wherein the mask tensor is a one-dimensional tensor.

22. The apparatus of claim 21, wherein the one-dimensional tensor sparsifies the input channel dimension of the neural network parameters.

23. The apparatus of claim 18, wherein the two-dimensional tensor sparsifies the input channel dimension and output channel dimension of the neural network parameters.

24. The apparatus of claim 18, wherein when the mask tensor is a two-dimensional tensor, the inverse mask tensor is generated by the computation circuit performing a dimensionality transformation on the mask tensor.

25. The apparatus of claim 18, wherein when the mask tensor is a one-dimensional tensor, the computation circuitry is configured to generate the mask tensor as follows: From every m data elements in a specified dimension of the neural network parameters, select n data elements with larger absolute values ​​as valid data elements, where m > n; and The mask tensor is determined based on the positions of the n valid data elements among the m data elements.

26. The apparatus of any one of claims 18-25, wherein the computational circuitry is configured to perform the operation in multiple iterations during the mask adjustment phase of sparse training.

27. The apparatus of claim 26, wherein the arithmetic circuitry is further configured to: during the mask adjustment phase, Determine whether the percentage of all element values ​​of the mask tensor that remain unchanged during multiple consecutive training iterations reaches a threshold; and If so, the mask adjustment phase ends.

28. The apparatus of claim 27, wherein the threshold is one of 80%, 90%, and 100%.

29. The apparatus of any one of claims 18-24, wherein the computational circuitry is configured to perform the operation in multiple iterations during a mask fixing phase of sparse training, and the mask tensor is fixed to the mask tensor finally determined in the previous phase.

30. The apparatus of claim 29, wherein the operational circuitry is further configured to: Update the parameters of the sparsed neural network.

31. The apparatus of claim 30, wherein the operational circuitry is further configured to: The gradient of the neuron is sparsified using the mask tensor; and The parameters of the sparsed neural network are updated based on the sparsed neuron gradients.

32. The apparatus of claim 31, wherein during the mask fixing phase, the storage circuit is configured to store the fixed mask tensor and the sparsed neural network parameters.

33. The apparatus of claim 31, wherein during the mask fixing phase, the storage circuit is configured to store the fixed mask tensor and the unsparsed neural network parameters.

34. A chip comprising a data processing apparatus according to any one of claims 18-33.

35. A circuit board comprising the chip according to claim 34.

Images