Computational kernels used in neural networks

Abstract

The present disclosure provides a computing core for a neural network. The computing core includes: a tensor engine, wherein the tensor engine calculates the reverse propagation excitation of another intermediate layer in a plurality of intermediate layers of the neural network using a first kernel function, and calculates the reverse propagation weight of the intermediate layer using a second kernel function, in the reverse direction of the neural network, wherein the other intermediate layer is one or more layers after the intermediate layer.

Description

Technical Field

[0001] The present disclosure generally relates to the field of neural network computing, and more specifically, to a computing kernel for use in a neural network. Background Art

[0002] Currently, neural networks have been widely used in fields such as image classification and object recognition. In these fields, neural networks can be trained using pre-acquired image samples to obtain corresponding trained neural network models. The trained neural network models can be used to identify or classify new image data.

[0003] The training of neural networks is a complex process. Small changes in the front layers of the network will be accumulated and amplified to the back layers, so the update of the training parameters of the front layers will lead to changes in the distribution of input data in the back layers. For this reason, the concept of batch normalization (BN) is introduced in neural networks. Before processing the input data of each layer, the input data is first batch normalized to forcibly transform the features of the input data to a mathematical model with a mean of 0 and a variance of 1.

[0004] The sizes of different layers of a neural network may be different. Calculating the back-propagation excitation and back-propagation weight of the same layer may cause unstable computational load and unbalanced demand for storage space.

[0005] In addition, always storing the output data of each sub-layer in the forward direction not only takes up more storage space, but also may increase the time overhead of reading these output data in the reverse direction. Summary of the invention

[0006] In response to the above problems, the present disclosure provides a computing core for use in a neural network, which can balance the computational load and the demand for storage space by alternately performing the calculation of back-propagation excitation and back-propagation weight.

[0007] According to one aspect of the present disclosure, a computing core for use in a neural network is provided. The computing core includes: a tensor engine, the tensor engine, in a reverse direction of the neural network, in one of a plurality of intermediate layers of the neural network, uses a first kernel function to calculate the back propagation excitation of another intermediate layer, and uses a second kernel function to calculate the back propagation weight of the intermediate layer, wherein the other intermediate layer is one or more layers after the intermediate layer.

[0008] In some embodiments, the first kernel function is different from the second kernel function.

[0009] In some embodiments, the computation of the tensor engine is a tensor or matrix operation, and the computing core further comprises: a vector engine that performs non-tensor and matrix operations in parallel with the tensor or matrix operations of the tensor engine.

[0010] In some embodiments, the vector engine updates parameters of the activation sublayer and the BN sublayer of the intermediate layer based on the back-propagation excitation in parallel with the calculation of the back-propagation weights.

[0011] In some embodiments, in the reverse convolution sublayer of the intermediate layer, the tensor engine calculates the back-propagation excitation of the other intermediate layer based on the weight value and the back-propagation excitation of the intermediate layer using the first kernel function, and calculates the back-propagation weight of the intermediate layer based on the excitation generated by the intermediate layer and the back-propagation weight of the previous intermediate layer of the intermediate layer using the second kernel function.

[0012] In some embodiments, the tensor engine further stores the back-propagation excitation of the intermediate layer in a register.

[0013] In some embodiments, the computing core also includes: an on-chip cache, the on-chip cache including a first cache for caching data required for calculating the back-propagation excitation, and a second cache for caching data required for calculating the back-propagation weights, and the tensor engine reads data from the first cache and the second cache in an interleaved manner.

[0014] In some embodiments, the computing core is configured to: in the forward direction of the neural network, determine whether to store the output data of an intermediate sublayer of the intermediate layer, based on the cost of loading the input data of the intermediate sublayer, the cost of calculating the output data using the input data of the intermediate sublayer, and the cost of loading the output data of the intermediate sublayer; if it is determined to store the output data of the intermediate sublayer, after obtaining the output data of the intermediate sublayer, cache the output data of the intermediate sublayer to the on-chip cache of the computing core and / or store it in an off-chip memory; and if it is determined not to store the output data of the intermediate sublayer, after obtaining the output data of the intermediate sublayer, cache the output data of the intermediate sublayer to the on-chip cache or register of the computing core for immediate use by the next intermediate layer or the next intermediate sublayer.

[0015] In some embodiments, the computing core further stores output data of another intermediate sublayer preceding the intermediate sublayer.

[0016] In some embodiments, in the reverse direction of the neural network, the computing core calculates the output data of the intermediate sub-layer using the stored output data of the other intermediate sub-layer.

[0017] In some embodiments, the intermediate sublayer is an activation sublayer of the intermediate layer, and the another intermediate sublayer is a convolutional sublayer or a BN sublayer of the intermediate layer.

[0018] In some embodiments, when the other intermediate sublayer is a convolution sublayer of the intermediate layer, in the reverse direction, the computing core calculates the output data of the reverse BN sublayer using the output data of the convolution sublayer to further calculate the gradient values ​​of the parameters of the BN sublayer of the intermediate layer.

[0019] In some embodiments, when the other intermediate sublayer is the BN sublayer of the intermediate layer, in the reverse direction, the computing core performs a BN inverse operation on the output data of the BN sublayer of the intermediate layer, and calculates the output data of the reverse BN sublayer based on the result of the BN inverse operation to further calculate the gradient value of the parameters of the BN sublayer of the intermediate layer.

[0020] In some embodiments, the computing core is configured to determine whether to store the output data of the intermediate sublayer based on whether the sum of the cost of loading the input data of the intermediate sublayer and the cost of calculating the output data of the intermediate sublayer is less than the cost of loading the output data of the intermediate sublayer.

[0021] In some embodiments, the cost of loading input data of the intermediate sublayer depends on the data volume of the input data and the storage location of the input data; the cost of loading output data of the intermediate sublayer depends on the data volume of the output data and the storage location of the output data; and the cost of calculating the output data depends on the total number of operations of the intermediate sublayer and the hardware data channel width of the computing core. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The present disclosure will be better understood and other objects, details, features and advantages of the present disclosure will become more apparent through the description of specific embodiments of the present disclosure given with reference to the following drawings.

[0023] Figure 1 A schematic diagram of the structure of a neural network according to the present disclosure is shown.

[0024] Figure 2 A schematic hardware structure diagram of a computing core according to an embodiment of the present invention is shown.

[0025] Figure 3 A schematic diagram of a back-propagation process according to the prior art is shown.

[0026] Figure 4 A schematic diagram of a back-propagation process according to an embodiment of the present invention is shown.

[0027] Figure 5A flow chart of a recalculation method performed in a neural network according to the present invention is shown.

[0028] Figure 6 A schematic diagram showing the data flow for storing the output data of a convolutional sublayer in the forward direction.

[0029] Figure 7 A schematic diagram showing the data flow of storing the output data of the BN sublayer in the forward direction.

[0030] Figure 8 A schematic diagram of a computing device according to the present invention is shown. DETAILED DESCRIPTION

[0031] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0032] As used herein, the term "including" and its variations mean open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "based at least in part on". The terms "one embodiment" and "some embodiments" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc. may refer to different or the same objects.

[0033] Figure 1 FIG. 1 shows a schematic diagram of the structure of a neural network 100 according to the present disclosure. Figure 1 As shown in FIG. 1 , the neural network 100 may include an input layer 110, a plurality of intermediate layers 120 ( Figure 1Schematically shows M intermediate layers 120-1, 120-2, 120-3, ... 120-M, where M is an integer greater than 2) and an output layer 130 implemented as a fully connected layer. Each intermediate layer 120 includes a convolution sublayer 122, a batch normalization (BN) sublayer 124, and an activation sublayer 126. In the forward direction, the convolution sublayer 122 is used to perform convolution processing on the input data, the BN sublayer 124 is used to batch normalize the input data, and the activation sublayer 126 is used to apply an activation function (such as a Relu function) to the normalized input data to generate activations. That is, in the forward direction, each intermediate layer 120 can process the input data to generate the activation of the intermediate layer as an output. In which, when the intermediate layer 120 is the first intermediate layer of the neural network 100, the input data of the intermediate layer 120 can be the training data input to the neural network 100; when the intermediate layer 120 is not the first intermediate layer of the neural network 100, the input data of the intermediate layer 120 is the excitation generated by the previous intermediate layer 120.

[0034] The input layer 110, the plurality of intermediate layers 120 and the output layer 130 of the neural network 100 may be implemented on a computing core, such as Figure 2 The computing core 200 shown in FIG. Figure 2 FIG. 2 shows a schematic hardware structure diagram of a computing core 200 according to an embodiment of the present invention. Figure 2 As shown in , the computing core 200 may include a tensor engine 210 and a vector engine 220. The tensor engine 210 is used to perform tensor calculations in the neural network 100 in the form of tensor input and tensor output, such as convolution operations of each convolution sublayer 122, and the vector engine 220 is used to perform vector operations in the neural network 100, such as batch normalization operations of each BN sublayer 124 and activation operations of each activation sublayer 126. In addition, the computing core 200 may also include one or more on-chip caches 230 for caching data generated by each layer of the neural network 100. The computing core 200 may also include registers (not shown in the figure) for storing output data of some intermediate sublayers or some back-propagation excitations, as described in detail below. It can be understood by those skilled in the art that, although not shown in the figure, the computing core 200 may also include other computing or control units, which can be used to perform judgments such as excitation recalculation described below, which will not be repeated here.

[0035] In addition, the neural network 100 may also utilize or include one or more off-chip memories 140. The on-chip cache 230 is only used to cache recently generated data. When new data is generated, the data that has not been read in the on-chip cache 230 will be transferred to the off-chip memory 140. Note that although Figure 1Only one off-chip memory 140 is shown in the figure, but in many implementations, the neural network 100 may include multiple off-chip memories 140, and these off-chip memories 140 may be located at different distances from the computing core 200 (or the tensor engine 210 and / or the vector engine 220) and / or use different memory technologies, thereby having different access costs (such as bandwidth and / or time).

[0036] During the training process of the neural network 100, the training data is processed in the forward direction through each layer of the neural network 100 to generate output data. From the perspective of the entire model, it is equivalent to

[0037] X out =W*X in +b

[0038] Among them, X in is the input training data, X out is the output data generated, W is the weight function of the neural network 100, b is the bias function of the neural network 100, and the purpose of training the neural network 100 is to continuously update its weight function W and bias function b to a convergence value. Here, the initial value of the weight function W can be set arbitrarily, or can be set based on experience.

[0039] In the output layer 130, various loss functions (such as the cross entropy loss function) can be used to calculate the loss value Loss of the output data and calculate its gradient value

[0040] For example, when using the cross entropy loss function, the loss value Loss can be expressed as:

[0041]

[0042] in Represents output data X out The probability that the sample label of each dimension j is 1, S j Represents output data X out The score of each dimension j of .

[0043] In this case, W M represents the weight function of the last intermediate layer 120 (Mth intermediate layer 120-M) of the neural network 100, Represents the weight gradient value of the intermediate layer.

[0044] Next, the weight function W of each intermediate layer 120 of the neural network 100 is adjusted by the back propagation algorithm. m(m=1, 2, ... M-1) are updated respectively, where M is the number of the middle layers 120 of the neural network 100. Specifically, the parameters of each sub-layer of the middle layers 120-(M-1), 120-(M-2), ... 120-1 are updated in sequence.

[0045] Back propagation includes back propagation of the gradient value of the excitation and back propagation of the gradient value of the weight. In this article, the gradient value of the excitation is also referred to as the back propagation excitation BPA, and the gradient value of the weight is referred to as the back propagation weight BPW.

[0046] Figure 3 FIG. 4 is a schematic diagram showing a back propagation process according to the prior art. Figure 1 Corresponding to the intermediate layer 120 in the reverse direction of the neural network 100, each intermediate layer 320 may include a reverse activation sublayer 326, a reverse BN sublayer 324, and a reverse convolution sublayer 322, wherein the reverse activation sublayer 326, the reverse BN sublayer 324, and the reverse convolution sublayer 322 respectively perform Figure 1 The reverse operation of the activation sublayer 126, BN sublayer 124 and convolution sublayer 122 is performed to update the parameters of the activation sublayer 126, BN sublayer 124 and convolution sublayer 122. Note that Figure 1 Different from the above, for the convenience of description, the reverse convolution sublayer 322, the reverse activation sublayer 326 and the reverse BN sublayer 324 are divided into an intermediate layer 320 in the order, and the ordering rule of the intermediate layer 320 is opposite to that of the intermediate layer 120 (that is, the former is numbered in the reverse direction and the latter is numbered in the forward direction). In addition, it should be noted that the reverse activation sublayer 326, the reverse BN sublayer 324 and the reverse convolution sublayer 322 of the intermediate layer 320 here are the same structure as the activation sublayer 126, the BN sublayer 124 and the convolution sublayer 122 in hardware, respectively, and the difference in name is only for the convenience of describing their respective functions.

[0047] like Figure 3 As shown in FIG. 1 , in the prior art, for each inverse convolution sublayer 322 of the intermediate layer 320 - m, the weight value W of the intermediate layer 320 - m may be first calculated based on the weight value W of the intermediate layer 320 - m. mThe back propagation excitation bpa_(m-1) of the current intermediate layer 320-m is calculated based on the back propagation excitation bpa_(m-1) of the previous intermediate layer 320-(m-1), and then the back propagation weight bpw_m of the current intermediate layer 320-m is calculated based on the excitation Acv_m generated by the current intermediate layer 320-m and the back propagation weight bpw_(m-1) of the previous intermediate layer 320-(m-1). The back propagation excitation bpa_m and the back propagation weight bpw_m of the intermediate layer 320-m are both calculated by the tensor engine 210 in the current intermediate layer 320-m. The back propagation excitation bpa_m is further passed to the reverse activation sublayer 326 and the reverse BN sublayer 324 of the current intermediate layer 320-m to further determine the gradient value of the parameter of the activation sublayer 126 and the gradient value of the parameter of the BN sublayer 124, wherein the operations of the reverse activation sublayer 326 and the reverse BN sublayer 324 are performed by the vector engine 220.

[0048] However, for each intermediate layer, the feature map size is different, and the back propagation excitation bpa_m and the back propagation weight bpw_m generated by the intermediate layer 320-m are calculated in the same intermediate layer 320-m, which will cause the computational load to be larger or smaller at the same time due to the same feature map size, thereby causing the computational load of the tensor engine 210 to be unstable. At the same time, due to the same reason, the data volume of the back propagation excitation bpa_m and the back propagation weight bpw_m generated by the intermediate layer 320-m will be larger or smaller at the same time, thereby making the storage space requirement for storing these data unbalanced.

[0049] In view of the above problems, in the solution according to the present disclosure, the back propagation excitation BPA and the back propagation weight BPW are executed in staggered layers to avoid excessive or too small computational load on the tensor engine 210 and balance the demand for storage space. Specifically, since the back propagation excitation BPA always needs to be calculated before the back propagation weight BPW, the back propagation excitation BPA can be configured to be calculated one or more layers ahead of the back propagation weight BPW.

[0050] Figure 4 FIG. 2 shows a schematic diagram of a back propagation process according to an embodiment of the present invention. Figure 3 Similarly, in the reverse direction of the neural network 100, each intermediate layer 320 may include a reverse activation sublayer 326, a reverse BN sublayer 324, and a reverse convolution sublayer 322, which respectively perform Figure 1 The activation sublayer 126, the BN sublayer 124, and the convolution sublayer 122 are reversed to update the parameters of the activation sublayer 126, the BN sublayer 124, and the convolution sublayer 122.

[0051] In an embodiment according to the present invention, in an intermediate layer 320, a kernel function may be used to calculate the back propagation weight BPW of the intermediate layer 320 and another kernel function may be used to calculate the back propagation activation BPA of another intermediate layer 320, wherein the other intermediate layer 320 is one or more layers after the intermediate layer 320 (in the reverse direction). For example, Figure 4 As shown in , assuming that the current intermediate layer 320 is the intermediate layer 320-m, the first kernel function can be used in the intermediate layer 320-m to calculate the back propagation weight bpw_m of the intermediate layer 320-m, and the second kernel function can be used to calculate the back propagation excitation bpa_(m+n) of the intermediate layer 320-(m+n), where n is an integer greater than or equal to 1.

[0052] For example, suppose n = 1. Figure 4 As shown in FIG. 1 , in the reverse convolution sublayer 322 of the current intermediate layer 320 - m, the first kernel function can be used based on the weight value W of the intermediate layer 320 - m. m The back propagation excitation bpa_(m+1) of the next intermediate layer 320-(m+1) is calculated by the back propagation excitation bpa_(m+1) and the back propagation excitation bpa_m. Then, the back propagation weight bpw_m of the current intermediate layer 320-m can be calculated based on the excitation Acv_m generated by the current intermediate layer 320-m and the back propagation weight bpw_(m-1) of the previous intermediate layer 320-(m-1) by using the second kernel function. The back propagation excitation bpa_(m+1) and the back propagation weight bpw_m are both calculated by the tensor engine 210 in the current intermediate layer 320-m. The back propagation excitation bpa_(m+1) is further passed to the reverse activation sublayer 326 and the reverse BN sublayer 324 of the current intermediate layer 320-m to update the parameters of the activation sublayer 126 and the BN sublayer 124, wherein the operations of the reverse activation sublayer 326 and the reverse BN sublayer 324 are performed by the vector engine 220.

[0053] Here, the first kernel function and the second kernel function are different. For example, the first kernel function is a 3*3 kernel function, and the second kernel function is a 1*1 kernel function.

[0054] Since the back propagation excitation BPA calculated in the current intermediate layer 320 is the back propagation excitation of a certain intermediate layer after the current intermediate layer 320, and the back propagation excitation BPA and the back propagation weight BPW calculated in the current intermediate layer 320 use kernel functions of different sizes, the computational loads of the two will not be superimposed, increased or decreased at the same time in the current intermediate layer 320, thereby avoiding the situation where the computational load of the tensor engine 210 is unstable. In addition, in different intermediate layers 320, the amount of input data required to calculate the back propagation excitation BPA and the amount of output data generated are usually different, so calculating the back propagation weight BPW of the current intermediate layer and the back propagation excitation BPA of the subsequent intermediate layer in the current intermediate layer can also balance the demand for storage space.

[0055] Further, in some embodiments, in order to better utilize all computing resources of the computing core 200, the back propagation excitation bpa_(m+1) obtained in the middle layer 320-m is stored in the register of the computing core 200, and while the tensor engine 210 calculates the back propagation weight bpw_m, the vector engine 220 can perform the operations of the reverse activation sublayer 326 and the reverse BN sublayer 324. In this way, the vector thread warp executed by the vector engine 220 can be executed in parallel with the tensor thread warp for calculating the back propagation excitation BPA by the tensor engine 210, thereby improving the computing resource utilization of the entire computing core 200.

[0056] In addition, in the neural network 100 implemented using the pipeline model, each intermediate layer 120 can be divided into multiple stages, and each stage can be mapped to multiple devices. In order to improve the utilization of device resources, after the batch data is divided into small batch data (mini-batch), the small batch data can be further divided into micro-batch data (micro-batch) so that each device can process different micro-batch data at the same time, and the processing results of different micro-batch data are added as the output data of the intermediate layer or the stage.

[0057] In this case, it is also possible to consider dividing the on-chip cache 230 to cache the back-propagation excitation BPA and the back-propagation weight BPW respectively. Figure 2 As shown in , a first cache 232 can be divided from the on-chip cache 230 to cache the data required for calculating the back-propagation excitation BPA, and a second cache 234 can be divided from the on-chip cache 230 to cache the data required for calculating the back-propagation weight BPW. Here, the first cache 232 and the second cache 234 can be obtained by dividing the same physical on-chip cache 230 or a part thereof by software, and the sizes of the first cache 232 and the second cache 234 can also be dynamically adjusted by software.

[0058] For example, as shown in Table 1 below, it is assumed that the first cache 232 and the second cache 234 (eg, 4 data blocks each) of the same size are allocated for back propagation excitation BPA and back propagation weight BPW, respectively.

[0059] Table 1

[0060] 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 bpa bpa bpa bpa bpw bpw bpw bpw bpa bpa bpa bpa bpw bpw bpw bpw

[0061] In this way, in combination Figure 4 In the calculation process of the back propagation excitation BPA and the back propagation weight BPW described, the back propagation excitation BPA and the back propagation weight BPW can be further calculated by first reading the data required for caching the back propagation excitation BPA from the first cache 232 to calculate the back propagation excitation BPA, and then reading the data required for caching the back propagation weight BPW from the second cache 234 to calculate the back propagation weight BPW. In other words, the calculation of the back propagation excitation BPA and the back propagation weight BPW is not only parallel in time, but also interleaved in storage space, so as to maximize the calculation load balancing of the tensor engine 210.

[0062] As mentioned above, in the middle layer 320-m, the back propagation weight bpw_m of the current middle layer 320-m needs to be calculated based on the activation Acv_m generated by the current middle layer 320-m and the back propagation weight bpw_(m-1) of the previous middle layer 320-(m-1). That is, when calculating the back propagation weight BPW in the reverse direction, the tensor engine 210 needs to use the activations (or Acv for short) generated by the middle layer 120 calculated in the forward direction.

[0063] In the prior art, the excitation generated by the middle layer 120 is first cached on the on-chip cache 230, and if it has not been read when new data is generated, it will be further transferred to the off-chip memory 140. Therefore, in the reverse process, the tensor engine 210 needs to read the excitation from the on-chip cache 230 or the off-chip memory 140 to calculate the back propagation weight BPW. However, in some cases, such as when the amount of data of the generated excitation is very large and / or the excitation is stored in the off-chip memory 140 (such as DDR) far away from the tensor engine 210, the overhead of the tensor engine 210 reading the excitation in the reverse process may be very large, even greater than the overhead of recalculating the excitation.

[0064] In addition, in a neural network 100 with branches (such as ResNet), there is also the possibility of storing only the output data of one branch without storing the output data of another branch in the forward direction, and inferring the output data of another branch based on the total output data and the output data of one branch in the reverse direction.

[0065] To this end, in another aspect of the present invention, in the forward direction, at an intermediate layer 120, the computing core 200 may determine whether to store the output data of an intermediate sub-layer of the intermediate layer 120 to determine whether to perform recomputation at the intermediate sub-layer in the reverse direction. For example, the intermediate sub-layer may be an activation sub-layer 126 that generates an activation of the current intermediate layer 120 as output data. Figure 5 A flowchart of a recomputation method 500 executed in the neural network 100 according to the present invention is shown.

[0066] As Figure 5 shown, at block 510, in the forward direction, the computing core 200 may determine whether to store the output data of the intermediate sub-layer based on the cost Load_input of loading the input data of the intermediate sub-layer, the cost R of computing the output data using the input data of the intermediate sub-layer, and the cost Load_output of loading the output data of the intermediate sub-layer.

[0067] If it is determined to store the output data of the intermediate sub-layer (the determination at block 510 is "yes"), then at block 520, the computing core 200 may cache the output data in the on-chip cache 230 and / or store it in the off-chip memory 140 after obtaining the output data of the intermediate sub-layer. For example, it may be determined whether the intermediate sub-layer is an activation sub-layer. In the case where the intermediate sub-layer is the activation sub-layer 126, the output data (activation) is both cached in the on-chip cache 230 for use by the next intermediate layer and stored in the off-chip memory for use in the reverse direction. Depending on the hardware capabilities of the computing core 200 (such as whether it supports direct writing to the off-chip memory), the output data of the intermediate layer may be stored directly or via the on-chip cache 230 in the off-chip memory 140.

[0068] On the other hand, if it is determined not to store the output data of the intermediate sub-layer (the determination at block 510 is "no"), then at block 540, the computing core 200 may cache the output data in the on-chip cache 230 or a register for immediate use by the next intermediate layer or the next intermediate sub-layer.

[0069] Specifically, at block 510, the computing core 200 may determine whether to store the output data of the intermediate sub-layer based on whether the sum of the cost Load_input of loading the input data of the intermediate sub-layer and the cost R of computing the output data is less than the cost Load_output of loading the output data of the intermediate sub-layer (i.e., whether Load_input + R < Load_output holds).

[0070] Here, the cost Load_input of loading the input data of the intermediate sublayer depends on the data volume and storage location of the input data; the cost Load_output of loading the output data of the intermediate sublayer depends on the data volume and storage location of the output data; the cost R of calculating the output data depends on the total number of operations of the intermediate sublayer and the hardware data channel width of the computing core 200.

[0071] For example, if the amount of input data of an intermediate sublayer is large and is located at a storage location far away from the computing core 200 (such as located at an off-chip memory 140 far away from the computing core 200), the total number of operations of the intermediate sublayer is also large, and the amount of output data is small and is located at a storage location close to the computing core 200 (such as located at an on-chip cache 230 or an off-chip memory 140 near the computing core 200), then it can be determined that the output data of the intermediate sublayer needs to be stored. Conversely, if the amount of input data of an intermediate sublayer is small and is located at a storage location close to the computing core 200 (such as located at an on-chip cache 230 or an off-chip memory 140 near the computing core 200), the total number of operations of the intermediate sublayer is also small, and the amount of output data is large and is located at a storage location far away from the computing core 200 (such as located at an off-chip memory 140 far away from the computing core 200), then it can be determined that the output data of the intermediate sublayer does not need to be stored.

[0072] In some embodiments, it can be determined quantitatively by calculation whether the output data of the intermediate sub-layer needs to be stored. For example, the cost Load_input of loading the input data of the intermediate sub-layer can be expressed as:

[0073] Load_input:=load_input_traffic / bandwidth_input

[0074] Among them, load_input_traffic represents the data volume of input data, and bandwidth_input represents the bandwidth from the storage location of the input data to the computing core 200.

[0075] The cost of loading the output data of the intermediate sublayer Load_output can be expressed as:

[0076] Load_output:=load_output_traffic / bandwidth_output

[0077] Among them, load_output_traffic represents the data volume of the output data, and bandwidth_output represents the bandwidth from the storage location of the output data to the computing core 200.

[0078] The cost R of using input data to calculate output data in the middle sub-layer can be expressed as:

[0079] R:=total_ops / hardware_datapath_width

[0080] Among them, total_ops represents the total number of operations of the intermediate sub-layer, and hardware_datapath_width represents the hardware data path width of the computing core 200.

[0081] In other embodiments, a lookup table may be set for each intermediate layer 120 based on the structure of the entire neural network 100. This is because, when the structure of the entire neural network 100 is determined, the size of each intermediate sublayer of each intermediate layer 120 is determined, and the position of the intermediate layer 120 in the entire neural network 100 is also determined, so it is easy to determine the relative size and relative storage position of the input data and output data of each intermediate sublayer in the intermediate layer 120 and the estimated value of the cost R of calculating the output data, so that the lookup table can be pre-compiled to indicate whether the output data needs to be stored.

[0082] In this way, in the forward direction, the computing core 200 can determine whether it is necessary to cache or store the output data of the intermediate sub-layer to minimize the time overhead in the reverse direction. Figure 4 The reverse propagation process according to the embodiment of the present invention is also applicable to other reverse propagation processes in the prior art, such as the reverse propagation process described above. Figure 3 The back propagation process described.

[0083] In method 500, at block 540, after the tensor engine 210 obtains the output data of the intermediate sub-layer, the computing core 200 may cache the output data to the on-chip cache 230 or register for immediate use by the next intermediate layer or the next intermediate sub-layer. In order to recalculate the output data of the intermediate sub-layer 122 in the reverse process, before block 540, the computing core 200 may store the output data of another intermediate sub-layer before the intermediate sub-layer, such as Figure 5 As shown in block 530. The output data of the other intermediate sub-layer may be stored in the on-chip cache 230 or the register of the computing core 200, for example, to be used for directly or indirectly calculating the output data of the intermediate sub-layer in the reverse direction.

[0084] Thereafter, in the reverse direction, the computing core 200 may calculate the output data of the intermediate sublayer using the output data of another intermediate sublayer before the intermediate sublayer stored in block 530 .

[0085] For example, in the reverse convolution sublayer 322, the tensor engine 210 may recalculate the excitation of the convolution sublayer 122 using the output data of another intermediate sublayer before the reverse convolution sublayer 322 stored in block 530, for further use in calculating the back-propagation weight BPW.

[0086] Here, the intermediate sublayer may be the activation sublayer 126 of the intermediate layer 120 , and thus another intermediate sublayer may be the convolution sublayer 122 or the BN sublayer 124 of the intermediate layer 120 . Figure 6 A schematic diagram showing the data flow for storing the output data of the convolutional sublayer 122 in the forward direction. Figure 7 A schematic diagram showing the data flow of storing output data of the BN sublayer 124 in the forward direction is shown.

[0087] like Figure 6 As shown in , in the forward direction, the output data of the convolution sublayer 122 is stored. The output data is used to calculate the output data of the reverse BN sublayer 324 in the reverse direction to further calculate the gradient value Relu_grad of the parameter of the BN sublayer 124.

[0088] like Figure 7 As shown in , in the forward direction, the output data of the BN sublayer 124 is stored. The output data is used to calculate the output data of the reverse BN sublayer 324 in the reverse direction to further calculate the gradient value Relu_grad of the parameter of the BN sublayer 124.

[0089] Note that with Figure 6 Different in Figure 7 In the data flow shown, in the reverse BN sublayer 324, it is first necessary to perform a BN inverse operation on the stored output data of the BN sublayer 124, and then calculate the gradient value Relu_grad of the parameter of the BN sublayer 124 based on the result of the BN inverse operation.

[0090] By determining in advance whether the output data of the intermediate sub-layer needs to be stored in the forward direction, not only can the storage space required to store the output data, such as the control of the video memory, be saved, but also in the reverse direction, when the tensor engine 210 or the vector engine 220 performs calculations on the intermediate sub-layer, the additional load caused by the driving data can be reduced.

[0091] In some embodiments, in combination with the above Figures 1 to 7 The described neural network 100 may be implemented as a card or other electronic device. Figure 8A schematic diagram of a computing device 800 according to the present invention is shown. The computing device 800 may include a computing core 200 and supporting components that execute the above-mentioned back-propagation process and / or recalculation method 500. The computing core 200 may be implemented as a single chip, for example. The supporting components may include one or more storage components 810, an interface component 820, and a control component 830.

[0092] Each storage component 810, for example, Figure 1 The off-chip memory 140 can be connected to the computing core 200 via a bus (such as a high-speed bus) 840 to store data transferred from the on-chip cache 230. Here, the storage component 810 can be a DDR SDRAM (Double Data Rate SDRAM). DDR SDRAM can double the access speed of SDRAM without increasing the clock frequency. Each storage component 810 can also include a plurality of DDR particles, such as DDR4 particles. The interface component 820 is connected to the computing core 200 to realize data transmission between the computing core 200 and other devices outside the computing device 800, such as a server or a host. The interface component 820 can be implemented, for example, using a standardized PCIE interface or a high-speed PCIE interface. For example, the interface component 820 can receive an operation instruction or data from an external host and transmit it to the computing core 200. Alternatively, the interface component 820 can transmit data in the storage component 810 to an external host.

[0093] The control component 830 may be connected to the computing core 200 to monitor the status of the computing core 200. Specifically, the control component 830 may be connected to the computing core 200 via a serial peripheral interface (SPI) to monitor the load of each computing unit in the computing core 200.

[0094] In addition, in some embodiments, the above-described back propagation process and / or recalculation method 500 may be implemented as a computer software program, which is tangibly included in a machine-readable medium. In some embodiments, part or all of the computer program may be loaded and / or installed on the computing core 200 to perform one or more operations of the back propagation process and / or recalculation method 500 described above.

[0095] Those skilled in the art will understand that Figure 8 The computing device 800 shown is exemplary only. In some embodiments, the computing device 800 may include more or fewer components.

[0096] The back propagation process and / or recalculation method 500 in the neural network 100 according to the present disclosure and the computing device 800 that can execute the method are described above in conjunction with the accompanying drawings. However, those skilled in the art will appreciate that the execution of the back propagation process and / or recalculation method 500 is not limited to the order shown in the figure and described above, but can be executed in any other reasonable order. In addition, the computing device 800 does not necessarily include Figure 8 All components shown in the figure may include only some of the components or more components necessary to perform the functions described in the present disclosure, and the connection method of these components is not limited to the form shown in the figure.

[0097] The present disclosure may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.

[0098] In one or more exemplary designs, the functions described in the present disclosure may be implemented using hardware, software, firmware, or any combination thereof. For example, if implemented using software, the functions may be stored as one or more instructions or codes on a computer-readable medium, or transmitted as one or more instructions or codes on a computer-readable medium.

[0099] The various units of the apparatus disclosed herein may be implemented using discrete hardware components or may be integrated on a hardware component, such as a processor. For example, the various exemplary logic blocks, modules, and circuits described in conjunction with the present disclosure may be implemented or executed using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof for performing the functions described herein.

[0100] Those skilled in the art should also understand that the various illustrative logical blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments of the present disclosure may be implemented as electronic hardware, computer software, or a combination of the two.

[0101] The above description of the present disclosure is intended to enable any person of ordinary skill in the art to implement or use the present disclosure. Various modifications of the present disclosure are obvious to those of ordinary skill in the art, and the general principles defined herein may also be applied to other variations without departing from the spirit and scope of the present disclosure. Therefore, the present disclosure is not limited to the examples and designs described herein, but is consistent with the broadest scope of the principles and novel features disclosed herein.

Claims

1. A computing kernel for use in a neural network, comprising: A tensor engine, wherein the tensor engine calculates, in a reverse direction of the neural network, a back-propagation excitation of another intermediate layer in a plurality of intermediate layers of the neural network using a first kernel function, and calculates a back-propagation weight of the intermediate layer using a second kernel function, wherein the further intermediate layer is one or more layers subsequent to the intermediate layer, In the reverse convolution sublayer of the intermediate layer, the tensor engine uses the first kernel function to calculate the back propagation excitation of the other intermediate layer based on the weight value of the intermediate layer and the back propagation excitation, and uses the second kernel function to calculate the back propagation weight of the intermediate layer based on the excitation generated by the intermediate layer and the back propagation weight of the previous intermediate layer of the intermediate layer.

2. The computational kernel of claim 1, wherein the first kernel function is different from the second kernel function.

3. The computing core of claim 1, wherein the calculation of the tensor engine is a tensor or matrix operation, and the computing core further comprises: A vector engine that performs non-tensor and matrix operations in parallel with the tensor or matrix operations of the tensor engine.

4. The computing core of claim 3, wherein the vector engine updates the parameters of the activation sublayer and the BN sublayer of the intermediate layer based on the back-propagation excitation in parallel with the calculation of the back-propagation weights.

5. The computing core of claim 1, wherein the tensor engine stores the back-propagation excitation of the intermediate layer in a register.

6. The computing core of claim 1, further comprising: an on-chip cache, the on-chip cache comprising a first cache for caching data required for calculating the back-propagation excitation, and a second cache for caching data required for calculating the back-propagation weight, and The tensor engine reads data from the first cache and the second cache in an interleaved manner.

7. The computing core of claim 1, wherein the computing core is configured to: In the forward direction of the neural network, determining whether to store output data of an intermediate sublayer of the intermediate layer based on a cost of loading input data of the intermediate sublayer, a cost of calculating output data using the input data of the intermediate sublayer, and a cost of loading the output data of the intermediate sublayer; If it is determined to store the output data of the intermediate sublayer, after obtaining the output data of the intermediate sublayer, the output data of the intermediate sublayer is cached in the on-chip cache of the computing core and / or stored in an off-chip memory; as well as If it is determined not to store the output data of the intermediate sublayer, after obtaining the output data of the intermediate sublayer, the output data of the intermediate sublayer is cached in the on-chip cache or register of the computing core for immediate use by the next intermediate layer or the next intermediate sublayer.

8. The computing core of claim 7, wherein the computing core also stores output data of another intermediate sublayer before the intermediate sublayer.

9. The computing core of claim 8, wherein in the reverse direction of the neural network, the computing core calculates the output data of the intermediate sub-layer using the stored output data of the other intermediate sub-layer.

10. The computing core of claim 7, wherein the intermediate sublayer is an activation sublayer of the intermediate layer, and the other intermediate sublayer is a convolutional sublayer or a BN sublayer of the intermediate layer.

11. The computing core of claim 10, wherein when the other intermediate sublayer is a convolutional sublayer of the intermediate layer, in the reverse direction, the computing core calculates the output data of the reverse BN sublayer using the output data of the convolutional sublayer to further calculate the gradient values ​​of the parameters of the BN sublayer of the intermediate layer.

12. The computing core of claim 10, wherein when the other intermediate sublayer is the BN sublayer of the intermediate layer, in the reverse direction, the computing core performs a BN inverse operation on the output data of the BN sublayer of the intermediate layer, and calculates the output data of the reverse BN sublayer based on the result of the BN inverse operation to further calculate the gradient value of the parameters of the BN sublayer of the intermediate layer.

13. The computing core of claim 6, wherein the computing core is configured to determine whether to store the output data of the intermediate sublayer based on whether the sum of the cost of loading the input data of the intermediate sublayer and the cost of calculating the output data of the intermediate sublayer is less than the cost of loading the output data of the intermediate sublayer.

14. The computing core of claim 13, wherein The cost of loading the input data of the intermediate sub-layer depends on the data volume of the input data and the storage location of the input data; The cost of loading the output data of the intermediate sub-layer depends on the data volume of the output data and the storage location of the output data; as well as The cost of computing the output data depends on the total number of operations of the intermediate sub-layer and the hardware data channel width of the computing core.

Images