Method for improving back propagation speed of GRU
By using CUDA kernel function to optimize GRU backpropagation calculation on the GPU side, the problem of slow training speed of GRU operators is solved, and more efficient GRU backpropagation and training speed is achieved, which significantly reduces training time.
Patent Information
- Application Number
- CN202310635242.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-31
- Publication Date
- 2024-12-03
AI Technical Summary
The existing GRU operators have a slower quantization training speed, resulting in a longer training time. Especially in deep learning model training, it is necessary to improve the inference and training speed of GRU operators to improve training efficiency.
By setting the index of backtraversal, using the CUDA kernel function to realize the calculation of GRU backpropagation on the GPU side, optimize the gradient calculation of gate_i and gate_h, and process bidirectional GRU calculations in parallel, avoid repeated calculations and slicing operations, and improve the gradient of weights and biases calculate.
It significantly improves the GRU backpropagation speed and quantitative training speed, saves training time, and improves the inference efficiency of the GRU operator.
Smart Images

Figure CN119067162A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of operator reasoning process of recurrent neural network, and in particular relates to a method for improving the back propagation speed of GRU. Background Art
[0002] In the prior art, the full name of GRU is Gate Recurrent Unit, which is a type of recurrent neural network (RNN). It is proposed to solve problems such as long-term memory and gradients in back propagation. RNN is a neural network used to process sequence data. Back propagation is a common method used in combination with optimization methods (such as gradient descent) to train artificial neural networks. In the existing training and quantization GRU process, there is a defect of slow training speed. In the deep learning training model, the training of the model requires the process of updating weights through forward propagation reasoning and back propagation. Among the products of Beijing Ingenic Integrated Circuit Co., Ltd. (referred to as Ingenic), Ingenic Magik is a full-stack deep learning open platform that integrates model training, optimization conversion, and deployment reasoning for end-side AI application scenarios. It improves the breadth and efficiency of the platform's training and quantization operator support. Recurrent neural networks are an important part that must be supported. Recurrent neural networks play an important role in processing audio and text recognition. GRU operators belong to a type of training neural network. As for improving the inference speed and training speed of the GRU operator on the Ingenic Magik platform, the current problem is that the existing GRU operator quantization training speed is much slower than the official (cudnn) speed, and its training speed must be improved to save training time. Summary of the invention
[0003] In order to solve the above problems, the purpose of this application is to propose a solution to improve the back propagation speed of the GRU operator.
[0004] Specifically, the present invention provides a method for improving the GRU back propagation speed, thereby improving the speed of GRU operator quantization training. The method comprises:
[0005] S0, set a reverse traversal index, the starting position is seq_length-1, the end index value is 0, and the step length is 1;
[0006] The specific implementation is completed using CUDA kernel functions. The kernel function refers to the code running on the GPU side, which specifies which data each thread of the GPU accesses and what calculations to perform;
[0007] CUDA logically divides GPU threads into three levels: grid, block, and thread; S1. Obtain the gradients of gate_i and gate_h and the gradient of old_h;
[0008] Kernel function implementation:
[0009] historys is the historical data saved by default during the forward inference of GRU and is used for gradient calculation. historys_gate_h saves the historical data of gate_h during the forward inference of GRU and is used for gradient calculation
[0010] Set grid_y to 2 if it is bidirectional, otherwise 1
[0011] Set BLOCK = 512
[0012] Loop index from seq_lenght - 1 to 0:
[0013] The gru_backward_kernel kernel function <<<grid_y, BLOCK>>> is passed index and the required parameters
[0014] The gru_backward_kernel kernel function mainly calculates the gradients of gate_i and gate_h. The specific process is as follows:
[0015] Set seq_step = batch_size * hidden_size * 3
[0016] Set his_step = batch_size * hidden_size * 4
[0017] Set offset = blockIdx.x * hidden_size * 3
[0018] Set history_offset = blockIdx.x * hidden_size * 4
[0019] Set i = threadIdx.x. F_h is the historical state value at this time for forward inference and is used for gradient calculation
[0020] If blockIdx.y == 0, then execute
[0021] If i < hidden_size, then execute:
[0022] history_s_offset = index * history_step,
[0023] rg = historys[history_s_offset + history_offset + i],
[0024] zg = historys[history_s_offset + history_offset + i + hidden_size],
[0025] ng = historys[history_s_offset + history_offset + i + hidden_size * 2],
[0026] Grad_h_data = output gradient[i * batch_size * hidden_size + blockIdx.x *
[0027] hidden_size + ] + gradient of state value[blockIdx.x * hidden_size + i],
[0028] D_n = Grad_h_data * (1 - zg) * (1 - ng * ng),
[0029] D_r = D_n * historys_gate_h[seq_offset_ + i + hidden_size * 2] * (1 - r_t) * r_t,
[0030] D_z = Grad_h_data * (f_h[index * batch_size * hidden_size + blockIdx.x * hidden_size + i] - ng) * (1 - zg) * zg,
[0031] D_h_old = Grad_h_data * zg,
[0032] d_gate_i[blockIdx.x * hidden_size * 3 + i] = D_r,
[0033] d_gate_i[blockIdx.x * hidden_size * 3 + i + hidden_size] = D_z,
[0034] d_gate_i[blockIdx.x * hidden_size * 3 + i + 2 * hidden_size] = D_n,
[0035] d_gate_h[blockIdx.x * hidden_size * 3 + i] = D_r,
[0036] d_gate_h[blockIdx.x*hidden_size*3+i+hidden_size]=D_z,
[0037] d_gate_h[blockIdx.x*hidden_size*3+i+2*hidden_size]=D_n*rg;
[0038] If blockIdx.y=1, it is bidirectional gru reverse gradient calculation. The process is the same as the forward gradient calculation process, so it is omitted here;
[0039] The gradients of gate_i and gate_h are denoted as d_gate_i, d_gate_h, respectively, with dimensions (batch, hidden_size*3); the gradient of old_h is denoted as D_h_old;
[0040] S2. Get the weight gradient;
[0041] The dimensions of W_ir, W_iz, and W_in in forward reasoning are all (input, hidden_size), so W_ir, W_iz, and W_in are processed together and collectively referred to as W_i. The dimension of W_i is (3×hidden_size, input_size).
[0042] Similarly, W_hr, W_hz, and W_hn are put together and collectively referred to as W_h, with a dimension of (3×hidden_size, hidden_size);
[0043] Kernel function implementation:
[0044] Set the thread block to (batch_size, direction). The direction value represents a unidirectional or bidirectional GRU and the value range is [1, 2]. The thread value is set to 512.
[0045] Set the pointer offset:
[0046] w_ih_offset=blockIdx.x*input_size,
[0047] w_hh_offset=blockIdx.x*hidden_size,
[0048] Max_size=max(input_size,hidden_size),
[0049] i = threadIdx.x
[0050] If i < max_size, then:
[0051] If i < input_size, d_W_i[w_ih_offset + i] += d_w_i_tmp[w_ih_offset + i]; if i < hidden_size, d_W_h[w_hh_offset + i] += d_w_h_tmp[w_hh_offset + i]; S3. Calculate the gradient of the bias;
[0052] In forward inference, the dimensions of b_ir, b_iz, and b_in are all (hidden_size). In the implementation, b_ir, b_iz, and b_in are processed together and are collectively referred to as b_i here. The dimension of b_i is (3 × hidden_size);
[0053] Similarly, b_hr, b_hz, and d_hn are also processed together and are collectively referred to as b_h here, with a dimension of (3 × hidden_size);
[0054] For the gradient of the bias, simply accumulate d_gate_i for all batches. The formula is The derivatives of d_bias_i and d_bias_h are d_gate_i and d_gate_h respectively;
[0055] The thread block is set to direction. The value of direction represents unidirectional or bidirectional GRU, and the value range is [1, 2]; the thread is set to 512,
[0056] Set i = threadIdx.x
[0057] Oc = 3 * hidden_size
[0058] If i < Oc
[0059] Execute d_i_sum = 0.0
[0060] Execute d_h_sum = 0.0
[0061] Enter the loop body. Set j = 0 and check if j < batch_size holds. If it holds, execute the loop body content: d_i_sum += d_gate_i[j * oc + i], d_h_sum += d_gate_h[j * oc + i], and j++; which is expressed as
[0062] for(int j = 0; j < batch_size; j++){
[0063] d_i_sum+=d_gate_i[j*oc+i]
[0064] d_h_sum+=d_gate_h[j*oc+i]
[0065] };
[0066] After jumping out of the loop body,
[0067] Execute d_bias_i+=d_i_sum
[0068] Execute d_bias_h+=d_h_sum;
[0069] Among them, d_bias_i is the gradient of b_i, and d_bias_h is the gradient of b_h;
[0070] S4. Get the gradient of the input;
[0071] The input gradient is calculated as:
[0072]
[0073]
[0074] In step S1, D_h_old is calculated.
[0075] Addmm means that matrix multiplication can be realized
[0076] Bias_add means that matrix addition operation can be realized
[0077] Let D_input_x be the gradient of x_t and D_h be the gradient of h, then
[0078] D_input_x=Addmm(d_gate_i,w_i)
[0079] D_h=Addmm(d_gate_i,h)
[0080] D_h=bias_add(D_h,D_h_old)
[0081] Let D_input_x be the gradient of x_t and D_h be the gradient of h, then
[0082] D_input_x=d_gate_i*w_i
[0083] D_h=d_gate_i*h+D_h_old;
[0084] The entire input `input_fx` has a dimension of (seq_length, batch_size, input_size); the implementation of the kernel function is as follows:
[0085] Set the thread block to (batch_size, direction), where the `direction` value represents unidirectional or bidirectional GRU, and its value range is [1, 2]; set the `thread` to 512.
[0086] Set the offset of the pointer
[0087] Offset = blockIdx.x * input_size
[0088] i = threadIdx.x
[0089] If i < input_size, then
[0090] Input_fx[index * batch_size * input_size + offset + i] = D_input_x[offset + i], where Input_fx is the gradient of the entire input.
[0091] In step S0, seq_lenght is 5, and the value of index changes to 4, 3, 2, 1, 0.
[0092] The forward inference of the GRU operator is as follows:
[0093] r_t = sigmoid(x_t × W_ir + b_ir + h × W_hr + b_hr),
[0094] z_t = sigmoid(x_t × W_iz + b_iz + h × W_hz + b_hz),
[0095] n_t = tanh(x_t × W_in + b_in + (h × W_hn + b_hn) × r_t),
[0096] h_t = (1 - z_t) × n_t + z_t × h;
[0097] Sigmoid: The function code implementation is 1 / (1 + expf(data))
[0098] tanh: The function code implementation is (expf(data) - expf(-data)) / (expf(data) + expf(-data))
[0099] In the actual implementation process, $W_{ir}$, $W_{iz}$, $W_{in}$ are stored in one variable, which we denote as $W_i$. $W_i = [W_{ir}, W_{iz}, W_{in}]$, and $W_h = [W_{hr}, W_{hz}, W_{hn}]$.
[0100] Similarly, $b_i = [b_{ir}, b_{iz}, b_{in}]$, and $b_h = [b_{hr}, b_{hz}, b_{hn}]$.
[0101] $gate_i = Addmm(x_t, W_i, b_i)$
[0102] Loop index from 0 to seq_length:
[0103] $gate_h = Addmm(h, W_h, b_h)$
[0104] The gru_forward_kernel kernel function <<<grid_y, BLOCK>>> is passed index, gate_i, gate_h and the required parameters.
[0105] The gru_forward_kernel kernel function mainly calculates $h_t$.
[0106] The specific process is as follows:
[0107] Set seq_step = batch_size * hidden_size * 3
[0108] Set offset = blockIdx.x * hidden_size * 3
[0109] Set i = threadIdx.x
[0110] If blockIdx.y == 0, then execute
[0111] If i < hidden_size, then execute:
[0112] seq_offset = s_index * seq_step
[0113] rg = gate_i[seq_offset + offset + i] + gate_h[offset + i];
[0114] zg = gate_i[seq_offset + offset + i + hidden_size] + gate_h[offset + i + hidden_size];
[0115] rg = sigmoid(rg);
[0116] zg = sigmoid(zg);
[0117] ng=gate_i[seq_offset+offset+i+hidden_size_x2]+rg*(gate_h[offset+i+hidden_size_x2]);
[0118] ng=tanh(ng);
[0119] h_t=ng+zg*(h[blockIdx.x*hidden_size+i]-ng);
[0120] The formula for GRU back propagation is as follows:
[0121] Available at time t GRU is an operator in a deep learning neural network. The gradient of the entire network is obtained by derivation of the final loss function, and then back propagation is performed layer by layer, that is, each operator calculates the gradient. Here is the gradient value of the next layer back-transferred to GRU,
[0122]
[0123]
[0124]
[0125]
[0126]
[0127] in the formula It is the input of the back-propagation calculation, denoted as grad_h, and its dimension is (seq_length, batch_size, hidden_size).
[0128] The gradient in step S1 is obtained by inserting the data used for the gradient into the following back propagation, i.e., chain derivation:
[0129]
[0130]
[0131]
[0132]
[0133]
[0134] historys are the historical data during the forward inference of the default saved GRU, which are used for calculating gradients; the historical data of gate_h during the forward inference of GRU are saved in historys_gate_h, which are used for gradient calculation;
[0135] Set grid_y. If it is bidirectional, grid_y = 2; otherwise, grid_y = 1
[0136] Set BLOCK = 512
[0137] Loop index from seq_lenght - 1 to 0:
[0138] The gru_backward_kernel kernel function <<<grid_y, BLOCK>>> is passed index and the required parameters
[0139] The gru_backward_kernel kernel function mainly calculates the gradients of gate_i and gate_h. The specific process is as follows:
[0140] Set seq_step = batch_size * hidden_size * 3
[0141] Set his_step = batch_size * hidden_size * 4
[0142] Set offset = blockIdx.x * hidden_size * 3
[0143] Set history_offset = blockIdx.x * hidden_size * 4
[0144] Set i = threadIdx.x. F_h is the historical state value during the forward inference at this time, which is used for gradient calculation
[0145] If blockIdx.y = 0, then execute
[0146] If i < hidden_size, then execute:
[0147] history_s_offset = index * history_step,
[0148] rg = historys[history_s_offset + history_offset + i],
[0149] zg = historys[history_s_offset + history_offset + i + hidden_size],
[0150] ng = historys[history_s_offset + history_offset + i + hidden_size * 2],
[0151] Grad_h_data = output gradient[i * batch_size * hidden_size + blockIdx.x *
[0152] hidden_size + ] + gradient of state value[blockIdx.x * hidden_size + i],
[0153] D_n = Grad_h_data * (1 - zg) * (1 - ng * ng),
[0154] D_r = D_n * historys_gate_h[seq_offset_ + i + hidden_size * 2] * (1 - r_t) * r_t,
[0155] D_z = Grad_h_data * (f_h[index * batch_size * hidden_size + blockIdx.x * hidden_size + i] - ng) * (1 - zg) * zg,
[0156] D_h_old = Grad_h_data * zg,
[0157] d_gate_i[blockIdx.x * hidden_size * 3 + i] = D_r,
[0158] d_gate_i[blockIdx.x * hidden_size * 3 + i + hidden_size] = D_z,
[0159] d_gate_i[blockIdx.x * hidden_size * 3 + i + 2 * hidden_size] = D_n,
[0160] d_gate_h[blockIdx.x * hidden_size * 3 + i] = D_r,
[0161] d_gate_h[blockIdx.x * hidden_size * 3 + i + hidden_size] = D_z,
[0162] d_gate_h[blockIdx.x*hidden_size*3+i+2*hidden_size]=D_n*rg;
[0163] In step S2, the calculation formula of the weight gradient is:
[0164]
[0165]
[0166] Let the gradient of W_i be d_W_i, and the gradient of W_i calculated at each moment index be d_w_i_tmp;
[0167] Let the gradient of W_h be d_W_h, and the gradient of W_h calculated at each moment index be d_w_h_tmp;
[0168] d_w_i_tmp=d_gate_i*x_t;
[0169] d_w_h_tmp=d_gate_h*h.
[0170] Therefore, the advantages of the present application are: the present method is simple, the existing GRU back-propagation process will use slicing operations, each training will traverse a sequence length, and an inference calculation will be performed each time, and when the bidirectional GRU calculation process is serial, the present method will save the data required for the back-propagation calculation during inference, and no longer perform repeated calculations. At the same time, slicing is no longer used to access data, but the data is directly operated through the kernel function index coordinates, and the bidirectional GRU calculation process is modified to be parallel, thereby improving the GRU inference speed. BRIEF DESCRIPTION OF THE DRAWINGS
[0171] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.
[0172] Figure 1 is a schematic diagram of the process of this method. DETAILED DESCRIPTION
[0173] In order to more clearly understand the technical content and advantages of the present invention, the present invention is now further described in detail in conjunction with the accompanying drawings.
[0174] like Figure 1 As shown, the present application provides a method for improving the GRU back propagation speed, wherein the calculation formula of the GRU operator is as follows:
[0175] r_t=sigmoid(x_t×W_ir+b_ir+h×W_hr+b_hr),
[0176] z_t=sigmoid(x_t×W_iz+b_iz+h×W_hz+b_hz),
[0177] n_t=tanh(x_t×W_in+b_in+(h×W_hn+b_hn)×r_t),
[0178] h_t=(1-z_t)×n_t+z_t×h;
[0179] For the convenience of writing, x_t×W_ir+b_ir+h×W_hr+b_hr is denoted as r;
[0180] x_t×W_iz+b_iz+h×W_hz+b_hz is denoted as z;
[0181] x_t×W_in+b_in+(h×W_hn+b_hn)×r_t is recorded as n;
[0182] The results of x_t×W_ir+b_ir,x_t×W_iz+b_iz,x_t×W_in+b_in are stored together, so they are uniformly recorded as gate_i;
[0183] h×W_hr+b_hr, h×W_hz+b_hz, h×W_hn+b_hn are similarly denoted as gate_h;
[0184] Si gmoid: Function code implements 1 / 1+expf(data)
[0185] tanh: Function code implementation (expf(data)-expf(-data)) / (expf(data)+expf(-data))
[0186] In the actual implementation process, Wi r Wi z Wi n is stored in a variable, which we represent as Wi = [W ir, W iz, W n], W h = [W hr, W hz, W hn]
[0187] Similarly, b_i=[b_ir,b_iz,b_in],b_i=[b_hr,b_hz,b_hn]
[0188] gate_i=Addmm(x_t,W_i,b_i)
[0189] Loop index from 0 to seq_length:
[0190] gate_h=Addmm(h,W_h,b_h)
[0191] The gru_forward_kernel kernel function is passed index, gate_i, gate_h and the required parameters <<<grid_y, BLOCK>>>
[0192] The gru_forward_kernel kernel function mainly calculates h_t
[0193] The specific process is as follows:
[0194] Set seq_step = batch_size * hidden_size * 3
[0195] Set offset = blockIdx.x * hidden_size * 3
[0196] Set i = threadIdx.x
[0197] If blockIdx.y == 0, then execute
[0198] If i < hidden_size, then execute:
[0199] seq_offset = s_index * seq_step
[0200] rg = gate_i[seq_offset + offset + i] + gate_h[offset + i];
[0201] zg = gate_i[seq_offset + offset + i + hidden_size] + gate_h[offset + i + hidden_size];
[0202] rg = sigmoid(rg);
[0203] zg = sigmoid(zg);
[0204] ng = gate_i[seq_offset + offset + i + hidden_size_x2] + rg * (gate_h[offset + i + hidden_size_x2]);
[0205] ng = tanh(ng);
[0206] h_t = ng + zg * (h[blockIdx.x * hidden_size + i] - ng);
[0207] The formula for GRU backpropagation is as follows:
[0208] historys are the historical data saved by default during the forward inference of GRU in the positive direction and are used for calculating gradients; historys_gate_h saves the historical data of gate_h during the forward inference of GRU and is used for gradient calculation;
[0209] Set grid_y. If it is bidirectional, grid_y = 2; otherwise, grid_y = 1;
[0210] Set BLOCK = 512;
[0211] Loop with index from seq_lenght - 1 to 0:
[0212] The gru_backward_kernel kernel function <<<grid_y, BLOCK>>> is passed index and the required parameters
[0213] The gru_backward_kernel kernel function mainly calculates the gradients of gate_i and gate_h. The specific process is as follows:
[0214] Set seq_step = batch_size * hidden_size * 3
[0215] Set his_step = batch_size * hidden_size * 4
[0216] Set offset = blockIdx.x * hidden_size * 3
[0217] Set history_offset = blockIdx.x * hidden_size * 4
[0218] Set i = threadIdx.x. F_h is the historical state value at this time during forward inference and is used for gradient calculation
[0219] If blockIdx.y = 0, then execute
[0220] If i < hidden_size, then execute:
[0221] history_s_offset = index * history_step,
[0222] rg = historys[history_s_offset + history_offset + i],
[0223] zg = historys[history_s_offset + history_offset + i + hidden_size],
[0224] ng = historys[history_s_offset + history_offset + i + hidden_size * 2],
[0225] Grad_h_data = 输出梯度[i * batch_size * hidden_size + blockIdx.x *
[0226] hidden_size + ] + 状态值的梯度[blockIdx.x * hidden_size + i],
[0227] D_n = Grad_h_data * (1 - zg) * (1 - ng * ng),
[0228] D_r = D_n * historys_gate_h[seq_offset_ + i + hidden_size * 2] * (1 - r_t) * r_t,
[0229] D_z = Grad_h_data * (f_h[index * batch_size * hidden_size + blockIdx.x * hidden_size + i] - ng) * (1 - zg) * zg,
[0230] D_h_old = Grad_h_data * zg,
[0231] d_gate_i[blockIdx.x * hidden_size * 3 + i] = D_r,
[0232] d_gate_i[blockIdx.x * hidden_size * 3 + i + hidden_size] = D_z,
[0233] d_gate_i[blockIdx.x * hidden_size * 3 + i + 2 * hidden_size] = D_n,
[0234] d_gate_h[blockIdx.x * hidden_size * 3 + i] = D_r,
[0235] d_gate_h[blockIdx.x * hidden_size * 3 + i + hidden_size] = D_z,
[0236] d_gate_h[blockIdx.x*hidden_size*3+i+2*hidden_size]=D_n*rg;
[0237] If blockIdx.y=1, it is bidirectional gru reverse gradient calculation. The general process is the same as the forward gradient calculation, so it is omitted here;
[0238] Available at time t
[0239]
[0240]
[0241]
[0242]
[0243]
[0244]
[0245] in the formula It is the input of the back-propagation calculation, denoted as grad_h, and its dimension is (seq_length, batch_size, hidden_size);
[0246] Step S0. Set a reverse traversal index, with the starting position being seq_length-1, the ending index value being 0, and the step length being 1 (for example, if seq_length is 5, the index value changes to 4, 3, 2, 1, 0);
[0247] The specific implementation is completed using CUDA kernel functions. (The kernel function refers to the code running on the GPU. In short, it specifies which data each GPU thread accesses and what calculations to perform.) CUDA logically divides GPU threads into three levels: grid, block, and thread.
[0248] Step S1. Calculate the gradients of gate_i and gate_h;
[0249] Kernel function implementation:
[0250] Set the thread block to (batch_size, direction). The direction value indicates unidirectional or bidirectional GRU and the value range is [1, 2].
[0251] Set the pointer offset value offset of the data to index * batch_size * hidden_size + blockIdx.x * hidden_size
[0252] Set i = threadIdx.x
[0253] If i < hidden_size, then calculate as follows:
[0254] Grad_h_data = grad_h[offset + i]
[0255] It can be seen from the backpropagation formula that the data used to calculate the gradient is: r_t, z_t, n_t, h at time t in the forward propagation, and also h × W_hn + b_hn. Therefore, in the forward propagation process, these required data are saved, avoiding the time-consuming problem of recalculation in the backpropagation, trading space for time.
[0256] historys is the historical data saved by default during the forward inference of gru and is used for calculating the gradient;
[0257] historys_gate_h saves the historical data of gate_h during the forward inference of gru and is used for gradient calculation;
[0258] Set grid_y. If it is bidirectional, grid_y = 2, otherwise = 1
[0259] Set BLOCK = 512
[0260] Loop index from seq_lenght - 1 to 0:
[0261] The gru_backward_kernel kernel function <<<grid_y, BLOCK>>> is passed index and the required parameters
[0262] The gru_backward_kernel kernel function mainly calculates the gradients of gate_i and gate_h. The specific process is as follows:
[0263] Set seq_step = batch_size * hidden_size * 3
[0264] Set his_step = batch_size * hidden_size * 4
[0265] Set offset = blockIdx.x * hidden_size * 3
[0266] Set history_offset = blockIdx.x * hidden_size * 4
[0267] Set i = threadIdx.x. F_h is the historical state value for forward inference at this time and is used for gradient calculation
[0268] If blockIdx.y == 0, then execute
[0269] If i < hidden_size, then execute:
[0270] history_s_offset = index * history_step
[0271] rg = historys[history_s_offset + history_offset + i]
[0272] zg = historys[history_s_offset + history_offset + i + hidden_size]
[0273] ng = historys[history_s_offset + history_offset + i + hidden_size * 2]
[0274] Grad_h_data = output gradient[i * batch_size * hidden_size + blockIdx.x *
[0275] hidden_size + ] + gradient of the state value[blockIdx.x * hidden_size + i]
[0276] D_n = Grad_h_data * (1 - zg) * (1 - ng * ng)
[0277] D_r = D_n * historys_gate_h[seq_offset_ + i + hidden_size * 2] * (1 - r_t) * r_t
[0278] D_z = Grad_h_data * (f_h[index * batch_size * hidden_size + blockIdx.x * hidden_size + i] - ng) * (1 - zg) * zg,
[0279] D_h_old = Grad_h_data * zg,
[0280] d_gate_i[blockIdx.x*hidden_size*3+i]=D_r,
[0281] d_gate_i[blockIdx.x*hidden_size*3+i+hidden_size]=D_z,
[0282] d_gate_i[blockIdx.x*hidden_size*3+i+2*hidden_size]=D_n,
[0283] d_gate_h[blockIdx.x*hidden_size*3+i]=D_r,
[0284] d_gate_h[blockIdx.x*hidden_size*3+i+hidden_size]=D_z,
[0285] d_gate_h[blockIdx.x*hidden_size*3+i+2*hidden_size]=D_n*rg;
[0286] If blockIdx.y=1, it is bidirectional gru reverse gradient calculation. The general process is the same as the forward gradient calculation, so it is omitted here.
[0287] The gradients of gate_i and gate_h are denoted as d_gate_i, d_gate_h, respectively, with dimensions of (batch, hidden_size*3); the gradient of old_h is denoted as D_h_old;
[0288] Substituting into the formula, we get:
[0289] D_n=Grad_h_data*(1-z_t)
[0290] D_n=D_n*(1-n_t*n_t)
[0291] D_r=D_n*h×W_hn+b_hn
[0292] D_r=D_r*(1-r_t)*r_t
[0293] D_z=Grad_h_data*(h-n_t)
[0294] D_z=D_z*(1-z_t)*z_t
[0295] D_h_old=Grad_h_data*z_t
[0296] d_gate_i[blockIdx.x*hidden_size*3+i]=D_r
[0297] d_gate_i[blockIdx.x*hidden_size*3+i+hidden_size]=D_z
[0298] d_gate_i[blockIdx.x*hidden_size*3+i+2*hidden_size]=D_n
[0299] d_gate_h[blockIdx.x*hidden_size*3+i]=D_r
[0300] d_gate_h[blockIdx.x*hidden_size*3+i+hidden_size]=D_z
[0301] d_gate_h[blockIdx.x*hidden_size*3+i+2*hidden_size]=D_n*r_t Step S2. Calculate the weight gradient;
[0302] Following the previous step, we get the gradients of gate_i and gate_h.
[0303] The dimensions of W_ir, W_iz, and W_in in the inference formula are all (input, hidden_size), so W_ir, W_iz, and W_in are processed together and collectively referred to as W_i. The dimension of W_i is (3×hidden_size, input_size).
[0304] Similarly, W_hr, W_hz, and W_hn are put together and collectively referred to as W_h, with a dimension of (3×hidden_size, hidden_size)
[0305] The calculation formula of weight gradient is:
[0306]
[0307]
[0308] Let the gradient of W_i be d_W_i, and the gradient of W_i calculated at each moment index be d_w_i_tmp;
[0309] Let the gradient of W_h be d_W_h, and the gradient of W_h calculated at each moment index be d_w_h_tmp;
[0310] d_w_i_tmp = d_gate_i * x_t;
[0311] d_w_h_tmp = d_gate_h * h;
[0312] Kernel function implementation:
[0313] Set the thread block to (batch_size, direction), where the direction value represents unidirectional or bidirectional GRU, and the value range is [1, 2]; set the thread to 512;
[0314] Set the pointer offset:
[0315] w_ih_offset = blockIdx.x * input_size,
[0316] w_hh_offset = blockIdx.x * hidden_size,
[0317] Max_size = max(input_size, hidden_size),
[0318] Set i = threadIdx.x
[0319] If i < max_size then:
[0320] If i < input_size, d_W_i[w_ih_offset + i] += d_w_i_tmp[w_ih_offset + i], if i < hidden_size, d_W_h[w_hh_offset + i] += d_w_h_tmp[w_hh_offset + i]; Step S3. Calculate the gradient of bias;
[0321] The dimensions of b_ir, b_iz, and b_in in the formula are all (hidden_size). In the implementation, b_ir, b_iz, and b_in are processed together and are collectively referred to as b_i here, and the dimension of b_i is (3 × hidden_size); similarly, b_hr, b_hz, and d_hn are also processed together and are collectively referred to as b_h here, with a dimension of (3 × hidden_size);
[0322] For the gradient of bias, directly accumulate d_gate_i for all batches. The formula is The derivatives of d_bias_i and d_bias_h are d_gate_i and d_gate_h;
[0323] The thread block is set to direction, where the value of direction represents unidirectional or bidirectional GRU, and the value range is [1, 2]; the thread is set to 512.
[0324] Let i = threadIdx.x
[0325] Oc = 3 * hidden_size
[0326] If i < Oc
[0327] Execute d_i_sum = 0.0
[0328] Execute d_h_sum = 0.0
[0329] Enter the loop body. Let j = 0 and judge whether j < batch_size holds. If it holds, execute the loop body content: d_i_sum += d_gate_i[j * oc + i], d_h_sum += d_gate_h[j * oc + i], and j++; which is expressed as
[0330] for(int j = 0; j < batch_size; j++){
[0331] d_i_sum += d_gate_i[j * oc + i]
[0332] d_h_sum += d_gate_h[j * oc + i]
[0333] };
[0334] After jumping out of the loop body,
[0335] Execute d_bias_i += d_i_sum
[0336] Execute d_bias_h += d_h_sum;
[0337] Among them, d_bias_i is the gradient of b_i, and d_bias_h is the gradient of b_h;
[0338] Step S4. Calculate the gradient of the input;
[0339] The calculation formula for the input gradient is:
[0340]
[0341]
[0342] In the above step S1, D_h_old is calculated.
[0343] Addmm represents matrix multiplication operation that can be achieved.
[0344] Bias_add represents the ability to perform matrix addition operations
[0345] Let D_input_x be the gradient of x_t and D_h be the gradient of h, then
[0346] D_input_x = Addmm(d_gate_i, w_i)
[0347] D_h = Addmm(d_gate_i, h)
[0348] D_h = bias_add(D_h, D_h_old)
[0349] The dimension of the entire input input_fx is (seq_length, batch_size, input_size);
[0350] Implementation of the kernel function
[0351] Set the thread block to (batch_size, direction), where the direction value represents unidirectional or bidirectional GRU, and the value range is [1, 2]; Set the thread to 512;
[0352] Set the offset of the pointer
[0353] Offset = blockIdx.x * input_size
[0354] i = threadIdx.x
[0355] If i < input_size, Input_fx[index * batch_size * input_size + offset + i] = D_input_x[offset + i], where Input_fx is the gradient of the entire input.
[0356] Taking a neural network as an example, it contains 2 GRU operators, and the specific definitions are as follows:
[0357] The following is a network operator structure. The network structure is executed from top to bottom. Con2d is a convolution operator, batchNorm2d is a normalization operator, Linear is a fully connected operator, and ConvTranpose2d is a transposed convolution operator.
[0358] Conv2d(2, 8, kernel_size=(2, 5), stride=(1, 2)
[0359] BatchNorm2d(8)
[0360] Conv2d(8,16,kernel_size=(2,3),stride=(1,2)
[0361] BatchNorm2d(16,)
[0362] Conv2d(16,32,kernel_size=(2,3),stride=(1,1))
[0363] GRU(32,32,batch_first=True,bidirectional=True),
[0364] Linear(64,32)
[0365] GRU(32,32,batch_first=True)
[0366] ConvTranspose2d(64,16,kernel_size=(2,3),stride=(1,1)
[0367] ConvTranspose2d(16,2,kernel_size=(2,5),stride=(1,2),padding=(1,2)
[0368] One of them has 2 GRU operators, one is a bidirectional GRU. When training this network, batch_size is set to 128, and the data volume is about 20G of voice data. The following is the test results of the quantization training of the old version of the original model and the quantization training table of this method:
[0369] Old quantization training (per epoch) This method quantizes training (per epoch) Original Model 1.5 hours (non-quantitative) 1.5 hours (non-quantitative) 32bit 3.5 hours 2 hours 8bit 15 hours 2.6 hours
[0370] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A method for improving the back propagation speed of GRU, characterized in that: The method includes: S0, setting an index for reverse traversal, with the starting position being seq_length - 1, the ending index value being 0, and the step size being 1; The specific implementation is completed using a CUDA kernel function. The kernel function refers to the code running on the GPU side, which specifies which data each thread of the GPU accesses and what calculations to perform; CUDA logically divides the GPU threads into three levels: grid, block, and thread; S1. Obtain the gradients of gate_i and gate_h and the gradient of old_h; Kernel function implementation: Set the block to (batch_size, direction), where the value of direction indicates unidirectional or bidirectional GRU, and the value range is [1, 2]; historys is the historical data saved by default during the forward inference of GRU and is used for calculating gradients; historys_gate_h saves the historical data of gate_h during the forward inference of GRU and is used for gradient calculation; Set grid_y to 2 if it is bidirectional, otherwise set it to 1, Set BLOCK = 512, Loop index from seq_lenght - 1 to 0: The gru_backward_kernel kernel function <<<grid_y, BLOCK>>> is passed the index and the required parameters The main process of the gru_backward_kernel kernel function for calculating the gradients of gate_i and gate_h is as follows: Set seq_step = batch_size * hidden_size * 3 Set his_step = batch_size * hidden_size * 4 Set offset = blockIdx.x * hidden_size * 3 Set history_offset = blockIdx.x * hidden_size * 4 Set i = threadIdx.x F_h is the historical state value of the forward inference at this time and is used for gradient calculation If blockIdx.y = 0, then execute If i < hidden_size, then execute: history_s_offset = index * history_step, rg = historys[history_s_offset + history_offset + i], zg = historys[history_s_offset + history_offset + i + hidden_size], ng = historys[history_s_offset + history_offset + i + hidden_size * 2], Grad_h_data = output gradient [i * batch_size * hidden_size + blockIdx.x * hidden_size+]+ gradient of state value [blockIdx.x*hidden_size+i], D_n=Grad_h_data*(1-zg)*(1-ng*ng), D_r=D_n*historys_gate_h[seq_offset_+i+hidden_size*2]*(1-r_t)*r_t, D_z=Grad_h_data*(f_h[index*batch_size*hidden_size+blockIdx.x*hidden_size+i]-ng)*(1-zg)*zg, D_h_old=Grad_h_data*zg, d_gate_i[blockIdx.x*hidden_size*3+i]=D_r, d_gate_i[blockIdx.x*hidden_size*3+i+hidden_size]=D_z, d_gate_i[blockIdx.x*hidden_size*3+i+2*hidden_size]=D_n, d_gate_h[blockIdx.x*hidden_size*3+i]=D_r, d_gate_h[blockIdx.x*hidden_size*3+i+hidden_size]=D_z, d_gate_h[blockIdx.x*hidden_size*3+i+2*hidden_size]=D_n*rg; If blockIdx.y=1, it is bidirectional gru reverse gradient calculation; the process is the same as the forward gradient calculation process; The gradients of gate_i and gate_h are denoted as d_gate_i, d_gate_h, respectively, with dimensions (batch, hidden_size*3); the gradient of old_h is denoted as D_h_old; S2. Get the weight gradient; The dimensions of weights W_ir, W_iz, and W_in are all (input, hidden_size), so W_ir, W_iz, and W_in are processed together and collectively referred to as W_i. The dimension of W_i is (3×hidden_size, input_size). Similarly, W_hr, W_hz, and W_hn are put together and collectively referred to as W_h, with a dimension of (3×hidden_size, hidden_size); Kernel function implementation: Set the thread block to (batch_size, direction). The direction value represents a unidirectional or bidirectional GRU and the value range is [1, 2]. The thread value is set to 512. Set the pointer offset: w_ih_offset=blockIdx.x*input_size, w_hh_offset = blockIdx.x * hidden_size, Max_size = max(input_size, hidden_size), Set i = threadIdx.x If i < max_size then: If i < input_size, d_W_i[w_ih_offset + i] += d_w_i_tmp[w_ih_offset + i], if i < hidden_size, d_W_h[w_hh_offset + i] += d_w_h_tmp[w_hh_offset + i]; S3. Calculate the gradient of bias; The dimensions of the values b_ir, b_iz, b_in used for bias are all (hidden_size). In the implementation, b_ir, b_iz, b_in are processed together. Here they are collectively referred to as b_i, and the dimension of b_i is (3 * hidden_size); Similarly, b_hr, b_hz, d_hn are also placed together. Here they are collectively referred to as b_h, with a dimension of (3 * hidden_size); The gradient of bias can be directly accumulated by adding up the d_gate_i of all batches. The formula is: The derivatives of d_bias_i and d_bias_h are d_gate_i and d_gate_h; The thread block is set to direction, and the value of direction represents unidirectional or bidirectional GRU. The value range is [1, 2]; the thread is set to 512, Set i = threadIdx.x Oc = 3 * hidden_size If i < Oc Execute d_i_sum = 0.0 Execute d_h_sum = 0.0 Enter the loop body. Set j = 0 and judge whether j < batch_size holds. If it holds, execute the loop body content: d_i_sum += d_gate_i[j * oc + i], d_h_sum += d_gate_h[j * oc + i], and j++; expressed as After jumping out of the loop body, Execute d_bias_i += d_i_sum Execute d_bias_h += d_h_sum; Among them, d_bias_i is the gradient of b_i, and d_bias_h is the gradient of b_h; S4. Obtain the gradient of the input; The calculation formula for the input gradient is: In the step S1, D_h_old is calculated, Addmm represents the operation that can implement matrix multiplication; Bias_add represents the operation that can implement matrix addition; Set D_input_x as the gradient of x_t and D_h as the gradient of h. Then D_input_x = Addmm(d_gate_i, w_i) D_h = Addmm(d_gate_i, h) D_h = bias_add(D_h, D_h_old) Find the dimension of the entire input input_fx is (seq_length, batch_size, input_size); Kernel function implementation: Set the thread block to (batch_size, direction). The value of direction represents unidirectional or bidirectional GRU, and the value range is [1, 2]; set the thread to 512; Set the offset of the pointer Offset = blockIdx.x * input_size i = threadIdx.x If i < input_size, then Input_fx[index * batch_size * input_size + offset + i] = D_input_x[offset + i], Input_fx is the gradient of the entire input.
2. A method for improving the GRU back propagation speed according to claim 1, characterized in that: In step S0, seq_lenght is 5, and the value of index changes to 4, 3, 2, 1, 0.
3. A method for improving the GRU back propagation speed according to claim 1, characterized in that: The The forward inference of the GRU operator is as follows: r_t = sigmoid(x_t × W_ir + b_ir + h × W_hr + b_hr), z_t = sigmoid(x_t × W_iz + b_iz + h × W_hz + b_hz), n_t = tanh(x_t × W_in + b_in + (h × W_hn + b_hn) × r_t), h_t = (1 - z_t) × n_t + z_t × h; Sigmoid: The function code implementation is 1 / (1 + expf(data)) tanh: The function code implementation is (expf(data) - expf(-data)) / (expf(data) + expf(-data)) In the actual implementation process, W_ir, W_iz, and W_in are stored in one variable. We represent it as W_i, W_i = [W_ir, W_iz, W_in], W_h = [W_hr, W_hz, W_hn] Similarly, b_i = [b_ir, b_iz, b_in], b_i = [b_hr, b_hz, b_hn] gate_i = Addmm(x_t, W_i, b_i) Loop index from 0 to seq_length: gate_h = Addmm(h, W_h, b_h) The gru_forward_kernel kernel function <<<grid_y, BLOCK>>> passes in index, gate_i, gate_h, and the required parameters The gru_forward_kernel kernel function mainly calculates h_t The specific process is as follows: Set seq_step = batch_size * hidden_size * 3 Set offset = blockIdx.x * hidden_size * 3 Set i = threadIdx.x If blockIdx.y = 0, then execute If i < hidden_size, then execute: seq_offset = s_index * seq_step rg = gate_i[seq_offset + offset + i] + gate_h[offset + i]; zg = gate_i[seq_offset + offset + i + hidden_size] + gate_h[offset + i + hidden_size]; rg = sigmoid(rg); zg = sigmoid(zg); ng = gate_i[seq_offset + offset + i + hidden_size_x2] + rg * (gate_h[offset + i + hidden_size_x2]); ng = tanh(ng); h_t = ng + zg * (h[blockIdx.x * hidden_size + i] - ng); The formula for GRU backpropagation is as follows: historys is the historical data saved by default during the forward inference of gru and is used for gradient calculation; historys_gate_h saves the historical data of gate_h during the forward inference of gru and is used for gradient calculation; Set grid_y. If it is bidirectional, grid_y = 2, otherwise = 1 Set BLOCK = 512 Loop index from seq_lenght - 1 to 0: The gru_backward_kernel kernel function <<<grid_y, BLOCK>>> is passed index and the required parameters The gru_backward_kernel kernel function mainly calculates the gradients of gate_i and gate_h. The specific process is as follows: Set seq_step = batch_size * hidden_size * 3 Set his_step = batch_size * hidden_size * 4 Set offset = blockIdx.x * hidden_size * 3 Set history_offset = blockIdx.x * hidden_size * 4 Set i = threadIdx.x F_h is the historical state value of the current forward inference and is used for gradient calculation; If blockIdx.y == 0, then execute If i < hidden_size, then execute: history_s_offset = index * history_step, rg = historys[history_s_offset + history_offset + i], zg = historys[history_s_offset + history_offset + i + hidden_size], ng = historys[history_s_offset + history_offset + i + hidden_size * 2], Grad_h_data = output gradient[i * batch_size * hidden_size + blockIdx.x * hidden_size+]+ gradient of state value [blockIdx.x*hidden_size+i], D_n=Grad_h_data*(1-zg)*(1-ng*ng), D_r=D_n*historys_gate_h[seq_offset_+i+hidden_size*2]*(1-r_t)*r_t, D_z=Grad_h_data*(f_h[index*batch_size*hidden_size+blockIdx.x*hidden_size+i]-ng)*(1-zg)*zg, D_h_old=Grad_h_data*zg, d_gate_i[blockIdx.x*hidden_size*3+i]=D_r, d_gate_i[blockIdx.x*hidden_size*3+i+hidden_size]=D_z, d_gate_i[blockIdx.x*hidden_size*3+i+2*hidden_size]=D_n, d_gate_h[blockIdx.x*hidden_size*3+i]=D_r, d_gate_h[blockIdx.x*hidden_size*3+i+hidden_size]=D_z, d_gate_h[blockIdx.x*hidden_size*3+i+2*hidden_size]=D_n*rg; If blockIdx.y=1, it is a bidirectional gru reverse gradient calculation, which is the same as the forward gradient calculation process; Available at time t GRU is an operator in a deep learning neural network. The gradient of the entire network is obtained by derivation of the final loss function, and then back propagation is performed layer by layer, that is, each operator calculates the gradient. Here is the gradient value of the next layer back-transferred to GRU, in the formula It is the input of the back-propagation calculation, denoted as grad_h, and its dimension is (seq_length, batch_size, hidden_size).
4. A method for improving the GRU back propagation speed according to claim 3, characterized in that: The gradient in step S1 is obtained by inserting the data used for the gradient into the following back propagation, i.e., chain derivation:
5. A method for improving the GRU back propagation speed according to claim 4, characterized in that: In step S2, the calculation formula of the weight gradient is: Let the gradient of W_i be d_W_i, and the gradient of W_i calculated at each moment index be d_w_i_tmp; Let the gradient of W_h be d_W_h, and the gradient of W_h calculated at each moment index be d_w_h_tmp; d_w_i_tmp=d_gate_i*x_t; d_w_h_tmp=d_gate_h*h.
