Transform architecture operator hybrid optimization method

Abstract

The invention discloses a Tranform architecture operator hybrid optimization method. The Tranform architecture operator hybrid optimization method comprises the following steps: step 1, operator mapping analysis and optimization demand identification; 2, the operation process of the operators is disassembled, and the calculation kernel of each operator is constructed by using OpenCL; and 3, taking an OpenCL kernel as a computational graph node, and constructing a computational graph by using OpenVX. According to the method, the OpenVX provides a powerful graphic processing optimization capability, a computational graph can be optimized into an efficient execution form, and the OpenCL accelerates the execution of operators by utilizing a multi-core parallel computing capability, so that the overall reasoning efficiency is improved.

Description

Technical Field

[0001] The present invention belongs to the field of AI accelerator applications and relates to a Transformer architecture operator hybrid optimization method. Background Art

[0002] As an advanced neural network structure, the Transformer architecture is widely used in natural language processing and other fields. However, its complex computational graph structure and large amount of computation have led to increasingly prominent problems of poor performance in the reasoning process. Summary of the invention

[0003] (I) Purpose of the invention

[0004] The purpose of the present invention is to provide a Transformer architecture operator hybrid optimization method, aiming to combine the characteristics of OpenVX and OpenCL to solve the technical challenges of operator mapping and optimization in the Transformer architecture.

[0005] (II) Technical solution

[0006] In order to solve the above technical problems, the present invention provides a Transformer architecture operator hybrid optimization method, which includes the following steps:

[0007] Step 1. Operator mapping analysis and optimization requirements identification;

[0008] Step 2. Disassemble the operation process of the operator and use OpenCL to build the calculation kernel of each operator respectively;

[0009] Step 3. Use OpenCL kernels as computational graph nodes and use OpenVX to build the computational graph.

[0010] Among them, in step 1, operator mapping analysis and optimization requirement identification include: analyzing the characteristics and computing requirements of each operator in the Transformer architecture model, and clarifying the operators that need to be mapped to OpenVX and OpenCL for hybrid optimization.

[0011] Among them, in step 2, the operators to be optimized in the Transformer architecture model include: position encoding, self-attention mechanism, fully connected layer, layer normalization, forward network, and residual connection.

[0012] Among them, in step 2, the position encoding operator kernel includes: position encoding matrix operation kernel; the self-attention mechanism operator kernel includes: linear change kernel, dot product operation kernel, scaling kernel, Softmax kernel, weighted sum kernel, multi-head splicing kernel; the fully connected layer operator kernel includes: matrix multiplication kernel, matrix addition kernel; the layer normalization operator kernel includes: mean kernel, variance kernel, normalization kernel, scaling and translation kernel; the forward network operator kernel includes: matrix multiplication kernel, matrix addition kernel, ReLU kernel; the residual connection operator kernel includes: residual connection kernel.

[0013] Among them, in step 2, the position coding matrix operation kernel is used to receive the pointer and dimension of the position coding matrix as parameters, calculate the sine and cosine values ​​of each position and each dimension in parallel, and generate the position coding.

[0014] Among them, in step 2, the linear transformation kernel is used to perform linear transformation on the input embedding to obtain the query vector Q, key vector K and value vector V; the dot product operation kernel is used to calculate the similarity between the query vector Q and the key vector K to obtain the attention score; the scaling kernel is used to scale the dot product operation result; the Softmax kernel is used to convert the scaling result into a normalized probability distribution value; the weighted summation kernel is used to apply the Softmax result to the value vector V for weighted summation; the multi-head splicing kernel is used to merge the calculation results of multiple self-attention heads to obtain the final multi-head attention output.

[0015] Among them, in step 2, the matrix multiplication kernel is used to perform the multiplication calculation of the weight matrix of the fully connected layer and the input vector; the matrix addition kernel is used to perform the addition calculation of the matrix multiplication result and the bias vector.

[0016] Among them, in step 2, the mean kernel is used to calculate the mean of the input tensor; the variance kernel is used to calculate the variance of the input tensor; the normalization kernel is used to normalize the input vector by the mean and variance; the scaling and translation kernel is used to apply scaling factors and offsets to the normalized output to enhance the expressiveness of the model.

[0017] Among them, in step 2, the matrix multiplication kernel is used to calculate the product of the input and the weight, and the product of the output after ReLU activation and the second set of weights; the matrix addition kernel is used to add a bias term after the matrix multiplication result; the ReLU kernel is used to apply the ReLU activation function; the residual connection kernel is used to add the output of a sublayer to the input of the layer to solve the gradient vanishing and gradient exploding problems of deep networks.

[0018] Wherein, step 3 includes the following sub-steps:

[0019] 3.1 Write the OpenVX computational graph description, including input, output, and nodes;

[0020] 3.2 Convert OpenCL kernel to OpenVX node;

[0021] 3.3 Build the OpenVX computation graph, verify and run the computation graph.

[0022] (III) Beneficial effects

[0023] The above technical solution provides a hybrid optimization method for Transformer architecture operators. OpenVX provides powerful graphics processing optimization capabilities, which can optimize the computational graph into an efficient execution form, while OpenCL uses multi-core parallel computing capabilities to accelerate the execution of operators, thereby improving the overall reasoning efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 Schematic diagram of the method of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the drawings and examples.

[0026] Reference Figure 1 As shown, the Transformer architecture operator hybrid optimization method of this embodiment includes the following steps:

[0027] Step 1. Operator mapping analysis and optimization requirements identification;

[0028] Step 2. Disassemble the operation process of the operator and use OpenCL to build the calculation kernel of each operator;

[0029] Step 3. Use OpenCL kernels as computational graph nodes and use OpenVX to build the computational graph.

[0030] In step 1, operator mapping analysis and optimization requirement identification include: analyzing the characteristics and computing requirements of each operator in the Transformer architecture model, and identifying the operators that need to be mapped to OpenVX and OpenCL for hybrid optimization.

[0031] The operators to be optimized in the Transformer architecture model include: position encoding, self-attention mechanism, fully connected layer, layer normalization, forward network, and residual connection.

[0032] In step 2, the position encoding operator kernel includes: position encoding matrix operation kernel; the self-attention mechanism operator kernel includes: linear change kernel, dot product operation kernel, scaling kernel, Softmax kernel, weighted sum kernel, multi-head splicing kernel; the fully connected layer operator kernel includes: matrix multiplication kernel, matrix addition kernel; the layer normalization operator kernel includes: mean kernel, variance kernel, normalization kernel, scaling and translation kernel; the forward network operator kernel includes: matrix multiplication kernel, matrix addition kernel, ReLU kernel; the residual connection operator kernel includes: residual connection kernel.

[0033] The calculation kernel construction process of each operator is as follows:

[0034] 2.1 The position coding matrix operation kernel is used to receive the pointer and dimension of the position coding matrix as parameters, calculate the sine and cosine values ​​of each position and each dimension in parallel, and generate the position coding.

[0035] 2.2 Linear Transformation The kernel performs a linear transformation on the input embedding to obtain the query vector (Q), key vector (K), and value vector (V);

[0036] The dot product kernel is used to calculate the similarity between the query vector (Q) and the key vector (K) to obtain the attention score;

[0037] The scaling kernel is used to scale the result of the dot product operation;

[0038] The role of the Softmax kernel is to convert the scaling result into a normalized probability distribution value;

[0039] The weighted sum kernel applies the Softmax result to the value vector (V) for weighted summation;

[0040] The function of the multi-head concatenation kernel is to merge the calculation results of multiple self-attention heads to obtain the final multi-head attention output.

[0041] 2.3 The matrix multiplication kernel performs the multiplication of the weight matrix of the fully connected layer and the input vector;

[0042] The matrix addition kernel performs the addition calculation of the matrix multiplication result and the bias vector.

[0043] 2.4 The mean kernel is used to calculate the mean of the input tensor;

[0044] The variance kernel is used to calculate the variance of the input tensor;

[0045] The normalization kernel acts to normalize the input vector by the mean and variance;

[0046] The scaling and translation kernels are used to apply scaling factors and offsets to the normalized output to enhance the expressiveness of the model.

[0047] The 2.5 matrix multiplication kernel is used to calculate the product of the input and the weight, and the product of the output after ReLU activation and the second set of weights;

[0048] The matrix addition kernel adds a bias term after the matrix multiplication result;

[0049] The ReLU kernel is used to apply the ReLU activation function.

[0050] 2.6 The residual connection kernel is used to add the output of a sub-layer to the input of the layer, solving the gradient vanishing and gradient exploding problems of deep networks.

[0051] Step 3 includes the following sub-steps:

[0052] 3.1 Write the OpenVX computational graph description, including input, output and nodes;

[0053] 3.2 Convert OpenCL kernel to OpenVX node;

[0054] 3.3 Build the OpenVX computation graph, verify and run the computation graph.

[0055] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A Transformer architecture operator hybrid optimization method, characterized in that: The following steps are involved: Step 1. Operator mapping analysis and optimization requirements identification; Step 2. Disassemble the operation process of the operator and use OpenCL to build the calculation kernel of each operator respectively; Step 3. Use OpenCL kernels as computational graph nodes and use OpenVX to build the computational graph.

2. The Transformer architecture operator hybrid optimization method according to claim 1, characterized in that: In step 1, operator mapping analysis and optimization requirement identification include: analyzing the characteristics and computing requirements of each operator in the Transformer architecture model, and identifying the operators that need to be mapped to OpenVX and OpenCL for hybrid optimization.

3. The Transformer architecture operator hybrid optimization method according to claim 2, characterized in that: In step 2, the operators to be optimized in the Transformer architecture model include: position encoding, self-attention mechanism, fully connected layer, layer normalization, forward network, and residual connection.

4. The Transformer architecture operator hybrid optimization method according to claim 3, characterized in that: In step 2, the position encoding operator kernel includes: position encoding matrix operation kernel; the self-attention mechanism operator kernel includes: linear change kernel, dot product operation kernel, scaling kernel, Softmax kernel, weighted sum kernel, multi-head splicing kernel; the fully connected layer operator kernel includes: matrix multiplication kernel, matrix addition kernel; the layer normalization operator kernel includes: mean kernel, variance kernel, normalization kernel, scaling and translation kernel; the forward network operator kernel includes: matrix multiplication kernel, matrix addition kernel, ReLU kernel; the residual connection operator kernel includes: residual connection kernel.

5. The Transformer architecture operator hybrid optimization method according to claim 4, characterized in that: In step 2, the position coding matrix operation kernel is used to receive the pointer and dimension of the position coding matrix as parameters, calculate the sine and cosine values ​​of each position and each dimension in parallel, and generate the position coding.

6. The Transformer architecture operator hybrid optimization method according to claim 5, characterized in that: In step 2, the linear transformation kernel performs a linear transformation on the input embedding to obtain the query vector Q, the key vector K, and the value vector V; the dot product kernel calculates the similarity between the query vector Q and the key vector K to obtain the attention score; The function of the scaling kernel is to scale the result of the dot product operation; the function of the Softmax kernel is to convert the scaling result into a normalized probability distribution value; the function of the weighted summation kernel is to apply the Softmax result to the value vector V for weighted summation; the function of the multi-head splicing kernel is to merge the calculation results of multiple self-attention heads to obtain the final multi-head attention output.

7. The Transformer architecture operator hybrid optimization method according to claim 6, characterized in that: In step 2, the matrix multiplication kernel performs the multiplication calculation of the weight matrix of the fully connected layer and the input vector; the matrix addition kernel performs the addition calculation of the matrix multiplication result and the bias vector.

8. The Transformer architecture operator hybrid optimization method according to claim 7, characterized in that: In step 2, the mean kernel is used to calculate the mean of the input tensor; the variance kernel is used to calculate the variance of the input tensor; The normalization kernel acts to normalize the input vector by the mean and variance; The scaling and translation kernels are used to apply scaling factors and offsets to the normalized output to enhance the expressiveness of the model.

9. The Transformer architecture operator hybrid optimization method according to claim 8, characterized in that: In step 2, the matrix multiplication kernel calculates the product of the input and the weight, and calculates the product of the output after ReLU activation and the second set of weights; the matrix addition kernel adds a bias term after the matrix multiplication result; The function of the ReLU kernel is to apply the ReLU activation function; the function of the residual connection kernel is to add the output of a sub-layer to the input of this layer to solve the gradient vanishing and gradient exploding problems of deep networks.

10. The Transformer architecture operator hybrid optimization method according to claim 9, characterized in that: Step 3 includes the following sub-steps: 3.1 Write the OpenVX computational graph description, including input, output and nodes; 3.2 Convert OpenCL kernel to OpenVX node; 3.3 Build the OpenVX computation graph, verify and run the computation graph.

Images