Method and device for processing vector operator based on Tensor Core and computer equipment

Abstract

The invention relates to a Tensor Core-based vector operator processing method and device and computer equipment, and relates to the technical field of artificial intelligence, and the method comprises the steps: converting a vector operator into an N * M to-be-operated matrix when the vector operator is received and the length of the vector operator is determined to be integer multiples of a preset value; n and M are integral multiples of k, and the preset value is k2; calling a matching operation matrix of which the number of elements is integer multiples of a preset value based on an operation demand of a vector operator; performing left multiplication on the matching operation matrix and the to-be-operated matrix to obtain a to-be-determined matrix Performing left multiplication on the to-be-determined matrix to match the operation matrix to obtain a target matrix, and determining any element in the target matrix as a result value of the operation demand of the vector operator; the vector operator to be operated is simply transformed, so that the Tensor Core performs operation processing on the non-matrix operation, and the utilization rate of GPU resources is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, apparatus and computer device for processing vector operators based on Tensor Core. Background Art

[0002] Tensor Core, or TC for short, is a custom hardware unit within a GPU (Graphic Processing Unit) that is specifically used to accelerate matrix multiplication operations. It is essentially an Application Specific Integrated Circuit (ASIC). Each Tensor Core can perform mixed-precision matrix multiplication operations of a specific size within one instruction. CUDA Core (a processing unit within the GPU) is capable of general-purpose computing, but its versatility limits its performance in specific calculations. Tensor Core is a computing unit that has implemented hardware-level optimization for matrix multiplication operations. Its matrix computing capabilities are far greater than those of CUDA Core, which is particularly critical for the training and reasoning of neural networks for AI (Artificial Intelligence) models.

[0003] However, since Tensor Core is specially designed to support only matrix multiplication operations, it cannot support other operations, that is, Tensor Core cannot directly process non-matrix operations. Summary of the invention

[0004] Based on this, it is necessary to provide a method, device, computer equipment, computer-readable storage medium and computer program product based on Tensor Core processing of vector operators, which can realize non-matrix operation processing through Tensor Core in order to solve the above technical problems.

[0005] In a first aspect, the present application provides a method for processing vector operators based on Tensor Cores, the method comprising:

[0006] When a vector operator is received and it is determined that the length of the vector operator is an integer multiple of a preset value, the vector operator is converted into an N×M matrix to be operated; wherein N and M are both integer multiples of k, and the preset value is k 2 ;

[0007] Based on the operation requirements of the vector operator, calling a matching operation matrix whose number of elements is an integer multiple of the preset value;

[0008] Multiply the matching operation matrix by the matrix to be operated to obtain a matrix to be determined;

[0009] The matrix to be determined is multiplied on the left by the matching operation matrix to obtain a target matrix, and any element in the target matrix is ​​determined to be a result value of the operation requirement of the vector operator.

[0010] In one of the embodiments, after receiving the vector operator, the method further comprises:

[0011] When it is determined that the length of the vector operator is not an integer multiple of a preset value, the vector operator is padded to a length that is an integer multiple of the preset value based on an element value of 0.

[0012] In one embodiment, the calling matching operation matrix whose number of elements is an integer multiple of the preset value includes:

[0013] The matching operation matrix is ​​called, and when it is determined that the number of elements of the matching operation matrix is ​​not an integer multiple of the preset value, the number of elements of the matching operation matrix is ​​filled to an integer multiple of the preset value based on the element value 0.

[0014] In one embodiment, calling a matching operation matrix whose number of elements is an integer multiple of the preset value based on the operation requirements of the vector operator includes:

[0015] When the operation requirement of the vector operator is summation, a k×k auxiliary matrix with each element value being 1 is called as a matching operation matrix.

[0016] In one of the embodiments, after receiving the vector operator, the method further comprises:

[0017] When it is determined that the vector operator is not a reduce-type vector operator, the vector operator is converted into a reduce-type vector operator.

[0018] In one of the embodiments, the operation process of calculating the target matrix reuses the register space of the operation process of calculating the matrix to be determined.

[0019] In a second aspect, the present application further provides a device for processing vector operators based on Tensor Cores, the device comprising:

[0020] A data conversion module is used to convert the vector operator into an N×M matrix to be operated when the vector operator is received and the length of the vector operator is determined to be an integer multiple of a preset value; wherein N and M are both integer multiples of k, and the preset value is k 2 ;

[0021] A matrix calling module, used for calling a matching operation matrix whose number of elements is an integer multiple of the preset value based on the operation requirements of the vector operator;

[0022] A matrix processing module, used for left-multiplying the matching operation matrix by the matrix to be operated to obtain a matrix to be determined;

[0023] The result output module is used to multiply the matrix to be determined by the matching operation matrix to obtain a target matrix, and determine that any element in the target matrix is ​​a result value of the operation requirement of the vector operator.

[0024] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0025] When a vector operator is received and it is determined that the length of the vector operator is an integer multiple of a preset value, the vector operator is converted into an N×M matrix to be operated; wherein N and M are both integer multiples of k, and the preset value is k 2 ;

[0026] Based on the operation requirements of the vector operator, calling a matching operation matrix whose number of elements is an integer multiple of the preset value;

[0027] Multiply the matching operation matrix by the matrix to be operated to obtain a matrix to be determined;

[0028] The matrix to be determined is multiplied on the left by the matching operation matrix to obtain a target matrix, and any element in the target matrix is ​​determined to be a result value of the operation requirement of the vector operator.

[0029] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0030] When a vector operator is received and it is determined that the length of the vector operator is an integer multiple of a preset value, the vector operator is converted into an N×M matrix to be operated; wherein N and M are both integer multiples of k, and the preset value is k 2 ;

[0031] Based on the operation requirements of the vector operator, calling a matching operation matrix whose number of elements is an integer multiple of the preset value;

[0032] Multiply the matching operation matrix by the matrix to be operated to obtain a matrix to be determined;

[0033] The matrix to be determined is multiplied on the left by the matching operation matrix to obtain a target matrix, and any element in the target matrix is ​​determined to be a result value of the operation requirement of the vector operator.

[0034] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0035] When a vector operator is received and it is determined that the length of the vector operator is an integer multiple of a preset value, the vector operator is converted into an N×M matrix to be operated; wherein N and M are both integer multiples of k, and the preset value is k 2 ;

[0036] Based on the operation requirements of the vector operator, calling a matching operation matrix whose number of elements is an integer multiple of the preset value;

[0037] Multiply the matching operation matrix by the matrix to be operated to obtain a matrix to be determined;

[0038] The matrix to be determined is multiplied on the left by the matching operation matrix to obtain a target matrix, and any element in the target matrix is ​​determined to be a result value of the operation requirement of the vector operator.

[0039] In the method, apparatus, computer device, computer-readable storage medium and computer program product for processing vector operators based on Tensor Core provided in the present application, the method for processing vector operators based on Tensor Core, by receiving the vector operator and determining that the length of the vector operator is an integer multiple of a preset value, converting the vector operator into an N×M matrix to be operated; wherein N and M are both integer multiples of k, and the preset value is k 2 ; Based on the operation requirements of the vector operator, call a matching operation matrix whose number of elements is an integer multiple of the preset value; multiply the matching operation matrix by the matrix to be operated on the left to obtain the matrix to be determined; multiply the matrix to be determined on the left by the matching operation matrix to obtain the target matrix, and determine that any element in the target matrix is ​​the result value of the operation requirements of the vector operator; It can be seen that this application realizes the Tensor Core to perform operation processing on non-matrix operations through a simple transformation of the vector operator to be operated, which is beneficial to improve the utilization rate of GPU resources and improve the overall performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 A schematic diagram of a Tensor Core unit in a GPU in the related art;

[0042] Figure 2 A schematic diagram of the MMA operation of the Tensor Core unit in the related art;

[0043] Figure 3 A schematic diagram of a process of processing vector operators based on Tensor Cores in one embodiment;

[0044] Figure 4 It is a schematic diagram of the acceleration effect of the Softmax operator based on Tensor Core acceleration in one embodiment;

[0045] Figure 5 It is a schematic diagram of the acceleration effect of the Softmax operator based on CUDA Core acceleration in one embodiment;

[0046] Figure 6 is a structural block diagram of a device for processing vector operators based on Tensor Cores in one embodiment;

[0047] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0049] In the modern computing environment, graphics processing units (GPUs) are widely used in scientific computing, artificial intelligence, image processing and other fields due to their powerful parallel computing capabilities. In typical AI (Artificial Intelligence) models, such as the ResNet (residual neural network) series for vision, the YOLO (a real-time object detection system) series for detection, and the NLPBert (a natural language processing model) series, matrix operations account for more than 90% of the operations. In modern large-model scenarios, this ratio is usually higher.

[0050] Please refer to Figure 1 , although CUDA Core already has powerful computing power, the number of CUDA Cores contained in the most advanced NVIDIA GPUs has reached tens of thousands (for example, NVIDIA L40S GPU contains 18176 CUDA Cores). Usually, we call the traditional multi-core CPU (Central Processing Unit) architecture a multi-core architecture, and GPUs are called many-core architectures because of their large number of CUDA Cores. The number of CUDA Cores usually indicates the computing power of the graphics card at fp32 data precision; it needs to be explained that float, a single-precision data type, occupies 4 bytes in the x86 platform memory. Because each byte in the computer occupies 8 bits, the float data type occupies 32 bits in the computer, so the float type is also called fp32, which is the abbreviation of float point 32bit. Simply put, the more CUDA Cores a GPU has, the stronger its single-precision (float or fp32) computing power is (of course, it is also related to the GPU main frequency). The versatility of CUDA Core and its large number make it have incomparable advantages over traditional CPUs in typical parallel computing scenarios. Figure 1 Instruction Cache is an instruction cache, warp scheduler is a warp scheduler, DispatchUnit is a dispatch unit, Register File is a register file, LD in "LD / ST" is load, ST is store, which is a read unit or arithmetic unit.

[0051] As mentioned earlier, since matrix operations account for more than 90% of the total computing power in AI models, NVIDIA has introduced a custom circuit specifically designed to accelerate matrix multiplication operations in GPU chips starting with the Volta architecture. This circuit is called the first-generation tensor core, which further accelerates deep learning applications and supports it from the programming model level starting with CUDA 9.0. With the latest Hopper architecture (super chip architecture), it has evolved into the fourth-generation Tensor Core. Each Tensor Core is an independent hardware unit that can perform 4x4x4 matrix multiplication and addition (Matrix Multiply and Accumulate, MMA) operations in a single instruction. Using CUDA Core requires multiple calls to the FMA (Fused Multiply–accumulate operation) instruction in a loop, such as Figure 2 shown.

[0052] Among them, the 4x4x4 matrix multiplication and accumulation operation is . TC (Tensor Core) plays an important role in deep learning applications such as autonomous driving, camera processing, real-time anomaly detection, as well as traditional linear algebra operations such as matrix multiplication of large matrices. The actual GPU Tensor Core performance can be an order of magnitude faster than the regular FP32 / INT32 GPU CUDA Core. However, it is not easy to adapt arbitrary algorithms to Tensor Core, because Tensor Core is different from the regular GPU CUDA Core core, which can execute the entire instruction set, and TC can only perform one operation (MMA), but much faster.

[0053] Although matrix operations account for a very high proportion of operations in AI models and are hardware accelerated by introducing TC in NVIDIA GPUs, the non-matrix operations in AI models cannot be ignored, especially in reasoning scenarios that pursue extreme performance.

[0054] The method for processing vector operators based on Tensor Core provided in the embodiment of the present application can be applied to a terminal, and the GPU of the terminal includes a customized hardware unit Tensor Core specifically used to accelerate matrix multiplication operations. Among them, the terminal can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc.

[0055] In order to clearly understand the present application, some nouns related to the present application are explained.

[0056] Graphics Processing Unit (GPU): It is a computer hardware specifically used for processing graphics and images. It accelerates graphics rendering, video editing and gaming experience by processing large amounts of data in parallel. GPU not only plays an important role in the gaming field, but also demonstrates powerful computing capabilities in scientific computing, artificial intelligence and deep learning. With the development of technology, the performance of GPU continues to improve, providing strong support for various high-performance computing tasks. Currently, GPU has become the standard hardware infrastructure for AI training and reasoning.

[0057] Compute Unified Device Architecture (CUDA) is a parallel computing platform and programming model. It allows developers to use NVIDIA's GPUs for general computing, not just graphics and image processing. CUDA enables ordinary C / C++ developers to use the powerful parallel processing capabilities of GPUs to accelerate compute-intensive tasks such as scientific simulations, deep learning, data analysis, etc. by providing a complete set of tools and libraries. CUDA supports multiple programming languages, including C, C++, Python, and Fortran (several programming languages), and is compatible with widely used development environments and compilers. With CUDA, developers can write efficient parallel code to make full use of the GPU's thousands of computing cores and significantly improve the performance of applications.

[0058] CUDA Core, or CC for short, is a key component of NVIDIA GPUs for high-performance computing. It provides users with powerful computing capabilities, enables complex computing tasks to be completed in a short period of time, and greatly promotes technological progress and innovation in various industries. The computing power of each CUDA Core in NVIDIA GPU is far less than that of a single CPU core, but due to its lightweight characteristics, a single NVIDIA GPU can carry tens of thousands of CUDA Cores. For typical parallel computing tasks such as image algorithms and deep learning, its extremely high degree of parallelism can fully utilize CUDA Core to improve computing efficiency. Compared with traditional CPU cores, it can improve computing efficiency by dozens, hundreds, or even thousands of times in such computing tasks.

[0059] Tensor Core, or TC for short, is a custom hardware unit in NVIDIA GPUs that is specifically used to accelerate matrix multiplication operations. It is essentially an application-specific integrated circuit (ASIC). Introduced in the Volta architecture GPU, each Tensor Core can perform mixed-precision matrix multiplication operations of a specific size in one instruction. CUDA Core is capable of general-purpose computing, but its versatility limits its performance in specific computing. Tensor Core is a computing unit that implements hardware-level optimization for matrix multiplication operations. Its matrix computing capabilities are far greater than those of CUDA Core, which is particularly critical for the training and reasoning of neural networks for AI models. This is because the operation that accounts for the largest proportion of computation in neural networks is matrix multiplication. Tensor Core significantly improves the performance and efficiency of deep learning models by optimizing matrix multiplication and accumulation operations. Tensor Core has become one of the core technologies that NVIDIA GPUs use to maintain their leading position in the field of AI computing, and it continues to push the boundaries of artificial intelligence technology.

[0060] Vector operator: AI models are built up by layers of operators. After the tensor data is calculated by each layer of operators, it flows into the next layer to continue calculating until the last layer outputs. As the name of the famous deep learning training framework TenorFlow (tensor flow), AI calculation is essentially the flow of tensors between operator layers. Therefore, the calculation process of AI models is usually represented by graphs. AI models contain a wide variety of operators, such as convolution, activation (ReLU, GeLU, Tanh), normalization (BatchNorm, Layer Norm, Group Norm), MHA (Multi Head Attention), etc. From the perspective of the ratio of the amount of calculation to the amount of data, operators can be divided into two types: computationally intensive and memory-intensive. Computationally intensive operators are operators that involve a much larger number of operations than the amount of data. Typical computationally intensive operators such as matrix multiplication and convolution usually have an order of magnitude in terms of the amount of calculation, while the number of operations is usually an order of magnitude. Typical memory-intensive operators include activation functions, normalization, Softmax (an activation function), etc. Memory-intensive operators are usually vector operators. The so-called vector operator is opposite to the matrix operator, or a pure element-wise calculation, that is, the operator operation only involves the correlation of a single element and does not involve the interaction between elements, such as the ReLU operator, or includes simple reduce (reduce sum, reduce max / min, etc.) interactions after the element-wise operation. Among them, element-wise operation means mathematical operations on elements in corresponding positions between vectors, matrices or tensors, such as addition, subtraction, multiplication and division. Among them, sum is for summation, Max is for maximum value, and Min is for minimum value.

[0061] In an exemplary embodiment, Figure 3 As shown, a method for processing vector operators based on Tensor Core is provided, and the method is applied to a terminal as an example for explanation, including the following steps 202 to 208. Among them:

[0062] Step 202: upon receiving a vector operator and determining that the length of the vector operator is an integer multiple of a preset value, convert the vector operator into an N×M matrix to be operated; wherein N and M are both integer multiples of k, and the preset value is k. 2 .

[0063] Among them, the vector operators in the present application may be non-matrix operations, such as vector reduce operators; wherein the reduce operators are used to perform specific calculations (such as sum, Max / Min) on a certain dimension of the input multi-dimensional tensor data, thereby achieving the purpose of reducing the dimension.

[0064] The N×M matrix to be calculated is also a matrix operation, which can be processed by Tensor Core.

[0065] For the preset value, the present application provides an optional embodiment of 16, and the corresponding k is 4.

[0066] Exemplarily, when a vector operator for a non-matrix operation is received, the present application needs to convert the vector operator into one that can be processed by the Tensor Core. At this time, the length of the received vector operator can be determined first to confirm whether the length of the vector operator is an integer multiple of the preset value of 16; when the length of the vector operator is an integer multiple of the preset value, the view of the vector operator is converted into an N×M matrix to be operated, for example, a 4×4 matrix to be operated.

[0067] Step 204: Based on the operation requirements of the vector operator, call a matching operation matrix whose number of elements is an integer multiple of a preset value.

[0068] Among them, the matching operation matrix can be an operation matrix that has an operation relationship with the vector operator, that is, the vector operator itself must operate with another operation matrix to obtain relevant operation results; in addition, the matching operation matrix can also be an operation matrix that is preset in advance based on the operation requirements of the vector operator in order to realize the related operations of the vector operator.

[0069] Exemplarily, based on the operational requirements of the vector operator, the matching operation matrix associated with the vector operator is called; for the matching operation matrix here, the present application provides an optional implementation method of calling a matching operation matrix whose number of elements is an integer multiple of a preset value, for example, calling a matching operation matrix whose number of elements is 16, that is, a 4×4 matching operation matrix.

[0070] Step 206, multiply the matching operation matrix by the matrix to be operated on the left to obtain the matrix to be determined.

[0071] Exemplarily, after obtaining the matrix to be operated corresponding to the vector operator and the matching operation matrix to be operated, the matching operation matrix is ​​multiplied by the matrix to be operated on the left, thereby obtaining the matrix to be determined.

[0072] Step 208, multiply the matrix to be determined by the matching operation matrix on the left to obtain a target matrix, and determine that any element in the target matrix is ​​a result value required by the operation of the vector operator.

[0073] Exemplarily, the matrix to be determined is then multiplied on the left by the matching operation matrix to obtain a target matrix associated with the vector operator, that is, the operation result obtained by the vector operator based on the Tensor Core, and then any element in the target matrix is ​​determined to be the result value of the operation requirement of the vector operator, for example, the first element in the target matrix is ​​determined to be the result value of the operation requirement of the vector operator.

[0074] It should be noted that the above steps 202 to 208 specifically implement the relevant operations through the Tensor Core of the GPU in the terminal.

[0075] In the above-mentioned method for processing vector operators based on Tensor Core, when a vector operator is received and it is determined that the length of the vector operator is an integer multiple of a preset value, the vector operator is converted into an N×M matrix to be operated; wherein N and M are both integer multiples of k, and the preset value is k 2 ; Based on the operation requirements of the vector operator, call a matching operation matrix whose number of elements is an integer multiple of the preset value; multiply the matching operation matrix by the matrix to be operated on the left to obtain the matrix to be determined; multiply the matrix to be determined on the left by the matching operation matrix to obtain the target matrix, and determine that any element in the target matrix is ​​the result value of the operation requirements of the vector operator; It can be seen that this application realizes the Tensor Core to perform operation processing on non-matrix operations through a simple transformation of the vector operator to be operated, which is beneficial to improve the utilization rate of GPU resources and improve the overall performance of the system.

[0076] In an exemplary embodiment, after receiving the vector operator, the method further includes: when it is determined that the length of the vector operator is not an integer multiple of a preset value, padding the vector operator to a length that is an integer multiple of the preset value based on the element value 0.

[0077] Exemplarily, for the vector operator received in step 202, when confirming its length, there may be a situation where the length is less than the preset value, or is not an integer multiple of the preset value. In order to make the matrix to be operated obtained by the subsequent conversion of the vector operator a 4×4 matrix to be operated, the present application provides an optional implementation method, in which, when it is determined that the length of the vector operator is not an integer multiple of the preset value, the vector operator can be padded to a length that is an integer multiple of the preset value based on the element value 0. For example, when the length of the vector operator is 14, 2 element values ​​0 are used to padded the vector operator to a length of 16.

[0078] By padding the length of the vector operator to an integer multiple of the preset value, it is helpful to make the matrix to be operated obtained by the subsequent conversion of the vector operator a 4×4 matrix to be operated, that is, a matrix that can be operated by Tensor Core.

[0079] It should also be added that, it is also possible to first convert the vector operator whose length is less than an integer multiple of the preset value into an N×M matrix to be operated, and then use the element value 0 to complete the missing elements in the N×M matrix to be operated.

[0080] However, the present application provides a method of first completing the length of the vector operator to an integer multiple of a preset value, and then converting the vector operator to a matrix to be operated, which can save more computing power requirements.

[0081] In an exemplary embodiment, calling a matching operation matrix whose number of elements is an integer multiple of a preset value includes: calling the matching operation matrix, and when it is determined that the number of elements of the matching operation matrix is ​​not an integer multiple of the preset value, padding the number of elements of the matching operation matrix to an integer multiple of the preset value based on the element value 0.

[0082] Exemplarily, for the matching operation matrix whose number of elements is an integer multiple of a preset value executed in the above-mentioned step 204, the present application provides an optional implementation method of calling a matching operation matrix, for example, a matrix that has been provided to be operated with a vector operator, and then determining the number of elements in the matching operation matrix to determine whether the number of elements in the matching operation matrix is ​​an integer multiple of a preset value; in the case where it is determined that the number of elements of the matching operation matrix is ​​not an integer multiple of the preset value, it is also possible to choose to fill the number of elements of the matching operation matrix to an integer multiple of the preset value based on the element value 0, so as to obtain a matrix (matching operation matrix) that can be operated on by Tensor Core.

[0083] In an exemplary embodiment, based on the operation requirements of the vector operator, a matching operation matrix whose number of elements is an integer multiple of a preset value is called, including: when the operation requirement of the vector operator is summation, calling a k×k auxiliary matrix with each element value being 1 as the matching operation matrix.

[0084] Exemplarily, for the content executed in the above-mentioned step 204, the present application also provides an optional implementation method. When the operation requirement of the vector operator is summation, that is, no matrix to be operated with the vector operator is originally provided, at this time, a pre-set k×k auxiliary matrix can be called as a matching operation matrix, and the value of each element in the auxiliary matrix is ​​1, for example, it can be a 4×4 auxiliary matrix whose element values ​​are all 1.

[0085] The auxiliary matrix enables the Tensor Core to perform the sum operation on the matrix to be operated after the vector operator is converted, and will not affect the result value after the sum operation of the vector operator, so as to ensure the accuracy of the result value obtained after the operation.

[0086] In an exemplary embodiment, after receiving the vector operator, the method further includes: if it is determined that the vector operator is not a reduce-type vector operator, converting the vector operator into a reduce-type vector operator.

[0087] Exemplarily, after receiving the vector operator executed in the above step 202, the present application also provides an optional implementation method: when it is determined that the vector operator is not a reduce-type vector operator, the vector operator can be first converted into a reduce-type vector operator, or at least part of the vector operators in the operation requirements can be converted into reduce-type vector operators; the vector operator converted into a reduce-type vector operator can perform related operations based on the method for processing vector operators based on Tensor Cores provided in the present application.

[0088] In an exemplary embodiment, the operation process of calculating the target matrix reuses the register space of the operation process of calculating the matrix to be determined. That is, an optional implementation provided by the present application is that the register space used in the operation process of calculating the target matrix can be the same as the register space used in the operation process of calculating the matrix to be determined, so as to avoid additional memory overhead.

[0089] From the above, it can be seen that this application transforms non-matrix operations into matrix operations through simple transformations, and uses TC to accelerate the operation, so that single operators and fused operators are suitable for training and reasoning scenarios respectively. The practice of the applicant has shown that vector operators optimized by TC have a performance improvement of 1 to 3 times. Specifically, the vector reduce operator can be converted into a matrix multiplication operation through a simple mathematical transformation to accelerate it using Tensor Core. This method can realize both single operator and fused operator versions, and is applied to the acceleration of AI model training and reasoning.

[0090] The technical effects achieved by this application include at least the following three points.

[0091] First, it squeezes GPU performance to the extreme: TC only supports matrix multiplication operations, and non-matrix multiplication operations can only be performed on CC. This application converts some vector operators into matrix operations through simple numerical transformations with negligible cost, and uses TC's powerful matrix operation capabilities to accelerate the operation of vector operators and improve the hardware resource utilization of the GPU.

[0092] Second, CC and TC hybrid programming: CC and TC are independent hardware units, and their collaborative work is a major research direction in academia and industry, with the goal of improving the overall resource utilization of GPUs. The transformation strategy proposed in this application can directly use TC to accelerate pure vector reduce operators, and can use CC and TC for hybrid programming of vector reduce operator variants, making full use of their respective advantages.

[0093] Third, improve AI training and reasoning performance: The performance of a single operator directly affects the time consumption and performance throughput of model training and reasoning. The performance of the AI ​​model depends not only on the performance parameter targets at the hardware level, but also on the ultimate optimization of hardware performance by efficient operators implemented by high-performance computing engineers. For training, especially large model training scenarios, efficient operators can greatly save training time, and less training time can also reduce the probability of hardware failure. For reasoning scenarios, better performance can save reasoning latency and improve service throughput, provide a better user experience and support a higher number of concurrent users.

[0094] This application solves the problem that non-matrix operators can only operate on CUDA Core. Through a simple method, some non-matrix operators can also use the powerful computing power of Tensor Core, thereby improving the utilization rate of GPU resources and improving the overall performance of the system.

[0095] For the method of processing vector operators based on Tensor Core provided in this application, an optional implementation is also provided here. Taking the float type vector summation operator with a length of 16 as an example, the specific method steps include steps S1 to S5, wherein:

[0096] Step S1, ReView: Assume that the vector to be summed is , change its view to treat it as a 4x4 matrix fragment, i.e. a fragment in MMA: In the field of AI, view only changes the view, not the actual memory layout of the tensor data, so there is no performance loss of additional operations.

[0097] Step S2, initialize a 4x4 auxiliary matrix b with all 1.0 through the Tensor Core MMA instruction fill_fragment: In actual encoding, since the value of b is fixed to 1.0, there is no need to declare data b of the same size as vector a. A single warp can share one fragment, so the memory overhead is negligible.

[0098] Step S3, multiply the left side of matrix b by matrix a to obtain the following matrix c.

[0099]

[0100] Step S4, use matrix c to multiply matrix b on the left to obtain the following matrix d.

[0101]

[0102] The calculation of matrix d can reuse the register space used for the calculation of matrix c, that is, c=c*b, without the need to declare the use of different register spaces separately, thus avoiding additional memory overhead.

[0103] Step S5, take out any value in the matrix d (usually the first one), which is the final calculation result.

[0104] From the above steps, we can find that by using Tensor Core and using two matrix multiplication instructions, namely steps S3 and S4, the sum operation of 16 elements is realized. If the shuffle instruction such as shfl_xor_sync is used in CC warp, 4 instruction calls are required. Compared with at least TC calculation, the number of instruction calls is reduced by half. If we consider the half type or int8 type, the multiplication of larger fragments can be performed in a single MMA instruction, which can save more instruction calls compared to the shuffle instruction of CC.

[0105] In AI algorithms, the most obvious typical reduce operator variants similar to this example are vector inner product and vector modulus. You only need to replace the above b vector with a real vector and then use the shuffle instruction on the diagonal elements of c. In addition, various normalization operators (Batch Norm, Layer Norm, RMSNorm, etc.), Softmax operators, etc., only need to make slight changes to the above transformations and combine them with CUDA Core for hybrid implementation, so that they can be extended to more scenarios.

[0106] like Figure 4 and Figure 5 As shown respectively, an acceleration effect of a Softmax operator based on TC acceleration and an acceleration effect of CUDA Core are provided for reference.

[0107] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0108] Based on the same inventive concept, the embodiment of the present application also provides a device for processing vector operators based on Tensor Cores for implementing the method for processing vector operators based on Tensor Cores involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of one or more devices for processing vector operators based on Tensor Cores provided below can be found in the limitations of the method for processing vector operators based on Tensor Cores above, and will not be repeated here.

[0109] In an exemplary embodiment, Figure 6 As shown, a device 800 for processing vector operators based on Tensor Core is provided, comprising: a data conversion module 81, a matrix calling module 82, a matrix processing module 83 and a result output module 84, wherein:

[0110] The data conversion module 81 is used to convert the vector operator into an N×M matrix to be operated when the vector operator is received and the length of the vector operator is determined to be an integer multiple of a preset value; wherein N and M are both integer multiples of k, and the preset value is k 2 ;

[0111] A matrix calling module 82, for calling a matching operation matrix whose number of elements is an integer multiple of a preset value based on the operation requirements of the vector operator;

[0112] A matrix processing module 83 is used to multiply the matching operation matrix by the matrix to be operated to obtain a matrix to be determined;

[0113] The result output module 84 is used to multiply the matrix to be determined by the matching operation matrix on the left to obtain the target matrix, and determine any element in the target matrix as the result value of the operation requirement of the vector operator.

[0114] In an exemplary embodiment, the data conversion module 81 is also used to, after receiving the vector operator, complete the vector operator to an integer multiple of the preset value based on the element value 0 when it is determined that the length of the vector operator is not an integer multiple of the preset value.

[0115] In an exemplary embodiment, the matrix calling module 82 is used to call a matching operation matrix whose number of elements is an integer multiple of a preset value. Specifically, it can be used to call a matching operation matrix. When it is determined that the number of elements of the matching operation matrix is ​​not an integer multiple of the preset value, the number of elements of the matching operation matrix is ​​completed to an integer multiple of the preset value based on the element value 0.

[0116] In an exemplary embodiment, the matrix calling module 82 is used to call a matching operation matrix whose number of elements is an integer multiple of a preset value based on the operation requirements of the vector operator. Specifically, it can be used to call a k×k auxiliary matrix with each element value being 1 as a matching operation matrix when the operation requirement of the vector operator is summation.

[0117] In an exemplary embodiment, the data conversion module 81 is further used to, after receiving the vector operator, convert the vector operator into a reduce-type vector operator if it is determined that the vector operator is not a reduce-type vector operator.

[0118] In an exemplary embodiment, the operation process of calculating the target matrix reuses the register space of the operation process of calculating the matrix to be determined.

[0119] Each module in the above-mentioned device for processing vector operators based on Tensor Core can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0120] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 7As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for processing vector operators based on Tensor Core is implemented.

[0121] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0122] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0123] When a vector operator is received and it is determined that the length of the vector operator is an integer multiple of a preset value, the vector operator is converted into an N×M matrix to be operated; wherein N and M are both integer multiples of k, and the preset value is k 2 ;

[0124] Based on the computational requirements of the vector operator, a matching computation matrix whose number of elements is an integer multiple of a preset value is called;

[0125] Multiply the matching operation matrix by the matrix to be operated on the left to obtain the matrix to be determined;

[0126] The matrix to be determined is multiplied on the left by the matching operation matrix to obtain the target matrix, and any element in the target matrix is ​​determined to be the result value of the operation requirement of the vector operator.

[0127] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0128] When a vector operator is received and it is determined that the length of the vector operator is an integer multiple of a preset value, the vector operator is converted into an N×M matrix to be operated; wherein N and M are both integer multiples of k, and the preset value is k 2 ;

[0129] Based on the computational requirements of the vector operator, a matching computation matrix whose number of elements is an integer multiple of a preset value is called;

[0130] Multiply the matching operation matrix by the matrix to be operated on the left to obtain the matrix to be determined;

[0131] The matrix to be determined is multiplied on the left by the matching operation matrix to obtain the target matrix, and any element in the target matrix is ​​determined to be the result value of the operation requirement of the vector operator.

[0132] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0133] When a vector operator is received and it is determined that the length of the vector operator is an integer multiple of a preset value, the vector operator is converted into an N×M matrix to be operated; wherein N and M are both integer multiples of k, and the preset value is k 2 ;

[0134] Based on the computational requirements of the vector operator, a matching computation matrix whose number of elements is an integer multiple of a preset value is called;

[0135] Multiply the matching operation matrix by the matrix to be operated on the left to obtain the matrix to be determined;

[0136] The matrix to be determined is multiplied on the left by the matching operation matrix to obtain the target matrix, and any element in the target matrix is ​​determined to be the result value of the operation requirement of the vector operator.

[0137] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0138] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0139] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0140] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for processing vector operators based on Tensor Core, characterized in that: The method comprises: When a vector operator is received and it is determined that the length of the vector operator is an integer multiple of a preset value, the vector operator is converted into an N×M matrix to be operated; wherein N and M are both integer multiples of k, and the preset value is k 2 ; Based on the operation requirements of the vector operator, calling a matching operation matrix whose number of elements is an integer multiple of the preset value; Multiply the matching operation matrix by the matrix to be operated to obtain a matrix to be determined; The matrix to be determined is multiplied on the left by the matching operation matrix to obtain a target matrix, and any element in the target matrix is ​​determined to be a result value of the operation requirement of the vector operator.

2. The method according to claim 1, characterized in that After receiving the vector operator, the method further includes: When it is determined that the length of the vector operator is not an integer multiple of a preset value, the vector operator is padded to a length that is an integer multiple of the preset value based on an element value of 0.

3. The method according to claim 1, characterized in that The matching operation matrix whose number of calling elements is an integer multiple of the preset value includes: The matching operation matrix is ​​called, and when it is determined that the number of elements of the matching operation matrix is ​​not an integer multiple of the preset value, the number of elements of the matching operation matrix is ​​completed to an integer multiple of the preset value based on the element value 0.

4. The method according to claim 1, characterized in that: The step of calling a matching operation matrix whose number of elements is an integer multiple of the preset value based on the operation requirement of the vector operator includes: When the operation requirement of the vector operator is summation, a k×k auxiliary matrix with each element value being 1 is called as a matching operation matrix.

5. The method according to claim 1, characterized in that After receiving the vector operator, the method further includes: When it is determined that the vector operator is not a reduce-type vector operator, the vector operator is converted into a reduce-type vector operator.

6. The method according to any one of claims 1 to 5, characterized in that: The operation process of calculating the target matrix reuses the register space of the operation process of calculating the matrix to be determined.

7. A device for processing vector operators based on Tensor Cores, characterized in that: The device comprises: A data conversion module is used to convert the vector operator into an N×M matrix to be operated when the vector operator is received and the length of the vector operator is determined to be an integer multiple of a preset value; wherein N and M are both integer multiples of k, and the preset value is k 2 ; A matrix calling module, used for calling a matching operation matrix whose number of elements is an integer multiple of the preset value based on the operation requirements of the vector operator; A matrix processing module, used for left-multiplying the matching operation matrix by the matrix to be operated to obtain a matrix to be determined; The result output module is used to multiply the matrix to be determined by the matching operation matrix to obtain a target matrix, and determine that any element in the target matrix is ​​a result value of the operation requirement of the vector operator.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images