Method for executing convolution operation in vector operation unit, computing equipment, computer readable storage medium and computer program product

Abstract

The invention provides a method for executing convolution operation in a vector operation unit, computing equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: loading a first matrix of convolution operation to a thread local register; loading the second matrix of the convolution operation to a thread bundle scalar register of the vector operation unit through the thread local register; multiplying and adding two elements of one sub-matrix of the first matrix read from the thread local register and two corresponding elements of the second matrix read from the thread bundle scalar register through a multiply-add instruction to obtain a result of the multiply-add instruction, wherein the multiply-add instruction is used for multiplying and adding two elements of one sub-matrix of the first matrix read from the thread local register and two corresponding elements of the second matrix read from the thread bundle scalar register; and adding a result of the multiply-add instruction and a product of a third element of the sub-matrix read from the thread local register and a third corresponding element of the second matrix read from the thread bundle scalar register to obtain one element of a result matrix of the convolution operation.

Description

Technical Field

[0001] The present disclosure generally relates to the field of processors, and more particularly, to a method, a computing device, a computer-readable storage medium, and a computer program product for performing a convolution operation in a vector operation unit. Background Art

[0002] Convolution is a common mathematical operation that is frequently used in artificial intelligence or neural networks. Artificial intelligence chips need to perform a large number of convolution operations when performing model training and inference. Typically, in artificial intelligence chips, a computing unit called a tensor computing core or tensor computing unit is used to perform convolution operations. However, in some cases, such as when the matrix size of the convolution operation is small or the convolution operation occurs less frequently, using tensor computing cores to perform convolution operations may result in a waste of computing power. In addition, during the execution of the convolution operation, the two input matrices participating in the convolution operation may need to be repeatedly loaded into the register, so that data loading is limited by the register bandwidth. Summary of the Invention

[0003] To address the above issues, the present disclosure provides a solution for using a vector operation unit to perform convolution operations. This solution can, for example, avoid wasting the computing power of tensor cores by using the vector operation unit to perform convolution operations when the matrix size of the convolution operation is small or the convolution operation occurs infrequently. Furthermore, the present disclosure mitigates register bandwidth limitations by designing the storage location of at least one input matrix for the convolution operation to avoid repeated loading of the input matrix.

[0004] According to one aspect of the present disclosure, a method for performing a convolution operation in a vector operation unit is provided. The method includes: loading a first matrix of the convolution operation into a thread-local register; loading a second matrix of the convolution operation into a warp scalar register of the vector operation unit via the thread-local register; multiplying two elements of a submatrix of the first matrix read from the thread-local register and two corresponding elements of the second matrix read from the warp scalar register by a multiply-add instruction to obtain a result of the multiply-add instruction; and adding the result of the multiply-add instruction to the product of a third element of the submatrix read from the thread-local register and a third corresponding element of the second matrix read from the warp scalar register to obtain an element of a result matrix of the convolution operation.

[0005] In some implementations, loading the second matrix of the convolution operation into the warp scalar register of the vector operation unit via the thread local register includes: loading the second matrix of the convolution operation into the thread local register; and transferring each element of the second matrix to the warp scalar register through a register transfer instruction.

[0006] In some implementations, obtaining an element of the result matrix of the convolution operation includes: storing the result of the multiply-add instruction in a shutdown register; and in the vector operation unit, performing a mixed multiplication on the third element of the submatrix of the first matrix, the third corresponding element of the second matrix, and the result of the multiply-add instruction by a fused multiply-add instruction to obtain an element of the result matrix of the convolution operation.

[0007] In some implementations, loading the first matrix of the convolution operation into thread-local registers includes: determining a multiple between a size of each register of the thread-local registers and an element size of the first matrix; and storing matrix elements of one or more channels of the first matrix in each register of the thread-local registers based on the multiple, wherein the number of channels stored in each register of the thread-local registers is equal to or less than the multiple.

[0008] In some implementations, loading the second matrix of the convolution operation into the thread warp scalar register of the vector operation unit via the thread local register includes: determining a multiple between the size of each register of the thread local register and the element size of the second matrix; based on the multiple, storing matrix elements of one or more channels of the second matrix in each register of the thread local register, wherein the number of channels stored in each register of the thread local register is equal to or less than the multiple; and transferring the matrix elements of each channel stored in each register of the thread local register to a thread warp scalar register through the register transfer instruction.

[0009] In some implementations, the first matrix is ​​an activation value matrix for a neural network model, and the second matrix is ​​a weight matrix for the neural network model.

[0010] In some implementations, elements of the first matrix and the second matrix are both 16-bit floating point numbers, and the sizes of the thread local register and the warp scalar register are 32 bits.

[0011] In some implementations, elements of the first matrix and the second matrix are both 8-bit integers or 8-bit floating-point numbers, and the thread local register and the warp scalar register are 32 bits in size.

[0012] According to another aspect of the present disclosure, a computing device is provided, including a vector computing unit, wherein the vector computing unit is configured to execute the method described above.

[0013] According to yet another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program code is stored. When the computer program code is executed, the method described above is executed.

[0014] According to yet another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program performs the method described above when executed by a machine. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The present disclosure will be better understood and other objects, details, features and advantages of the present disclosure will become more apparent through the following description of specific embodiments of the present disclosure given with reference to the accompanying drawings.

[0016] Figure 1 An exemplary structural diagram of a computing device is shown.

[0017] Figure 2 A schematic diagram of a computing device according to some embodiments of the present invention is shown.

[0018] Figure 3 Schematic diagram showing two input matrices for a convolution operation.

[0019] Figure 4 An exemplary flow chart of a method for performing a convolution operation in a vector operation unit according to some embodiments of the present invention is shown.

[0020] Figure 5A A schematic diagram illustrating storage of a first matrix in a thread-local register according to some embodiments of the present invention is shown.

[0021] Figure 5B FIG. 4 shows a schematic diagram of storing a second matrix in a thread local register and a warp scalar register according to some embodiments of the present invention. DETAILED DESCRIPTION

[0022] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0023] As used herein, the term "including" and its variations represent open inclusion, i.e., "including but not limited to." Unless otherwise stated, the term "or" means "and / or." The term "based on" means "based at least in part on." The terms "one embodiment" and "some embodiments" mean "at least one example embodiment." The term "another embodiment" means "at least one additional embodiment." The terms "first," "second," etc. may refer to different or identical objects.

[0024] Figure 1 FIG. 1 shows an exemplary structural diagram of a computing device 100. Figure 1 As shown in FIG, the computing device 100 may include a plurality of general computing units 110 ( Figure 1 4 general computing units 110) and one or more tensor computing units 120 ( Figure 1 1 tensor computing unit 120 is exemplarily shown in FIG. 1 ). Typically, the tensor computing unit 120 is used to perform matrix operations such as matrix multiplication and addition and convolution, and the general computing unit 110 is used to perform other general computing operations other than matrix operations, such as vector operations and scalar operations.

[0025] Each general computing unit 110 may include multiple execution units 130 ( Figure 1 , each execution unit 130 may further include a vector operation unit 112, a scalar operation unit 114, a thread local register (TLR) 116, and a warp scalar register (WSR) 118 (see Figure 2 The vector operation unit 112 can execute issued instructions (i.e., Single Instruction Multiple Data (SIMD) instructions) on a warp basis to perform vector operations, such as 8-bit or 16-bit vector multiplication. The scalar operation unit 114 can be used to perform scalar operations, such as 16-bit or 32-bit scalar addition. The thread-local register 116 is a register group consisting of multiple registers, which is used to store input data required by and / or output data generated by each thread of the warp running on each vector operation unit 112. Each register in the thread-local register 116 can store an element of a matrix (e.g., a 16-bit floating point or an 8-bit integer). The warp scalar register 118 is used to store input data required by and / or output data generated by the scalar operation unit 114.

[0026] The computing device 100 is also typically connected to a global memory 200 to retrieve data required by the general-purpose computing unit 110 and the tensor computing unit 120 from external memory, and to store data generated by the general-purpose computing unit 110 and the tensor computing unit 120. Global memory 200 is typically located off-chip from the computing device 100 and connected to the computing device 100 via a protocol such as PCIE. It has a large storage capacity but a low transmission bandwidth to and from the computing device 100. The computing device 100 can be a basic unit of a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), or a tensor processing unit (TPU), and can be called a stream processing cluster (SPC). A GPU, a GPGPU, or a TPU can be composed of multiple computing devices 100.

[0027] Typically, the tensor computing unit 120 in the computing device 100 is used to perform convolution operations, but in some cases, such as when the matrix size of the convolution operation is small or the convolution operation occurs less frequently, using the tensor computing unit 120 to perform convolution operations may result in a waste of computing power.

[0028] To address this situation, the present disclosure provides a method for performing a convolution operation using the general computing unit 110, particularly the vector operation unit 112 in the general computing unit 110. In this method, the storage location of at least one input matrix for the convolution operation is designed to avoid repeated loading of the input matrix, thereby alleviating register bandwidth limitations.

[0029] Figure 2 Schematic diagram of a computing device 100 according to some embodiments of the present invention is shown. Figure 2 The computing device 100 and Figure 1 The structure of the computing device 100 is the same as that of the present invention, and components that are not directly related to the implementation of the present invention are omitted for illustrative purposes only, where the direction of the arrow represents the flow of data.

[0030] Figure 3 Figure 2 shows a schematic diagram of two input matrices for convolution operation. Figure 3As shown in , assume that the two input matrices are a first matrix A and a second matrix W. In artificial intelligence or neural network models, the first matrix A can be an activation matrix, such as a 9*9 activation matrix, and the second matrix W can be a weight matrix, such as a 3*3 or 5*5 weight matrix, also known as a convolution kernel. Typically, the first matrix A is larger than the second matrix W.

[0031] The first matrix A and the second matrix W can also be three-dimensional matrices, where the third dimension is the channel. Figure 3 In [1], the first matrix A can be expressed as [A0 A1 A2…], and the second matrix W can be expressed as [W0 W1 W2…]. Each element A0, A1, A2… represents a channel of the first matrix A, and each element W0, W1, W2… represents a channel of the second matrix W. The first channel W0 of the second matrix W is a 3*3 matrix with elements [w0, w1, w2, w3, w4, w5,w6, w7, w8]. Correspondingly, the elements of a 3*3 submatrix of the first channel A0 of the first matrix A are [a0, a1,a2, a3, a4, a5, a6, a7, a8].

[0032] like Figure 3 As shown in , the convolution operation of the first matrix A and the second matrix W includes a multiplication-addition operation on each channel. For example, the convolution operation of the first channel A0 of the first matrix A and the first channel W0 of the second matrix W includes taking a submatrix of the same size as the first channel W0 of the second matrix W from the first channel A0 of the first matrix A to perform matrix multiplication-addition.

[0033] The matrix multiplication and addition process can be expressed as: acc = a0 * w0 + a1 * w1 + a2 * w2 + a3 * w3 + a4 * w4 + a5 * w5 + a6 * w6 + a7 * w7 + a8 * w8 Among them, a0, a1...a8 are matrix elements of the submatrix of the first channel A0 of the first matrix A, w0, w1...w8 are matrix elements of the first channel W0 of the second matrix W, and acc is the result of matrix multiplication and addition.

[0034] acc can be split into the following types according to the number of rows: acc_row0 = a0 * w0 + a1 * w1 + a2 * w2 acc_row1 = a3 * w3 + a4 * w4 + a5 * w5 acc_row2 = a6 * w6 + a7 * w7 + a8 * w8 acc = acc_row0 + acc_row1 + acc_row2 Among them, acc_row0, acc_row1, and acc_row2 represent the three rows of calculation results of the first channel A0 and W0 respectively.

[0035] Assume dot_product_0 = a0 * w0 + a1 * w1, then acc_row0 = dot_product_0 +a2 * w2 For the vector operation unit 112, dot_product_0 can be implemented by a single multiply-add instruction, and acc_row0 can be implemented by a fused multiply-add instruction. Here, the multiply-add instruction is a hardware atomic instruction of the vector operation unit 112, for example, it can be expressed as hdp2(a, b, c, d) = a*c+b*d, where a, b, c, and d are the input elements of the multiply-add instruction. The fused multiply-add instruction is another hardware atomic instruction of the vector operation unit 112, for example, it can be expressed as fmadw(a, b, c) = (axb) + c, where a, b, and c are the input elements of the fused multiply-add instruction.

[0036] In this way, the convolution operation of two matrices can be converted into a vector operation, which can be implemented by hardware atomic instructions executable by the vector operation unit 112 without using tensor cores (such as the tensor computation unit 120).

[0037] Similarly, for the second channel A1 of the first matrix A and the second channel W1 of the second matrix W, the results of matrix multiplication and addition can be similarly obtained: acc' = a0' * w0' + a1' * w1' + a2' * w2' + a3' * w3' + a4' * w4' + a5' * w5' + a6' * w6' + a7' * w7' + a8' * w8' acc' can be split into the following according to the number of rows: acc_row0' = a0' * w0' + a1' * w1' + a2' * w2' acc_row1' = a3' * w3' + a4' * w4' + a5' * w5' acc_row2' = a6' * w6' + a7' * w7' + a8' * w8' acc' = acc_row0' + acc_row1' + acc_row2' Among them, acc_row0', acc_row1', and acc_row2' represent the three rows of calculation results of the second channels A1 and W1 respectively.

[0038] Assume dot_product_0' = a0' * w0' + a1' * w1' Then, acc_row0' = dot_product_1 + a2' * w2' Similarly, dot_product_0 ′ may be implemented by a single multiply-add instruction of the vector arithmetic unit 112 , and acc_row0 ′ may be implemented by a single fused multiply-add instruction of the vector arithmetic unit 112 .

[0039] Figure 4 An exemplary flow chart of a method 400 for performing a convolution operation in the vector operation unit 112 according to some embodiments of the present invention is shown.

[0040] like Figure 4 As shown in , at 410, the first matrix A for the convolution operation may be loaded into the thread local register 116. The thread local register 116 may be a dedicated register reserved for each thread of a warp running on a vector operation unit 112, and different vector operation units 112 may use different sets of thread local registers 116 to avoid contention for general purpose registers.

[0041] At 420 , the second matrix W of the convolution operation may be loaded into the warp scalar registers 118 of the vector operation unit 112 via the thread local registers 116 .

[0042] Specifically, 420 may further include: first loading the second matrix W of the convolution operation into the thread local register 116 , and then transferring each element of the second matrix W to the warp scalar register 118 through a register transfer instruction.

[0043] Since the vector operation unit 112 itself is for each thread of the thread warp running on the vector operation unit 112, the two input matrices of the convolution operation should be read into the thread local register 116. The thread warp scalar register 118 can be used to store thread warp level scalar data, which can be shared by all threads in the same thread warp and can be accessed by all threads in the thread warp running when the vector operation unit 112 executes a SIMD instruction.

[0044] Considering that the second matrix W, which serves as the weight matrix, is frequently used in convolution operations and thus needs to be repeatedly loaded from the thread-local register 116 to the vector operation unit 112, the second matrix W can be transferred to the warp scalar register 118. In this way, when the vector operation unit 112 performs a convolution operation on the first matrix A and the second matrix W, only the elements of the first matrix A need to be read from the thread-local register 116, thereby reducing the access pressure on the thread-local register 116.

[0045] Here, the transfer of the second matrix W from the vector operation unit 112 to the thread-local register 116 can be performed using a register transfer instruction. The register transfer instruction is a hardware atomic instruction of the vector operation unit 112 and is used to transfer data between the thread-local register 116 and the warp scalar register 118. For example, it can be expressed as r2q(dst, scr1, scr2), where dst is the destination address of the data in the warp scalar register 118, scr1 is the address of the thread-local register 116 where the data is located (i.e., the address of the register file), and scr2 is the address of the specific register in the thread-local register 116 where the data is located.

[0046] Next, at 430, a multiply-add instruction (such as the hdp2 instruction described above) is used to multiply two elements (e.g., a0, a1) of a submatrix of the first matrix A (e.g., the submatrix consisting of [a0, a1, a2, a3, a4, a5, a6, a7, a8]) read from the thread-local register 116 by two corresponding elements (e.g., w0, w1) of the second matrix W read from the warp scalar register 118 to obtain a result of the multiply-add instruction. Here, the two corresponding elements (e.g., w0, w1) of the second matrix W are elements that need to be multiplied with the two elements (e.g., a0, a1) of the submatrix of the first matrix A when performing the convolution operation.

[0047] That is, the vector operation unit 112 may execute hdp2(a0, a1, w0, w1) to obtain dot_product_0 = a0 * w0 + a1 * w1.

[0048] As above combined Figure 3 As described above, the convolution operation can be converted into a vector operation, so the vector operation unit 112 can be used to perform a vector operation multiplication and addition instruction on the elements in the first matrix A and the corresponding elements in the second matrix W.

[0049] At 440, the vector operation unit 112 may add the result of the multiply-add instruction (e.g., dot_product_0) to the product of the third element (e.g., a2) of the submatrix of the first matrix A (e.g., the submatrix consisting of [a0, a1, a2, a3, a4, a5, a6, a7, a8]) read from the thread-local register 116 and the third corresponding element (e.g., w2) of the second matrix W read from the warp scalar register 118 to obtain an element (i.e., acc) of the result matrix of the convolution operation. Here, the two elements (e.g., a0 and a1) of the submatrix of the first matrix A (e.g., the submatrix consisting of [a0, a1, a2, a3, a4, a5, a6, a7, a8]) are referred to as the first element and the second element, respectively, and the two corresponding elements (e.g., w0 and w1) of the second matrix W are referred to as the first corresponding element and the second corresponding element, respectively. In this case, the element next to the two elements (e.g., a0 and a1) of the submatrix of the first matrix A is referred to as the third element. Correspondingly, the third corresponding element of the second matrix W (such as w2) refers to the element that needs to be multiplied by the third element of the above submatrix of the first matrix A (such as a2) when performing the convolution operation.

[0050] In some embodiments, the result of the multiply-add instruction (e.g., dot_product_0) can be stored in a shutdown register (not shown). The shutdown register is used to temporarily store specific values ​​to ensure pipeline operations for subsequent data. In some embodiments, the shutdown register can be, for example, a separate register located in the execution unit 130, which does not have port conflicts and can be exclusively occupied while the thread warp is running. The size of each shutdown register can be, for example, 32 bits * 32 threads, and each thread warp can use up to four shutdown registers of this size. Those skilled in the art will understand that the present invention is not limited to this. The shutdown register can also be located in other locations in the computing device 100 or have other sizes, as long as it can temporarily store specific values ​​and be exclusively occupied while the thread warp is running.

[0051] In the vector operation unit 112, a mixed multiplication and addition instruction (such as the fmadw instruction described above) can be used to perform a mixed multiplication and addition on the third element a2 of the submatrix of the first matrix A, the third corresponding element w2 of the second matrix W, and the result of the multiplication and addition instruction (such as dot_product_0) to obtain an element of the result matrix of the convolution operation.

[0052] That is, the vector operation unit 112 may execute fmadw(a2, w2, dot_product_0) to obtain acc.

[0053] In some cases, depending on the size of registers (eg, each of the thread local registers 116 and the warp scalar registers 118 ), data for multiple channels of a matrix may also be multiplexed in one register.

[0054] In this case, 410 may further include determining a multiple of a size of each register of the thread local registers 116 and an element size of the first matrix A, and storing matrix elements of one or more channels of the first matrix A in each register of the thread local registers 116 based on the multiple, wherein the number of channels stored in each register of the thread local registers 116 is equal to or less than the multiple.

[0055] For example, assume that each thread-local register 116 is 32 bits. If the elements of the first matrix A are 16-bit floating-point numbers (such as half-precision floating-point numbers fp16 or Brain floating-point numbers f16), each register can store a maximum of 2 (32 / 16=2) matrix elements, i.e., matrix elements for 2 channels. If the elements of the first matrix A are 8-bit integers or 8-bit floating-point numbers, each register can store a maximum of 4 (32 / 8=4) matrix elements, i.e., matrix elements for 4 channels. To maximize the capacity of each register and improve the parallel processing efficiency of the vector operation unit 112, the number of channels stored in each thread-local register 116 can be equal to the above multiple.

[0056] On the other hand, for the second matrix W, similar processing can be performed.

[0057] For example, in some embodiments, a multiple of the size of each register of the thread-local registers 116 and the element size of the second matrix W may be determined. Based on the multiple, matrix elements of one or more channels of the second matrix W may be stored in each register of the thread-local registers 116, where the number of channels stored in each register of the thread-local registers 116 is equal to or less than the multiple. The matrix elements of each channel stored in each register of the thread-local registers 116 are then transferred to a warp scalar register 118 using the register transfer instruction.

[0058] Figure 5A The following diagram illustrates the storage of a first matrix A in thread-local registers 116 according to some embodiments of the present invention. Assuming each register in thread-local registers 116 is 32 bits and the elements of first matrix A are 16-bit floating-point numbers, each register can store two elements of two channels A0 and A1 of first matrix A. For example, register r0 can store a0 and a0', and register r1 can store a1 and a1'. Figure 5BA schematic diagram illustrates the storage of the second matrix W in thread-local registers 116 and warp scalar registers 118 according to some embodiments of the present invention. Assuming each register in thread-local registers 116 and warp scalar registers 118 is 32-bit, and the matrix elements of the second matrix W are 16-bit floating-point numbers, each register can store two elements of two channels, W0 and W1, of the second matrix W. For example, register r9 can store w0 and w0', and register r10 can store w1 and w1'. The two elements in each register can be transferred to the warp scalar register 118, for example, using a register transfer instruction. For example, elements w0 and w0' stored in register r9 can be transferred to register q0, and elements w1 and w1' stored in register r10 can be transferred to register q1. If each of the thread local registers 116 and the warp scalar register 118 has 32 bits and the matrix elements of the first matrix A and the second matrix W have 8 bits, then each register can store matrix elements of up to 4 channels.

[0059] By decomposing convolution operations into vector operations, the disclosed solution enables convolution operations to be performed using vector computation units, thereby avoiding wasting computing power on tensor computation units. Furthermore, by transferring an input matrix (such as a weight matrix) for the convolution operation to a warp scalar register, thread-local register access pressure is reduced.

[0060] The method and computing device for performing a convolution operation in a vector operation unit according to the present disclosure are described above in conjunction with the accompanying drawings. However, it will be understood by those skilled in the art that the execution of the above method is not limited to the order shown in the figure and described above, but can be performed in any other reasonable order. In addition, the computing device does not necessarily include all the components shown in the figure. It may only include some or more components necessary to perform the functions described in the present disclosure, and the connection method of these components is not limited to the form shown in the figure.

[0061] The present disclosure may be implemented as a method, a computing device, a system, and / or a computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions for performing various aspects of the present disclosure. The computing device may include at least one processor and at least one memory coupled to the at least one processor, the memory may store instructions for execution by the at least one processor. When the instructions are executed by the at least one processor, the computing device may perform the above-described method.

[0062] In one or more exemplary designs, the functions described in this disclosure may be implemented using hardware, software, firmware, or any combination thereof. For example, if implemented using software, the functions may be stored as one or more instructions or codes on a computer-readable medium, or transmitted as one or more instructions or codes on a computer-readable medium.

[0063] The various units of the apparatus disclosed herein may be implemented using discrete hardware components or may be integrated into a single hardware component, such as a processor. For example, the various exemplary logic blocks, modules, and circuits described in conjunction with the present disclosure may be implemented or executed using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof for performing the functions described herein.

[0064] Those skilled in the art should also understand that the various illustrative logical blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments of the present disclosure may be implemented as electronic hardware, computer software, or a combination of both.

[0065] The above description of the present disclosure is intended to enable any person skilled in the art to implement or use the present disclosure. Various modifications of the present disclosure will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit and scope of the present disclosure. Therefore, the present disclosure is not limited to the examples and designs described herein, but is intended to be consistent with the widest scope of the principles and novel features disclosed herein.

Claims

1. A method for performing a convolution operation in a vector operation unit, comprising: Loading a first matrix of the convolution operation into a thread-local register; Loading a second matrix of the convolution operation into a warp scalar register of the vector operation unit via the thread local register; multiplying two elements of a submatrix of the first matrix read from the thread local register and two corresponding elements of the second matrix read from the warp scalar register by a multiply-add instruction to obtain a result of the multiply-add instruction; as well as Adding a result of the multiply-add instruction to a product of a third element of the submatrix read from the thread local register and a third corresponding element of the second matrix read from the warp scalar register to obtain an element of a result matrix of the convolution operation.

2. The method of claim 1 , wherein loading the second matrix of the convolution operation into the warp scalar register of the vector operation unit via the thread local register comprises: Loading a second matrix of the convolution operation into the thread-local register; as well as Each element of the second matrix is ​​transferred to the warp scalar register by a register transfer instruction.

3. The method of claim 1 , wherein obtaining an element of a result matrix of the convolution operation comprises: storing the result of the multiply-add instruction in a shutdown register; as well as In the vector operation unit, a mixed multiplication is performed on the third element of the submatrix of the first matrix, the third corresponding element of the second matrix, and the result of the multiplication and addition instruction by a fused multiplication and addition instruction to obtain an element of the result matrix of the convolution operation.

4. The method of claim 1 , wherein loading the first matrix of the convolution operation into a thread-local register comprises: determining a multiple of a size of each of the thread-local registers and an element size of the first matrix; as well as Based on the multiple, matrix elements of one or more channels of the first matrix are stored in each of the thread local registers, wherein the number of channels stored in each of the thread local registers is equal to or less than the multiple.

5. The method of claim 1 , wherein loading the second matrix of the convolution operation into a warp scalar register of the vector operation unit via the thread local register comprises: determining a multiple of a size of each of the thread-local registers and an element size of the second matrix; storing, based on the multiple, matrix elements of one or more channels of the second matrix in each of the thread local registers, wherein the number of channels stored in each of the thread local registers is equal to or less than the multiple; as well as The matrix element of each channel stored in each register of the thread local register is transferred to a warp scalar register by using a register transfer instruction.

6. The method of claim 1, wherein the first matrix is ​​an activation value matrix for a neural network model, and the second matrix is ​​a weight matrix for the neural network model. 7 . The method of claim 1 , wherein elements of the first matrix and the second matrix are both 16-bit floating point numbers, and sizes of the thread local register and the warp scalar register are 32 bits. 8 . The method of claim 1 , wherein elements of the first matrix and the second matrix are both 8-bit integers or 8-bit floating-point numbers, and the thread local register and the warp scalar register are 32 bits in size. 9 . A computing device comprising a vector computing unit, wherein the vector computing unit is configured to execute the method according to claim 1 .

10. A computer-readable storage medium having computer program code stored thereon, wherein the computer program code executes the method according to any one of claims 1 to 8 when executed.

11. A computer program product comprising a computer program, which, when executed by a machine, performs the method according to any one of claims 1 to 8.

Images