Methods, computing devices, computer-readable storage media, and computer program products for performing convolution operations in vector processing units.

Abstract

This disclosure provides a method, computing device, computer-readable storage medium, and computer program product for performing convolution operations in a vector processing unit. The method includes: loading a first matrix of the convolution operation into a thread-local register; loading a second matrix of the convolution operation into a thread-beam scalar register of the vector processing unit via the thread-local register; multiplying two elements of a submatrix of the first matrix read from the thread-local register and two corresponding elements of the second matrix read from the thread-beam scalar register using a multiply-add instruction to obtain a result of the multiply-add instruction; and adding the result of the multiply-add instruction to the product of a third element of the submatrix read from the thread-local register and a third corresponding element of the second matrix read from the thread-beam scalar register to obtain an element of the result matrix of the convolution operation.

Description

Technical Field

[0001] This disclosure generally relates to the field of processors, and more specifically, to a method, computing device, computer-readable storage medium, and computer program product for performing convolution operations in a vector operation unit. Background Technology

[0002] Convolution is a common mathematical operation frequently used in artificial intelligence and neural networks. AI chips require a large number of convolution operations during model training and inference. Typically, AI chips use computational units called tensor kernels or tensor computation units to perform convolution operations. However, in some cases, such as when the matrix size of the convolution operation is small or when convolution operations are infrequent, using tensor kernels may result in wasted computational resources. Furthermore, during convolution, the two input matrices involved in the operation may need to be repeatedly loaded into registers, thus limiting data loading due to register bandwidth constraints. Summary of the Invention

[0003] To address the aforementioned problems, this disclosure provides a scheme for performing convolution operations using vector operation units. For example, this allows convolution operations to be performed using vector operation units when the matrix size is small or convolution operations are infrequent, thus avoiding wasted computational power of tensor computation kernels. Furthermore, this disclosure alleviates register bandwidth limitations by designing the storage location of at least one input matrix for the convolution operation to avoid repeated loading of the input matrix.

[0004] According to one aspect of this disclosure, a method for performing a convolution operation in a vector processing unit is provided. The method includes: loading a first matrix of the convolution operation into a thread-local register; loading a second matrix of the convolution operation into a thread-binding scalar register of the vector processing unit via the thread-local register; multiplying two elements of a submatrix of the first matrix read from the thread-local register and two corresponding elements of the second matrix read from the thread-binding scalar register using a multiply-add instruction to obtain a result of the multiply-add instruction; and adding the result of the multiply-add instruction to the product of a third element of the submatrix read from the thread-local register and a third corresponding element of the second matrix read from the thread-binding scalar register to obtain an element of the result matrix of the convolution operation.

[0005] In some implementations, loading the second matrix of the convolution operation into the thread-bundle scalar register of the vector operation unit via the thread-local register includes: loading the second matrix of the convolution operation into the thread-local register; and transferring each element of the second matrix into the thread-bundle scalar register via a register transfer instruction.

[0006] In some implementations, obtaining an element of the result matrix of the convolution operation includes: storing the result of the multiply-accumulate instruction in a shutdown register; and in the vector operation unit, performing a hybrid multiplication on the third element of the submatrix of the first matrix and the third corresponding element of the second matrix, along with the result of the multiply-accumulate instruction, to obtain an element of the result matrix of the convolution operation.

[0007] In some implementations, loading the first matrix of the convolution operation into a thread-local register includes: determining a multiple between the size of each register of the thread-local register and the element size of the first matrix; and storing matrix elements of one or more channels of the first matrix in each register of the thread-local register based on the multiple, wherein the number of channels stored in each register of the thread-local register is equal to or less than the multiple.

[0008] In some implementations, loading the second matrix of the convolution operation into the thread bundle scalar register of the vector operation unit via the thread local register includes: determining a multiple between the size of each register of the thread local register and the element size of the second matrix; storing matrix elements of one or more channels of the second matrix in each register of the thread local register based on the multiple, wherein the number of channels stored in each register of the thread local register is equal to or less than the multiple; and transferring the matrix elements of each channel stored in each register of the thread local register to a thread bundle scalar register via the register transfer instruction.

[0009] In some implementations, the first matrix is ​​an activation value matrix for a neural network model, and the second matrix is ​​a weight matrix for the neural network model.

[0010] In some implementations, the elements of both the first and second matrices are 16-bit floating-point numbers, and the thread-local register and the thread-beam scalar register are 32-bit in size.

[0011] In some implementations, the elements of the first matrix and the second matrix are both 8-bit integers or 8-bit floating-point numbers, and the thread local register and the thread bundle scalar register are 32 bits in size.

[0012] According to another aspect of this disclosure, a computing device is provided, including a vector computing unit configured to perform the method described above.

[0013] According to another aspect of this disclosure, a computer-readable storage medium is provided having computer program code stored thereon, which, when run, performs the method described above.

[0014] According to another aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a machine, performs the method described above. Attached Figure Description

[0015] This disclosure will be better understood by referring to the following description of specific embodiments given in the accompanying drawings, and other objects, details, features, and advantages of this disclosure will become more apparent.

[0016] Figure 1 An exemplary structural schematic diagram of a computing device is shown.

[0017] Figure 2 A schematic diagram of a computing device according to some embodiments of the present invention is shown.

[0018] Figure 3 A schematic diagram of the two input matrices for the convolution operation is shown.

[0019] Figure 4 An exemplary flowchart of a method for performing convolution operations in a vector operation unit according to some embodiments of the present invention is shown.

[0020] Figure 5A A schematic diagram of the storage of a first matrix in a thread-local register according to some embodiments of the present invention is shown.

[0021] Figure 5B A schematic diagram illustrating the storage of a second matrix in a thread-local register and a thread-bundle scalar register according to some embodiments of the present invention is shown. Detailed Implementation

[0022] Preferred embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0023] The term "comprising" and its variations as used herein signify open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "at least partially based on". The terms "one embodiment" and "some embodiments" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc., may refer to different or the same objects.

[0024] Figure 1 An exemplary structural schematic diagram of a computing device 100 is shown. Figure 1 As shown, computing device 100 may include a plurality of general-purpose computing units 110 ( Figure 1 The diagram exemplarily illustrates four general-purpose computing units 110 and one or more tensor computing units 120. Figure 1 An example of a tensor computation unit 120 is shown. Typically, the tensor computation unit 120 is used to perform matrix operations such as matrix multiplication and addition, convolution, while the general-purpose computation unit 110 is used to perform other general-purpose computations besides matrix operations, such as vector operations, scalar operations, etc.

[0025] Each general-purpose computing unit 110 may include multiple execution units 130 ( Figure 1 The example shows four execution units 130, each of which may further include a vector operation unit 112, a scalar operation unit 114, a thread local register (TLR) 116, and a warp scalar register (WSR) 118 (see [link to example]). Figure 2 The vector operation unit 112 can execute issued instructions (i.e., Single Instruction Multiple Data (SIMD) instructions) on a thread-by-thread basis to perform vector operations, such as 8-bit or 16-bit vector multiplication. The scalar operation unit 114 can be used to perform scalar operations, such as 16-bit or 32-bit scalar addition. The thread local register 116 is a register group consisting of multiple registers, used to store the input data and / or output data required by each thread of the thread bundle running on each vector operation unit 112. Each register in the thread local register 116 can store an element of a matrix (such as a 16-bit floating-point or 8-bit integer). The thread bundle scalar register 118 is used to store the input data required by the scalar operation unit 114 and / or the output data generated.

[0026] The computing device 100 is typically also connected to a global memory 200 to retrieve data required by the general-purpose computing unit 110 and the tensor computing unit 120 from external memory, and to store data generated by the general-purpose computing unit 110 and the tensor computing unit 120. The global memory 200 is typically located off-chip within the computing device 100 and is connected to the computing device 100 via a protocol such as PCIe. It has a large storage capacity but a low transmission bandwidth with the computing device 100. The computing device 100 can be a basic unit of a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), or a tensor processing unit (TPU), for example, it can be called a stream processing cluster (SPC). A GPU, a GPGPU, or a TPU can be composed of multiple computing devices 100.

[0027] Typically, the tensor computation unit 120 in the computing device 100 is used to perform convolution operations. However, in some cases, such as when the matrix size of the convolution operation is small or when convolution operations occur infrequently, using the tensor computation unit 120 to perform convolution operations may result in a waste of computing power.

[0028] To address this situation, this disclosure provides a method for performing convolution operations using a general-purpose computing unit 110, particularly a vector operation unit 112 within the general-purpose computing unit 110. In this method, the storage location of at least one input matrix for the convolution operation is designed to avoid repeated loading of the input matrix, thereby mitigating register bandwidth limitations.

[0029] Figure 2 A schematic diagram of a computing device 100 according to some embodiments of the present invention is shown. Wherein, Figure 2 Computing device 100 and Figure 1 The structure is the same as that of the computing device 100. For illustrative purposes only, components that are not directly related to the implementation of the present invention are omitted, wherein the direction of the arrows represents the flow of data.

[0030] Figure 3 A schematic diagram of the two input matrices for the convolution operation is shown. (For example...) Figure 3As shown in the diagram, assume the two input matrices are a first matrix A and a second matrix W. In artificial intelligence or neural network models, the first matrix A can be an activation matrix, such as a 9x9 activation matrix, and the second matrix W can be a weight matrix, such as a 3x3 or 5x5 weight matrix, also known as a convolution kernel. Typically, the first matrix A is larger than the second matrix W.

[0031] The first matrix A and the second matrix W can also be three-dimensional matrices, where the third dimension represents the channels. For example, in Figure 3 In the diagram, the first matrix A can be represented as [A0 A1 A2……], and the second matrix W can be represented as [W0 W1 W2……]. Each element A0, A1, A2…… represents a channel of the first matrix A, and each element W0, W1, W2…… represents a channel of the second matrix W. The first channel W0 of the second matrix W is a 3*3 matrix with elements [w0, w1, w2, w3, w4, w5, w6, w7, w8]. Correspondingly, the elements of a 3*3 submatrix of the first channel A0 of the first matrix A are [a0, a1, a2, a3, a4, a5, a6, a7, a8].

[0032] like Figure 3 As shown, the convolution operation between the first matrix A and the second matrix W includes multiplication and addition operations on each channel. For example, the convolution operation between the first channel A0 of the first matrix A and the first channel W0 of the second matrix W includes taking a submatrix of the same size as the first channel W0 of the second matrix W from the first channel A0 of the first matrix A and performing matrix multiplication and addition.

[0033] The matrix multiplication and addition process can be represented as:

[0034] acc = a0 * w0 + a1 * w1 + a2 * w2 + a3 * w3 + a4 * w4 + a5 * w5 + a6 * w6 + a7 * w7 + a8 * w8

[0035] Where a0, a1...a8 are matrix elements of the submatrix of the first channel A0 of the first matrix A, w0, w1...w8 are matrix elements of the first channel W0 of the second matrix W, and acc is the result of matrix multiplication and addition.

[0036] acc can be split into the following based on the number of rows:

[0037] acc_row0 = a0 * w0 + a1 * w1 + a2 * w2

[0038] acc_row1 = a3 * w3 + a4 * w4 + a5 * w5

[0039] acc_row2 = a6 * w6 + a7 * w7 + a8 * w8

[0040] acc = acc_row0 + acc_row1 + acc_row2

[0041] Here, acc_row0, acc_row1, and acc_row2 represent the three rows of calculation results for the first channel A0 and W0, respectively.

[0042] Assuming dot_product_0 = a0 * w0 + a1 * w1, then acc_row0 = dot_product_0 + a2 * w2

[0043] For the vector operation unit 112, `dot_product_0` can be implemented using a single multiply-accumulate instruction, while `acc_row0` can be implemented using a fused multiply-accumulate instruction. Here, the multiply-accumulate instruction is a hardware atomic instruction of the vector operation unit 112, for example, it can be represented as `hdp2(a, b, c, d) = a*c+b*d`, where a, b, c, and d are the input elements of this multiply-accumulate instruction. The fused multiply-accumulate instruction is another hardware atomic instruction of the vector operation unit 112, for example, it can be represented as `fmadw(a, b,c)=(axb)+c`, where a, b, and c are the input elements of this fused multiply-accumulate instruction.

[0044] In this way, the convolution operation of two matrices can be converted into a vector operation, which can be implemented by the hardware atomic instructions that the vector operation unit 112 can run, without the need to use a tensor kernel (such as the tensor computation unit 120).

[0045] Similarly, for the second channel A1 of the first matrix A and the second channel W1 of the second matrix W, the result of matrix multiplication and addition can be obtained in a similar manner:

[0046] acc' = a0' * w0' + a1' * w1' + a2' * w2' + a3' * w3' + a4' * w4' + a5' * w5' + a6' * w6' + a7' * w7' + a8' * w8'

[0047] acc' can be split into the following based on the number of lines:

[0048] acc_row0' = a0' * w0' + a1' * w1' + a2' * w2'

[0049] acc_row1' = a3' * w3' + a4' * w4' + a5' * w5'

[0050] acc_row2' = a6' * w6' + a7' * w7' + a8' * w8'

[0051] acc' = acc_row0' + acc_row1' + acc_row2'

[0052] Here, acc_row0', acc_row1', and acc_row2' represent the three rows of calculation results for the second channels A1 and W1, respectively.

[0053] Assume dot_product_0' = a0' * w0' + a1' * w1'

[0054] Then, acc_row0' = dot_product_1 + a2' * w2'

[0055] Similarly, dot_product_0' can be implemented by a single multiply-accumulate instruction of the vector operation unit 112, and acc_row0' can be implemented by a single fused multiply-accumulate instruction of the vector operation unit 112.

[0056] Figure 4 An exemplary flowchart of a method 400 for performing convolution operations in a vector operation unit 112 according to some embodiments of the present invention is shown.

[0057] like Figure 4 As shown, at 410, the first matrix A of the convolution operation can be loaded into the thread-local register 116. The thread-local register 116 may be a dedicated register reserved for each thread of the thread bundle running on the vector operation unit 112, and different vector operation units 112 use different sets of thread-local registers 116 to avoid contention for general-purpose registers.

[0058] At 420, the second matrix W of the convolution operation can be loaded into the thread bundle scalar register 118 of the vector operation unit 112 via the thread local register 116.

[0059] Specifically, 420 may further include: first loading the second matrix W of the convolution operation into the thread local register 116, and then transferring each element of the second matrix W to the thread bundle scalar register 118 through a register transfer instruction.

[0060] Since the vector operation unit 112 is designed for each thread within a warp running on the vector operation unit 112, both input matrices for the convolution operation should be read into the thread-local register 116. The warp scalar register 118 can be used to store warp-level scalar data, which can be shared by all threads within the same warp. Therefore, it can be accessed by all threads within the warp running when the vector operation unit 112 executes a SIMD instruction.

[0061] Considering that the second matrix W, serving as the weight matrix, is frequently used in convolution operations, thus requiring repeated loading from thread-local register 116 to vector operation unit 112, this paper proposes transferring the second matrix W to thread bundle scalar register 118. This way, when vector operation unit 112 performs convolution operations on the first matrix A and the second matrix W, it only needs to read elements of the first matrix A from thread-local register 116, thereby reducing the access pressure on thread-local register 116.

[0062] Here, the transfer of the second matrix W from the vector operation unit 112 to the thread local register 116 can be executed using a register transfer instruction. The register transfer instruction is a hardware atomic instruction of the vector operation unit 112, used to perform data transfer between the thread local register 116 and the thread bundle scalar register 118. For example, it can be represented as r2q(dst,scr1, scr2), where dst is the destination address of the data in the thread bundle scalar register 118, scr1 is the address of the thread local register 116 where the data is located (i.e., the address of the register set), and scr2 is the address of the specific register within the thread local register 116 where the data is located.

[0063] Next, at 430, the result of the multiply-add instruction is obtained by multiplying two elements (e.g., a0, a1) of a submatrix of the first matrix A (e.g., a submatrix composed of [a0, a1, a2, a3, a4, a5, a6, a7, a8]) read from the thread local register 116 and two corresponding elements (e.g., w0, w1) of the second matrix W read from the thread bundle scalar register 118. Here, the two corresponding elements (e.g., w0, w1) of the second matrix W refer to the elements that need to be multiplied by the two elements (e.g., a0, a1) of the aforementioned submatrix of the first matrix A when performing the convolution operation.

[0064] In other words, the vector operation unit 112 can execute hdp2(a0, a1, w0, w1) to obtain dot_product_0 = a0 * w0 + a1 * w1.

[0065] As described above Figure 3 As mentioned above, the convolution operation can be converted into a vector operation, so the vector operation unit 112 can be used to perform vector operation multiplication and addition instructions on the elements in the first matrix A and the corresponding elements in the second matrix W.

[0066] At 440, the vector operation unit 112 can add the result of the multiplication-addition instruction (e.g., dot_product_0) to the product of the third element (e.g., a2) of the submatrix of the first matrix A (e.g., the submatrix composed of [a0, a1, a2, a3, a4, a5, a6, a7, a8]) read from the thread local register 116 and the third corresponding element (e.g., w2) of the second matrix W read from the thread bundle scalar register 118 to obtain an element (i.e., acc) of the result matrix of the convolution operation. Here, the two elements (e.g., a0, a1) of the submatrix of the first matrix A (e.g., the submatrix composed of [a0, a1, a2, a3, a4, a5, a6, a7, a8]) are referred to as the first element and the second element, respectively, and the two corresponding elements (e.g., w0, w1) of the second matrix W are referred to as the first corresponding element and the second corresponding element, respectively. In this case, the next element after the two elements (e.g., a0, a1) of the aforementioned submatrix of the first matrix A is referred to as the third element. Correspondingly, the third corresponding element of the second matrix W (e.g., w2) refers to the element that needs to be multiplied by the third element (e.g., a2) of the aforementioned submatrix of the first matrix A when performing the convolution operation.

[0067] In some embodiments, the result of a multiply-accumulate instruction (such as dot_product_0) can be stored in a shutdown register (not shown in the figure). The shutdown register is used to temporarily store specific values ​​to ensure pipelined operation of subsequent data. In some embodiments, the shutdown register can be, for example, a separate register located in execution unit 130, which is free from port conflicts and can be exclusively used during thread bundle execution. The size of each shutdown register can be, for example, 32 bits * 32 threads, and each thread bundle can use up to four shutdown registers of this size. Those skilled in the art will understand that the invention is not limited thereto; the shutdown register can also be located in other locations within computing device 100 or have other sizes, as long as it can temporarily store specific values ​​and be exclusively used during thread bundle execution.

[0068] In the vector operation unit 112, a hybrid multiplication can be performed on the third element a2 of the submatrix of the first matrix A and the third corresponding element w2 of the second matrix W, as well as the result of the multiplication and addition instruction (such as dot_product_0), to obtain an element of the result matrix of the convolution operation.

[0069] In other words, the vector operation unit 112 can execute fmadw(a2, w2, dot_product_0) to obtain acc.

[0070] In some cases, depending on the size of the registers (such as each register of thread-local register 116 and thread-bundle scalar register 118), data from multiple channels of a matrix can also be multiplexed in a single register.

[0071] In this case, 410 may further include: determining a multiple between the size of each register of the thread local register 116 and the element size of the first matrix A, and storing matrix elements of one or more channels of the first matrix A in each register of the thread local register 116 based on the multiple, wherein the number of channels stored in each register of the thread local register 116 is equal to or less than the multiple.

[0072] For example, assume each register of thread-local register 116 is 32 bits. If the elements of the first matrix A are 16-bit floating-point numbers (such as half-precision floating-point number fp16 or Brain floating-point number f16), then each register can store at most 2 (32 / 16=2) matrix elements, that is, store matrix elements of 2 channels. If the elements of the first matrix A are 8-bit integers or 8-bit floating-point numbers, then each register can store at most 4 (32 / 8=4) matrix elements, that is, store matrix elements of 4 channels. To maximize the utilization of the capacity of each register and improve the parallel processing efficiency of vector operation unit 112, the number of channels stored in each register of thread-local register 116 can be equal to a multiple of the above.

[0073] On the other hand, similar processing can be performed on the second matrix W.

[0074] For example, in some embodiments, a multiple between the size of each register of thread local register 116 and the element size of the second matrix W can be determined. Based on this multiple, matrix elements of one or more channels of the second matrix W are stored in each register of thread local register 116, wherein the number of channels stored in each register of thread local register 116 is equal to or less than this multiple. Then, the matrix elements of each channel stored in each register of thread local register 116 are transferred to a thread bundle scalar register 118 using the aforementioned register transfer instruction.

[0075] Figure 5A A schematic diagram illustrating the storage of a first matrix A in a thread-local register 116 according to some embodiments of the present invention is shown. As shown, assuming each register in the thread-local register 116 is 32 bits and the matrix elements of the first matrix A are 16-bit floating-point numbers, each register can store two elements from the two channels A0 and A1 of the first matrix A. For example, register r0 can store a0, a0', register r1 can store a1, a1', and so on.

[0076] Figure 5B A schematic diagram illustrating the storage of a second matrix W in thread-local register 116 and thread-beam scalar register 118 according to some embodiments of the present invention is shown. As shown, assuming each register in thread-local register 116 and thread-beam scalar register 118 is 32 bits, and the matrix elements of the second matrix W are 16-bit floating-point numbers, then each register can store two elements from the two channels W0 and W1 of the second matrix W. For example, register r9 can store w0, w0', register r10 can store w1, w1', and so on.

[0077] For example, two elements in each register can be transferred to the thread bundle scalar register 118 using register transfer instructions. For instance, elements w0 and w0' stored in register r9 can be transferred to register q0, and elements w1 and w1' stored in register r10 can be transferred to register q1, and so on.

[0078] If each register of thread local register 116 and thread bundle scalar register 118 is 32 bits, and the matrix elements of the first matrix A and the second matrix W are 8 bits, then each register can store matrix elements of up to 4 channels.

[0079] By using the scheme disclosed herein, convolution operations can be decomposed into vector operations, enabling the execution of convolution operations using vector computation units, thereby avoiding the waste of computational power in tensor computation units. Furthermore, by transferring one input matrix (such as the weight matrix) of the convolution operation to the thread bundle scalar register, the access pressure on thread-local registers is reduced.

[0080] The method and computing apparatus for performing convolution operations in a vector operation unit according to the present disclosure have been described above with reference to the accompanying drawings. However, those skilled in the art will understand that the execution of the above method is not limited to the order shown in the figures and described above, but can be performed in any other reasonable order. Furthermore, the computing apparatus does not necessarily include all the components shown in the figures; it may include only some or more of the components necessary to perform the functions described in this disclosure, and the connection of these components is not limited to the form shown in the figures.

[0081] This disclosure can be implemented as a method, a computing device, a system, and / or a computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure. A computing device may include at least one processor and at least one memory coupled to the at least one processor, the memory storing instructions for execution by the at least one processor. When executed by the at least one processor, the computing device may perform the methods described above.

[0082] In one or more exemplary designs, the functions described herein may be implemented using hardware, software, firmware, or any combination thereof. For example, if implemented in software, the functions may be stored as one or more instructions or code on a computer-readable medium, or transmitted as one or more instructions or code on a computer-readable medium.

[0083] The various units of the apparatus disclosed herein can be implemented using discrete hardware components or integrated on a single hardware component, such as a processor. For example, they can be implemented or perform the various exemplary logic blocks, modules, and circuits described herein using a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof for performing the functions described herein.

[0084] Those skilled in the art should also understand that the various exemplary logic blocks, modules, circuits, and algorithm steps described in connection with embodiments of this disclosure can be implemented as electronic hardware, computer software, or a combination of both.

[0085] The foregoing description of this disclosure is intended to enable any person skilled in the art to implement or use this disclosure. Various modifications to this disclosure will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other variations without departing from the spirit and scope of this disclosure. Therefore, this disclosure is not limited to the examples and designs described herein, but is consistent with the broadest scope of the principles and novel features disclosed herein.

Claims

1. A method for performing convolution operations in a vector operation unit, comprising: The first matrix of the convolution operation is loaded into a thread-local register, wherein the thread-local register is used to store the input data required and / or the output data generated by each thread of the thread bundle running on the vector operation unit; The second matrix of the convolution operation is loaded into the thread bundle scalar register of the vector operation unit via the thread local register, wherein the thread bundle scalar register is used to store scalar data at the thread bundle level and is shared by all threads within the same thread bundle; The result of the multiply-add instruction is obtained by multiplying two elements of a submatrix of the first matrix read from the thread local register and two corresponding elements of the second matrix read from the thread bundle scalar register, wherein the multiply-add instruction is a hardware atomic instruction of the vector operation unit. as well as The result of the multiplication-addition instruction is added to the product of the third element of the submatrix read from the thread local register and the third corresponding element of the second matrix read from the thread bundle scalar register to obtain an element of the result matrix of the convolution operation.

2. The method of claim 1, wherein loading the second matrix of the convolution operation into the thread bundle scalar register of the vector operation unit via the thread-local register comprises: The second matrix of the convolution operation is loaded into the thread-local register; as well as Each element of the second matrix is ​​transferred to the thread bundle scalar register using a register transfer instruction.

3. The method of claim 1, wherein obtaining an element of the result matrix of the convolution operation comprises: The result of the multiply-accumulate instruction is stored in the shutdown register; as well as In the vector operation unit, a hybrid multiplication is performed on the third element of the submatrix of the first matrix and the third corresponding element of the second matrix, along with the result of the multiplication instruction, to obtain an element of the result matrix of the convolution operation.

4. The method of claim 1, wherein loading the first matrix of the convolution operation into a thread-local register comprises: Determine the multiple between the size of each register in the thread-local register and the element size of the first matrix; as well as Based on the multiple, matrix elements of one or more channels of the first matrix are stored in each register of the thread-local register, wherein the number of channels stored in each register of the thread-local register is equal to or less than the multiple.

5. The method of claim 1, wherein loading the second matrix of the convolution operation into the thread bundle scalar register of the vector operation unit via the thread-local register comprises: Determine the multiple between the size of each register in the thread-local register and the element size of the second matrix; Based on the multiplier, matrix elements of one or more channels of the second matrix are stored in each register of the thread local register, wherein the number of channels stored in each register of the thread local register is equal to or less than the multiplier; as well as The matrix elements of each channel stored in each register in the thread-local register are transferred to a thread-bundle scalar register using a register transfer instruction.

6. The method of claim 1, wherein the first matrix is ​​an activation value matrix for a neural network model, and the second matrix is ​​a weight matrix for the neural network model.

7. The method of claim 1, wherein the elements of the first matrix and the second matrix are both 16-bit floating-point numbers, and the thread local register and the thread bundle scalar register are 32-bit in size.

8. The method of claim 1, wherein the elements of the first matrix and the second matrix are both 8-bit integers or 8-bit floating-point numbers, and the thread local register and the thread bundle scalar register are 32 bits in size.

9. A computing device comprising a vector computing unit configured to perform the method as claimed in any one of claims 1 to 8.

10. A computer-readable storage medium having stored thereon computer program code, which, when run, performs the method as described in any one of claims 1 to 8.

11. A computer program product comprising a computer program that, when executed by a machine, performs the method as described in any one of claims 1 to 8.

Images