Processor operation method and device, electronic device and program product

Abstract

The present invention provides a processor operation method, a processor operation device, an electronic device, and a computer program product. The method comprises: loading at least a first part of a data block of a tensor from a global memory into a shared memory according to a sliding window; in the process of performing a matrix multiplication operation, extracting a mask value for a position of a data block in the shared memory where edge filling of the first part of the data block of the tensor is to be performed according to the sliding window to perform a matrix multiplication operation.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of processors, and more particularly to a processor operating method, a processor operating device, an electronic device, and a computer program product. Background Art

[0002] Convolutional Neural Network (CNN) is a deep learning algorithm mainly used in the field of computer vision. It has been widely used in image recognition, speech recognition, video recognition, natural language processing and other fields.

[0003] The convolution layer is the core part of CNN. It is used to extract features from input data such as images. The calculation range of convolution is performed within the spatial neighborhood of the pixel point, so the spatial information of the input data can be used. The main function of the convolution layer is to calculate the features of the input data. The convolution layer consists of several convolution kernels and possible bias terms. The convolution kernel is used to perform sliding window operations on the input data (such as images) to calculate the features of the input data. In image processing, the convolution kernel is usually a small matrix (or filter, template), whose size, shape and value determine the type of features to be calculated. Different convolution kernels can be used to calculate different features. For example, a convolution kernel is used to calculate edge features, a convolution kernel is used to calculate brightness features, and so on. The convolution kernel generates a pixel value on the output feature map by performing element-wise multiplication and summing (sometimes including a bias term) on the input data area covered by its sliding window. This process is repeated until the convolution kernel traverses the entire input data. Each time the convolution kernel (the sliding window) is moved, it interacts with a local region of the input data, computing features specific to that region.

[0004] By stacking multiple convolutional layers, CNN is able to learn more complex and abstract feature representations. Each convolutional layer receives the output of the previous layer as input data and applies a set of different convolution kernels to calculate higher-level features. This hierarchical feature calculation method enables CNN to perform well in tasks such as image recognition, object detection, and image segmentation.

[0005] In a traditional fully connected neural network, each neuron is connected to all neurons in the previous layer, which causes the number of network parameters to increase dramatically as the number of network layers increases. In contrast, in a convolutional neural network, the convolution layer implements local connectivity through the convolution kernel, that is, each neuron (or convolution kernel) is only connected to a local area of ​​the input data. This local connection feature greatly reduces the number of network parameters because the weights of each convolution kernel are shared and only related to a small area of ​​the input data. The direct benefit of the convolution kernel is to reduce the number of connections between network layers while reducing the risk of overfitting, which brings advantages to convolutional neural networks.

[0006] Edge padding is a common operation in image processing. It is mainly used to add extra pixels around the edges of an image (usually by repeating the values ​​of edge pixels, setting them to fixed values ​​such as 0 or 255, or other calculated values) so that the integrity of the image information can be maintained when performing certain image processing operations (such as convolution, etc.), preventing edge information loss or producing undesirable edge effects. Summary of the invention

[0007] According to one aspect of the present disclosure, at least one embodiment provides a processor operation method, including: loading at least a first portion of data blocks of a tensor from a global memory into a shared memory according to a sliding window; during a matrix multiplication operation, extracting a mask value for a position of a data block in the shared memory where edge filling of the first portion of data blocks of the tensor is to be performed according to the sliding window to perform a matrix multiplication operation.

[0008] According to another aspect of the present disclosure, at least one embodiment provides a processor operation device, comprising: a loading device, configured to load at least a first portion of data blocks of a tensor from a global memory into a shared memory according to a sliding window; a mask device, configured to perform a matrix multiplication operation according to a sliding window by extracting a mask value for a position of a data block in the shared memory where the edge of the first portion of data blocks of the tensor are to be filled.

[0009] According to another aspect of the present disclosure, at least one embodiment provides an electronic device including a processor operating device according to at least one embodiment of the present disclosure.

[0010] According to another aspect of the present disclosure, at least one embodiment provides a processor operating device, including: a memory for storing computer instructions; a processing unit for executing the computer instructions in the memory to implement a processor operating method according to at least one embodiment of the present disclosure.

[0011] According to another aspect of the present disclosure, at least one embodiment provides a computer program product, including computer instructions, wherein when the computer instructions are executed by a processor, the processor executes the processor operation method according to at least one embodiment of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or related technologies, the drawings required for use in the embodiments or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0013] Figure 1 A schematic diagram of multi-level storage of a graphics processing unit (GPU) for performing convolution calculations according to at least one embodiment of the present disclosure is shown.

[0014] Figure 2A It shows the edge padding of the input tensor when the convolution kernel is convolved with the input tensor in FULL mode. Figure 2B It shows the edge padding of the input tensor when the convolution kernel is convolved with the input tensor in VALID mode. Figure 2C It shows the edge padding of the input tensor when the convolution kernel is convolved with the input tensor in the SAME mode.

[0015] Figure 3 A schematic diagram showing the conversion process of img2col for an example.

[0016] Figure 4 A flowchart of a processor operating method according to at least one embodiment of the present disclosure is shown.

[0017] Figure 5A-5D A schematic diagram of a convolution method according to at least one embodiment of the present disclosure is shown.

[0018] Figure 6 A schematic diagram is shown in which, according to an embodiment of the present disclosure, when applied to back-propagation convolution, for a convolution kernel, the convolution kernel can be directly loaded into a corresponding position in a shared memory in a rotated form (rotated 180 degrees clockwise) through coordinate mapping.

[0019] Figure 7 A schematic diagram of a process of implementing a convolution calculation according to at least one embodiment of the present disclosure using a producer-consumer model of one or more warps according to at least one embodiment of the present disclosure is shown.

[0020] Figure 8 A block diagram of a processor operating apparatus according to at least one embodiment of the present disclosure is shown.

[0021] Fig. 9 A block diagram of a processor operating apparatus according to at least one embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0022] Reference will now be made in detail to specific embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. Although the present application will be described in conjunction with specific embodiments, it will be understood that it is not intended to limit the present application to the described embodiments. On the contrary, it is intended to cover changes, modifications, and equivalents included within the spirit and scope of the present disclosure. It should be noted that the method steps described herein can all be implemented by any functional block or functional arrangement, and any functional block or functional arrangement can be implemented as a physical entity or a logical entity, or a combination of the two.

[0023] The term "multiple" as used herein refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0024] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0025] Convolutional Neural Networks (CNN) are a type of feedforward neural network that includes convolution calculations and has a deep structure. It is one of the representative algorithms of deep learning. In CNN, the convolution kernel (or filter) is designed to perform a sliding window operation on the input data (such as an image) to calculate the local features of the data. Specifically, the convolution kernel slides on the input data with a certain stride and performs local connection and weight sharing operations at each position. At each position, the convolution kernel is element-wise multiplied and summed with the corresponding local area (i.e., window) of the input data (in some cases, an offset is added) to obtain a single point on the output feature map. As the convolution kernel slides on the input data, the entire output feature map of the image is calculated.

[0026] Artificial intelligence processors play an important role in convolutional computing, especially in the fields of deep learning, computer vision, and image processing. Artificial intelligence processors can include any of the following: Graphics Processing Unit (GPU), Tensor Processing Unit (TPU), Neural Network Processing Unit (NPU), Deep Learning Processing Unit (DPU), Accelerated Processing Unit (APU), and General-Purpose computing on Graphics Processing Unit (GPGPU).

[0027] The description of the embodiments herein is illustrated by taking a GPU as an example, but the embodiments of the present disclosure are not limited thereto.

[0028] Figure 1 A schematic diagram of multi-level storage of GPU 100 for performing convolution calculations according to at least one embodiment of the present disclosure is shown.

[0029] The multi-level storage structure of GPU 100 generally includes registers, caches (such as L1 cache, L2 cache), global memory (such as video memory in GPU 100), and possible external memory of GPU 100 (such as a hard disk) (not shown in the figure).

[0030] The basic processing unit of the GPU 100 is a streaming multiprocessor. Figure 1 The streaming multiprocessor 0 110 - 1 and the streaming multiprocessor 1 110 - 2 are shown, but this figure is only an example, and there may be more or fewer streaming multiprocessors in practice.

[0031] Each streaming multiprocessor has a register that is used to store temporary data and instructions. It has limited capacity but extremely fast access speed. Each streaming multiprocessor also has an L1 cache (optional) and shared memory. The L1 cache and shared memory are located between the registers and the global memory. The L1 cache can be responsible for caching memory addresses, while the shared memory can be responsible for caching recently accessed data blocks, including partial areas of the input tensor and convolution kernels.

[0032] The streaming multiprocessor may also have an L2 cache 120 (optional) and a global memory (or video memory) 130 outside the streaming multiprocessor to store data that can be shared between multiple streaming multiprocessors. The L2 cache can save a copy of a portion of the global memory that is often used.

[0033] Before the GPU 100 processes the convolution calculation, the central processing unit (CPU) 140 stores the input tensor and convolution kernel to be convolved in the hard disk of the CPU 140 (not shown in the figure) and sends the input tensor and convolution kernel to the global memory 130 of the GPU 100 (via the PCIe bus (or similar high-speed interface)), so that the input tensor and convolution kernel can be loaded from the hard disk of the CPU 140 to the global memory 130 of the GPU 100. Once these data are loaded into the global memory, they are ready for access by the streaming multiprocessor 110-1 or 110-2 of the GPU. When performing the convolution calculation, the threads of the GPU 100 will cooperate to load the relevant parts of the input tensor and convolution kernel in the global memory 130 into the shared memory 113-1 or 113-2 for calculation. In the convolution calculation, some parts of the input tensor may be reused by multiple convolution kernels, so using shared memory to store these data can improve the data reuse rate.

[0034] After the streaming multiprocessor 110 - 1 or 110 - 2 of the GPU 100 completes the convolution calculation, the convolution result is stored in the global memory 130 , and the CPU 140 can read the convolution result from the global memory 130 .

[0035] Three different convolution modes are usually implemented in CNN of deep learning framework, namely SAME, VALID, and FULL. The core difference between these three modes is that the moving area of ​​the convolution kernel is different, which leads to different sizes of output tensors.

[0036] Figure 2A It shows the edge padding of the input tensor when the convolution kernel is convolved with the input tensor in FULL mode. Figure 2B It shows the edge padding of the input tensor when the convolution kernel is convolved with the input tensor in VALID mode. Figure 2C The edge padding of the input tensor is shown when the convolution kernel is convolved with the input tensor in SAME mode. Note that in the application scenario of image recognition, the input tensor here can be the pixels of the image. In the application scenario of speech recognition, the input tensor here can be audio data. In the application scenario of video recognition, the input tensor here can be video data. In the application scenario of natural language processing, the input tensor here can be natural language text.

[0037] 1. In FULL mode, the convolution kernel starts convolution from the point where it intersects with the input tensor at one point. Figure 2A As shown, the square above represents the convolution kernel, the central gray square below represents the input tensor, and the white dotted square at the edge below represents the edge filling. Figure 2A It shows that the convolution kernel starts convolution with the upper left corner of the input tensor. Therefore, in FULL mode, the edges of the input tensor need to be padded (edge ​​padding) for convolution, and the size of the padding is relatively large (such as Figure 2A 8 blocks are shown). Therefore, the convolution kernel movement area in FULL mode is the largest, and the size of the output after convolution is also the largest. Edge padding refers to filling the edges of the input tensor with 0 or other values ​​during the convolution operation to control the size and shape of the output feature map.

[0038] 2. VALID mode is opposite to FULL mode. Convolution operation starts at the place where the entire convolution kernel overlaps with the input tensor, so there is no need to do edge padding, and the output size is also the smallest. Figure 2B As shown, the squares on the top represent the convolution kernels, and the grayscale squares on the bottom represent the input tensors without edge padding.

[0039] 3. SAME mode is the most commonly used mode. SAME means that the size of the output after convolution is consistent with the size of the input tensor (assuming that the sliding step size (stride) of the sliding window is 1). The starting position of the convolution kernel is determined by aligning the center of the convolution kernel with the first point of the input tensor, and then filling the corresponding edge padding bits. Figure 2C As shown, the upper square represents the convolution kernel, the central gray square below represents the input tensor, and the edge white dotted square below represents edge filling. Figure 2C and Figure 2A The difference is that Figure 2C The number of pixels filled at the edge is small (such as Figure 2C 5 squares are shown).

[0040] In convolutional neural networks, the implementation of convolutional layers is close to the cross-correlation operation defined in mathematics. By making certain changes to the convolution kernel and the input tensor, convolution can be equivalently converted to matrix and vector multiplication, which can improve the efficiency of convolution calculation. There are two conversion methods here, one is to expand and fill the convolution kernel and flatten the input tensor, and the other is to flatten the convolution kernel and expand and fill the input tensor. This article takes the expansion and filling of the input tensor and the flattening of the convolution kernel as an example for description, but the conversion method of expanding and filling the convolution kernel and flattening the input tensor can be derived by referring to the description in this article.

[0041] img2col (image to column) is an accelerated computing strategy for implementing convolution, which converts the convolution operation (Conv) into a matrix multiplication (Gemm) operation, thereby maximizing the reduction of convolution calculation time. In img2col, each local block (or local window, or receptive field) of each convolution of the input tensor is expanded into a column vector (column), and then all these column vectors are stacked into a large matrix. Each column of this matrix corresponds to a specific block in the original input tensor. Assuming the input tensor is x(NHWC) and the convolution kernel is w(RSKC), the converted matrix has two main dimensions: N (or Gemm_N): represents the number of all blocks, usually equal to (the height of the input tensor - the height of the convolution kernel + 1) × (the width of the input tensor - the width of the convolution kernel + 1) × (the number of channels of the input tensor, if it is a multi-dimensional convolution); K (or Gemm_K): represents the number of elements in each block, that is, the size of the convolution kernel multiplied by the number of channels of the convolution kernel (if it is a multi-dimensional convolution, it is the depth of the convolution kernel). img2col usually needs to establish a storage area (workspace) in the global memory (video memory), that is, it needs to increase the global memory, and convert the input tensor x(NHWC) into a matrix form of Gemm_N(NHW)×Gemm_K(RSC) in advance by reshaping and transposing each block (which increases the time consumption), and then it can be completely regarded as a matrix for matrix multiplication calculation between the two.

[0042] Figure 3 A schematic diagram showing the conversion process of img2col for an example.

[0043] Consider the convolution y = conv(x,w). The input tensor 310 is x(NHWC), where the batch size (N), height (H), width (W), and number of channels (C). The convolution kernel (or weight) 320 is w(RSKC), where R and S are the height and width of the convolution kernel, K is the number of output channels (i.e. the number of convolution kernels), and C is the number of input channels (the same as the number of channels in x). Figure 3 As shown, the number of input channels C is 2, the width W is 3, the height H is 3, and the number of output channels K is 2. Therefore, the channels are C0 and C1. The output tensor 330 of the convolution is y(NPQK), where P and Q are the height and width of the output feature map, respectively, and N and K are the same as the batch size and the number of output channels of the input tensor x, respectively. Figure 3 It is assumed that N=1.

[0044] During the img2col conversion process, each local block in x that matches the size of the convolution kernel (i.e., a block of RxS, where R and S are the height and width of the convolution kernel) is converted into a column vector. These column vectors are then stacked into a large matrix with a shape of (RSC, NPQ), where: RSC is the number of elements in each column vector, i.e., the convolution kernel size multiplied by the number of channels. NPQ is the total number of elements in the output feature map, i.e., the batch size N multiplied by the output height P multiplied by the output width Q.

[0045] Therefore, the convolution can be transformed into Gemm_M(K)×Gemm_N(NPQ)×Gemm_K(RSC) by img2col.

[0046] The advantage of img2col is that it simplifies the convolution operation into a matrix multiplication operation. When implementing it, you only need to focus on the implementation and performance optimization of the matrix multiplication operation. However, the disadvantage is that it increases the global memory (video memory) to store the converted matrix and increases the time consumption.

[0047] In addition to the explicit img2col, there is also an implicit matrix multiplication conversion (ImplicitGemm), which is to first store the original input tensor (activation area) (not expanded) and the convolution kernel (weight) to the global memory, and then load the convolution kernel (weight) and the corresponding input tensor (activation area) from the global memory to the shared memory in a matrix form through coordinate mapping to complete the corresponding matrix multiplication Gemm operation. Any convolution kernel size is converted into a 1x1 convolution for calculation, that is, each convolution kernel is calculated separately, and there are R×S accumulations. Specifically, ImplicitGemm needs to map the coordinates of the input tensor (after each sliding window movement) to the matrix index of the Gemm operation, so that when the part of the input tensor covered by each sliding window is directly loaded into the shared memory, the part of the input tensor covered by each sliding window can be directly loaded into the mapped corresponding coordinates of the shared memory in a matrix form, which is equivalent to performing a Gemm operation on the original data. In terms of memory usage, ImplicitGemm does not require the additional video memory space occupied by expanding the input tensor, but the calculation of the mapping coordinates is added, which increases the amount of calculation.

[0048] In the above img2col and ImplicitGemm methods, when edge padding in FULL mode and SAME mode is involved, the input tensor needs to be edge-padded first, loaded into global memory and / or shared memory, and then subsequent conversion and calculation are performed. The edge-padded data (for example, filling with 0 at some positions) causes additional storage space of global memory and / or shared memory to be occupied.

[0049] The defects existing in the solutions in the above-mentioned prior art are also the results obtained by the inventor after careful research through practice and creative labor. The process of discovering the above-mentioned problems and the solutions proposed for the above-mentioned problems in at least one embodiment disclosed in this text are all creative contributions of the inventor in the invention process.

[0050] At least one embodiment of the present disclosure can first load at least the first part of the data blocks of the input tensor (without edge filling) from the global memory to the shared memory according to the sliding window; in the process of performing the matrix multiplication operation, according to the sliding window, the mask value is extracted for the position of the data block in the shared memory to perform the edge filling of the first part of the data blocks of the input tensor to perform the matrix multiplication operation, so there is no need to perform edge filling on the input tensor in advance, and no additional global memory and / or shared memory storage space is added to store the edge filling data itself, so as to implement the convolution operation. At least one embodiment of the present disclosure can use implicit matrix multiplication conversion in the process of loading the input tensor from the global memory to the shared memory, without the need for prior expansion, so as to further reduce the occupation of the global memory and / or shared memory. At least one embodiment of the present disclosure can be applicable to a general non-strided convolution operator, and can be applicable to convolution operators and back propagation algorithms (BPA) with any input tensor size, any convolution kernel size, and any edge filling. At least one embodiment of the present disclosure can adopt a collaborative hardware scheduling strategy, use pipelines to implement data input / output and calculation, hide memory access latency, and maximize hardware parallelism. At least one embodiment of the present disclosure can make full use of the multi-level storage space of the hardware, and reduce memory access latency and memory access requests by using L2 cache, shared memory, L1 cache, and registers.

[0051] Figure 4 A flowchart of a processor operating method according to at least one embodiment of the present disclosure is shown.

[0052] Note that the execution subject of the method provided in the embodiment of the present disclosure may be a computer device with certain computing capabilities (eg, a GPU, etc.).

[0053] like Figure 4 As shown, the processor operation method includes step 410 and step 420. The processor operation method here is mainly a processor operation for convolution operation. Since the convolution operation can be converted into a matrix multiplication operation in the manner described above, the following description of the steps does not mention "convolution", but the ultimate purpose of these steps is to complete the convolution operation.

[0054] In step 410, at least a first portion of the data blocks of the tensor are loaded from the global memory into the shared memory according to the sliding window.

[0055] In the case of computing a forward propagation convolution, the tensor may be an input tensor and / or a convolution kernel. Alternatively, in the case of computing a backward propagation convolution, the tensor may be an output tensor and / or a rotated convolution kernel. This document describes the tensor as an input tensor, but the tensor may include a convolution kernel, an output tensor, and / or a convolution kernel rotated 180 degrees clockwise. Here, the "first" indication in "tensor" is used to refer to any tensor, rather than to limit a specific order or number.

[0056] Here, before the convolution calculation begins, the input data (such as an image or a feature map) and / or the convolution kernel (or filter) are usually loaded from the global memory to the shared memory. The global memory or shared memory can be a dynamic random access memory (DRAM), a static random access memory (SRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDR SDRAM), or a synchronous dynamic graphics random access memory (DDR SGRAM).

[0057] Here, in some embodiments, according to the sliding window, step 410 of loading at least a first portion of the data block of the tensor from the global memory into the shared memory may include not padding the edges of the tensor. In some embodiments, according to the sliding window, step 410 of loading at least a first portion of the data block of the tensor from the global memory into the shared memory may include loading the first portion of the data block next to the data stored in the shared memory. The stored data here may also be valid data of other tensors.

[0058] In this way, shared memory can be utilized to the maximum extent to store valid data, such as input tensors and / or convolution kernels themselves, rather than edge padding values ​​(such as 0, other constants, or replicated edge pixel values, etc.).

[0059] In step 420, during the matrix multiplication operation, according to the sliding window, a mask value is extracted for the position of the data block in the shared memory where the edge filling of the first part of the data blocks is to be performed to perform the matrix multiplication operation.

[0060] The above Figure 2A and 2CAs shown, for FULL mode and SAME mode, when the sliding window moves to the edge of the input tensor, the input tensor may need to be edge-filled in order to be convolved with the convolution kernel. However, since the input data and / or convolution kernels stored in the shared memory are valid data themselves, there is no edge-filled value. In some embodiments, the data stored at the location of the data block in the shared memory where the first part of the data block of the tensor is to be edge-filled can be valid data of the second part of the data block of the tensor or other tensors. Therefore, in the case where the input tensor needs to be edge-filled to be convolved with the convolution kernel, if the elements near the edge of the input tensor in the shared memory are directly extracted according to the sliding window, the elements of other valid data stored next to the edge of the input tensor will be extracted for matrix multiplication, resulting in an error in the result of the convolution.

[0061] Therefore, in some embodiments, the positions of the data blocks in the shared memory where the edge filling of the first part of the data blocks of the tensor is to be performed may be masked according to the sliding window to form the mask value.

[0062] In some embodiments, according to the sliding window, the step of masking the position of the data block in the shared memory to fill the edge of the first part of the data block of the tensor to form the mask value may include: determining the coordinates of the position of the data block included in the sliding window to fill the edge of the first part of the data block of the tensor according to the offset between the starting coordinates of the sliding window and the starting coordinates of the tensor; masking the coordinates of the position of the data block to fill the edge of the first part of the data block of the tensor to generate a mask value.

[0063] In some embodiments, the step of masking the position of the data block in the shared memory to perform edge filling on the first part of the data block of the tensor according to the sliding window to form the mask value can be implemented by software through the (convolution) software instruction of the computer. That is, there is no need to actually perform edge filling in hardware or actually overwrite the elements of the original valid data with the edge filling value, but in the implementation process of the (convolution) software instruction for extracting at least the first part of the data block of the tensor from the shared memory to perform the matrix multiplication operation, according to the sliding window, the position of the data block in the shared memory to perform edge filling on the first part of the data block of the tensor is masked by the (convolution) software instruction of the computer to form a mask value (the mask value is configurable by software and is usually 0).

[0064] Note that since real data is not actually extracted from the position of the data block to be edge-filled, the actual starting logical coordinates of the extraction instruction must be converted into the coordinates of the actual valid data of the first part of the data block of the tensor in the shared memory to extract at least the first part of the data block of the tensor from the shared memory, and the extraction of the mask value from the position of the data block in the shared memory where the edge of the first part of the data block of the tensor is to be filled can be virtually completed through software instructions, without extracting actual valid data from the shared memory.

[0065] In this way, the embodiment of the present disclosure first loads at least the first part of the data blocks of the input tensor (without edge filling) from the global memory to the shared memory according to the sliding window; in the process of performing the matrix multiplication operation, according to the sliding window, the mask value is extracted for the position of the data block in the shared memory where the edge of the first part of the data block of the input tensor is to be filled, so that the convolution operation is realized without having to fill the edges of the input tensor in advance and without increasing the storage space of the additional global memory and / or shared memory to store the edge-filled data itself.

[0066] Note that in some embodiments, the process of loading the tensor from the global memory into the shared memory can be performed by the aforementioned implicit matrix multiplication conversion ImplicitGemm, that is, the various parts of the input tensor (activation area) to be convolved corresponding to the convolution kernel (weight) are loaded from the global memory into the shared memory directly in the form of a matrix through coordinate mapping, and then (streaming multiprocessor) reads data from the shared memory to complete the corresponding matrix multiplication Gemm operation. According to at least one embodiment of the present disclosure, in the process of loading the input tensor from the global memory into the shared memory, an implicit matrix multiplication conversion can be used without prior expansion, further reducing the occupation of the global memory and / or shared memory. In this way, the embodiment of the present disclosure combines the extraction method of the implicit matrix multiplication conversion ImplicitGemm, and loads the first part of the data block of the tensor to the corresponding position in the shared memory through coordinate mapping to complete the implicit matrix multiplication conversion, which can achieve the effect of the implicit matrix multiplication conversion ImplicitGemm without the need to fill the edges of the input tensor in advance, and without adding additional global memory and / or shared memory storage space to store the edge filling data itself. That is, the solution according to at least one embodiment of the present disclosure can very efficiently utilize memory space to implement convolution calculations with edge padding.

[0067] In addition, at least one embodiment of the present disclosure may be Figure 1 The multi-level storage shown is implemented in the graphics processing unit, so the multi-level storage space of the hardware can be fully utilized by utilizing the Figure 1The global memory, L2 cache, shared memory, L1 cache and / or registers shown are used to reduce memory access latency and memory access requests.

[0068] Note that at least one embodiment of the present disclosure may be applicable to convolution calculations of forward propagation as well as convolution calculations of backward propagation.

[0069] The following describes the process of the convolution method according to at least one embodiment of the present disclosure by taking the forward propagation convolution calculation as an example.

[0070] For example, for the forward pass, y=conv(x,w), where the input tensor is x(NHWC), the convolution kernel is w(RSKC), and the output tensor is y(NPQK). The dimensions of the input tensor are batch size (N), height (H), width (W), and number of channels (C). The dimensions of the convolution kernel are: R and S are the height and width of the convolution kernel, K is the number of output channels (i.e. the number of convolution kernels), and C is the number of input channels (the same as the number of channels in x). The dimensions of the output tensor are: P and Q are the height and width of the output feature map, respectively, and N and K are the same as the batch size and number of output channels of the input tensor x.

[0071] Figures 5A-5D A schematic diagram of a convolution method according to at least one embodiment of the present disclosure is shown.

[0072] Assume that in this example, the input tensor x(NHWC) is (N,10,9,64), the convolution kernel w(RSKC) is (3,3,K,64), the output tensor y(NPQK) (N,10,9,K), the sliding window sliding step (stride) is (1,1), the edge padding (padding) is (1,1,1,1) (indicates that 1 unit of zero padding is added on both sides of the height and width of the input tensor x), and the dilation or hole convolution (dilation) is (1,1), indicating no dilation, that is, using a standard convolution kernel. The input tensor and convolution kernel are stored on the CPU's hard disk.

[0073] Figure 5A Shows storing tensors from an external hard disk to global memory. Figure 5A The 3×3 block above represents the convolution kernel (weight), where the gray block represents w(0,0) in the convolution kernel (weight). Here, for example, for the pixel in the input tensor x corresponding to w(0,0) in the convolution kernel (weight), Figure 5A The dotted boxes represent pixels stored in the global memory that need to be multiplied with w(0,0), where N=0, there are 10×9 boxes, N=1, there are 10×9 boxes, and so on. Figures 5A-5D Each box in represents 1 pixel × 128 channels.

[0074] Figure 5B Indicates the situation where two data (N=0 and N=1) are loaded from global memory to shared memory according to a sliding window (e.g., Tensor Memory Access Copy (TmaCopy)). First, the N, H, and W dimensions of the input tensor are synthesized into a continuous pixel dimension, that is, a pixel dimension of N×10×9. Then, the valid data of the pixel dimension of the input tensor is continuously loaded into the shared memory (or TmaCopy), where the valid data does not include the edge padding value. For example, with a granularity of 64 pixels per copy, 64 consecutive pixels are copied to the shared memory each time in the height or width direction (or both).

[0075] like Figure 5B As shown, it can be seen that the two data are stored continuously, with no empty space or edge filling values ​​in between.

[0076] Next, during the matrix multiplication operation, the positions of the data blocks in the shared memory where the edge padding of the first part of the data blocks of the tensor is to be performed are masked according to the sliding window to form mask values.

[0077] For example, Figure 5C Representing data of two input tensors x copied to shared memory, the coordinates of the position of the data block included in the sliding window for edge filling of the input tensor are determined through information such as the starting coordinates of the pixel dimension data of the input tensor x in the shared memory (for example, (0,0,0)) and the offset between the starting coordinates of the sliding window and the starting coordinates of the input tensor (for example, (-1,-1,0)); a mask value is generated for the coordinates of the position of the data block for edge filling of the input tensor (i.e., masking is performed), and the mask value is used as the edge filling data block of the pixel dimension data included in the sliding window to perform matrix multiplication with the valid data of the pixel dimension of the convolution kernel using the convolution instruction supported by the hardware.

[0078] like Figure 5C It can be seen that the coordinates of the data blocks for edge filling of the input tensor are (outside the upper edge) (-1,-1,0), (-1,-1,1), (-1,-1,2), (-1,-1,3), (-1,-1,4), (-1,-1,5), (-1,-1,6), (-1,-1,7), (-1,-1,8). Originally, at these coordinates, there were second-part data blocks of the input tensor copied from the global memory. However, according to an embodiment of the present disclosure, these coordinates are masked so that when performing matrix multiplication, the masked values ​​(instead of the true values ​​at these coordinates) are used to perform matrix multiplication with w(0,0) in the convolution kernel.

[0079] Likewise, operations of masking coordinates outside the left edge, masking coordinates outside the right edge, and masking coordinates outside the bottom edge (not shown) are similar to the above-described masking operations.

[0080] Figure 5D Indicates the equivalent data arrangement after masking. That is, the position of the mask is equivalent to the effect of filling the edge with 0.

[0081] In this way, the embodiment of the present disclosure first loads the original tensor (without edge filling) into the shared memory, and in the process of executing matrix multiplication, generates a mask for part of the valid data in the shared memory as part of the edge filling of the tensor to take the tensor out of the shared memory for matrix multiplication calculation. Therefore, there is no need to fill the edges of the tensor in advance, and no additional global memory and / or shared memory storage space is added to store the edge filling data itself, so as to implement the convolution operation.

[0082] The method of matrix multiplication for each of the other w(0,1), w(0,2), w(1,0), w(1,1), w(2,1), w(2,0), w(2,1), and w(2,2) in the convolution kernel (weight) is the same as Figures 5A-5D The description is similar and will not be described here one by one.

[0083] For back propagation, dx = conv(dy, rotated(w)), where the output tensor is dx(NHWC), the input tensor is dy(NPQK), the convolution kernel is w(RSKC), H = P, W = Q, and rotate(w) is to rotate w(RSKC) 180 degrees clockwise. All convolution kernels are equivalent to the calculation of the convolution kernel 1x1x1. If conv(dy, rotated(w)) is converted to gemm_M(C)×gemm_N(NHW)×gemm_K(RSK) by the img2col algorithm, the left matrix of the matrix multiplication is the convolution kernel (weight), and the right matrix is ​​the output tensor (activation function).

[0084] According to an embodiment of the present disclosure, the output tensor and the convolution kernel can be loaded into a shared memory. During the loading process into the shared memory, for the output tensor, the various parts of the output tensor to be convolved can be directly loaded into the corresponding positions in the shared memory through coordinate mapping, and for the convolution kernel, the convolution kernel can be directly loaded into the corresponding position in the shared memory in a rotated form (rotated 180 degrees clockwise) through coordinate mapping. Figure 6FIG. 1 is a schematic diagram showing that, according to an embodiment of the present disclosure, when a convolution kernel is applied to a back-propagation convolution, the convolution kernel can be directly loaded into a corresponding position in a shared memory in a rotated form (rotated 180 degrees clockwise) through coordinate mapping. Figure 6 As shown, the original convolution kernel is Figure 6 The arrangement on the left side becomes: Figure 6 The arrangement on the right allows for rotation when loaded into shared memory.

[0085] The subsequent masking steps are the same as those in the forward propagation case and will not be repeated here.

[0086] Figure 5A-5D as well as Figure 6 This is only an example. In fact, tensors of other shapes and sizes, convolution calculations of convolution kernels, and other edge filling situations can all be applicable to the embodiments of the present disclosure.

[0087] In this way, at least one embodiment of the present disclosure can be applicable to general convolution calculations, and can be applicable to convolution calculations and forward / backward propagation algorithms with any input tensor size, any convolution kernel size, and any edge padding.

[0088] During the entire convolution process, pipeline technology can realize parallel processing of tasks by decomposing complex tasks into multiple stages.

[0089] In some embodiments, loading tensors into shared memory can be implemented by a first thread bundle that loads input tensors or output tensors and / or a second thread bundle that loads convolution kernels or rotated convolution kernels, extracting at least input tensors or output tensors from shared memory can be implemented by a third thread bundle, extracting convolution kernels or rotated convolution kernels from shared memory to perform matrix multiplication operations with input tensors or output tensors can be implemented by a fourth thread bundle, and storing the results of the matrix multiplication operation can be implemented by a fifth thread bundle.

[0090] A warp usually contains 32 threads, which can execute the same instructions in lock-step to reduce the overhead of thread scheduling and management and increase the parallelism of instruction execution.

[0091] Each warp can be synchronized through a pipeline producer-consumer model, so that operations of other warps can be performed during the delay of a warp accessing memory, thereby improving parallel execution efficiency.

[0092] Figure 7A schematic diagram of a process of implementing a convolution calculation according to at least one embodiment of the present disclosure using a producer-consumer model of one or more warps according to at least one embodiment of the present disclosure is shown.

[0093] like Figure 7 As shown, warp 0 (producer) may be responsible for loading a portion of the input tensor or output tensor (e.g., A data) to be convolved into shared memory, warp 1 (producer) may be responsible for loading a portion of the convolution kernel or the rotated convolution kernel (e.g., B data) to be convolved into shared memory, and warp 2 (warp2) (producer) may be responsible for loading another portion of the input tensor or output tensor (e.g., A data) to be convolved into shared memory. Warp groups 0 and 1 (multiple warps in warp group 0 and warp group 1, respectively) (as consumers) are responsible for extracting a portion of the input tensor or output tensor (e.g., A data) to be convolved and a portion of the convolution kernel or the rotated convolution kernel (e.g., B data) to be convolved (as well as performing necessary masking of edge padding portions), performing matrix multiplication, storing matrix multiplication results, and the like.

[0094] In this way, at least one embodiment of the present disclosure may adopt a cooperative hardware scheduling strategy, use pipeline thread warps to implement data input / output and calculation, hide memory access latency, and maximize hardware parallelism.

[0095] In summary, at least one embodiment of the present disclosure first loads the original tensor (without edge padding) into the shared memory, and generates a mask for updating part of the valid data in the memory during the process of performing matrix multiplication as part of the edge padding of the tensor to take the tensor out of the shared memory for matrix multiplication calculation, so there is no need to pad the edges of the tensor in advance, and no additional global memory and / or shared memory storage space is added to store the edge-filled data itself, so as to implement the convolution operation. According to at least one embodiment of the present disclosure, in the process of loading the input tensor from the global memory to the shared memory, an implicit matrix multiplication conversion can be used without prior expansion, further reducing the occupation of the global memory and / or shared memory. At least one embodiment of the present disclosure can be applied to a general non-strided convolution operator, and can be applied to convolution operators and back propagation algorithms with any input tensor size, any convolution kernel size, and any edge padding. At least one embodiment of the present disclosure can adopt a collaborative hardware scheduling strategy, use pipelines to implement data input / output and calculation, hide memory access delays, and maximize hardware parallelism. At least one embodiment of the present disclosure can fully utilize the multi-level storage space of hardware and reduce memory access latency and memory access requests by utilizing L2 cache, shared memory, L1 cache, and registers.

[0096] Figure 8 A block diagram of a processor operating apparatus 800 is shown in accordance with at least one embodiment of the present disclosure.

[0097] like Figure 8 As shown, the apparatus 800 may include a loading device 810 and a masking device 820 .

[0098] The loading device 810 may be configured to load at least a first portion of data blocks of the tensor from the global memory to the shared memory according to the sliding window.

[0099] The mask device 820 may be configured to perform a matrix multiplication operation by extracting a mask value for a position of a data block in the shared memory where edge filling of the first part of the data blocks is to be performed according to a sliding window during the matrix multiplication operation.

[0100] In some embodiments, the mask device 820 may also be configured to mask the position of the data block in the shared memory where the edge filling of the first part of the data blocks is to be performed according to the sliding window to form a mask value.

[0101] In some embodiments, the mask value is configurable by software and is typically 0.

[0102] In some embodiments, according to the sliding window, the positions of the data blocks in the shared memory where the edge filling of the first part of the data blocks is to be performed are masked to form mask values, which is implemented by software instructions.

[0103] In some embodiments, the mask device 820 can be configured to determine the coordinates of the position of the data block in the sliding window where the edge filling of the first part of the data block is to be performed based on the offset between the starting coordinates of the sliding window and the starting coordinates of the tensor; and mask the coordinates of the position of the data block where the edge filling of the first part of the data block is to be performed to generate a mask value.

[0104] In some embodiments, the loader 810 may be configured to not perform edge padding on the tensor.

[0105] In some embodiments, the loading device 810 may be configured to load the first portion of data blocks next to the data stored in the shared memory.

[0106] In some embodiments, the data stored at the location of the data block in the shared memory where the edge padding of the first portion of the data block is to be performed is valid data of the second portion of the data block of the tensor or other tensors.

[0107] In some embodiments, the loading device 810 may be configured to load the first portion of the data block to a corresponding position in the shared memory through coordinate mapping to complete an implicit matrix multiplication transformation.

[0108] In some embodiments, the tensor may be an input tensor and / or a convolution kernel, or the tensor may be an output tensor and / or a rotated convolution kernel.

[0109] In some embodiments, loading tensors into shared memory is implemented by loading the first thread bundle of the input tensor or the output tensor and / or loading the convolution kernel or the rotated convolution kernel by the second thread bundle, extracting at least the input tensor or the output tensor from the shared memory is implemented by the third thread bundle, extracting the convolution kernel or the rotated convolution kernel from the shared memory to perform a matrix multiplication operation with the input tensor or the output tensor is implemented by the fourth thread bundle, and storing the result of the matrix multiplication operation is implemented by the fifth thread bundle.

[0110] In this way, at least one embodiment of the present disclosure first loads the original tensor (without edge filling) into the shared memory, and in the process of executing matrix multiplication, generates a mask for part of the valid data in the shared memory as part of the edge filling of the tensor to take the tensor out of the shared memory for matrix multiplication calculation. Therefore, there is no need to fill the edges of the tensor in advance, and no additional global memory and / or shared memory storage space is added to store the edge filling data itself, so as to implement the convolution operation.

[0111] Fig. 9 A block diagram of a processor operating apparatus 900 is shown in accordance with at least one embodiment of the present disclosure.

[0112] like Fig. 9 As shown, the processor operation device 900 may include a processing unit 910 and a memory 920, wherein the memory 920 is coupled to the processing unit 910 and stores computer instructions therein for performing the steps of each method of at least one embodiment of the present disclosure when executed by the processing unit 910.

[0113] The processing unit 910 may include, but is not limited to, one or more processors or microprocessors.

[0114] The memory 920 may include, but is not limited to, for example, random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, computer storage media (e.g., hard disk, floppy disk, solid-state drive, removable disk, CD-ROM, DVD-ROM, Blu-ray disk, etc.).

[0115] In addition, the processor operating device may also include (but not limited to) a data bus, an input / output (I / O) bus, a display, and input / output devices (eg, keyboard, mouse, speaker, etc.), etc.

[0116] The processing unit 910 can communicate with an external display and input / output devices through an I / O bus.

[0117] In one embodiment, the at least one computer instruction may also be compiled into or constitute a computer program product or software product, wherein one or more computer instructions, when executed by a processor, perform the various functions and / or method steps in the embodiments described in the present technology.

[0118] At least one embodiment of the present disclosure further provides a processor, including: a processing unit and a control unit configured to control the processing unit to execute a method according to at least one embodiment of the present disclosure. The processing unit may include Figure 1 The streaming multiprocessor shown, or other processor, and the control unit (not shown in the figure) can be hardware, software, firmware or a combination of the above for controlling the operation of the processing unit.

[0119] At least one embodiment of the present disclosure also provides an electronic device including a processor operating device according to at least one embodiment of the present disclosure.

[0120] At least one embodiment of the present disclosure also provides a non-transitory computer-readable storage medium. Instructions are stored on the non-transitory computer-readable storage medium, and the instructions are, for example, computer instructions. When the computer instructions are executed by the processor, the various methods described above can be executed. The non-transitory computer-readable storage medium includes, but is not limited to, for example, random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, register, computer storage medium (such as a hard disk, a floppy disk, a solid-state hard disk, a removable disk, a CD-ROM, a DVD-ROM, a Blu-ray disc, etc.), etc. For example, the non-transitory computer-readable storage medium can be connected to a computing device such as a computer, and then, when the computing device executes the computer instructions stored on the computer-readable storage medium, the various methods described above can be performed.

[0121] The present disclosure may also include a computer program product, wherein the computer program product can perform the methods, steps, and operations presented herein. For example, such a computer program product can be a computer software package, computer code instructions, a computer-readable tangible medium having computer instructions tangibly stored (and / or encoded) thereon, the instructions being executable by a processor to perform the operations described herein. The computer program product can include packaging materials.

[0122] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably therewith. The words "such as / for example" used herein refer to the phrase "such as / for example but not limited to", and can be used interchangeably therewith.

[0123] The step flow charts and the above method descriptions in this disclosure are only illustrative examples and are not intended to require or imply that the steps of each embodiment must be performed in the order given. As will be appreciated by those skilled in the art, the order of the steps in the above embodiments can be performed in any order. Words such as "thereafter", "then", "next", etc. are not intended to limit the order of steps; these words are only used to guide the reader through the description of these methods. In addition, any reference to a singular element, such as using the article "one", "one", or "the" is not to be construed as limiting the element to the singular.

[0124] In addition, the steps and devices in the various embodiments of this document are not limited to being implemented in a certain embodiment. In fact, according to the concept of this disclosure, relevant partial steps and partial devices in the various embodiments of this document can be combined to conceive new embodiments, and these new embodiments are also included in the scope of this disclosure.

[0125] The above methods may be implemented in hardware, software, firmware or any combination thereof.

[0126] In addition, modules and / or other appropriate means for performing the methods and techniques described herein can be downloaded from a server via wireless communication when appropriate. Alternatively, the various methods described herein can be provided via a storage component so that the various methods are available when coupled to the storage component. In addition, any other appropriate technology for providing the methods and techniques described herein to a device can be utilized.

[0127] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit at least one embodiment of the present disclosure to the form disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A processor operation method, comprising: Loading at least a first portion of a block of the tensor from a global memory into a shared memory according to the sliding window without edge padding the tensor; In the process of performing the matrix multiplication operation, according to the sliding window, a mask value is extracted for the position of the data block in the shared memory where the edge filling of the first part of the data blocks is to be performed to perform the matrix multiplication operation; According to the sliding window, masking the position of the data block in the shared memory where the edge filling of the first part of the data blocks is to be performed to form the mask value; The step of masking the position of the data block in the shared memory where the edge of the first part of the data blocks is to be filled according to the sliding window to form the mask value includes: Determine, according to an offset between a starting coordinate of the sliding window and a starting coordinate of the tensor, a coordinate of a position of the data block included in the sliding window where edge filling of the first portion of the data blocks is to be performed; The coordinates of the positions of the data blocks where the edge filling of the first portion of the data blocks is to be performed are masked to generate the mask values.

2. The processor operating method according to claim 1, wherein: The mask value is configurable through software.

3. The processor operating method according to claim 1, wherein: The step of masking the positions of the data blocks in the shared memory where the first part of the data blocks are to be edge-filled according to the sliding window to form the mask value is implemented by software instructions.

4. The processor operating method according to claim 1, wherein: The loading of at least a first portion of the data block of the tensor from the global memory to the shared memory according to the sliding window includes not performing edge padding on the tensor.

5. The processor operating method according to claim 1, wherein: Loading at least a first portion of data blocks of the tensor from the global memory to the shared memory according to the sliding window includes loading the first portion of data blocks next to data stored in the shared memory.

6. The processor operating method according to claim 1, wherein: The data stored at the location of the data block in the shared memory where the edge filling of the first part of the data block is to be performed is the valid data of the second part of the data block of the tensor or other tensors.

7. The processor operating method according to claim 1, wherein: The step of loading at least the first portion of data blocks from the global memory to the shared memory according to the sliding window includes: The first part of the data block is loaded into the corresponding position in the shared memory through coordinate mapping to complete the implicit matrix multiplication conversion.

8. The processor operating method according to claim 1, wherein: The tensor is an input tensor and / or a convolution kernel, or the tensor is an output tensor and / or a rotated convolution kernel.

9. The processor operating method according to claim 8, wherein: The loading of the tensor into the shared memory is implemented by loading a first thread warp of an input tensor or an output tensor and / or loading a second thread warp of a convolution kernel or a rotated convolution kernel, Extracting at least the input tensor or the output tensor from the shared memory is implemented by a third thread warp, Extracting the convolution kernel or the rotated convolution kernel from the shared memory to perform the matrix multiplication operation with the input tensor or the output tensor is implemented by a fourth thread warp, Storing the result of the matrix multiplication operation is implemented by the fifth warp.

10. A processor operating device, comprising: A loading device configured to load at least a first portion of the data blocks of the tensor from the global memory into the shared memory according to the sliding window without edge padding the tensor; The mask device is configured to extract a mask value according to the sliding window and perform a matrix multiplication operation on a position of a data block in the shared memory where the edge filling of the first part of the data blocks is to be performed during the matrix multiplication operation, The masking device is configured to mask the position of the data block in the shared memory where the edge filling of the first part of the data blocks is to be performed according to the sliding window to form the mask value; Wherein, the mask device is configured to determine the coordinates of the position of the data block to be edge-filled for the first part of the data block in the sliding window according to the offset between the starting coordinates of the sliding window and the starting coordinates of the tensor; and mask the coordinates of the position of the data block to be edge-filled for the first part of the data block to generate the mask value.

11. A processor operating device, comprising: Memory, for storing computer instructions; A processing unit, configured to execute the computer instructions in the memory to implement the processor operation method according to any one of claims 1-9.

12. An electronic device comprising the processor operating device according to claim 10 or 11.

13. A computer program product storing computer instructions, in, When the computer instructions are executed by a processor, the processor is caused to perform the processor operation method according to any one of claims 1 to 9.

Images