Data processing method and device, processor, electronic device and storage medium

Abstract

A data processing method and device, a processor, an electronic device and a storage medium. The data processing method comprises: performing L first operations for L types of operation areas respectively, so as to load the output gradient tensor from the memory to the cache component of the processor, and using the thread bundle to perform the back propagation operation of strided convolution on the output gradient tensor, to obtain the input gradient tensor and store it in the memory; wherein, the back propagation operation result of the same type of operation area is obtained by performing the first operation once, and the input gradient tensor is obtained by sequentially splicing the back propagation operation results of the L types of operation areas, and when performing the first operation for any type of operation area in the L types of operation areas, it includes performing the back propagation operation of at least one convolution kernel element, and the at least one convolution kernel element is the convolution kernel element of the convolution kernel tensor corresponding to any type of operation area on the plane formed by the R dimension and the S dimension.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to a data processing method, a data processing apparatus, a processor, an electronic device, and a non-transitory computer-readable storage medium. Background Art

[0002] Strided convolution is a special convolution operation in which the convolution kernel (or filter) does not move one pixel at a time on the input feature map, but skips some pixels. The number of pixels skipped is called the stride, and for strided convolution, the stride is greater than 1. Strided convolution has the effect of downsampling, which can reduce the size of the feature map while retaining important feature information.

[0003] The reverse feature gradient calculation is part of the back-propagation algorithm in the neural network, which is used to calculate the gradient of the loss function with respect to the input feature map. This gradient is used to update the weights and biases of the neural network in order to minimize the loss function and improve the accuracy of the model.

[0004] In strided convolution, the process of reverse feature gradient calculation is similar to standard convolution, but because the stride will cause a size mismatch between the input feature map and the output feature map, the gradient needs to be appropriately upsampled (or interpolated) during back propagation to align it with the size of the input feature map. Summary of the invention

[0005] At least one embodiment of the present disclosure provides a data processing method, comprising: performing L first operations for L types of operation areas respectively, so as to load an output gradient tensor from a memory into a cache component of a processor, and using a thread warp to perform a strided convolution back-propagation operation on the output gradient tensor to obtain an input gradient tensor and store it in the memory, wherein the output gradient tensor is expressed as [N, P, Q, K], where the N dimension is used to represent the number of batches, the P dimension is used to represent the height of the output gradient, the Q dimension is used to represent the width of the output gradient, and the K dimension is used to represent the number of output channels; a convolution kernel tensor used for the strided convolution is expressed as [R, S, K, C], where the R dimension is used to represent the height of the convolution kernel, the S dimension is used to represent the width of the convolution kernel, and the C dimension is used to represent the number of input channels; the input gradient tensor is expressed as [N, H, W, C], where the W dimension is used to represent the width of the input gradient, and the H dimension is used to represent the width of the input gradient. The dimension is used to represent the height of the input gradient; the input gradient tensor is divided into the L types of operation areas at least on the plane formed by the H dimension and the W dimension, and the operation areas of the same type are separated by s elements on the H dimension and s elements on the W dimension, each operation area includes 1 element, s is a positive integer and is the step size of the strided convolution, and L is a positive integer and is equal to s×s; wherein, when performing the first operation on any type of operation area in the L types of operation areas, it includes performing a backpropagation operation of at least one convolution kernel element, and the at least one convolution kernel element is a convolution kernel element of the convolution kernel tensor corresponding to the any type of operation area on the plane formed by the R dimension and the S dimension, and the at least one convolution kernel element is used to obtain the backpropagation operation result of the any type of operation area through the backpropagation operation.

[0006] For example, in the data processing method provided by at least one embodiment of the present disclosure, for any type of operation area, the first operation includes the following operations: loading the first matrices corresponding to the at least one convolution kernel element from the memory to the cache component; loading at least one second matrix corresponding to each first matrix to the cache component; using the thread warp to perform matrix multiplication and accumulation operations between the loaded first matrix and the second matrix to obtain the operation results corresponding to the at least one convolution kernel element; accumulating the operation results corresponding to the at least one convolution kernel element to obtain the back propagation operation results of any type of operation area; and striding and storing the back propagation operation results of any type of operation area to the memory.

[0007] For example, in the data processing method provided by at least one embodiment of the present disclosure, the first matrices corresponding to the at least one convolution kernel element are loaded from the memory to the cache component, including: for any convolution kernel element among the at least one convolution kernel element, the first matrix corresponding to the any convolution kernel element is loaded to the cache component, wherein the shape size of the first matrix is ​​A×B, the B elements located in the same row of the first matrix are B consecutive elements in the convolution kernel tensor that have the same coordinates as the any convolution kernel element in the R dimension, the S dimension and the C dimension, but different coordinates in the K dimension, different row elements in the first matrix have different coordinates in the C dimension, and A and B are positive integers.

[0008] For example, in the data processing method provided by at least one embodiment of the present disclosure, loading at least one second matrix corresponding to each first matrix into the cache component includes: loading at least one second matrix corresponding to the first matrix into the cache component, wherein the shape size of each second matrix is ​​B×A, and the A elements located in the same row in the second matrix are A consecutive elements in the first sequence corresponding to the convolution kernel element corresponding to the first matrix, and the first sequence is obtained by expanding the output gradient tensor into one dimension in the H dimension, the W dimension and the N dimension, and the first element in the first sequence is related to the convolution kernel element, and the coordinates of the elements in different rows of the second matrix in the K dimension are different, and A and B are positive integers.

[0009] For example, in the data processing method provided by at least one embodiment of the present disclosure, the thread warp is used to perform a matrix multiplication and accumulation operation between the loaded first matrix and the second matrix to obtain the operation results corresponding to the at least one convolution kernel element, including: for any convolution kernel element among the at least one convolution kernel element, the thread warp is used to respectively perform the matrix multiplication and accumulation operation between the first matrix corresponding to the any convolution kernel element and M second matrices to obtain M intermediate results as the operation results corresponding to the any convolution kernel element, wherein the M second matrices are M second matrices corresponding to the first matrix corresponding to the any convolution kernel element, each intermediate result is the output result of the matrix multiplication and accumulation operation between the first matrix and one of the M second matrices, and M is a positive integer.

[0010] For example, in the data processing method provided by at least one embodiment of the present disclosure, the operation results corresponding to the at least one convolution kernel element are accumulated to obtain the back propagation operation result of any type of operation area, including: adding the M intermediate results corresponding to each of the at least one convolution kernel elements to obtain the back propagation operation result of any type of operation area.

[0011] For example, in the data processing method provided by at least one embodiment of the present disclosure, in response to obtaining the operation result corresponding to any one of the at least one convolution kernel elements, the M intermediate results included in the operation result are respectively stored in M ​​register units, and in response to obtaining the operation result corresponding to other convolution kernel elements again, the M intermediate results included in the operation results corresponding to the other convolution kernel elements are respectively accumulated with the data in the corresponding register units, and the accumulated results are stored in the corresponding register units.

[0012] For example, in the data processing method provided by at least one embodiment of the present disclosure, the back propagation operation result of any type of operation area includes M operation results, each operation result is expressed as [NHW, C], wherein NHW represents expansion from H dimension, W dimension and N dimension to one dimension, the shape size of the operation result is A×A, A is a positive integer, and the back propagation operation result is strided and stored in the memory, including: sequentially storing the M operation results to the corresponding positions in the input gradient tensor located in the memory; wherein, for each operation result, the A elements on the NHW dimension of the operation result are stored one by one in the order of W dimension first, H dimension second, and N dimension last, to the corresponding positions in the input gradient tensor, and the A elements are stored at intervals of s steps in the H dimension and the W dimension.

[0013] For example, in the data processing method provided by at least one embodiment of the present disclosure, at least some elements in the first sequence corresponding to different convolution kernel elements are the same, and the same elements are cached in the global cache of the processor when loaded from the memory to the cache component, and when the same elements are loaded into the cache component again, they are loaded from the global cache.

[0014] For example, in the data processing method provided by at least one embodiment of the present disclosure, in response to s=2, R=3, S=3, the input gradient tensor is divided into four types of operation areas, including a first type of operation area, a second type of operation area, a third type of operation area and a fourth type of operation area, at least one convolution kernel element corresponding to the first type of operation area includes a first convolution kernel element located at the center of the plane formed by the R dimension and the S dimension in the convolution kernel tensor, at least one convolution kernel element corresponding to the second type of operation area includes convolution kernel elements at the upper left corner, lower left corner, upper right corner and lower right corner of the first convolution kernel element, at least one convolution kernel element corresponding to the third type of operation area includes convolution kernel elements on the upper and lower sides of the first convolution kernel element, and at least one convolution kernel element corresponding to the fourth type of operation area includes convolution kernel elements on the left and right sides of the first convolution kernel element.

[0015] For example, in the data processing method provided by at least one embodiment of the present disclosure, in response to the output gradient tensor needing to perform boundary filling, when performing a matrix multiplication-accumulation operation between the first matrix and the second matrix, a mask operation is performed on some elements to simulate the boundary filling.

[0016] For example, in the data processing method provided by at least one embodiment of the present disclosure, first matrices corresponding to the at least one convolution kernel element are loaded from the memory to the cache component, including: loading multiple first matrices corresponding to each convolution kernel element from the memory to the cache component, wherein different row elements in the multiple first matrices have different coordinates in the C dimension; using the thread warp to perform a matrix multiplication and accumulation operation between the loaded first matrix and the second matrix, including: using multiple thread warps to perform the matrix multiplication and accumulation operation between the multiple first matrices and the loaded second matrix in parallel.

[0017] At least one embodiment of the present disclosure provides a processor, comprising an instruction parsing unit, an execution unit, a cache component and a memory, wherein the instruction parsing unit is used to receive and parse data processing instructions, wherein the data processing instructions are used to load an output gradient tensor from the memory to the cache component, and use a thread warp to perform a strided convolution back-propagation operation on the output gradient tensor to obtain an input gradient tensor and store it in the memory, and the execution unit executes the data processing method according to any embodiment of the present disclosure after the instruction parsing unit parses the data processing instruction.

[0018] At least one embodiment of the present disclosure provides a data processing device, comprising an execution module, wherein the execution module is configured to execute L first operations for L types of operation areas respectively, so as to load an output gradient tensor from a memory into a cache unit of a processor, and use a thread warp to perform a back-propagation operation of a strided convolution on the output gradient tensor to obtain an input gradient tensor and store it in the memory; wherein the output gradient tensor is represented as [N, P, Q, K], where the N dimension is used to represent the number of batches, the P dimension is used to represent the height of the output gradient, the Q dimension is used to represent the width of the output gradient, and the K dimension is used to represent the number of output channels; a convolution kernel tensor used for the strided convolution is represented as [R, S, K, C], where the R dimension is used to represent the height of the convolution kernel, the S dimension is used to represent the width of the convolution kernel, and the C dimension is used to represent the number of input channels; the input gradient tensor is represented as [N, H, W, C], where the W dimension is used to represent the width of the input gradient, and the H dimension is used to represent the height of the input gradient; the input gradient tensor is divided into the L dimensions at least on the plane formed by the H dimension and the W dimension. The embodiment of the present invention relates to a type of operation region, wherein the operation regions of the same type are spaced by s elements in the H dimension and also spaced by s elements in the W dimension, each operation region includes 1 element, s is a positive integer and is the step size of the strided convolution, and L is a positive integer and is equal to s×s; wherein the execution module is configured to execute the first operation once to obtain the back propagation operation result of the same type of operation region, and the input gradient tensor is obtained by sequentially splicing the back propagation operation results of the L type of operation regions, wherein when the execution module executes the first operation for any type of operation region in the L type of operation regions, it includes performing a back propagation operation of at least one convolution kernel element, and the at least one convolution kernel element is a convolution kernel element of the convolution kernel tensor corresponding to the any type of operation region on the plane formed by the R dimension and the S dimension, and the at least one convolution kernel element is used to obtain the back propagation operation result of the any type of operation region through the back propagation operation.

[0019] At least one embodiment of the present disclosure provides an electronic device, comprising: a memory, which non-transitorily stores computer-executable instructions; and a processor, which is configured to execute the computer-executable instructions, wherein the computer-executable instructions, when executed by the processor, implement the data processing method according to any embodiment of the present disclosure.

[0020] At least one embodiment of the present disclosure provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the data processing method according to any embodiment of the present disclosure is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, but are not intended to limit the present disclosure.

[0022] Figure 1 A schematic structural diagram of a general-purpose graphics processing unit (GPGPU);

[0023] Figure 2 A schematic diagram of the implementation process of a back-propagation operator of a strided convolution;

[0024] Figure 3 A schematic flow chart of a data processing method provided for at least one embodiment of the present disclosure;

[0025] Figure 4A A schematic diagram of an input gradient tensor;

[0026] Figure 4B A schematic diagram of a convolution kernel tensor;

[0027] Figure 5A A schematic diagram of a computing area provided in an embodiment of the present disclosure;

[0028] Figure 5B A schematic diagram of a computing area provided for another embodiment of the present disclosure;

[0029] Fig. 6A A schematic diagram of a convolution kernel provided in one embodiment of the present disclosure;

[0030] Figure 6B A schematic diagram of a convolution kernel provided for another embodiment of the present disclosure;

[0031] Figure 7 A schematic flowchart of step S10 provided for at least one embodiment of the present disclosure;

[0032] Figure 8 A schematic diagram of a first matrix provided in an embodiment of the present disclosure;

[0033] Fig.9A A schematic diagram of stride storage provided by an embodiment of the present disclosure;

[0034] Fig. 9B A schematic diagram of stride storage provided for another embodiment of the present disclosure;

[0035] Fig.10 A process diagram of a data processing method provided by an embodiment of the present disclosure;

[0036] Fig.11 A schematic block diagram of a data processing device provided for at least one embodiment of the present disclosure;

[0037] Fig.12 A schematic structural diagram of a processor provided for at least one embodiment of the present disclosure;

[0038] Fig.13 A schematic block diagram of an electronic device provided in accordance with an embodiment of the present disclosure;

[0039] Fig.14 A schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0041] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly. In order to keep the following description of the embodiments of the present disclosure clear and concise, the present disclosure omits the detailed description of some known functions and known components.

[0042] In a convolutional neural network, the backward feature gradient calculation involves the following steps:

[0043] First, based on the loss function and the current output feature map, the gradient of the loss function to the output feature map is calculated. This gradient is usually a matrix of the same size as the output feature map.

[0044] Then, according to the convolution kernel and the step size, the gradient of the loss function on the output feature map is back-propagated to the input feature map. In this process, it is necessary to consider the flipping of the convolution kernel (for some types of convolution, such as standard convolution operations, the convolution kernel needs to be flipped during back propagation; but for some variants, such as cross-correlation operations, flipping is not required) and the impact of the step size.

[0045] Finally, according to the gradient obtained by back-propagation, the weights and biases of the convolutional layer are updated. This update process usually uses the gradient descent algorithm or its variants (such as stochastic gradient descent, Adam, etc.).

[0046] Figure 1 A schematic structural diagram of a general-purpose graphics processing unit (GPGPU).

[0047] like Figure 1 As shown, the general-purpose graphics processor is actually an array of programmable multiprocessors. For example, the programmable multiprocessor can be a streaming processor cluster (SPC), for example, including Figure 1 The stream processor clusters 1, ..., and stream processor cluster M are shown, where M is a positive integer greater than 1. In a general-purpose graphics processor, one stream processor cluster processes one computing task, or multiple stream processor clusters process one computing task. Multiple stream processor clusters share data through a global cache or a global memory.

[0048] like Figure 1 As shown, taking stream processor cluster 1 as an example, one stream processor cluster includes multiple computing units, such as Figure 1 In the figure, there are computing units 1, 2, ..., N, where N is a positive integer. Each computing unit (CU) is used to perform arithmetic and logical operations, such as accumulation, reduction, conventional addition, subtraction, multiplication, and division. A computing unit includes multiple cores (also called computing cores or computing cores), each of which includes an arithmetic logic unit (ALU), a floating-point computing unit, etc. The computing core is used to perform specific computing tasks. In addition, the computing unit also includes registers (e.g. Figure 1 The register file in the computing unit and the shared memory are used to hierarchically store source data and destination data related to computing tasks. The shared memory in a computing unit is used to share data between the cores of the computing unit.

[0049] In parallel computing, computing tasks are generally performed by multiple threads. These threads are divided into multiple thread blocks before being executed in a general-purpose graphics processor (or parallel computing processor), and then distributed through the thread block distribution module ( Figure 1The CPU (not shown) distributes multiple thread blocks to each computing unit. All threads in a thread block must be assigned to the same computing unit for execution. At the same time, the thread block is split into minimum execution thread bundles (or simply thread bundles, warps), each of which contains a fixed number of threads (or less than this fixed number), for example, 32 threads. Multiple thread blocks can be executed in the same computing unit or in different computing units.

[0050] In each compute unit, the warp scheduling / dispatching module ( Figure 1 The thread warps are scheduled and allocated (not shown) so that multiple computing cores of the computing unit can run the thread warps. Depending on the number of computing cores in the computing unit, multiple thread warps in a thread block can be executed simultaneously or in time-sharing. Multiple threads in each thread warp will execute the same instruction. The memory execution instruction will be emitted to the shared memory in the computing unit or further emitted to the intermediate level cache or global cache or global memory for read and write operations, etc.

[0051] The back-propagation operation is essentially similar to the standard convolution process. Take standard convolution as an example. For example, for multi-channel convolution, the parameters of the convolution kernels on different channels are different. In essence, for multi-channel convolution, the calculation is still performed according to the calculation process of two-dimensional convolution. For example, for a 3-channel input image, the convolution kernel is also a 3-channel convolution kernel. The 3-channel convolution kernel corresponds to the 3-channel input image on each channel one by one, and finally an output feature map is obtained through convolution multiplication. Specifically, a single-channel sliding window operation is performed on each channel according to the two-dimensional convolution to obtain three intermediate output results, and the corresponding position elements of the three intermediate output results are added to obtain the final output feature map.

[0052] The current common strided convolution backpropagation calculation (for example, using the strided convolution backpropagation operator) usually loads the output gradient tensor into on-chip memory (such as Figure 1 When the shared memory is used as shown in the figure, (stride-1) zeros are filled between each element of the output gradient tensor to expand it, and then the usual unstride convolution is performed on the expanded output gradient tensor according to the convolution kernel to obtain the input gradient tensor.

[0053] Figure 2 Schematic diagram of the implementation process of a back-propagation operator of a strided convolution.

[0054] exist Figure 2In the example, the convolution kernel is a 3×3 convolution kernel, the padding size is (1,1,1,1), the stride is (2,2), and the expansion factor is (1,1). The output gradient tensor is represented as (2,7,6,128), the convolution kernel tensor is represented as (3,3,128,128), and the input gradient tensor is represented as (2,14,12,128).

[0055] For example, Figure 2 As shown, in the process of loading the output gradient tensor into the cache component, 1 0 is filled between each element, such as Figure 2 As shown in , taking batch0 of the output gradient tensor as an example, after the output gradient tensor is loaded into the cache component, the dark gray boxes represent the elements in the original output gradient tensor, and the light gray boxes represent the inserted 0s and the 0s filled at the boundaries.

[0056] Afterwards, a regular unstride convolution operation is performed on the output gradient tensor to obtain batch0 of the input gradient tensor on the right.

[0057] This method has obvious flaws. First, it will occupy a large amount of on-chip memory space, and most of the data is invalid, and only a small part is valid data that actually participates in the calculation; second, this method wastes a lot of computing power, and most of the operations are multiplication with 0, which are invalid operations. Therefore, this method leads to low operator performance of the existing strided convolution backpropagation operator.

[0058] In addition, in terms of hardware scheduling strategy, the current strided convolution back propagation calculation uses a warp specialized scheduling strategy, which requires that data loading and calculation are both performed by one thread warp and cannot be performed in parallel by multiple thread warps. This strategy also cannot fully utilize the parallelism of hardware scheduling.

[0059] At least one embodiment of the present disclosure provides a data processing method, a processor, an electronic device, and a non-transitory computer-readable storage medium. The data processing method includes: performing L first operations for L types of operation areas respectively to load the output gradient tensor from the memory to the cache unit of the processor, and using the thread bundle to perform the back propagation operation of the strided convolution on the output gradient tensor to obtain the input gradient tensor and store it in the memory, wherein the output gradient tensor is represented as [N, P, Q, K], the N dimension is used to represent the batch number, the P dimension is used to represent the height of the output gradient, the Q dimension is used to represent the width of the output gradient, and the K dimension is used to represent the number of output channels; the convolution kernel tensor used for strided convolution is represented as [R, S, K, C], the R dimension is used to represent the height of the convolution kernel, the S dimension is used to represent the width of the convolution kernel, and the C dimension is used to represent the number of input channels; the input gradient tensor is represented as [N, H, W, C], the W dimension is used to represent the width of the input gradient, and the H dimension is used to represent the height of the input gradient; the input gradient tensor is divided into L dimensions on the plane formed by the H dimension and the W dimension. Class operation regions, the same type of operation regions are spaced by s elements in the H dimension and also spaced by s elements in the W dimension, each operation region includes 1 element, s is a positive integer and is the step size of the strided convolution, L is a positive integer and is equal to s×s; wherein, the back propagation operation result of the same type of operation region is obtained by executing the first operation once, and the input gradient tensor is obtained by sequentially splicing the back propagation operation results of the L type operation regions, wherein, when executing the first operation for any type of operation region in the L type operation regions, it includes performing a back propagation operation of at least one convolution kernel element, at least one convolution kernel element is a convolution kernel element of the convolution kernel tensor corresponding to any type of operation region on the plane formed by the R dimension and the S dimension, and at least one convolution kernel element is used to obtain the back propagation operation result of any type of operation region through the back propagation operation.

[0060] In the data processing method provided in at least one embodiment of the present disclosure, a new implementation method of the back propagation operation of strided convolution in a processor (especially a parallel processor) is provided. According to the characteristics of the back propagation operation of strided convolution, the implementation method divides the input gradient tensor as the output result into multiple types of operation areas, and the multiple types of operation areas are regularly spaced in a mesh, and the interval is s elements. Correspondingly, the R×S convolution kernel elements of the convolution kernel tensor on the plane formed by the R dimension and the S dimension are classified, and each operation area corresponds to at least one convolution kernel element, and these convolution kernel elements are used to obtain the back propagation operation result of the corresponding type of operation area through the back propagation operation (for example, by adding the product results of these convolution kernel elements and the corresponding elements in the output gradient tensor to obtain the back propagation operation result of the operation area). As a result, there is no need to supplement invalid data (for example, 0) when loading, avoiding the loading and operation of invalid data, and the cache components are all used for the storage of valid data, and the computing power is all used for the operation of valid data, thereby improving the utilization rate of the storage components (such as cache components or memory) and hardware computing units.

[0061] Moreover, the data processing method provided by at least one embodiment of the present disclosure is applicable to the general back-propagation operation of strided convolution, and has no restrictions on the size of the output gradient tensor, the input gradient tensor, the size of the convolution kernel tensor, the stride length, the padding method, etc., and has good versatility.

[0062] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings, but the present disclosure is not limited to these specific embodiments.

[0063] Figure 3 A schematic flowchart of a data processing method provided for at least one embodiment of the present disclosure.

[0064] like Figure 3 As shown, the data processing method provided by at least one embodiment of the present disclosure includes step S10.

[0065] In step S10, the first operation is performed L times for each of the L types of operation areas to load the output gradient tensor from the memory to the cache component of the processor, and the thread warp is used to perform a strided convolution back-propagation operation on the output gradient tensor to obtain the input gradient tensor and store it in the memory.

[0066] For example, the cache component is used to cache the output gradient tensor and send the output gradient tensor to the corresponding register when the computing core performs the back propagation operation on the output gradient tensor. Figure 1 Shared memory in .

[0067] For example, the output gradient tensor is represented as [N, P, Q, K], where the N dimension is used to represent the number of batches, the P dimension is used to represent the height of the output gradient, the Q dimension is used to represent the width of the output gradient, and the K dimension is used to represent the number of output channels.

[0068] For example, the convolution kernel tensor for strided convolution is expressed as [R, S, K, C], where the R dimension is used to represent the convolution kernel height, the S dimension is used to represent the convolution kernel width, the C dimension is used to represent the number of input channels, and the K dimension is used to represent the number of output channels.

[0069] For example, the input gradient tensor is represented as [N,H,W,C], where the N dimension is used to represent the number of batches, the W dimension is used to represent the width of the input gradient, the H dimension is used to represent the height of the input gradient, and the C dimension is used to represent the number of input channels.

[0070] Figure 4A A schematic diagram of an input gradient tensor.

[0071] like Figure 4A As shown in Figure 1, the input gradient tensor shows a batch, but the input gradient tensor for the actual convolution operation may have multiple batches, the dimensions of multiple batches are N dimensions, and the structure of each batch is the same as Figure 4A Same as shown.

[0072] like Figure 4A As shown, a batch consists of 21×6×8 elements, H=21, W=6, C=8, Figure 4A A small cube in a batch represents an element. For example, an element can have a data width of 8 bits, 16 bits, etc.

[0073] For example, in Figure 4A The H dimension, W dimension, and C dimension are shown in . Assume that the element pointed by the arrow is the origin, C, W, and H represent the three coordinate axes, row is the coordinate value in the height H direction (also the column direction), col is the coordinate value in the width W direction (also the row direction), and c is the coordinate value in the channel C direction (also the depth direction). The coordinates of the element pointed by the arrow in the input data are: c=0, row=0, and col=0. The channel, width, and height values ​​in the coordinates of other elements increase along the direction of the arrow.

[0074] Figure 4B A schematic diagram of a convolution kernel tensor.

[0075] like Figure 4BAs shown in the figure, in the back propagation operation of the strided convolution, the convolution kernel tensor used in the forward propagation process needs to be rotated 180 degrees before performing the conventional convolution operation. The convolution kernel tensor shows a batch. Similarly, the convolution kernel tensor for the actual convolution operation may have multiple batches. The dimensions of multiple batches are C dimensions. The structure of each batch is the same as Figure 4B Same as shown.

[0076] like Figure 4B As shown in the figure, a batch consists of 3×3×7 elements, R=3, S=3, K=7, and the plane formed by the R dimension and the S dimension is a 3×3 convolution kernel. Figure 4B A small cube in a batch represents an element. For example, an element can have a data width of 8 bits, 16 bits, etc.

[0077] For example, in Figure 4B The R dimension, S dimension, and K dimension are shown in . Similarly, assuming that the element pointed by the arrow is the origin, R, S, and K represent the three coordinate axes, row is the coordinate value in the height S direction, col is the coordinate value in the width R direction, and c is the coordinate value in the output channel K direction. The coordinates of the element pointed by the arrow in the input data are: c=0, row=0, and col=0. The channel, width, and height values ​​in the coordinates of other elements increase along the direction of the arrow.

[0078] The structure of the output gradient tensor can be referenced Figure 4A and Figure 4B The structures of the examples shown are similar and will not be described again here.

[0079] For example, the input gradient tensor is divided into L types of operation regions at least on the plane formed by the H dimension and the W dimension. The same type of operation regions are separated by s elements in the H dimension and s elements in the W dimension. Each operation region includes 1 element, s is a positive integer and is the step size of the strided convolution, and L is a positive integer and is equal to s×s.

[0080] Figure 5A A schematic diagram of a computing area provided in an embodiment of the present disclosure.

[0081] For example, Figure 5A The plane formed by the input gradient tensor in the H dimension and the W dimension is shown, but in fact the operation region itself can cover the C dimension, that is, elements with the same H dimension and W dimension coordinates but different C dimension coordinates can be considered to belong to the same type of operation region. However, for the convenience of description, the plane is used as an example for description here.

[0082] like Figure 5A As shown, when the step size s=2, L=4, Figure 5AFour operation areas are shown, marked as 1, 2, 3, and 4 respectively. Figure 5A Each box in represents a computation region, and a computation region includes an element in the input gradient tensor. For example, Figure 5A As shown, taking operation area 1 as an example, the operation areas 1 marked as 1 are separated by 2 elements in both the H dimension and the W dimension, which is reflected in the coordinate as a difference of s (=2).

[0083] Figure 5B A schematic diagram of a computing area provided in another embodiment of the present disclosure.

[0084] like Figure 5B As shown, when the step size s=3, L=9, Figure 5B 9 operation areas are shown in the figure, marked as 1, 2, 3, 4, 5, 6, 7, 8, and 9 respectively. Figure 5B Each box in represents an element, that is, a computing region. For example, taking computing region 1 as an example, the computing regions 1 marked as 1 are separated by 3 elements in both the H dimension and the W dimension, which is reflected in the difference of 3 in the coordinates.

[0085] For example, the first operation is executed once to obtain the back propagation operation results of the same type of operation area, and the input gradient tensor is obtained by sequentially splicing the back propagation operation results of L types of operation areas.

[0086] For example, Figure 5A For example, the first operation is performed four times to obtain the back propagation operation results of the four operation areas. For example, in the first first operation, the back propagation operation results of all operation areas 1 are obtained, and in the second first operation, the back propagation operation results of all operation areas 2 are obtained, and so on. The different types of back propagation operation results are sequentially spliced. For example, after the back propagation operation results of operation area 1 are obtained, they are filled into the corresponding operation area 1 in the input gradient tensor, and after the back propagation operation results of operation area 2 are obtained, they are filled into the corresponding operation area 2 in the input gradient tensor, and so on, to obtain the input gradient tensor.

[0087] The process is similar for FIG. 5B and will not be described again here.

[0088] For example, when executing the first operation for any type of operation area in the L type operation areas, it includes performing a back propagation operation of at least one convolution kernel element. Here, the at least one convolution kernel element is a convolution kernel element corresponding to any type of operation area on the plane formed by the R dimension and the S dimension of the convolution kernel tensor, and the at least one convolution kernel element is used to obtain the back propagation operation result of any type of operation area through a back propagation operation (for example, performing a convolution operation with a corresponding element in the output gradient tensor).

[0089] Fig. 6A A schematic diagram of a convolution kernel provided according to an embodiment of the present disclosure.

[0090] For example, Fig. 6A The convolution kernel shown is a schematic diagram of the convolution kernel tensor on the plane formed by the R dimension and the S dimension. Fig. 6A In the example, R=3, S=3. Fig. 6A As shown, the convolution kernel tensor includes 9 convolution kernel elements, namely, convolution kernel element w22, convolution kernel element w21, and convolution kernel element w20.

[0091] Similarly, similar to the operation area, in fact, the convolution kernel elements corresponding to the operation area described here can also cover K dimensions, but for the convenience of description, the plane is used as an example for description here.

[0092] For example, different convolution kernel elements act on different operation areas. Fig. 6A , convolution kernel element w22, convolution kernel element w20, convolution kernel element w02, and convolution kernel element w0 act on operation area 4, and are used to obtain the back propagation operation result of operation area 4 through back propagation operation, that is, after the convolution kernel element w22, convolution kernel element w20, convolution kernel element w02, and convolution kernel element w0 are convolved with the elements in the corresponding output gradient tensor, the input gradient elements of operation area 4 are obtained.

[0093] Convolution kernel element w21 and convolution kernel element w01 act on operation area 3. Convolution kernel element w21 and convolution kernel element w01 are used to obtain the back propagation operation result of operation area 3. Convolution kernel element w12 and convolution kernel element w10 act on operation area 2. Convolution kernel element w12 and convolution kernel element w10 are used to obtain the back propagation operation result of operation area 2. Convolution kernel element w11 acts on operation area 1. Convolution kernel element w11 is used to obtain the back propagation operation result of operation area 1.

[0094] Therefore, the convolution kernel elements can be classified according to the category of the operation area. Each operation area corresponds to at least one convolution kernel element. For example, operation area 4 corresponds to Fig. 6A The convolution kernel elements w22, w20, w02, and w0 in the black rectangle; Operation area 2 corresponds to Fig. 6A The convolution kernel element w12 and w10 in the white rectangle; the operation area 3 corresponds to Fig. 6A Convolution kernel element w21 and convolution kernel element w01 in the midpoint background rectangle; operation area 1 corresponds to Fig. 6A The convolution kernel element w11 in.

[0095] Figure 6B A schematic diagram of a convolution kernel provided for another embodiment of the present disclosure.

[0096] For example, Figure 6B The convolution kernel shown is a schematic diagram of the convolution kernel tensor on the plane formed by the R dimension and the S dimension. Figure 6B In the example, R=5, S=5. Figure 6B As shown, the convolution kernel tensor includes 25 convolution kernel elements, including convolution kernel element w44, convolution kernel element w43, and convolution kernel element w42.

[0097] Similarly, similar to the operation area, in fact, the convolution kernel elements corresponding to the operation area described here can also cover K dimensions, but for the convenience of description, the plane is used as an example for description here.

[0098] For example, different convolution kernel elements act on different operation areas. Figure 6B , the convolution kernel element w22 acts on the operation area 1 and is used to obtain through the back propagation operation Figure 6B The result of the back propagation operation of the operation area 1, that is, the convolution kernel element w22 is convolved with the element in the corresponding output gradient tensor to obtain the input gradient element of the operation area 1.

[0099] Convolution kernel element w23 and convolution kernel element w20 act on operation area 2. Convolution kernel element w23 and convolution kernel element w20 are used to obtain the back propagation operation result of operation area 2.

[0100] Convolution kernel element w24 and convolution kernel element w21 act on operation area 3. Convolution kernel element w24 and convolution kernel element w21 are used to obtain the back propagation operation result of operation area 3.

[0101] Convolution kernel element w32 and convolution kernel element w02 act on operation area 4. Convolution kernel element w32 and convolution kernel element w02 are used to obtain the back propagation operation result of operation area 4.

[0102] Convolution kernel element w33, convolution kernel element w30, convolution kernel element w03, and convolution kernel element w00 act on operation area 5. Convolution kernel element w33, convolution kernel element w30, convolution kernel element w03, and convolution kernel element w00 are used to obtain the back propagation operation result of operation area 5.

[0103] Convolution kernel element w34, convolution kernel element w31, convolution kernel element w04, and convolution kernel element w01 act on operation area 6. Convolution kernel element w34, convolution kernel element w31, convolution kernel element w04, and convolution kernel element w01 are used to obtain the back propagation operation result of operation area 6.

[0104] The convolution kernel element w42 and the convolution kernel element w12 act on the operation area 7. The convolution kernel element w42 and the convolution kernel element w12 are used to obtain the back propagation operation result of the operation area 7.

[0105] Convolution kernel element w43, convolution kernel element w40, convolution kernel element w13, and convolution kernel element w10 act on operation area 8. Convolution kernel element w43, convolution kernel element w40, convolution kernel element w13, and convolution kernel element w10 are used to obtain the back propagation operation result of operation area 8.

[0106] Convolution kernel element w44, convolution kernel element w41, convolution kernel element w14, and convolution kernel element w11 act on operation area 9. Convolution kernel element w44, convolution kernel element w41, convolution kernel element w14, and convolution kernel element w11 are used to obtain the back propagation operation result of operation area 9.

[0107] Therefore, the convolution kernel elements can be classified according to the category of the operation area. Each operation area corresponds to at least one convolution kernel element. For example, operation area 1 corresponds to Figure 6B The convolution kernel element w22 in the middle corresponds to the operation area 2 Figure 6B The convolution kernel element w23 and the convolution kernel element w20, etc.

[0108] Therefore, according to the characteristics of the back-propagation operation of the strided convolution, the input gradient tensor is divided into multiple types of operation areas according to the step size of the strided convolution, and the multiple types of operation areas are regularly spaced in a mesh, and are spaced by s elements in both the H dimension and the W dimension. Correspondingly, the R×S convolution kernel elements of the convolution kernel tensor on the plane formed by the R dimension and the S dimension are also classified, and each operation area corresponds to at least one convolution kernel element, and these convolution kernel elements are used to obtain the back-propagation operation results of the corresponding types of operation areas through back-propagation operation (for example, by performing convolution operation on these convolution kernel elements and corresponding elements in the output gradient tensor to obtain the back-propagation operation results of the operation area). Therefore, there is no need to supplement invalid data (for example, 0) when loading, which avoids the loading and operation of invalid data, and the cache components are all used for the storage of valid data, and the computing power is all used for the operation of valid data, thereby improving the utilization of storage components and hardware computing units.

[0109] The specific execution process of the first operation is described in detail below with reference to the accompanying drawings.

[0110] Figure 7 A schematic flow chart of step S10 is provided for at least one embodiment of the present disclosure. Figure 7 As shown, step S10 may include steps S101 - S105 .

[0111] For example, steps S101-S105 may be executed sequentially or in other adjusted orders. The embodiments of the present disclosure do not limit the execution order of the steps and may be adjusted according to actual conditions. For example, in some examples, the data processing method provided by at least one embodiment of the present disclosure may selectively execute some steps in steps S101-S105, or may execute some additional steps in addition to steps S101-S105. The embodiments of the present disclosure do not specifically limit this.

[0112] In step S101, a first matrix corresponding to at least one convolution kernel element is loaded from a memory to a cache component.

[0113] For example, in some embodiments, step S101 may include: for any convolution kernel element of at least one convolution kernel element, loading a first matrix corresponding to any convolution kernel element into a cache component, wherein the shape size of the first matrix is ​​A×B, the B elements located in the same row of the first matrix are consecutive B elements in the convolution kernel tensor that have the same coordinates as any convolution kernel element in the R dimension, S dimension, and C dimension, but different coordinates in the K dimension, different row elements in the first matrix have different coordinates in the C dimension, and A and B are positive integers.

[0114] Figure 8 A schematic diagram of a first matrix provided according to an embodiment of the present disclosure.

[0115] like Figure 8 As shown, for example, Figure 8 The 3D graph on the left represents batch0, and the 3D graph on the right represents batch1. Of course, the convolution kernel tensor can also include more batches in the C dimension. Figure 8 Not shown in FIG.

[0116] Take the convolution kernel element in the upper left corner (for example, Fig. 6A For example, the first matrix corresponding to the convolution kernel element may include the elements shown as black cubes in the convolution kernel tensor. For example, the first row of elements in the first matrix may be the consecutive B elements shown as black cubes in batch0, the second row of elements in the first matrix may be the consecutive B elements shown as black cubes in batch1, and so on.

[0117] The convolution kernel element to the right of the upper left corner (for example, reference Fig. 6AFor example, the first matrix corresponding to the convolution kernel element may include the elements shown as gray cubes in the convolution kernel tensor. For example, the first row of elements in the first matrix may be B consecutive elements shown as gray cubes in batch0, the second row of elements in the first matrix may be B consecutive elements shown as gray cubes in batch1, and so on.

[0118] For example, when using thread warps to perform backpropagation operations, if the size of the input data is large, a large image can be divided into multiple small blocks, each of which is called a "tile". Each tile contains a portion of the image data, which can be processed in an independent processing unit. The specific values ​​of A and B can be determined according to actual needs, for example, based on the tile size supported by the processor, the processing speed of the processor, load balancing, etc.

[0119] When loading, the number of first matrices to be loaded may be determined according to the storage capacity of the cache component. For example, if the cache component has sufficient storage space, all first matrices corresponding to the at least one convolution kernel element may be loaded into the cache component. For example, if the storage space of the cache component is insufficient, first matrices corresponding to some convolution kernel elements may be loaded, and after these first matrices are used up, other first matrices may be loaded.

[0120] In step S102, at least one second matrix corresponding to each first matrix is ​​loaded into a cache component.

[0121] For example, in some embodiments, step S102 may include: loading at least one second matrix corresponding to the first matrix into a cache component, wherein the shape size of each second matrix is ​​B×A, the A elements located in the same row of the second matrix are consecutive A elements in a first sequence corresponding to the convolution kernel element corresponding to the first matrix, the first sequence is obtained by expanding the output gradient tensor into one dimension in H dimensions, W dimensions and N dimensions, the first element in the first sequence is related to the convolution kernel element, the coordinates of elements in different rows of the second matrix in the K dimension are different, and A and B are positive integers.

[0122] For example, refer to Figure 2 Schematic diagram of the output gradient tensor in , the output gradient tensor is expanded first in Q dimension, then in P dimension, and finally in N dimension. The expanded output gradient tensor can be expressed as NPQ (pixel) × K, where NPQ (pixel) means expanded into one dimension in P dimension, Q dimension, and N dimension.

[0123] According to the different elements of the convolution kernel, different elements in the expanded output gradient tensor are selected as the starting position of the first sequence, and all the remaining elements starting from this element constitute the first sequence. For example, for a convolution kernel element, it can correspond to k first sequences, and the N-dimensional, P-dimensional, and Q-dimensional coordinates of the first element in the k first sequences are the same, but the K-dimensional coordinates are different, and k is the specific value of the number of output channels.

[0124] The positions of the first elements in the first sequence corresponding to different convolution kernel elements in the output gradient tensor are all different. The first element in the first sequence corresponding to each convolution kernel element is determined according to the position of the convolution kernel element, for example, the corresponding element in the input gradient tensor when the convolution kernel element performs the first matrix multiplication and accumulation operation.

[0125] For example, Fig. 6A Taking the convolution kernel shown as an example, for the convolution kernel element w11, the first element in each corresponding first sequence is the element at position H=0, W=0, N=0 in the output gradient tensor. For the convolution kernel element w10, the first element in each corresponding first sequence is the element at position H=0, W=1, N=0 in the output gradient tensor. For the convolution kernel element w01, the first element in each corresponding first sequence is the element at position H=1, W=0, N=0 in the output gradient tensor. The method for determining the first element of the first sequence of other convolution kernel elements is similar and will not be repeated here.

[0126] For example, assuming that NPQ=4×A after expansion into one dimension, that is, the first matrix corresponding to a certain convolution kernel element corresponds to 4 second matrices, for example, including second matrix 1, second matrix 2, second matrix 3, and second matrix 4. The convolution kernel element also corresponds to k first sequences, for example, including first sequence 1, .., first sequence k, and the first sequence is determined with reference to the above process.

[0127] For example, the first row of elements in the second matrix 1 may be the first A elements in the first sequence 1, the second row of elements in the second matrix 1 may be the first A elements in the first sequence 2, the third row of elements in the second matrix 1 may be the first A elements in the first sequence 3, the fourth row of elements in the second matrix 1 may be the first A elements in the first sequence 4, and so on.

[0128] For example, the first row of elements in the second matrix 2 may be the A+1th element to the 2×Ath element in the first sequence 1, the second row of elements in the second matrix 2 may be the A+1th element to the 2×Ath element in the first sequence 2, the third row of elements in the second matrix 2 may be the A+1th element to the 2×Ath element in the first sequence 3, and so on.

[0129] The same is true for the elements included in the second matrix 3 and the second matrix 4, which will not be described here.

[0130] Similar to the loading process of the first matrix, during loading, the number of second matrices to be loaded can be determined according to the storage capacity of the cache component. For example, if the cache component has sufficient storage space, all second matrices corresponding to the at least one convolution kernel element can be loaded into the cache component. For example, if the storage space of the cache component is insufficient, the second matrices corresponding to some convolution kernel elements can be loaded, and after these second matrices are used up, other second matrices can be loaded.

[0131] Moreover, for the second matrix corresponding to different convolution kernel elements, the starting elements are different, but most of the elements are repeated, that is, at least some elements in the first sequence corresponding to different convolution kernel elements are the same. When loading the second matrix from the memory for the first time, refer to Figure 1 , it can be loaded into the cache component after passing through the global cache, so the loaded second matrix has a backup in the global cache. Based on the loading method for the second matrix provided in the embodiment of the present disclosure, when the second matrix corresponding to another convolution kernel element is loaded into the cache component, these same elements can be loaded from the global cache at this time without having to be loaded from the memory. Since most of the elements of the first sequence corresponding to different convolution kernel elements are actually the same, this method can greatly reduce loading delays, reduce read and write overhead, and improve operator efficiency.

[0132] In step S103, the thread warp is used to perform a matrix multiplication-accumulation operation between the loaded first matrix and the second matrix to obtain a calculation result corresponding to at least one convolution kernel element.

[0133] For example, in some embodiments, step S103 may include: for any convolution kernel element of at least one convolution kernel element, using thread warps to respectively perform matrix multiplication and accumulation operations between a first matrix corresponding to any convolution kernel element and M second matrices, and obtaining M intermediate results as the operation results corresponding to any convolution kernel element, wherein the M second matrices are M second matrices corresponding to the first matrix corresponding to any convolution kernel element, each intermediate result is the output result of the matrix multiplication and accumulation operation between the first matrix and a second matrix, and M is a positive integer.

[0134] Here, the number of M is related to the length of the first sequence.

[0135] For example, in one embodiment, a convolution kernel element corresponds to a first matrix A00, and the first matrix A00 corresponds to four second matrices, namely, a second matrix B00, a second matrix B01, a second matrix B02, and a second matrix B03. For the first matrix and the second matrix, please refer to the above content, which will not be repeated here.

[0136] For example, in step S103, a thread warp is used to perform a matrix multiplication and accumulation operation on the first matrix A00 and the second matrix B00 to obtain an intermediate result R00; a thread warp is used to perform a matrix multiplication and accumulation operation on the first matrix A00 and the second matrix B01 to obtain an intermediate result R01; a thread warp is used to perform a matrix multiplication and accumulation operation on the first matrix A00 and the second matrix B02 to obtain an intermediate result R02; a thread warp is used to perform a matrix multiplication and accumulation operation on the first matrix A00 and the second matrix B03 to obtain an intermediate result R03.

[0137] For example, in some embodiments, the intermediate results obtained may be cached in a register unit, such as a thread-local register (TLR). For example, the intermediate result R00 may be cached in register unit TLR1, the intermediate result R01 may be cached in register unit TLR2, the intermediate result R02 may be cached in register unit TLR2, and the intermediate result R03 may be cached in register unit TLR3.

[0138] The matrix multiplication and accumulation operations of the first matrix and the second matrix corresponding to other convolution kernel elements are similar and will not be repeated here.

[0139] For example, if the output gradient tensor needs to be border-filled, when performing a matrix multiplication-accumulation operation between the first matrix and the second matrix, a mask operation is performed on some elements to simulate the border filling (padding), for example, some elements are masked by a mask operation and replaced with 0, thereby simulating the border filling.

[0140] For example, in at least one embodiment of the present disclosure, a collaborative hardware scheduling strategy and a persistent kernel programming model are adopted. The principle of the collaborative hardware scheduling strategy is to concurrently schedule multiple thread warps and interleave to complete convolution operations. For example, multiple thread warps can be used to perform back propagation operations in parallel.

[0141] For example, step S101 may include: loading multiple first matrices corresponding to each convolution kernel element from the memory to the cache component, wherein different row elements in the multiple first matrices have different coordinates in the C dimension.

[0142] For example, the first first matrix performs convolution operations on the first A batches of K dimensions, the second first matrix performs convolution operations on the A+1th to 2×Ath batches of K dimensions, and so on.

[0143] For example, at this time, step S103 may include: using multiple warps to perform matrix multiplication and accumulation operations between multiple first matrices and the loaded second matrix in parallel.

[0144] Therefore, a collaborative hardware scheduling strategy is used to give full play to the parallelism of hardware scheduling.

[0145] Moreover, multiple first matrices corresponding to the same convolution kernel element are convolved with the same second matrix, which can also avoid repeated data loading and increase data reusability. These second matrices only need to be loaded from the memory once.

[0146] In addition, in at least one embodiment of the present disclosure, a persistent kernel programming model is used, which requires that new tasks be restarted after all thread warps in the computing unit have completed their tasks. Tasks are divided in the C dimension and the NPQ dimension, and the task division is more reasonable, and the hardware load is balanced.

[0147] In step S104, the operation results corresponding to at least one convolution kernel element are accumulated to obtain the back propagation operation result of any type of operation area.

[0148] For example, in some embodiments, step S104 may include: adding M intermediate results corresponding to each of at least one convolution kernel elements to obtain the back propagation operation result of any type of operation area.

[0149] For example, in some embodiments, in response to obtaining the operation result corresponding to any convolution kernel element, the M intermediate results included in the operation result are respectively stored in M ​​register units, and in response to obtaining the operation result corresponding to other convolution kernel elements again, the M intermediate results included in the operation result corresponding to other convolution kernel elements are respectively accumulated with the data in the corresponding register units, and the accumulated results are stored in the corresponding register units.

[0150] For example, in one embodiment, Fig. 6A Taking the convolution kernel shown as an example, for the four convolution kernel elements corresponding to the operation area 4, the convolution kernel element w22 corresponds to the first matrix A00, and the first matrix A00 corresponds to four second matrices, namely the second matrix B00, the second matrix B01, the second matrix B02 and the second matrix B0; the convolution kernel element w20 corresponds to the first matrix A01, and the first matrix A01 corresponds to four second matrices, namely the second matrix B10, the second matrix B11, the second matrix B12 and the second matrix B13; the convolution kernel element w02 corresponds to the first matrix A02, and the first matrix A02 corresponds to four second matrices, namely the second matrix B20, the second matrix B21, the second matrix B22 and the second matrix B23; the convolution kernel element w03 corresponds to the first matrix A03, and the first matrix A03 corresponds to four second matrices, namely the second matrix B30, the second matrix B31, the second matrix B32 and the second matrix B33.

[0151] For example, referring to step S101 and step S102 , the first matrix and the second matrix are loaded, and the specific process is not repeated here.

[0152] For example, through step S103, the intermediate results of the matrix multiplication and accumulation operations of the first matrix A00 and the corresponding four second matrices, such as intermediate results R00, intermediate results R01, intermediate results R02, and intermediate results R03, are obtained and stored in register unit TLR1, register unit TLR2, register unit TLR3, and register unit TLR4, respectively.

[0153] For example, through step S103, the intermediate results of the matrix multiplication and accumulation operation of the first matrix A01 and the corresponding four second matrices are obtained, such as intermediate results R10, intermediate results R11, intermediate results R12, and intermediate results R13. Then, the intermediate result R10 is accumulated into the register unit TLR1, that is, the intermediate result R10 is added to the intermediate result R00 and the accumulated result is stored in the register unit TLR1. The process for other intermediate results is similar and will not be repeated here.

[0154] For example, through step S103, the intermediate results of the matrix multiplication and accumulation operation of the first matrix A02 and the corresponding four second matrices are obtained, such as intermediate results R20, intermediate results R21, intermediate results R22, and intermediate results R23. Then, the intermediate result R20 is accumulated into the register unit TLR1, that is, the intermediate result R20 is added to the data stored in the register unit TLR1 and the accumulated result is stored in the register unit TLR1. The process for other intermediate results is similar and will not be repeated here.

[0155] For example, through step S103, the intermediate results of the matrix multiplication and accumulation operation of the first matrix A03 and the corresponding four second matrices are obtained, such as intermediate results R30, intermediate results R31, intermediate results R32, and intermediate results R33. Then, the intermediate result R30 is accumulated into the register unit TLR1, that is, the intermediate result R30 is added to the data stored in the register unit TLR1 and the accumulated result is stored in the register unit TLR1. The process for other intermediate results is similar and will not be repeated here.

[0156] After the above process, the operation result C00 stored in the register unit TLR1, the operation result C01 stored in the register unit TLR2, the operation result C02 stored in the register unit TLR3, and the operation result C03 stored in the register unit TLR4 constitute the reverse propagation operation result of the operation area 4.

[0157] Of course, other feasible implementations may also be used for the accumulation process, and the present disclosure does not impose any specific limitation thereto.

[0158] In step S105, the back propagation operation result of any type of operation area is strided and stored in the memory.

[0159] For example, when the result of the back-propagation operation is stored from the register unit to the shared memory, stride storage may not be performed; however, when the result of the back-propagation operation needs to be stored to the memory, stride storage is required.

[0160] For example, the back-propagation operation result includes M operation results, each of which is represented by [NHW, C], where NHW represents expansion from H dimension, W dimension and N dimension to one dimension, and the shape size of the operation result is A×A, where A is a positive integer.

[0161] For example, if the result of the back-propagation operation needs to be stored in the memory, the following operation can be performed: the M operation results are sequentially stored in the corresponding positions of the input gradient tensor in the memory; wherein, for each operation result, the A elements on the NHW dimensions of the operation result are stored one by one in the order of W dimension, then H dimension, and finally N dimension to the corresponding positions in the input gradient tensor, and the A elements are stored at intervals of s steps in both the H dimension and the W dimension.

[0162] For example, the M operation results are stored in the corresponding positions one by one from NHW dimensions according to the relative position relationship in the first sequence. Specifically, the W dimension is stored first, then the H dimension, and finally the N dimension in the corresponding position in the input gradient tensor, and for the positions with the same coordinates in the W dimension, H dimension, and N dimension, A elements are stored in the C dimension.

[0163] For an operation result, the A elements in the NHW dimension are stored with a step size of s in the H dimension and the W dimension, that is, the elements need to be stored in the corresponding operation area.

[0164] Fig.9A A schematic diagram of stride storage provided by an embodiment of the present disclosure. Fig.9A For Fig. 6A The stride storage of calculation area 4 is shown.

[0165] Fig.9A Two batches are shown, each of which represents an element in a box, and the gray box represents the location where the operation result is stored. Of course, for each gray box, A elements are actually stored in the C dimension, but for ease of description, only the plane diagram is shown.

[0166] like Fig.9AAs shown in the figure, after obtaining A×A operation results, the A elements of NHW dimension are stored in the corresponding position of the input gradient tensor in the order of W dimension first, H dimension second, and N dimension last. When storing, they are strided and stored with an interval of 2 elements in H dimension and W dimension. Figure 5A , the operation result is stored in the location of operation area 4.

[0167] The M operation results are stored in the input gradient tensor in the memory in the same logical order.

[0168] Fig. 9B A schematic diagram of stride storage provided by another embodiment of the present disclosure. Fig. 9B For Fig. 6A The stride storage of calculation area 3 is shown.

[0169] Fig. 9B Two batches are shown, each of which represents an element in a box, and the gray box represents the location where the operation result is stored. Of course, for each gray box, A elements are actually stored in the C dimension, but for ease of description, only the plane diagram is shown.

[0170] like Fig. 9B As shown in the figure, after obtaining A×A operation results, the A elements of NHW dimension are stored in the corresponding position of the input gradient tensor in the order of W dimension first, H dimension second, and N dimension last. When storing, they are strided and stored with an interval of 2 elements in H dimension and W dimension. Figure 5A , the operation result is stored in the location of operation area 3.

[0171] The M operation results are stored in the input gradient tensor in the memory in the same logical order.

[0172] Fig.10 A process diagram of a data processing method provided by an embodiment of the present disclosure.

[0173] For example, Fig.10 The embodiment shown is based on Figure 5A The operation area and Fig. 6A The convolution kernel tensor shown is used to describe the convolution kernel, that is, the convolution kernel size is 3×3 and the step size is 2. Based on this premise, the execution process of the entire data processing method is described.

[0174] exist Fig.10 In the illustrated embodiment, two warps are used to perform back propagation operations in parallel. For example, warp 0 performs operations on the first matrix A00, etc., and warp 1 performs operations on the first matrix A10, etc. For the first matrix A00 being equal to the first matrix A10, etc., please refer to the description of the above-mentioned related contents, which will not be repeated here.

[0175] Referring to the foregoing content, the input gradient tensor is divided into four operation regions according to the stride length, and the first operation is performed four times to obtain the back propagation operation results of the four operation regions.

[0176] For example, in Fig.10 In the example, the first operation for operation area 4 is performed first. Of course, the present disclosure does not limit the execution order of the operation areas. As mentioned above, operation area 4 corresponds to 4 convolution kernel elements, namely convolution kernel element w22, convolution kernel element w20, convolution kernel element w02, and convolution kernel element w00.

[0177] For example, Fig.10 As shown, the first matrix A00 and the first matrix A10 corresponding to the convolution kernel element w22 are loaded into the cache component. The specific process of loading the first matrix A00 and the first matrix A10 can refer to the specific description of the aforementioned step S101, which will not be repeated here.

[0178] As mentioned above, the number of first matrices loaded here is related to the storage space capacity of the cache component. For example, if the storage space of the cache component is sufficient, the first matrix A01 and the like may continue to be loaded into the cache component.

[0179] For example, Fig.10 As shown, the second matrix B00, the second matrix B01, the second matrix B02, the second matrix B03, etc. corresponding to the first matrix A00 and A10 are loaded into the cache component.

[0180] For the specific process of loading the second matrix, reference may be made to the related description of the aforementioned step S102, which will not be repeated here.

[0181] Afterwards, thread warp 0 is used to perform a matrix multiplication and accumulation operation between the loaded first matrix A00 and the second matrix B00, and an intermediate result R00 is obtained, which is stored in register unit 00; thread warp 0 is used to perform a matrix multiplication and accumulation operation between the loaded first matrix A00 and the second matrix B01, and an intermediate result R01 is obtained, which is stored in register unit 01; thread warp 0 is used to perform a matrix multiplication and accumulation operation between the loaded first matrix A00 and the second matrix B02, and an intermediate result R02 is obtained, which is stored in register unit 02; thread warp 0 is used to perform a matrix multiplication and accumulation operation between the loaded first matrix A00 and the second matrix B03, and an intermediate result R03 is obtained, which is stored in register unit 03.

[0182] At the same time, thread warp 1 is used to perform a matrix multiplication and accumulation operation between the loaded first matrix A10 and the second matrix B00, obtaining an intermediate result R10, which is stored in register unit 10; thread warp 1 is used to perform a matrix multiplication and accumulation operation between the loaded first matrix A10 and the second matrix B01, obtaining an intermediate result R11, which is stored in register unit 11; thread warp 1 is used to perform a matrix multiplication and accumulation operation between the loaded first matrix A10 and the second matrix B02, obtaining an intermediate result R12, which is stored in register unit 12; thread warp 1 is used to perform a matrix multiplication and accumulation operation between the loaded first matrix A10 and the second matrix B03, obtaining an intermediate result R13, which is stored in register unit 13.

[0183] The calculation results corresponding to the convolution kernel element w22 include the intermediate results R00 and R10 obtained above.

[0184] Afterwards, if the first matrix A01, the first matrix A11, the second matrix B10, etc. are not loaded, refer to the aforementioned step S101 and step S102 to load the first matrix and the second matrix.

[0185] Afterwards, thread warp 0 is used to perform a matrix multiplication and accumulation operation between the loaded first matrix A01 and the second matrix B10, and an intermediate result R00 is obtained, which is accumulated in register unit 00; thread warp 0 is used to perform a matrix multiplication and accumulation operation between the loaded first matrix A01 and the second matrix B11, and an intermediate result R01 is obtained, which is accumulated in register unit 01; thread warp 0 is used to perform a matrix multiplication and accumulation operation between the loaded first matrix A01 and the second matrix B12, and an intermediate result R02 is obtained, which is accumulated in register unit 02; thread warp 0 is used to perform a matrix multiplication and accumulation operation between the loaded first matrix A01 and the second matrix B13, and an intermediate result R03 is obtained, which is accumulated in register unit 03. The process of accumulating the corresponding register unit here can refer to the relevant description of the aforementioned step S104, which will not be repeated here.

[0186] At the same time, thread warp 1 is used to perform a matrix multiplication and accumulation operation between the loaded first matrix A11 and the second matrix B10, and an intermediate result R10 is obtained, which is accumulated in register unit 10; thread warp 1 is used to perform a matrix multiplication and accumulation operation between the loaded first matrix A11 and the second matrix B11, and an intermediate result R11 is obtained, which is accumulated in register unit 11; thread warp 1 is used to perform a matrix multiplication and accumulation operation between the loaded first matrix A11 and the second matrix B12, and an intermediate result R12 is obtained, which is accumulated in register unit 12; thread warp 1 is used to perform a matrix multiplication and accumulation operation between the loaded first matrix A11 and the second matrix B13, and an intermediate result R13 is obtained, which is accumulated in register unit 13.

[0187] Afterwards, the back propagation operation process of the first matrix A02, the first matrix A03, etc. is continued with reference to the above process.

[0188] After completing the above processes of the first matrix A02, the first matrix A03, the first matrix A12, and the first matrix A13, the operation results C00, the operation results C01, etc. stored in the register unit 00, the register unit 01, etc. are the back propagation operation results of the operation area 4.

[0189] Refer to step S105 and Fig.9A , the operation results C00 stored in register unit 00, the operation results C01 stored in register unit 01, the operation results C02 stored in register unit 02, and the operation results C03 stored in register unit 03 are strided and stored to the position of operation area 4. Specifically, the operation results C00, the operation results C01, the operation results C02, and the operation results C03 are strided and stored in the corresponding positions of the input gradient tensor in the NHW dimension sequence. The stride storage here refers to the interval of s elements in both the H dimension and the W dimension.

[0190] For operation result c10, it is stored in the K dimension of operation result c00, and continues to be stored in dimension K. For operation result c11, it is stored in the K dimension of operation result c01, and continues to be stored in dimension K. The same is true for operation results c12 and c13, which will not be repeated here.

[0191] At this point, the first operation on the operation area 4 is completed.

[0192] Then, the first operation is performed on the operation area 2.

[0193] For example, Fig.10 , the first matrix A08, the first matrix A09, the first matrix A18, and the first matrix A19 involved in the operation area 2 are shown. The first operation process for the operation area 2 is similar to the above process and will not be repeated here.

[0194] In this embodiment, no invalid data is involved in loading and computing, all on-chip memory is used to store valid data, and all computing power is used to compute valid data, thereby improving the utilization of memory and hardware computing units; it can be used for any feature map size, any convolution kernel size, any stride length, and any filling rule, and is more versatile; a collaborative hardware scheduling strategy is used to give full play to the parallelism of hardware scheduling, increase data reusability, improve data loading efficiency, and reduce memory access overhead; a persistent kernel programming model is used to make task division reasonable and hardware load balanced.

[0195] At least one embodiment of the present disclosure further provides a data processing device. Fig.11 A schematic block diagram of a data processing device provided for at least one embodiment of the present disclosure.

[0196] like Fig.11 As shown, the data processing device 100 includes an execution module 110 .

[0197] The execution module 110 is configured to execute L first operations for L types of operation areas respectively, so as to load the output gradient tensor from the memory to the cache component of the processor, and use the thread warp to perform a strided convolution back-propagation operation on the output gradient tensor to obtain the input gradient tensor and store it in the memory.

[0198] For example, the output gradient tensor is expressed as [N, P, Q, K], where the N dimension is used to represent the number of batches, the P dimension is used to represent the height of the output gradient, the Q dimension is used to represent the width of the output gradient, and the K dimension is used to represent the number of output channels; the convolution kernel tensor used for strided convolution is expressed as [R, S, K, C], where the R dimension is used to represent the height of the convolution kernel, the S dimension is used to represent the width of the convolution kernel, and the C dimension is used to represent the number of input channels; the input gradient tensor is expressed as [N, H, W, C], where the W dimension is used to represent the width of the input gradient, and the H dimension is used to represent the height of the input gradient. For the specific description of the output gradient tensor, the input gradient tensor, and the convolution kernel tensor, please refer to the relevant description in the aforementioned data processing method, which will not be repeated here.

[0199] The input gradient tensor is divided into L types of operation regions on the plane formed by the H dimension and the W dimension. The same type of operation regions are separated by s elements in the H dimension and s elements in the W dimension. Each operation region includes 1 element. s is a positive integer and is the step length of the strided convolution. L is a positive integer and is equal to s×s. For the description of the operation region, please refer to the relevant description in the aforementioned data processing method, which will not be repeated here.

[0200] For example, when the execution module 110 executes the first operation for any type of operation area in the L type operation areas, it includes performing a back-propagation operation on at least one convolution kernel element, and at least one convolution kernel element is a convolution kernel element corresponding to any type of operation area on the plane formed by the R dimension and the S dimension of the convolution kernel tensor, and at least one convolution kernel element is used to obtain the back-propagation operation result of any type of operation area through a back-propagation operation.

[0201] For example, the execution module 110 includes codes and programs stored in the memory. The execution module 110 is implemented as a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities. The processing unit can be a general-purpose processor, and can also be a single-chip microcomputer, a microprocessor, a digital signal processor, a dedicated image processing chip, or a field programmable logic array, etc. The execution module 110 executes the code and program to implement some or all of the functions of the execution module 110 described above. For example, the execution module 110 can be a circuit board or a combination of multiple circuit boards, which is used to implement the functions described above. In an embodiment of the present application, the circuit board or the combination of multiple circuit boards may include: (1) one or more processors; (2) one or more non-temporary memories connected to the processor; and (3) firmware stored in the memory that can be executed by the processor.

[0202] like Fig.11 As shown, the execution module 110 includes a first loading module 1011 , a second loading module 1012 , an operation module 1013 , an accumulation module 1014 , and a storage module 1015 .

[0203] For example, for any type of operation area, the first loading module 1011 is configured to load the first matrix corresponding to at least one convolution kernel element from the memory to the cache component.

[0204] The second loading module 1012 is configured to load at least one second matrix corresponding to each first matrix into the cache component.

[0205] The operation module 1013 is configured to use the thread warp to perform a matrix multiplication-accumulation operation between the loaded first matrix and the second matrix to obtain operation results corresponding to at least one convolution kernel element.

[0206] The accumulation module 1014 is configured to accumulate the operation results corresponding to at least one convolution kernel element to obtain the back propagation operation result of any type of operation area.

[0207] The storage module 1015 is configured to store the back-propagation operation results of any type of operation area in the memory by stride.

[0208] It should be noted that the execution module 110 can be used to implement Figure 3 Step S10 shown; the first loading module 1011 can be used to implement Figure 7 Step S101 shown; the second loading module 1012 can be used to implement Figure 7 Step S102 shown; operation module 1013 can be used to implement Figure 7 Step S103 shown; the accumulation module 1014 can be used to implement Figure 7Step S104 shown; the storage module 1015 can be used to implement Figure 7 As shown in step S105. Therefore, for the specific description of the functions that can be realized by the execution module 110, reference can be made to the relevant description of step S10 in the embodiment of the above-mentioned data processing method; for the specific description of the functions that can be realized by the first loading module 1011, reference can be made to the relevant description of step S101 in the embodiment of the above-mentioned data processing method; for the specific description of the functions that can be realized by the second loading module 1012, reference can be made to the relevant description of step S102 in the embodiment of the above-mentioned data processing method; for the specific description of the functions that can be realized by the operation module 1013, reference can be made to the relevant description of step S103 in the embodiment of the above-mentioned data processing method; for the specific description of the functions that can be realized by the accumulation module 1014, reference can be made to the relevant description of step S104 in the embodiment of the above-mentioned data processing method; for the specific description of the functions that can be realized by the storage module 1015, reference can be made to the relevant description of step S105 in the embodiment of the above-mentioned data processing method, and the repeated parts will not be repeated here. In addition, the data processing device 100 can achieve technical effects similar to those of the above-mentioned data processing method, which will not be repeated here.

[0209] It should be noted that in at least one embodiment of the present disclosure, the data processing device 100 may include more or fewer circuits or units, and the connection relationship between the various circuits or units is not limited and can be determined according to actual needs. The specific configuration of each circuit or unit is not limited and can be composed of analog devices according to circuit principles, or can be composed of digital chips, or can be composed in other applicable ways.

[0210] For example, the data processing device 100 may be implemented in hardware, software, or a combination of hardware and software, and the present disclosure does not impose any specific limitations on this.

[0211] The data processing method provided by at least one embodiment of the present disclosure can achieve a technical effect similar to the data processing method described above, and will not be described in detail here.

[0212] The data processing method and data processing device provided in at least one embodiment of the present disclosure can be applied to different systems or devices, such as Fig.13The electronic device 300 shown. The electronic device 300 can be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an AR device, a VR device, a vehicle terminal, etc., and can also be a server, etc. The data processing method provided in at least one embodiment of the present disclosure can be applied to scenarios involving CPU, high performance computing (HPC) and artificial intelligence (AI) in the electronic device 300, such as tensor computing units, etc. Regarding convolution operations. Of course, the present disclosure is not limited to this, and any scenario, device, apparatus, etc. involving back-propagation calculations can adopt the data processing method or data processing apparatus provided by at least one embodiment of the present disclosure.

[0213] In some embodiments, the data processing device provided by at least one embodiment of the present disclosure may be a chip, for example, the chip is a system-on-a-chip (SoC). The system-on-a-chip includes a processor, which may be a single-core processor or a multi-core processor, a memory, and an I / O interface, etc. After the processor loads the data and application programs in the memory, it processes the data, for example, performs computational processing of back propagation calculations.

[0214] It should be noted that the output gradient tensor and the input gradient tensor of the data processing method or data processing device provided in at least one embodiment of the present disclosure may have different specific physical meanings according to different application scenarios. For example, the data processing method provided in at least one embodiment of the present disclosure may be applied in the fields of speech processing, image processing, text processing, video processing, etc.

[0215] For example, in the field of speech processing, the output gradient tensor and the input gradient tensor can be the input and output parameters of back-propagation calculations involved in tasks such as feature extraction, speech enhancement, and speech recognition.

[0216] For example, in the field of image processing, the output gradient tensor and the input gradient tensor can be the input and output parameters of back-propagation calculations involved in tasks such as image recognition, feature extraction, image segmentation, target detection, image classification, and scene reconstruction.

[0217] For example, in the field of text processing, the output gradient tensor and the input gradient tensor can be the input and output parameters of back-propagation calculations involved in tasks such as text classification, sentiment analysis, and text generation.

[0218] For example, in the field of video processing, the output gradient tensor and the input gradient tensor can be related parameters in the image processing field as described above, or input and output parameters that are specific to the video processing field, such as the optical flow operator (used to estimate motion between video frames), the target tracking operator (used to track specific targets in the video), etc., involving back-propagation calculations.

[0219] Of course, the present disclosure is not limited to this, and for other application scenarios or fields, as long as back propagation calculation is required, the data processing method described in at least one embodiment of the present disclosure can be applied, and will not be described one by one here.

[0220] Fig.12 A schematic structural diagram of a processor provided in at least one embodiment of the present disclosure. Fig.12 As shown, the processor 200 includes an instruction parsing unit 201 , an execution unit 202 , a memory 203 and a cache unit 204 .

[0221] For example, the instruction parsing unit 201 is used to receive and parse data processing instructions.

[0222] For example, the data processing instruction is used to load the output gradient tensor from the memory 203 to the cache unit 204 of the processor, and use the thread warp to perform a strided convolution back-propagation operation on the output gradient tensor to obtain the input gradient tensor and store it in the memory 203.

[0223] For example, the cache component 204 may include a shared memory. For example, the processor 200 may use the aforementioned Figure 1 The graphics processor or general graphics processor architecture shown.

[0224] For example, the output gradient tensor is expressed as [N, P, Q, K], where the N dimension is used to represent the batch size, the P dimension is used to represent the width of the output gradient, the Q dimension is used to represent the height of the output gradient, and the K dimension is used to represent the number of output channels; the convolution kernel tensor used for strided convolution is expressed as [R, S, K, C], where the R dimension is used to represent the width of the convolution kernel, the S dimension is used to represent the height of the convolution kernel, and the C dimension is used to represent the number of input channels; the input gradient tensor is expressed as [N, H, W, C], where the W dimension is used to represent the width of the input gradient, and the H dimension is used to represent the height of the input gradient. For the specific description of the output gradient tensor, the input gradient tensor, and the convolution kernel tensor, please refer to the relevant description in the aforementioned data processing method, which will not be repeated here.

[0225] The input gradient tensor is divided into L types of operation regions on the plane formed by the H dimension and the W dimension. The same type of operation regions are separated by s elements in the H dimension and s elements in the W dimension. Each operation region includes 1 element. s is a positive integer and is the step length of the strided convolution. L is a positive integer and is equal to s×s. For the description of the operation region, please refer to the relevant description in the aforementioned data processing method, which will not be repeated here.

[0226] For example, the execution unit 202 executes the data processing instruction after the instruction parsing unit parses the data processing instruction.

[0227] For example, after receiving a data processing instruction, the processor parses the data processing instruction, such as decoding the data processing instruction, generates a microinstruction and sends the microinstruction to the instruction allocation unit; the instruction allocation unit sends the microinstruction to the corresponding scheduling queue according to the microinstruction category; the scheduler in the processor selects the appropriate microinstruction from the scheduling queue for execution based on the priority, dependency and other factors of the microinstruction; in response to the microinstruction, when all or the required input data is prepared, the execution unit reads the data and executes the relevant operations of the data processing instruction.

[0228] For example, the execution unit may include hardware involved in executing data processing instructions, such as a multiplier, a register, a queue, an ALU (Arithmetic Logic Unit), etc.

[0229] For example, when the execution unit 202 executes a data processing instruction, it includes performing a back-propagation operation on at least one convolution kernel element, at least one convolution kernel element is a convolution kernel element corresponding to any type of operation area on the plane formed by the R dimension and the S dimension of the convolution kernel tensor, and at least one convolution kernel element is used to obtain a back-propagation operation result of any type of operation area through a back-propagation operation.

[0230] Specifically, when processor-based upper-level software (for example, AI applications, HPC applications, and scientific computing applications, etc.) can send data processing instructions for computing processing to a processor (such as a CPU or GPU) through a uniformly packaged function library, the data processing instructions can carry input parameters such as storage addresses; when the processor receives the data processing instructions, the instruction parsing unit 201 parses the data processing instructions, obtains the input parameters, loads the data indicated by the storage address from the memory to the register, and caches them in the cache component.

[0231] Regarding the specific process of using the execution unit 202 to execute the data processing instruction, reference may be made to step S10 in the aforementioned data processing method, and the repeated parts will not be repeated here.

[0232] The processor provided in at least one embodiment of the present disclosure can achieve technical effects similar to those of the aforementioned data processing method, and the repeated parts will not be repeated.

[0233] Fig.13 The following is a schematic block diagram of an electronic device provided by an embodiment of the present disclosure. Fig.13 As shown, the electronic device 300 is suitable for implementing the data processing method provided by the embodiment of the present disclosure. It should be noted that Fig.13 The components of the electronic device 300 shown are merely exemplary and non-limiting. The electronic device 300 may also have other components according to actual application requirements.

[0234] like Fig.13 As shown, the electronic device 300 may include a processing device (eg, a central processing unit, a graphics processing unit, etc.) 301, which may perform various appropriate actions and processes according to non-transitory computer-readable instructions stored in a memory to implement various functions.

[0235] For example, when the computer readable instructions are executed by the processing device 301, one or more steps in the data processing method described in any of the above embodiments may be executed. It should be noted that for a detailed description of the processing process of the data processing method, reference may be made to the relevant description in the above data processing method embodiments.

[0236] For example, the memory may include any combination of one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) 303 and / or cache memory (cache), etc. For example, computer-readable instructions may be loaded from storage device 308 into random access memory (RAM) 303 to run computer-readable instructions. Non-volatile memory may include, for example, read-only memory (ROM) 302, hard disk, erasable programmable read-only memory (EPROM), portable compact disk read-only memory (CD-ROM), USB memory, flash memory, etc. Various applications and various data, such as style images, and various data used and / or generated by applications, etc., may also be stored in the computer-readable storage medium.

[0237] For example, the processing device 301 , the ROM 302 , and the RAM 303 are connected to one another via a bus 304 . An input / output (I / O) interface 305 is also connected to the bus 304 .

[0238] Typically, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, a flash memory, etc.; and a communication device 309. The communication device 309 may allow the electronic device 300 to communicate with other electronic devices wirelessly or by wire to exchange data. Although Fig.13An electronic device 300 having various devices is shown, but it should be understood that it is not required to implement or have all the devices shown, and the electronic device 300 may alternatively implement or have more or fewer devices. For example, the processing device 301 can control other components in the electronic device 300 to perform the desired functions. The processing device 301 can be a device with data processing capabilities and / or program execution capabilities such as a central processing unit (CPU), a tensor processing unit (TPU) or a graphics processing unit GPU. The central processing unit (CPU) can be an X86, ARM, RISC-V architecture, etc. The GPU can be directly integrated into the SOC, directly integrated into the motherboard, or built into the north bridge chip of the motherboard.

[0239] Fig.14 A schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure. Fig.14 As shown, the storage medium 400 may be a non-transitory computer-readable storage medium, and one or more computer-readable instructions 401 may be non-transitory stored on the storage medium 400. For example, when the computer-readable instructions 401 are executed by a processor, one or more steps in the data processing method described above may be executed.

[0240] For example, the storage medium 400 may be applied in the electronic device 300 . For example, the storage medium 400 may include the storage device 308 in the electronic device 300 .

[0241] For example, the storage device may include any combination of one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory, etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disk read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer-readable instructions may be stored on the computer-readable storage medium, and the processor may run the computer-readable instructions to implement various functions of the processor. Various applications and various data may also be stored in the storage medium.

[0242] For example, the storage medium may include a memory card of a smart phone, a cache component of a tablet computer, a hard disk of a personal computer, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other applicable storage media.

[0243] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0244] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0245] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0246] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0247] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0248] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

[0249] There are a few points to note about this disclosure:

[0250] (1) The drawings of the embodiments of the present disclosure only relate to the structures related to the embodiments of the present disclosure. Other structures may refer to the general design.

[0251] (2) In the absence of conflict, the embodiments of the present disclosure and the features therein may be combined with each other to obtain new embodiments.

[0252] The above description is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. The protection scope of the present disclosure shall be based on the protection scope of the claims.

Claims

1. A data processing method, comprising: Performing L first operations for L types of operation areas respectively to load the output gradient tensor from the memory to the cache component of the processor, and using the thread warp to perform a strided convolution back-propagation operation on the output gradient tensor to obtain the input gradient tensor and store it in the memory, The output gradient tensor is expressed as [N, P, Q, K], where the N dimension is used to represent the number of batches, the P dimension is used to represent the height of the output gradient tensor, the Q dimension is used to represent the width of the output gradient tensor, and the K dimension is used to represent the number of output channels; the convolution kernel tensor used for the strided convolution is expressed as [R, S, K, C], where the R dimension is used to represent the height of the convolution kernel, the S dimension is used to represent the width of the convolution kernel, and the C dimension is used to represent the number of input channels; the input gradient tensor is expressed as [N, H, W, C], where the W dimension is used to represent the width of the input gradient tensor, and the H dimension is used to represent the height of the input gradient tensor; The input gradient tensor is divided into the L types of operation regions at least on the plane formed by the H dimension and the W dimension, the operation regions of the same type are separated by s elements in the H dimension and s elements in the W dimension, each operation region includes 1 element, s is a positive integer and is the step length of the strided convolution, and L is a positive integer and equal to s×s; Wherein, when executing the first operation for any type of operation area in the L type operation areas, it includes performing a back-propagation operation operation of at least one convolution kernel element, and the at least one convolution kernel element is the convolution kernel element corresponding to the any type of operation area on the plane formed by the R dimension and the S dimension of the convolution kernel tensor, and the at least one convolution kernel element is used to obtain the back-propagation operation result of the any type of operation area through the back-propagation operation.

2. The data processing method according to claim 1, wherein: For any type of operation area, the first operation includes the following operations: Loading the first matrix corresponding to the at least one convolution kernel element from the memory to the cache component; Loading at least one second matrix corresponding to each first matrix into the cache component; Using the thread warp to perform a matrix multiplication-accumulation operation between the loaded first matrix and the second matrix to obtain operation results corresponding to the at least one convolution kernel element respectively; Accumulating the operation results corresponding to the at least one convolution kernel element respectively to obtain the back propagation operation result of any type of operation area; The back propagation operation result of any type of operation area is strided and stored in the memory.

3. The data processing method according to claim 2, wherein: Loading the first matrix corresponding to the at least one convolution kernel element from the memory to the cache component includes: For any convolution kernel element of the at least one convolution kernel element, loading a first matrix corresponding to the any convolution kernel element into the cache component, Among them, the shape size of the first matrix is ​​A×B, the B elements located in the same row of the first matrix are B consecutive elements in the convolution kernel tensor that have the same coordinates as any convolution kernel element in the R dimension, the S dimension and the C dimension, but different coordinates in the K dimension, different rows of elements in the first matrix have different coordinates in the C dimension, and A and B are positive integers.

4. The data processing method according to claim 2, wherein: Loading at least one second matrix corresponding to each first matrix into the cache component includes: loading at least one second matrix corresponding to the first matrix into the cache component, Among them, the shape size of each second matrix is ​​B×A, the A elements located in the same row in the second matrix are consecutive A elements in the first sequence corresponding to the convolution kernel elements corresponding to the first matrix, the first sequence is obtained by expanding the output gradient tensor into one dimension in the H dimension, the W dimension and the N dimension, the first element in the first sequence is related to the convolution kernel element corresponding to the first sequence, the coordinates of different rows of elements in the second matrix in the K dimension are different, and A and B are positive integers.

5. The data processing method according to claim 2, wherein: Using the thread warp to perform a matrix multiplication-accumulation operation between the loaded first matrix and the second matrix to obtain operation results corresponding to the at least one convolution kernel element, including: For any convolution kernel element of the at least one convolution kernel element, use the thread warp to respectively perform the matrix multiplication and accumulation operations between the first matrix corresponding to the any convolution kernel element and the M second matrices to obtain M intermediate results as the operation results corresponding to the any convolution kernel element, Among them, the M second matrices are M second matrices corresponding to the first matrix corresponding to any convolution kernel element, each intermediate result is the output result of the matrix multiplication and accumulation operation of the first matrix and one of the M second matrices, and M is a positive integer.

6. The data processing method according to claim 5, wherein: Accumulating the operation results corresponding to the at least one convolution kernel element to obtain the back propagation operation result of any type of operation area, including: The M intermediate results corresponding to each of the at least one convolution kernel elements are added together to obtain the back propagation operation result of any type of operation area.

7. The data processing method according to claim 2, wherein: In response to obtaining a calculation result corresponding to any convolution kernel element of the at least one convolution kernel element, M intermediate results included in the calculation result are stored in M ​​register units respectively, where M is a positive integer. In response to obtaining the operation results corresponding to other convolution kernel elements again, the M intermediate results included in the operation results corresponding to the other convolution kernel elements are respectively accumulated with the data in the corresponding register units, and the accumulated results are stored in the corresponding register units.

8. The data processing method according to claim 2, wherein: The back propagation operation result of any type of operation area includes M operation results, each of which is expressed as [NHW, C], wherein NHW represents expansion from H dimension, W dimension and N dimension into one dimension, and the shape size of the operation result is A×A, A and M are positive integers, Storing the back-propagation operation result in the memory by striding, comprising: Sequentially store the M operation results to corresponding positions in the input gradient tensor in the memory; Among them, for each operation result, the A elements on the NHW dimension in the operation result are stored one by one in the corresponding positions in the input gradient tensor in the order of the W dimension first, the H dimension second, and the N dimension last, and the A elements are stored at intervals of s steps in both the H dimension and the W dimension.

9. The data processing method according to claim 4, wherein: At least some of the elements in the first sequence corresponding to different convolution kernel elements are the same, and the same elements are cached in the global cache of the processor when loaded from the memory to the cache component, When the same element is loaded into the cache component again, it is loaded from the global cache.

10. The data processing method according to claim 2, wherein: In response to s=2, R=3, S=3, the input gradient tensor is divided into 4 types of operation regions, including a first type of operation region, a second type of operation region, a third type of operation region, and a fourth type of operation region. The at least one convolution kernel element corresponding to the first type of operation area includes a first convolution kernel element located at the center of a plane formed by the R dimension and the S dimension in the convolution kernel tensor, The at least one convolution kernel element corresponding to the second type of operation area includes the convolution kernel elements at the upper left corner, lower left corner, upper right corner and lower right corner of the first convolution kernel element, The at least one convolution kernel element corresponding to the third type of operation area includes the convolution kernel elements above and below the first convolution kernel element. The at least one convolution kernel element corresponding to the fourth type of operation area includes the convolution kernel elements on the left and right sides of the first convolution kernel element.

11. The data processing method according to claim 2, wherein: In response to the output gradient tensor needing to be border-filled, when performing a matrix multiplication-accumulation operation between the first matrix and the second matrix, a mask operation is performed on some elements to simulate the border-filling.

12. The data processing method according to any one of claims 2 to 11, wherein: Loading the first matrix corresponding to the at least one convolution kernel element from the memory to the cache component includes: Loading a plurality of first matrices corresponding to each convolution kernel element from the memory to the cache component, wherein different row elements in the plurality of first matrices have different coordinates in the C dimension; Using the thread warp to perform a matrix multiplication-accumulation operation between the loaded first matrix and the second matrix, comprising: The matrix multiplication-accumulation operation between the plurality of first matrices and the loaded second matrix is ​​performed in parallel using a plurality of thread warps.

13. A processor comprising an instruction parsing unit, an execution unit, a cache component and a memory, wherein: The instruction parsing unit is used to receive and parse data processing instructions, wherein the data processing instructions are used to load the output gradient tensor from the memory to the cache component, and use the thread warp to perform a back propagation operation of strided convolution on the output gradient tensor to obtain an input gradient tensor and store it in the memory, The execution unit executes the data processing method according to any one of claims 1 to 12 after the instruction parsing unit parses the data processing instruction.

14. A data processing device, comprising an execution module, The execution module is configured to execute L first operations for L types of operation areas respectively, so as to load the output gradient tensor from the memory to the cache component of the processor, and use the thread warp to perform a back propagation operation of strided convolution on the output gradient tensor to obtain the input gradient tensor and store it in the memory; in, The output gradient tensor is expressed as [N, P, Q, K], where the N dimension is used to indicate the number of batches, the P dimension is used to indicate the height of the output gradient tensor, the Q dimension is used to indicate the width of the output gradient tensor, and the K dimension is used to indicate the number of output channels; the convolution kernel tensor used for the strided convolution is expressed as [R, S, K, C], where the R dimension is used to indicate the height of the convolution kernel, the S dimension is used to indicate the width of the convolution kernel, and the C dimension is used to indicate the number of input channels; the input gradient tensor is expressed as [N, H, W, C], where the W dimension is used to indicate the width of the input gradient tensor, and the H dimension is used to indicate the height of the input gradient tensor; The input gradient tensor is divided into the L types of operation regions at least on the plane formed by the H dimension and the W dimension, the operation regions of the same type are separated by s elements in the H dimension and also separated by s elements in the W dimension, each operation region includes 1 element, s is a positive integer and is the step size of the strided convolution, and L is a positive integer and equal to s×s; The execution module is configured to execute the first operation once to obtain the back propagation operation result of the same type of operation area, and the input gradient tensor is obtained by sequentially splicing the back propagation operation results of the L type of operation areas. Among them, when the execution module executes the first operation for any type of operation area in the L type operation areas, it includes performing a back-propagation operation of at least one convolution kernel element, and the at least one convolution kernel element is a convolution kernel element of the convolution kernel tensor corresponding to the any type of operation area on the plane formed by the R dimension and the S dimension, and the at least one convolution kernel element is used to obtain the back-propagation operation result of the any type of operation area through the back-propagation operation.

15. An electronic device, comprising: A memory non-transitorily stores computer executable instructions; a processor configured to execute the computer executable instructions, Wherein, when the computer executable instructions are executed by the processor, the data processing method according to any one of claims 1-12 is implemented.

16. A non-transitory computer-readable storage medium, wherein: The non-transitory computer-readable storage medium stores computer-executable instructions, When the computer executable instructions are executed by a processor, the data processing method according to any one of claims 1 to 12 is implemented.

Images