Data processing method and device, processor, electronic device and storage medium

Abstract

A data processing method and device, processor, electronic device, and storage medium. The data processing method includes: performing a first operation multiple times to obtain multiple weight gradient sub-tensors, each weight gradient sub-tensor being a portion of the weight gradient tensor divided according to a 1×n granularity; arranging the multiple weight gradient sub-tensors to obtain a weight gradient tensor, wherein the first operation includes: loading multiple first elements of the input gradient tensor and multiple second elements of the activation tensor from a first cache to a second cache, wherein the multiple second elements are multiple elements in the activation tensor corresponding to a target sub-tensor; reading the multiple first elements from the second cache to a tensor calculation module; and having the tensor calculation module reuse the multiple second elements to perform a second operation n times to obtain a target sub-tensor, wherein each second operation is used to obtain a target element in the target sub-tensor. This method can reduce cache access and improve cache utilization.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to a data processing method, a data processing apparatus, a processor, an electronic device, and a non-transitory computer-readable storage medium. Background Art

[0002] Convolutional Neural Network (CNN) is a deep learning algorithm mainly used in the field of computer vision. It has been widely used in image recognition, speech recognition, video recognition, natural language processing and other fields. Summary of the Invention

[0003] At least one embodiment of the present disclosure provides a data processing method, which at least partially alleviates the technical problem of low performance of backpropagation operations, and achieves the technical effects of improving the performance of backpropagation operations, reducing cache access, and improving cache utilization.

[0004] At least one embodiment of the present disclosure provides a data processing method for performing operations on an input gradient tensor and an activation tensor during backpropagation to obtain a weight gradient tensor, the method comprising: performing a first operation multiple times to obtain a plurality of weight gradient sub-tensors, wherein the weight gradient tensor includes a height direction and a width direction, and each weight gradient sub-tensor is a part of the weight gradient tensor divided according to a 1×n granularity within a two-dimensional matrix formed by the height direction and the width direction, where n is an integer greater than 1; and arranging the plurality of weight gradient sub-tensors to obtain the weight gradient tensor, each time performing the first operation to obtain a target sub-tensor in the plurality of weight gradient sub-tensors, the first The operation includes: loading multiple first elements of the input gradient tensor and multiple second elements of the activation tensor from the first cache to the second cache, wherein the multiple second elements are multiple elements in the activation tensor corresponding to the target sub-tensor; reading the multiple first elements from the second cache to the tensor calculation module; the tensor calculation module reuses the multiple second elements to perform n second operations to obtain the target sub-tensor, wherein each second operation is used to obtain the target element in the target sub-tensor, and the second operation includes: reading the multiple second elements to obtain multiple third elements required for the second operation; and determining the target element based on the multiple first elements and the multiple third elements.

[0005] At least one embodiment of the present disclosure further provides a processor, comprising an instruction parsing unit and an execution unit, wherein the instruction parsing unit is used to receive and parse data processing instructions, wherein the data processing instructions are used to perform operations on input gradient tensors and activation tensors to obtain weight gradient tensors during backpropagation, and the execution unit executes the data processing method provided by any embodiment of the present disclosure after the instruction parsing unit parses the data processing instructions.

[0006] At least one embodiment of the present disclosure further provides a data processing device for performing operations on an input gradient tensor and an activation tensor during backpropagation to obtain a weight gradient tensor, the device comprising: an execution module configured to perform multiple first operations to obtain multiple weight gradient sub-tensors, wherein the weight gradient tensor includes a height direction and a width direction, and each weight gradient sub-tensor is a part of the weight gradient tensor divided according to a granularity of 1×n in the two-dimensional matrix formed by the height direction and the width direction, where n is an integer greater than 1; and arranging the multiple weight gradient sub-tensors to obtain the weight gradient tensor, wherein each execution of the first operation obtains a target sub-tensor in the multiple weight gradient sub-tensors, the execution module comprising: a loading module and a tensor calculation module, The loading module is configured to load multiple first elements of the input gradient tensor and multiple second elements of the activation tensor from the first cache to the second cache, wherein the multiple second elements are multiple elements in the activation tensor corresponding to the target sub-tensor; the tensor calculation module is configured to read the multiple first elements from the second cache to the tensor calculation module; and reuse the multiple second elements to perform n second operations to obtain the target sub-tensor, wherein each second operation is used to obtain the target element in the target sub-tensor, and the tensor calculation module includes: a reading unit, configured to: read the multiple second elements to obtain the multiple third elements required for the second operation, and a calculation unit, configured to: determine the target element based on the multiple first elements and the multiple third elements.

[0007] At least one embodiment of the present disclosure further provides an electronic device, comprising: a memory, which non-transiently stores computer-executable instructions; and a processor, configured to execute the computer-executable instructions, wherein the computer-executable instructions, when executed by the processor, implement the data processing method provided by any embodiment of the present disclosure.

[0008] At least one embodiment of the present disclosure further provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the data processing method provided by any embodiment of the present disclosure is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, rather than limiting the present disclosure.

[0010] Figure 1 A schematic diagram of a convolution operation is shown;

[0011] Figure 2 A schematic diagram showing the use of implicit GEMM to implement convolution;

[0012] Figure 3 A schematic structural diagram of a general-purpose graphics processing unit (GPGPU);

[0013] Figure 4A A schematic flow chart of a data processing method provided by at least one embodiment of the present disclosure is shown;

[0014] Figure 4B A schematic diagram of an activation tensor;

[0015] Figure 4C A schematic diagram of a weight gradient tensor;

[0016] Figure 4D It shows that at least one embodiment of the present disclosure provides Figure 4C The weight gradient tensor shown is divided into multiple weight gradient sub-tensors according to the 1×3 granularity;

[0017] Figure 5A A schematic diagram of performing a first operation to obtain a target sub-tensor provided by at least one embodiment of the present disclosure is shown;

[0018] Figure 5B A schematic diagram of an input gradient sub-tensor obtained by dividing the input gradient tensor into [m,n'] granularity is shown;

[0019] Figure 5C A schematic diagram illustrating multiple second elements in an activation tensor provided by at least one embodiment of the present disclosure is shown;

[0020] Figure 5D A schematic diagram of loading a plurality of first elements and a plurality of second elements from a global cache into a shared memory provided by at least one embodiment of the present disclosure is shown;

[0021] Figure 6 A flow chart of a method for determining starting positions of a plurality of second elements provided by at least one embodiment of the present disclosure is shown;

[0022] Figure 7A schematic diagram of loading multiple second elements in an activation tensor provided by at least one embodiment of the present disclosure is shown;

[0023] Figure 8 A flow chart of a method for reading a plurality of second elements to obtain a plurality of third elements required for a second operation, provided by at least one embodiment of the present disclosure, is shown;

[0024] Figure 9A A schematic diagram of a mask provided by at least one embodiment of the present disclosure is shown;

[0025] Figure 9B A schematic diagram showing another mask provided by at least one embodiment of the present disclosure is shown;

[0026] Figure 10 A schematic diagram of a ping-pong strategy process according to an embodiment of the present disclosure;

[0027] Figure 11 A schematic block diagram of a data processing device provided in at least one embodiment of the present disclosure;

[0028] Figure 12 A schematic structural diagram of a processor provided for at least one embodiment of the present disclosure;

[0029] Figure 13 A schematic block diagram of an electronic device provided in accordance with an embodiment of the present disclosure; and

[0030] Figure 14 A schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] To make the purpose, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0032] Unless otherwise defined, the technical or scientific terms used in this disclosure should have the usual meanings understood by people with ordinary skills in the field to which this disclosure belongs. The words "first", "second" and similar words used in this disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "one", "an" or "the" do not indicate a quantity limitation, but rather indicate the existence of at least one. Words such as "include" or "comprise" mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connect" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0033] The present disclosure is described below through several specific embodiments. To keep the following description of the embodiments of the present disclosure clear and concise, the present disclosure omits detailed descriptions of known functions and known components. When any component of an embodiment of the present disclosure appears in more than one figure, the component is represented by the same or similar reference numeral in each figure.

[0034] Convolutional Neural Networks (CNNs) are a type of feedforward neural network with a deep architecture that incorporates convolutional computations. They are a representative algorithm for deep learning. In CNNs, convolution kernels (or filters) are designed to perform a sliding window operation on an input tensor (such as an image) to compute local features of the tensor. Specifically, the convolution kernel slides across the input tensor with a specific stride, performing local connections and weight sharing at each location. At each location, the convolution kernel is element-wise multiplied with the corresponding local region (i.e., window) of the input tensor and the sum is calculated (in some cases, a bias is added). This yields a single point in the output tensor (feature map). As the convolution kernel slides across the input tensor, the entire output tensor of the image is computed. In image recognition applications, the input tensor can be image pixels. In speech recognition applications, the input tensor can be audio data. In video recognition applications, the input tensor can be video data. In the application scenario of natural language processing, the input tensor here can be natural language text.

[0035] CNNs in deep learning frameworks often perform edge padding based on different requirements. For example, to ensure that the output tensor and the input tensor have the same size, the center of the convolution kernel is usually aligned with the first point of the input tensor to determine the starting position of the convolution kernel, and then the corresponding edge padding bits are added.

[0036] Figure 1 A schematic diagram of a convolution operation is shown.

[0037] like Figure 1 As shown in the figure, the shaded area filled with diagonal lines is the input tensor, and the size of the convolution kernel J is 3×3. In order to make the output tensor the same size as the input tensor, the edges of the input tensor are padded so that the center E'0 of the convolution kernel J is aligned with the first point E0 of the input tensor. Since the size of the convolution kernel J is 3×3, the padding parameter is (1,1) (that is, the input tensor is padded with 1 pixel size on all sides, Figure 1 The white dashed square in the middle) can align the center E'0 of the convolution kernel J with the first point E0 of the input tensor. For example, Figure 1 The white dotted square in the middle is filled with 0.

[0038] CNN of deep learning framework is not limited to Figure 1 In the manner shown, for example, the sliding step size of the convolution kernel J can be arbitrary and not limited to 1.

[0039] The convolution operation formula can be expressed as Output = Conv(Weight, Activation), where Activation represents the activation tensor (also called the "input tensor" described above). Its tensor shape can be expressed as [N,H,W,C], where the N dimension represents the batch size, the W dimension represents the width of the activation tensor, the H dimension represents the height of the activation tensor, and the C dimension represents the number of input channels. Weight represents the convolution kernel tensor, and its tensor shape can be expressed as [R,S,K,C], where the R dimension represents the height of the convolution kernel, the S dimension represents the width of the convolution kernel, the C dimension represents the number of input channels, and the K dimension represents the number of output channels. Output represents the output tensor, and its tensor shape can be expressed as [N,P,Q,K], where the P dimension represents the height of the output tensor, the Q dimension represents the width of the output tensor, and the K dimension represents the number of output channels. Specifically, during the convolution operation (Conv(Weight, Activation)), the edges of the activation tensor are typically padded so that the size of the output data after the convolution is generally consistent with the size of the input data. After padding, convolution in convolutional neural networks can be equivalently converted to matrix-vector multiplication, which improves convolution computation efficiency. Currently, convolution operations are typically implemented using the Implicit GEMM (Generalized Matrix-Matrix Multiply) algorithm. The Implicit GEMM algorithm formulates the convolution operation as a generalized matrix-matrix multiplication (GEMM). During the convolution process, the activation tensor is applied to a sliding filter (also known as a convolution kernel) to produce an output tensor. The Implicit GEMM algorithm avoids explicitly converting the activation tensor and convolution kernel into matrix form during computation through coordinate mapping, thereby saving storage space.

[0040] Figure 2 Figure 2 shows a schematic diagram of using implicit GEMM to implement convolution. Figure 2 As shown, the response convolution operation performs sliding window dot products on N input tensors of size C×H×W using K convolution kernels of size C×R×S. The resulting output is N output tensors of size K×P×Q. The implicit GEMM algorithm treats the convolution kernel as a two-dimensional matrix of size K×(C×R×S). The input tensor is transformed into a two-dimensional matrix of size (C×R×S)×(N×H×W) based on the position of the convolution kernel sliding window. The two matrices are then multiplied to obtain the output data. This conversion method converts the input data from [N,H,W,C] to [NHW,RSC], increasing the data size by a factor of R×S.

[0041] To achieve robust performance, convolutional neural networks (CNNs) require a robust training process. Trained CNNs can be effectively applied to applications such as image processing and machine translation. CNN training typically consists of two phases: forward propagation and backpropagation. During the forward propagation phase, the input tensor passes through layers such as convolutional layers, pooling layers, and fully connected layers. Through layer-by-layer linear transformations (such as convolution and matrix multiplication) and nonlinear activations (such as ReLU and Sigmoid functions), the network generates an output. This output is compared with the true label, and a loss function (such as cross-entropy loss or mean squared error loss) is used to calculate the training error, which provides an optimization target for backpropagation. Backpropagation is the core mechanism for optimizing network parameters. The core idea of backpropagation is to use the chain rule to calculate the gradient of each layer's parameters (primarily weights and biases) with respect to the loss function. The parameters are then updated using optimization algorithms such as gradient descent to minimize the loss function. The calculation of the gradient of the weight with respect to the loss function (also called "weight gradient") is a key step in backpropagation, and its accuracy directly affects the efficiency and performance of network training. The weight gradient is the partial derivative of the loss function with respect to the weight parameters of each layer in the network. It reflects the degree of influence of the change of the weight parameters on the loss function. The following describes the calculation method of the weight gradient. Assume that in a convolutional neural network, the following basic symbolic definitions are made: the activation tensor of the lth layer (that is, the output tensor of the l-1 layer); the weight W (l) represents the weight matrix of layer l; bias b (l) Represents the bias vector of the lth layer, and the loss function L is, for example, mean square error or cross entropy.

[0042] First, forward propagation is performed to obtain the output tensor z of the lth layer (l) = W (l) Activation + b (l) .

[0043] Then, the gradient tensor Gradient of the lth layer is calculated, which is the gradient tensor Gradient of the loss function L on the output tensor z (l) The gradient of the output tensor is usually a matrix of the same size as the output tensor, for example, the size of the gradient tensor Gradient is [N, P, Q, K].

[0044] Finally, the gradient tensor Gradient and the activation tensor Activation are cross-correlated to obtain the weight gradient .Right now:

[0045]

[0046] The essence of the cross-correlation operation is a weighted summation of the local region of the activation tensor using a sliding window. Similar to the convolution operation described above, the cross-correlation operation between the gradient tensor and the activation tensor can be viewed as a convolution of the transpose of the gradient tensor and the activation tensor. This can also be implemented using the implicit GEMM algorithm. The gradient tensor is considered a two-dimensional matrix of K × (N × P × Q). The activation is transformed into a two-dimensional matrix of (N × P × Q) × (R × S × C) based on the position of the gradient tensor sliding window. The output data is then obtained by performing a matrix multiplication of the two matrices. This conversion method converts the input data from [N, H, W, C] to [NPQ, RSC], increasing the data volume by a factor of R × S.

[0047] On the one hand, since the activation tensor changes from [N, H, W, C] to [NPQ, RSC], the data volume increases by R×S times (usually P=H, Q=W). This data volume expansion basically occurs in the global cache or the second-level cache, and the operator performance is easily limited by the memory access bandwidth of the second-level cache or the global cache. On the other hand, the method of padding the edges with zeros will occupy a large amount of on-chip memory space, and most of the data is invalid, and only a small part is valid data that is actually involved in the calculation. In addition, most operations are multiplications with 0, which wastes a lot of computing power. These two points lead to the low performance of existing backpropagation operations.

[0048] Artificial intelligence processors play a key role in convolutional computing, especially in deep learning, computer vision, and image processing. They can include graphics processing units (GPUs), tensor processing units (TPUs), neural network processing units (NPUs), deep learning processing units (DPUs), accelerated processing units (APUs), and general-purpose computing on graphics processing units (GPGPUs).

[0049] The embodiments herein are described using GPGPU as an example, but the embodiments of the present disclosure are not limited thereto.

[0050] Figure 3 A schematic structural diagram of a general-purpose graphics processing unit (GPGPU).

[0051] like Figure 3 As shown, the general purpose graphics processor is actually an array of programmable multiprocessors. For example, the programmable multiprocessor can be a streaming processor cluster (SPC), including Figure 3 Streaming processor clusters 1, ..., and M are shown, where M is a positive integer greater than 1. In a general-purpose graphics processor, one streaming processor cluster processes one computing task, or multiple streaming processor clusters process one computing task. Multiple streaming processor clusters share data through a global cache or global memory.

[0052] like Figure 3 As shown, taking stream processor cluster 1 as an example, a stream processor cluster includes multiple computing units, such as Figure 3 In the CU, there are CU 1, CU 2, ..., CU N, where N is a positive integer. Each CU is used to perform arithmetic and logical operations, such as accumulation, reduction, conventional addition, subtraction, multiplication, and division. A CU includes multiple cores (also called computing cores or computing cores), each of which includes an arithmetic logic unit (ALU), a floating-point computing unit, etc. The computing core is used to perform specific computing tasks. In addition, the CU also includes registers (e.g. Figure 3 The register file in the computing unit and the shared memory are used to hierarchically store source data and destination data related to computing tasks. The shared memory in a computing unit is used to share data between the cores of the computing unit.

[0053] In parallel computing, computing tasks are generally executed by multiple threads. These threads are divided into multiple thread blocks before being executed in a general-purpose graphics processor (or parallel computing processor), and then distributed through the thread block distribution module ( Figure 3 (Not shown) Multiple thread blocks are distributed to various CUs. All threads in a thread block must be assigned to the same CU for execution. Thread blocks are also split into minimum execution warps (or simply warps). Each warp contains a fixed number (or less) of threads, for example, 32 threads. Multiple thread blocks can execute in the same CU or in different CUs.

[0054] In each computational unit, the warp scheduling / dispatching module ( Figure 3 (not shown) schedules and allocates thread warps so that the multiple compute cores of the compute unit can execute the warps. Depending on the number of compute cores in the compute unit, multiple warps in a thread block can execute simultaneously or in a time-sharing manner. Multiple threads in each warp execute the same instruction. Memory execution instructions are sent to the shared memory within the compute unit or further to the mid-level cache, global cache, or global memory for read and write operations.

[0055] The defects existing in the solutions in the above-mentioned prior art are also the results obtained by the inventor after careful research through practice and creative labor. The discovery process of the above-mentioned problems and the solutions proposed for the above-mentioned problems in at least one embodiment disclosed in this text are all creative contributions of the inventor in the invention process.

[0056] At least one embodiment of the present disclosure provides a data processing method for performing an operation on an input gradient tensor and an activation tensor during backpropagation to obtain a weight gradient tensor, the method comprising: performing a first operation multiple times to obtain a plurality of weight gradient sub-tensors, the weight gradient tensor including a height direction and a width direction, each weight gradient sub-tensor being a part of the weight gradient tensor divided according to a 1×n granularity in a two-dimensional matrix in the height direction and the width direction, where n is an integer greater than 1; and arranging the plurality of weight gradient sub-tensors to obtain a weight gradient tensor, each time performing the first operation to obtain an object in the plurality of weight gradient sub-tensors. The first operation includes: loading multiple first elements of the input gradient tensor and multiple second elements of the activation tensor from the first cache to the second cache, where the multiple second elements are multiple elements in the activation tensor corresponding to the target sub-tensor; reading the multiple first elements from the second cache to the tensor calculation module; the tensor calculation module reuses the multiple second elements to perform the second operation n times to obtain the target sub-tensor, each second operation is used to obtain the target element in the target sub-tensor, and the second operation includes: obtaining multiple third elements required for the second operation from the multiple second elements; and determining the target element based on the multiple first elements and the multiple third elements.

[0057] In the data processing method provided by at least one embodiment of the present disclosure, the weight gradient tensor is split into multiple weight gradient sub-tensors according to the granularity of 1×n, and the multiple first elements of the input gradient tensor and the multiple second elements in the corresponding activation tensor are loaded from the first cache to the second cache. When performing cross-correlation operations, the multiple first elements and the corresponding multiple second elements are read from the second cache to the tensor calculation module, so that during specific execution, the multiple second elements can be reused, the computing intensity is improved, the memory access overhead of the first cache is reduced, the repeated loading of the activation tensor is reduced and avoided, and no invalid data is involved in the loading and operation. The second cache is used entirely for storing valid data, and the computing power is used entirely for the operation of valid data, thereby improving the utilization rate of the first cache, the second cache and the hardware operation unit.

[0058] Figure 4A A schematic flow chart of a data processing method provided by at least one embodiment of the present disclosure is shown.

[0059] At least one embodiment of the present disclosure provides a data processing method for performing operations on an input gradient tensor and an activation tensor during backpropagation to obtain a weight gradient tensor. The input gradient tensor is, for example, the gradient tensor Gradient described above. During backpropagation, this gradient tensor Gradient serves as the input to the first layer and is therefore referred to as the input gradient tensor.

[0060] For example, the activation tensor is represented as [N, H, W, C], where N represents the batch size, W represents the width, H represents the height, and C represents the number of input channels. The weight gradient tensor is represented as [R, S, K, C]. The structure of the weight gradient tensor is the same as that of the convolution kernel used in forward propagation. R represents the weight gradient tensor and the height of the convolution kernel, S represents the weight gradient tensor and the width of the convolution kernel, K represents the number of output channels, and C represents the number of input channels.

[0061] For example, in one example, the shape of the input gradient tensor Gradient is [N, P, Q, K] = [2, 9, 14, 256], where P and Q represent the height and width of the input gradient tensor Gradient, and K represents the number of output channels. The shape of the activation tensor Activation is [N, H, W, C] = [2, 9, 14, 512], where H and W represent the height and width of the activation tensor, respectively, and C represents the number of input channels. The shape of the weight gradient tensor Weight is [R, S, K, C] = [3, 3, 256, 512], where R and S represent the height and width of the weight gradient tensor, respectively. The stride size (stride=(1,1) represents the sliding stride of the convolution kernel in the height and width dimensions during forward propagation, respectively. The dilation coefficient (dilation=(1,1)) represents the dilation coefficient of the convolution kernel in the height and width dimensions, respectively. The padding size (padding=(1,1)) represents the padding size of the input tensor in the height and width dimensions, respectively. Unless otherwise specified, the following description uses the parameters in this embodiment as an example to illustrate the embodiments of the present disclosure.

[0062] Figure 4B A schematic diagram of an activation tensor.

[0063] like Figure 4B As shown, the activation tensor shows a batch, but the input gradient tensor for the actual cross-correlation operation may have multiple batches, and the dimensions of multiple batches are N dimensions. The structure of each batch is the same as Figure 4B Same as shown.

[0064] like Figure 4B As shown, a batch consists of 9×14×8 elements, H=9, W=14, C=8, Figure 4B In the example, C=8 is used as an example for convenience.

[0065] For example, in Figure 4B The H, W, and C dimensions are shown in Figure 1. Assume that the element pointed by the arrow is the origin, C, W, and H represent the three coordinate axes, row represents the coordinate value in the height direction (also the column direction), col represents the coordinate value in the width direction (also the row direction), and c represents the coordinate value in the channel direction (also the depth direction). The coordinates of the element pointed by the arrow in the input data are: c = 0, row = 0, and col = 0. The channel, width, and height values in the coordinates of other elements increase in the direction of the arrow.

[0066] It should be noted that Figure 4B This is just an example of the activation tensor, and does not mean that in actual operation, the structure of the activation tensor is Figure 4B In practice, H, W, and C can be any integers, such as C=512 or C=256.

[0067] The structure of the input gradient tensor [N, P, Q, K] is similar to the activation tensor, for example, P = H = 9, Q = W = 14, and K can be equal to or different from C, for example, K = 256, C = 512.

[0068] Figure 4C A schematic diagram of a weight gradient tensor.

[0069] like Figure 4C As shown in the back propagation operation of the convolutional neural network, the weight gradient tensor shows a batch. Similarly, the weight gradient tensor for the actual cross-correlation operation may have multiple batches, and the dimensions of multiple batches are K dimensions. The structure of each batch is the same as Figure 4C The structure of the weight gradient tensor is the same as that of the convolution kernel during forward propagation.

[0070] like Figure 4C As shown in the figure, a batch consists of 3×3×8 elements, R=3, S=3, C=8, and the two-dimensional matrix formed by the R dimension and the S dimension is 3×3. Figure 4C A small cube in a batch represents an element. For example, an element can have a data width of 8 bits, 16 bits, etc.

[0071] For example, in Figure 4C The R, S, and C dimensions are shown in Figure 1. Similarly, assuming the element pointed by the arrow is the origin, R, S, and C represent the three coordinate axes, row represents the height coordinate value in the S direction, col represents the height coordinate value in the R direction, and c represents the input channel coordinate value in the C direction. The coordinates of the element pointed by the arrow in the input data are: c = 0, row = 0, and col = 0. The channel, width, and height values in the coordinates of other elements increase in the direction of the arrow.

[0072] like Figure 4A As shown, the data processing method includes steps S10 and S20.

[0073] Step S10: Execute the first operation multiple times to obtain multiple weight gradient sub-tensors. The weight gradient tensor includes a height direction and a width direction. Each weight gradient sub-tensor is a part divided according to a 1×n granularity within the two-dimensional matrix formed by the gradient tensor in the height direction and the width direction, where n is an integer greater than 1.

[0074] Step S20: Arrange multiple weight gradient sub-tensors to obtain a weight gradient tensor.

[0075] For step S10, the structure of the weight gradient tensor is consistent with the structure of the convolution kernel during forward propagation, so the structure of the weight gradient tensor can be known during backward propagation. For example, the structure of the weight gradient tensor is Figure 4C As shown. During back propagation, the weight gradient tensor is transformed in the height direction ( Figure 4C R direction) and width direction ( Figure 4C The two-dimensional matrix (in the S direction) is divided into multiple weight gradient sub-tensors according to the 1×n granularity. For example, n can be the number of pixels of the weight gradient tensor in the S direction, or less than the number of pixels S in the S direction, and the number of pixels S in the width direction is an integer multiple of a. If a is the number of pixels in the width direction of the weight gradient tensor, the activation tensor can be reused to the greatest extent. For example, in Figure 4C In the weight gradient tensor shown, the weight gradient tensor has 3 pixels in the S direction. The two-dimensional matrix formed by the weight gradient tensor in the R and S directions can be divided into 1×3 granularity. It should be noted that in this disclosure, a direction and the size of the tensor in that direction are marked with the same identifier. For example, S indicates both the S direction and the size of the tensor in the S direction.

[0076] Figure 4D It shows that at least one embodiment of the present disclosure provides Figure 4C The weight gradient tensor shown is divided into multiple weight gradient sub-tensors according to the granularity of 1×3.

[0077] like Figure 4D As shown, Figure 4C The two-dimensional matrix formed by the weight gradient tensor in the R direction and the S direction is divided into three weight gradient sub-tensors according to the granularity of 1×3, namely weight gradient sub-tensor 401, weight gradient sub-tensor 402 and weight gradient sub-tensor 403, and their sizes are all [1, 3, 8].

[0078] In other examples, for example, in the S direction, the weight gradient tensor has 9 pixels, then a can be equal to 3. That is, it is divided into 3 parts in the R direction according to the granularity 1 and in the S direction according to the granularity 3.

[0079] Each execution of the first operation is used to obtain a weight gradient sub-tensor. For example, weight gradient sub-tensor 401, weight gradient sub-tensor 402 and weight gradient sub-tensor 403 are obtained respectively through three first operations.

[0080] For step S20, for example, how the weight gradient tensor is divided, then according to the operations corresponding to the division, multiple weight gradient sub-tensors are combined into one weight gradient tensor. For example, if the weight gradient tensor is divided as 1×S, where S represents the size in the width direction (i.e., sliced in the height direction with a granularity of 1 pixel in the height direction), then multiple weight gradient sub-tensors are arranged in sequence in the height direction to obtain the weight gradient tensor. For example, after obtaining weight gradient sub-tensor 401, weight gradient sub-tensor 402, and weight gradient sub-tensor 403, weight gradient sub-tensor 401, weight gradient sub-tensor 402, and weight gradient sub-tensor 403 are arranged in sequence in the R direction to obtain Figure 4C the weight gradient tensor shown.

[0081] For another example, if the weight gradient tensor is divided as 1×n, where n < S and S represents the size in the width direction (i.e., sliced in the height direction with a granularity of 1 pixel and sliced in the width direction with a granularity of n pixels), then multiple weight gradient sub-tensors are arranged in sequence in the width direction and the height direction.

[0082] In some embodiments of the present disclosure, n is the size of the weight gradient tensor in the width direction. According to the division method of 1×n granularity, multiple weight gradient sub-tensors are arranged to obtain the weight gradient tensor, including: arranging n weight gradient sub-tensors in sequence in the height direction to obtain an n×n weight gradient tensor.

[0083] Each time the first operation is executed, a target sub-tensor in multiple weight gradient sub-tensors is obtained.

[0084] Figure 5A shows a schematic diagram of obtaining a target sub-tensor by performing a first operation provided in at least one embodiment of the present disclosure.

[0085] As Figure 5A shown, this first operation includes steps S501 to S503.

[0086] Step S501: Load multiple first elements of the input gradient tensor and multiple second elements of the activation tensor from the first cache to the second cache. The multiple second elements are multiple elements in the activation tensor corresponding to the target sub-tensor.

[0087] Step S502: Read multiple first elements from the second cache to the tensor calculation module.

[0088] Step S503: The tensor calculation module reuses multiple second elements to perform n second operations to obtain the target sub-tensor. Each second operation is used to obtain a target element in the target sub-tensor.

[0089] For step S501, the first cache is, for example, Figure 3The global cache or memory shown. For example, if the data processing method is applied to a GPU, the first cache can be the GPU's video memory. The second cache is, for example, Figure 3 The shared memory shown is shown. In the following, the embodiments of the present disclosure are described by taking the first cache as a global cache and the second cache as a shared memory as an example.

[0090] For example, when performing cross-correlation or convolution operations, if the input gradient tensor and activation tensor are large, the large image can be divided into multiple smaller blocks, each of which is called a "tile." Each tile contains a portion of the elements of a tensor (also called data), which can be processed in independent processing units. In step S501, the first elements of the input gradient tensor can refer to a tile.

[0091] For example, the input gradient tensor can be represented as [K, NPQ]. Here, NPQ refers to the input gradient tensor [N, P, Q, K] being expanded into one dimension along the batch, height, and width dimensions. Specifically, the input gradient tensor is expanded along the width dimension, then the height dimension, and finally the batch dimension. The expanded output tensor can be represented as K × NPQ (pixel), where NPQ (pixel) represents the one-dimensional expansion along the P, Q, and N dimensions. The multiple first elements of the input gradient tensor are one of multiple input gradient sub-tensors obtained by partitioning the input gradient tensor in the PQ plane according to the granularity [m, n']. That is, each input gradient sub-tensor includes these multiple first elements. Each input gradient sub-tensor is considered a "tile." The specific values of m and n' can be determined based on practical needs, such as the tile size supported by the processor, the processor's processing speed, load balancing, data type, and so on. For example, in one embodiment, m = 32 and n' = 64, although this disclosure is not limited to this.

[0092] Figure 5B A schematic diagram of an input gradient sub-tensor obtained by dividing the input gradient tensor according to the [m,n'] granularity is shown.

[0093] like Figure 5B As shown in the figure, the shaded part is the input gradient sub-tensor obtained by dividing the input gradient tensor into [1, 64]. Among them, m=1 is shown only for the convenience of reading. In actual applications, m can usually be equal to 32, 64, etc. If m=32, there are 32 rows in the K direction, and each row includes Figure 5B 64 elements shown.

[0094] In step S501 , a plurality of second elements in the activation tensor are used to operate with a tile of the input gradient tensor (ie, the input gradient sub-tensor) to obtain elements required for n target elements included in the target sub-tensor.

[0095] Figure 5C A schematic diagram of multiple second elements in an activation tensor provided by at least one embodiment of the present disclosure is shown.

[0096] like Figure 5C As shown, the 66 elements shown in the shaded part are used as multiple second elements. The 66 elements are Figure 5B The 64 elements shown are subjected to the second operation described below to obtain, for example Figure 4D A weight gradient sub-tensor 401 is shown, which is an example of a target sub-tensor.

[0097] In step S501, for example, a Tensor Memory Access (TMA) instruction is used to load multiple first elements of the input gradient tensor and multiple second elements of the activation tensor from the global cache into shared memory. The TMA instruction is a hardware-based asynchronous data handling unit designed for efficient processing of tensor data access. It achieves parallelism between data loading and computation through hardware-level optimization, significantly improving the efficiency of deep learning and high-performance computing.

[0098] For example, a TMA instruction is used to load multiple first elements of the input gradient tensor from the GPGPU video memory to the shared memory, and multiple second elements of the activation tensor corresponding to the input gradient sub-tensor are loaded from the GPGPU video memory to the shared memory.

[0099] Step S501 includes: loading multiple elements of the input gradient tensor from the first cache to the second cache as multiple first elements according to the first shape size; determining a second shape size of the multiple second elements based on the first shape size and n of at least a portion of the tensor; determining a starting position of the multiple second elements; and loading multiple elements of the activation tensor from the first cache to the second cache as the multiple second elements according to the second shape size and the starting position.

[0100] For example, the input gradient tensor is represented as [N, P, Q, K], where P represents the height, Q represents the width, N represents the batch size, K represents the number of output channels, and the first shape size is [K1, D], where K1 represents the size of multiple first elements (i.e., the input gradient sub-tensor) in the output channel number dimension, and D represents the size of multiple first elements in the first channel dimension. The first channel dimension is the input gradient tensor expanded into one dimension in the height dimension, width dimension, and batch number dimension, and K1 is less than or equal to K.

[0101] For example, the input gradient tensor Gradiernt is represented as [N, P, Q, K] = [2, 9, 14, 256]. The input gradient tensor Gradiernt is expanded into one dimension along the height, width, and batch size dimensions. This dimension serves as the first channel dimension, and the output channel K serves as the other dimension. That is, the input gradient tensor Gradiernt can be represented as a matrix containing the first channel dimension and the output channel dimension. For example, the first channel dimension serves as the matrix column, and the output channel dimension serves as the matrix row. This matrix is represented as Gradiernt[256, 252]. This can be understood as sequentially arranging the K output channels of the input gradient tensor according to the NPQ arrangement. For example, the PQ pixels of one output channel in the first batch are arranged first, followed by the PQ pixels of that output channel in the second batch; then the PQ pixels of another output channel in the first batch are arranged; then the PQ pixels of that output channel in the second batch are arranged; and so on, until the pixels of the Kth channel in the second batch are arranged.

[0102] The first shape size of the multiple first elements is, for example, [K1, D], that is, the matrix Gradient [256, 252] is divided into multiple input gradient sub-tensors with a granularity of [K1, D], for example, K1=32, D=64. When loading multiple first elements from the global cache, they are loaded at a granularity of K1=32, D=64. For example, if P=9 and Q=14, each time the TMA instruction is executed, 32 channels of the input gradient tensor are loaded from the global cache, and each channel includes 4 rows (4×14=56) plus 8 pixels (a total of 64 pixels) in the PQ plane. For example, each channel includes Figure 5B The 64 elements shown in the shaded area. K1 can be any integer less than or equal to K. K1 can be determined based on the processing speed of the processor, load balancing, data type, etc.

[0103] The activation tensor is represented as [N, H, W, C], where W represents width, H represents height, and C represents the number of input channels. The second shape size is [D+n-1, C1], where D+n-1 represents the size of the activation tensor in the second channel dimension. The second channel dimension is the activation tensor in the height, width, and batch size dimensions expanded into one dimension. C1 represents the size of the activation tensor in the input channel dimension, where C1 is less than or equal to C. Similar to the input gradient tensor described above, the activation tensor can be expanded into one dimension in the height, width, and batch size dimensions to obtain the second channel dimension. The method for expanding the activation tensor to obtain the second channel dimension is similar to that of the input gradient tensor described above and is not repeated here.

[0104] For example, if D=64 and n=3, the second shape size is [66, C1], where C1 can be any integer less than or equal to C. C1 can be determined based on the processor's processing speed, load balancing, data type, etc. For example, each time a TMA instruction is executed, C1 (e.g., 64, 128, etc.) channels of the activation tensor are loaded from memory, and each channel includes 4 rows (4×14=56) plus 10 pixels (a total of 66 pixels) in the HW plane. For example, each channel includes Figure 5C The 66 elements are shown in the shaded areas.

[0105] Determine the starting position of the multiple second elements corresponding to the weight gradient sub-tensor 401, and continuously load D+n-1 elements in the second channel dimension of the activation tensor from the starting position. It should be noted that since each element in the weight gradient sub-tensor 401 needs to be obtained through a cross-correlation operation, it is necessary to load D+n-1 elements for each input channel dimension to obtain all elements in the weight gradient sub-tensor 401. For the method of determining the starting position of multiple second elements, please refer to the following Figure 6 Description.

[0106] Figure 5D A schematic diagram of loading multiple first elements and multiple second elements from a global cache into a shared memory provided by at least one embodiment of the present disclosure is shown.

[0107] like Figure 5D As shown in the figure, in the shared memory, the input gradient tensor is stored as input gradient sub-tensors such as A00, A01, and A02, and the activation tensor is stored as data blocks such as B00, B01, and B02.

[0108] In the process of loading multiple first elements from the global cache to the shared memory, the input gradient tensor is regarded as a matrix [K, NPQ], where the output channel dimension of the matrix [K, NPQ] includes, for example, 256 rows (K0-K255) and the first channel dimension includes, for example, 252 columns (P0-P251). It is divided into multiple input gradient sub-tensors according to the granularity of [32, 64]. For example, each TMA instruction is used to load the input gradient tensor of the granularity of [32, 64] from the global cache to the shared memory. Figure 5D As shown in the figure, the input gradient tensor is divided into input gradient sub-tensors such as A00, A01, and A02. The input gradient sub-tensors such as A00, A01, and A02 are loaded into the shared memory in sequence.

[0109] In the process of loading multiple second elements from the global cache to the shared memory, the activation tensor is regarded as a matrix [NHW, C], where the second channel dimension of the matrix [NHW, C] includes, for example, 252 rows (H0-H251) and the input channel dimension direction includes, for example, 512 columns (R0-R511). The multiple second elements include, for example, the elements of size [66, 128] in the activation tensor. For example, each TMA instruction is used to load multiple second elements of size [66, 128] from the global cache to the shared memory. Figure 5D As shown in the figure, the activation tensor is divided into data blocks such as B00, B01, and B02, and each TMA instruction is used to load a data block. It should be noted that if the activation tensor is loaded according to the size [66,128], the sequence number of data block B00 in the second channel dimension is H0 to H65, the sequence number of data block B01 in the second channel dimension is H64 to H129, and the sequence number of data block B02 in the second channel dimension is H128 to H193.

[0110] For example, use the TMA instruction to load the input gradient sub-tensor A00 into the shared cache, then use the TMA instruction to load the data block B00 into the shared cache, and then execute steps S502 and S503 for the input gradient sub-tensor A00 and data block B00. Similarly, use the TMA instruction to load the input gradient sub-tensor A01 into the shared memory, then use the TMA instruction to load the data block B01 into the shared memory, and then execute steps S502 and S503 for A01 and B01. This process continues in this way until all tiles of the input gradient tensor have executed steps S502 and S503.

[0111] It should be noted that in some other embodiments of the present disclosure, if the tile size supported by the processor is greater than or equal to the size of the input gradient tensor, the input gradient tensor may not be partitioned according to the partitioning granularity. That is, all elements in the input gradient tensor may be loaded into the shared cache simultaneously, in which case the multiple first elements are all elements in the input gradient tensor.

[0112] In step S502, a plurality of first elements are read from the second cache to the tensor calculation module. For example, A00 is read from the second cache to the tensor calculation module.

[0113] In step S503, since the weight gradient tensor is divided into multiple weight gradient sub-tensors at a 1×n granularity, each weight gradient sub-tensor includes 1×n elements in a two-dimensional matrix consisting of height and width. When calculating the n elements in the weight gradient sub-tensor, the tensor calculation module performs the second operation n times on the multiple second elements in the shared cache, that is, the n second operations reuse the multiple second elements in the shared cache.

[0114] The second operation includes: reading the plurality of second elements to obtain the plurality of third elements required for the second operation; and determining a target element based on the plurality of first elements and the plurality of third elements.

[0115] In some embodiments of the present disclosure, the plurality of first elements are elements in one of a plurality of input gradient sub-tensors obtained by partitioning the input gradient tensor according to a partitioning granularity, and the plurality of first elements correspond one-to-one to the plurality of third elements. Determining a target element based on the plurality of first elements and the plurality of third elements includes: calculating the sum of products of a plurality of elements corresponding to the plurality of first elements and the plurality of third elements; and adding the sums of the plurality of products corresponding to the plurality of input gradient sub-tensors to obtain the target element.

[0116] For example, in Figure 5D In the example, after loading the input gradient sub-tensor A00 and data block B00 into the shared cache, the TMA instruction is used to load the input gradient sub-tensor A00 from the shared cache into the tensor computation module. Furthermore, the TMA instruction is used to load multiple third elements of data block B00, numbered H0 to H63, along the second channel dimension, from the shared cache. After loading both the input gradient sub-tensor A00 and the multiple third elements into the tensor computation module, the tensor computation module performs dot products on the corresponding elements between the multiple first elements and the multiple third elements, and calculates the sum of the products of the dot products, which forms part of W00. Subsequently, multiple third elements, numbered H1 to H64, along the second channel dimension of data block B00 are loaded from the shared cache. The sum of the products of the input gradient sub-tensor A00 and the multiple third elements corresponding to these elements, H1 to H64, are then calculated, which forms part of W01. Then, multiple third elements corresponding to the second channel dimension of the data block B00 with serial numbers H2~H65 are loaded from the shared cache, and then the sum of the products of the input gradient sub-tensor A00 and the multiple third elements corresponding to H2~H65 are calculated. The sum of the products is part of W02.

[0117] Afterwards, similar first and second operations are performed on the input gradient sub-tensor A01 and data block B01 according to the steps of the first and second operations on the input gradient sub-tensor A00 and data block B00 .

[0118] In some embodiments of the present disclosure, since the input gradient tensor is divided into multiple input gradient sub-tensors, the sum of the products corresponding to the multiple input gradient sub-tensors is added to obtain the target element. For example, the sum of the multiplications and additions obtained by multiple tiles is accumulated again to obtain the target element. For example, A00*B00 + A01*B01 + … is calculated, and the output of the accumulation result is the target element W00, the target element W01, and the target element W02. The data in B00, B01, … used when calculating the target element W00, the target element W01, and the target element W02 are different. For example, the elements with sequence numbers 0-63 in B00, B01, … are used to calculate W00; the elements with sequence numbers 1-64 in B00, B01, … are used to calculate W01; and the elements with sequence numbers 2-65 in B00, B01, … are used to calculate W02. It should be noted that “*” represents the sum of the products of the corresponding elements. For example, according to Figure 5D Similarly, by calculating A00*B10 + A01*B11 + …, the accumulated results are output as target elements W10, W11, and W12.

[0119] In some other embodiments of the present disclosure, in an embodiment where the input gradient tensor is not divided according to the division granularity, for example, the size of another input gradient tensor is the same as Figure 5D The input gradient subtensor A00 in is the same as that in , so the size of the loaded activation tensor is, for example, Figure 5D The data block B00 is transferred to the global memory, and then the second operation is performed three times on the data block B00 to obtain A00*B00. The value of A00*B00 is the target element W00 in the weight gradient sub-tensor.

[0120] Figure 6 A flow chart of a method for determining starting positions of multiple second elements provided by at least one embodiment of the present disclosure is shown.

[0121] like Figure 6 As shown, the method includes steps S601 to S603.

[0122] Step S601: Get the filling parameters of the activation tensor.

[0123] Step S602: Determine the minimum index value of the weight gradient sub-tensor in the height direction and the width direction.

[0124] Step S603: Determine the starting position based on the minimum index value and the filling parameter.

[0125] For step S601, the padding parameter refers to, for example, the padding size, which can be the same as the padding size of the input tensor during forward propagation. The padding size is, for example, 1 row of pixels each in the front, back, left, and right directions. Figure 1As shown, the fill parameter is (1,1).

[0126] For step S602, for example, the first element in the weight gradient sub-tensor is the minimum index in the height direction and the width direction. Figure 4D In the example, the first element of the weight gradient sub-tensor 401 is W00, and the index of element W00 in the height direction and width direction is (0, 0); the first element of the weight gradient sub-tensor 402 is W10, and the index of element W10 in the height direction and width direction is (1, 0); the first element of the weight gradient sub-tensor 403 is W20, and the index of element W20 in the height direction and width direction is (2, 0).

[0127] For step S603, for example, the weight gradient tensor is expressed as [R, S, K, C], R represents the height, S represents the width, the index is [R1, S1, K2, C2], K2 is the output channel where the first element is located, and the filling parameter is (f, g). Step S603 includes: determining the starting coordinates of multiple second elements as [N1, R1-f, S1-g, C2], N1, f, g, R1, S1, C2 are all integers greater than or equal to 0, N1 is the batch where the element corresponding to the minimum index value (for example, the first element mentioned above) is located, and C2 is the input channel where the element corresponding to the minimum index value is located; and transforming the starting coordinates into the valid range of the activation tensor.

[0128] For example, if the padding parameter is (1, 1), the minimum index value in the height direction and width direction is [0, 0]. For any input channel number dimension and output channel number dimension, the coordinate of the target element with the minimum index value is marked as [0, 0, K2, C2]. Then the starting coordinates of the multiple second elements are [N1, -1, -1, C2]. N1 is an arbitrary batch, and C2 is an arbitrary channel. That is, for the element with index 00 in each channel in each batch (for example, Figure 4C For element W00 in each batch, the starting coordinates of the second elements in the activation tensor required for element W00 are calculated to be [-1,-1] in both the height and width directions. Similarly, for the element with index (1,0) in each channel in each batch, the starting coordinates of the second elements in the activation tensor required for W10 are calculated to be [0,-1] in both the height and width directions. Based on [N1, R1-f, S1-g, C2], the starting coordinates of the second elements in the activation tensor corresponding to the element with index (0,0), the element with index (1,0), and the element with index (2,0) in each channel in each batch can be calculated. I will not go into details here.

[0129] Since the activation tensor needs to be padded around when performing cross-correlation or convolution operations and converting them into matrix multiplication and addition operations, the starting coordinates of multiple second elements are obtained based on the index minus the padding parameter. Figure 4B If the coordinate system of the activation tensor is established in the manner shown, the starting coordinates calculated in this way may not be within the range of the activation tensor. For example, elements with negative H or W values are not within the range of the activation tensor. For example, the starting coordinates of multiple second elements in the activation tensor required by element W00 in the height and width directions are [-1,-1], [-1,-1] with H=-1 and W=-1, which is not within the range of the activation tensor. Therefore, a coordinate transformation is required to transform it to within the range of the activation tensor.

[0130] For example, transforming the starting coordinates into the valid range of the activation tensor includes: converting the coordinate values of the starting coordinates in the width dimension and the coordinate values in the height dimension into coordinate values greater than or equal to 0 in the order of width dimension and height dimension.

[0131] For example, determine whether S1-g is less than 0; in response to S1-g<0, transform the starting coordinates [N1, R1-f, S1-g, C2] into [N1, R1-f-t1, S1-g+t1×W, C2], so that S1-g+t1×W>=0, where t1=|S1-g| / W is rounded up; and in response to R1-f-t1<0, further transform [N1, R1-f-1, S1-g+ t1×W, C2] into [N1-t2, R1-f-t1+t2×H, S1-g+t1×W, C2], so that R1-f-t1+t2×H>=0, where t2=|R1-f-t1| / H is rounded up.

[0132] For another example, in response to S1-g>=0, it is determined whether R1-f is less than 0. In response to S1-g>=0 and R1-f<0, the starting coordinates [N1, R1-f, S1-g, C2] are transformed to [N1-t3, R1-f+ t3×H, S1-g, C2], such that R1-f+t3×H>=0, where t3=|R1-f| / H is rounded up.

[0133] For example, execute in the following order.

[0134] Step S1: Determine whether S1-g is less than 0.

[0135] Step S2: If S1-g < 0, transform the starting coordinates [N1, R1-f, S1-g, C2] to [N1, R1-f - t1, S1-g + t1 × W, C2], such that S1-g + t1 × W > 0, where t1 = |S1-g| / W (rounded up). If S1-g > 0, jump to step S5.

[0136] Step S3: Determine whether R1-f-t1 is less than 0.

[0137] Step S4: If R1-f-t1 < 0, further transform [N1, R1-f-t1, S1-g + t1×W, C2] to [N1-t2, R1-f-t1+t2×H, S1-g + t1×W, C2], such that R1-f – t1+t2×H >= 0, where t2 = |R1-f-t1| / H (rounded up). The starting position is [N1-t2, R1-f-t1+t2×H, S1-g + t1×W, C2].

[0138] If R1-f-t1>=0, the starting position is [N1, R1-f-t1, S1-g+ t1×W, C2].

[0139] Step S5: Determine whether R1-f is less than 0.

[0140] Step S6: If R1-f<0, transform the starting coordinates [N1, R1-f, S1-g, C2] to [N1-t3, R1-f + t3×H, S1-g, C2], so that R1-f + t3×H >= 0, t3 = |R1-f| / H rounded up, and the starting position is [N1-t3, R1-f + t3×H, S1-g, C2].

[0141] If R1-f>=0, the starting position is [N1, R1-f, S1-g, C2]. After determining the second shape size and starting position, multiple elements of the activation tensor of the second shape size are loaded into the shared memory starting from the starting position as multiple second elements.

[0142] In the above embodiment of loading multiple second elements into the shared memory, no invalid data is involved in the loading and calculation, the shared memory is entirely used to store valid data, and the computing power is entirely used to calculate valid data, thereby improving the utilization of the shared memory and the hardware computing unit.

[0143] Figure 7 A schematic diagram of loading multiple second elements in an activation tensor provided by at least one embodiment of the present disclosure is shown.

[0144] like Figure 7For ease of understanding, Figure 7 Only the last elements of batch1 and batch0 are shown; the activation tensor can also include more batches. Figure 7 The activation tensor is shown as flat, showing the NHW dimensions, but actually has C elements per box position in the depth direction. Figure 7 In p(i,j,q), i represents the batch number, j represents the coordinate in the height direction H, and q represents the coordinate in the width direction W. For example, p(0,8,0) represents the element in the 8th row and 0th column of batch 0; p(1,0,0) represents the element in the 0th row and 0th column of batch 1. Each box contains C elements in the depth direction. For example, Figure 7 Each block shown in may be referred to as a sequence unit, and each sequence unit includes C elements in a depth direction (C direction).

[0145] For example, to calculate batch1 Figure 4D The weight gradient sub-tensor 401, the element W00 corresponding to the minimum index value of the weight gradient sub-tensor 401 has a starting coordinate Z of [-1, -1] in the height direction and width direction. The starting coordinate is the offset relative to the coordinate of 0 in the height coordinate and 0 in the width coordinate of the activation tensor. For example, [-1, -1] is relative to Figure 7 In the example, p(1,0,0) is offset by 1 pixel upward in height and 1 pixel left in width. The starting coordinates are then transformed to fit within the activation tensor. The starting coordinates Z[1, -1, -1, C2] are first transformed to [1, -2, 13, C2], and then [1, -2, 13, C2] is transformed to [0, 7, 13, C2]. That is, the starting coordinates Z are transformed to p(0,7,13). Sixty-six elements are sequentially loaded from p(0,7,13) into shared memory as multiple second elements. C2 is the coordinate value of the input channel dimension. If the processor supports a granularity of C1 on the input channel, then C1 input channel dimensions are loaded starting from coordinate value C2. That is, for the activation tensor, a data block of shape [66, C1] is loaded into the tensor computation module starting from the starting position p(0,7,13). For example, load p(0,7,13), p(0,8,0), p(0,8,1), ..., p(1,3,8) into the tensor computation module in sequence.

[0146] For example, to calculate batch1 Figure 4DThe weight gradient sub-tensor 402 shown in FIG, then the element W10 corresponding to the minimum index value of the weight gradient sub-tensor 402 has a starting coordinate of [0, -1] in the height direction and width direction. The starting coordinate is the offset relative to the coordinate of 0 in the height coordinate and 0 in the width coordinate of the activation tensor. For example, [0, 1] is relative to Figure 7 In the example, p(1,0,0) is offset 1 pixel to the left in the width direction. The starting coordinates [0, -1] are then converted to the range of the activation tensor. The starting coordinates [1, 0, -1, C2] are transformed to [0, 8, 13, C2]. For one depth, 66 elements are loaded sequentially from p(0,8,13) as multiple second elements into shared memory. For example, p(0,8,13), p(1,0,0), ..., p(1,4,9) are loaded sequentially into the tensor computation module.

[0147] For example, Figure 4D The calculation method of the starting position of the weight gradient sub-tensor 403 shown in is similar to that of the weight gradient sub-tensor 401 and the weight gradient sub-tensor 402, and will not be repeated here.

[0148] Figure 8 A flow chart of a method for reading a plurality of second elements to obtain a plurality of third elements required for a second operation, provided by at least one embodiment of the present disclosure, is shown.

[0149] like Figure 8 As shown, the method includes steps S801-S802.

[0150] Step S801: Determine positions of a plurality of third elements in a plurality of second elements, where the shape and size of the plurality of third elements are [D, C1], where D is the size in the second channel dimension.

[0151] Step S802: obtaining a plurality of third elements based on data blocks of a shape and size [D, C1] at positions in the plurality of second elements.

[0152] In some embodiments of the present disclosure, determining the positions of multiple third elements in multiple second elements includes: determining the starting point of the multiple third elements according to the index of the target element in the height direction and the width direction in the weight gradient tensor, and loading the data blocks in the multiple second elements with a shape size of [D, C1] from the starting point. The method of determining the starting point of the multiple third elements according to the index of the target element in the height direction and the width direction in the weight gradient tensor is similar to the method of determining the starting position of the first element in the weight gradient sub-tensor mentioned above. For example, the index of the starting point is [R1, S1, K2, C2], and the filling parameter is (f, g), then the starting coordinates of the starting point in the activation tensor are [N1, R1-f, S1-g, C2], and then the coordinate transformation is performed. For example, determine whether S1-g is less than 0; in response to S1-g<0, transform the starting coordinates [N1, R1-f, S1-g, C2] into [N1, R1-f-t1, S1-g+t1×W, C2], so that S1-g+t1×W>=0, where t1=|S1-g| / W is rounded up, and in response to R1-f-t1<0, further transform [N1, R1-f-t1, S1-g+t1×W, C2] into [N1-t2, R1-f-t1+t2×H, S1-g+t1×W, C2], so that R1-f-t1+t2×H>=0, where t2=|R1-f-t1| / H is rounded up; in response to S1-g>=0, it is determined whether R1-f is less than 0; in response to S1-g>=0 and R1-f<0, the starting coordinates [N1, R1-f, S1-g, C2] are transformed to [N1-t3, R1-f+ t3×H, S1-g, C2], so that R1-f+t3×H>=0, where t3=|R1-f| / H is rounded up.

[0153] For example, in Figure 7 In the example, to calculate batch 1 Figure 4C The element W01 with index (0, 1) shown above has a starting coordinate of [-1,0] in the height and width directions, which is an offset from the coordinates in the activation tensor where the height coordinate is 0 and the width coordinate is 0. For example, [-1,0] is relative to Figure 7 In the example, p(1,0,0) is offset upward by 1 pixel in the height direction. The starting coordinate [-1,0] is then converted to the activation tensor's range. The starting coordinate [1, -1, 0, C2] is transformed to [0, 8, 0, C2]. For one depth, 64 elements are loaded sequentially from p(0,8,0) into the tensor computation module. For example, p(0,8,0), p(0,8,1), ..., p(1,3,7) are loaded into the tensor computation module.

[0154] For example, to calculate batch1 Figure 4C For element W02 with index (0, 2), the starting coordinates in the height and width directions are [-1, 1]. For a depth of 1, the starting coordinates [1, -1, 1] are transformed to [0, 8, 1]. 64 elements are sequentially loaded from p(0, 8, 1) as multiple third elements into the tensor computation module. For example, p(0, 8, 0), p(0, 8, 1), ..., p(1, 3, 8) are sequentially loaded into the tensor computation module.

[0155] In another embodiment of the present disclosure, a data block with a shape size of [D, C1] can also be loaded into the tensor calculation module based on the starting position and sliding step size of the first element of the weight gradient sub-tensor. For example, if the sliding step size is 1, starting from the starting position, each time the second operation is performed, a 1-pixel size is slid to the right to obtain a data block with a shape size of [D, C1] loaded into the tensor calculation module.

[0156] In some embodiments of the present disclosure, multiple third elements are obtained based on a data block with a shape size of [D, C1] at a position in multiple second elements, including: determining the filling position in the data block for filling according to the minimum index value and the filling parameter; and masking the elements at the filling position to obtain multiple third elements.

[0157] When the tensor calculation module reads multiple second elements, the element value at the filling position is modified to the default value according to the index information and the filling parameter and then loaded into the tensor calculation module. The filling position is indicated by the index and the filling parameter. For example, the filling position refers to the offset of the position to be masked relative to the index of the target element. For example, offset = index - filling parameter. For example, for Figure 4C For element W00 in , the offset = (0,0)-(1,1) = (-1,-1), so the filling position is the first column on the left and the first row above the data block. For example, for Figure 4C For element W01 in the data block, offset = (0,1) - (1,1) = (-1,0), so the padding position is the first column from the left in the data block. If offset = (0,1), the padding position is the last column from the right in the data block. If offset = (1,0), the padding position is the last row below in the data block.

[0158] Thus, the data at the padding position of the original convolution is processed to a default value (for example, the default value is 0) through a mask operation, and then loaded into the tensor calculation module as multiple third elements. That is, the data block after setting the elements at the multiple padding positions in the data block to the default value of 0 is used as multiple third elements. In this way, the padding process can be simulated during the loading process, and the loading of the activated tensor can still be performed through the automatic loading process and the padding process is automatically simulated, thereby accelerating the loading process.

[0159] Figure 9A A schematic diagram of a mask provided by at least one embodiment of the present disclosure is shown.

[0160] like Figure 9A As shown, the multiple second elements in the shared memory are from P0 to P65. Figure 9A The activation tensor is shown flat, with only the NHW dimension shown.

[0161] When obtaining multiple third elements corresponding to element W00 from the shared memory, data block DA1 is loaded from P0 to P63 in the shared memory. The two-dimensional matrix formed by data block DA1 in the height direction and width direction has a total of 64 elements, and because the offset = (-1, -1), the first row and the first column of data block DA1 are masked to obtain multiple third elements corresponding to W00. The multiple third elements corresponding to element W00 are loaded into the tensor calculation module. The coordinates of P0 are, for example, Figure 7 For example, the coordinates of P(0,7,13) and P63 are Figure 7 P(1,3,6) in.

[0162] When obtaining multiple third elements for element W01 from the shared memory, data block DA2 is loaded from P1 to P64 in the shared memory. The two-dimensional matrix formed by data block DA2 in the height direction and width direction has a total of 64 elements. Since the offset = (-1, 0), the first row of data block DA2 is masked to obtain multiple third elements corresponding to element W01. The multiple third elements corresponding to element W01 are loaded into the tensor calculation module. The coordinates of P1 are, for example, Figure 7 For example, the coordinates of P(0,8,0) and P64 are Figure 7 P(1,3,7) in.

[0163] When obtaining multiple third elements for element W02 from the shared memory, data block DA3 is loaded from P2 to P65 in the shared memory. The two-dimensional matrix formed by data block DA3 in the height direction and width direction has a total of 64 elements. Since the offset = (-1, 1), the first row and the last column of data block DA3 are masked to obtain multiple third elements corresponding to W02. The multiple third elements corresponding to element W02 are loaded into the tensor calculation module. The coordinates of P2 are, for example, Figure 7 For example, the coordinates of P(0,8,1) and P65 are Figure 7 P(1,3,8) in.

[0164] It should be noted that although Figure 9A Although shown in the figure as a two-dimensional matrix, each block contains C elements in the depth direction. Each block can be called a sequence unit, and each sequence unit contains C elements in the depth direction (C direction). Therefore, loading from shared memory starting from P0 (0, 7, 13) to P63 (1, 3, 6) actually loads 64 subsequence units. The size of a subsequence unit in the depth direction can be determined based on the processor's capabilities. A subsequence unit can contain C elements or less. The index of the target element is essentially the index of each element in the sequence unit, that is, the index of elements in the same sequence unit is the same. For example, the index (0, 0) of element W00 is the common index of the C elements in the sequence unit where element W00 resides.

[0165] exist Figure 9A In the figure, the shaded part is the filling position that needs to be masked.

[0166] from Figure 9A During the calculation process, it can be found that after this embodiment splits the solution granularity of the weight gradient tensor into 1x3, multiple second elements of the activation tensor required to solve the weight gradient tensor 1x3 are shared in the same shared memory. Compared with ImplicitGemm, the memory access requests of the global cache or the secondary cache can be reduced to 1 / 3.

[0167] Figure 9B A schematic diagram of another mask provided by at least one embodiment of the present disclosure is shown.

[0168] like Figure 9B As shown, the elements in the shared memory are from P'0 to P'65.

[0169] When obtaining multiple third elements corresponding to element W10 from the shared memory, data block DA'1 is loaded from P'0 to P'63 in the shared memory. The two-dimensional matrix formed by data block DA'1 in the height direction and width direction has a total of 64 elements, and because the offset = (0, -1), the first column of data block DA'1 is masked to obtain multiple third elements corresponding to W10. The multiple third elements corresponding to element W10 are loaded into the tensor calculation module. The coordinates of P'0 are, for example, Figure 7 For example, the coordinates of P(0,8,13) in P'63 are Figure 7 P(1,4,6) in.

[0170] When obtaining multiple third elements for element W11 from the shared memory, data block DA'2 is loaded from P'1 to P'64 in the shared memory. The two-dimensional matrix formed by data block DA'2 in the height direction and width direction has a total of 64 elements, and because the offset = (0,0), data block DA'2 does not need to be masked. Data block DA'2 is the multiple third elements corresponding to element W11. The multiple third elements corresponding to element W11 are loaded into the tensor calculation module. The coordinates of P1 are, for example, Figure 7 For example, the coordinates of P(1,0,0) and P64 are Figure 7 P(1,4,7) in.

[0171] When obtaining multiple third elements for element W12 from the shared memory, data block DA'3 is loaded from P'2 to P'65 in the shared memory. The two-dimensional matrix formed by data block DA'3 in the height direction and width direction has a total of 64 elements. Since the offset = (0, 1), the last column of data block DA'3 is masked to obtain multiple third elements corresponding to W12. The multiple third elements corresponding to element W12 are loaded into the tensor calculation module. The coordinates of P'2 are, for example, Figure 7 For example, the coordinates of P(1,0,1) and P65 are Figure 7 P(1,4,8) in.

[0172] In some embodiments of the present disclosure, a producer thread warp is used to load multiple first elements of an input gradient tensor and multiple second elements of an activation tensor from a first cache to a second cache; and a consumer thread warp is used to read the multiple first elements from the second cache to a tensor computing module, and the tensor computing module reuses the multiple second elements to perform n second operations to obtain a target sub-tensor.

[0173] Producer warpgroups are mainly used for data loading, while consumer warpgroups are mainly used for data computation.

[0174] For example, the consumer thread bundle includes a first thread bundle and a second thread bundle, the first thread bundle and the second thread bundle use the tensor calculation module using the ping-pong strategy, and the first thread bundle and the second thread bundle are used to obtain different weight gradient sub-tensors. Figure 4D The weight gradient subtensor 401 in the second thread warp is used to obtain Figure 4D The weight gradient subtensor 402 in .

[0175] For example, first, the producer thread bundle uses the TMA instruction to copy the input gradient tensor from the global cache to the shared memory; at the same time, the TMA instruction is used to copy the required data of the activation tensor from the global cache to the shared memory according to the 1x3 granularity of the weight gradient tensor; then in the consumer thread bundle, the first thread bundle, the calculation tensor module reads in, for example Figure 5D For data (A00, B00) and (A01, B01), we calculate A00*B00 + A01*B01 + …, and write the accumulated result to the accumulator. After the accumulation is complete, the result is copied to shared memory. The second warp is similar. Finally, the result is written to the global cache using the TMA instruction.

[0176] It's worth noting that the primary operations of a consumer warp include calling the tensor computation module to perform backpropagation weight update calculations, then using the vector computation module to register and store the backpropagation weight update results into shared memory, performing accumulation, and other operations (referred to as "computation processing"). To fully parallelize the computational resources of the tensor computation module and the vector computation module, some embodiments of the present disclosure initiate two consumer warps, synchronizing operations so that the two consumer warps utilize the tensor computation module in a ping-pong strategy. That is, at a certain point in time, the first warp and the second warp, respectively, call the tensor computation module and the vector computation module to perform backpropagation weight update calculations and computation processing, fully leveraging the powerful computing power of the tensor computation module while minimizing the time consumed by the vector computation module.

[0177] The data processing method also includes: calling a vector calculation module to perform further calculation processing on the weight gradient sub-tensor; while the first thread warp performs the first operation, the second thread warp performs calculation processing; while the second thread warp performs the first operation, the first thread warp performs calculation processing.

[0178] From a certain point in time, the first thread warp and the second thread warp respectively call the vector calculation module and the tensor calculation module to perform cross-correlation calculation and further calculation processing, making full use of the powerful computing power of the tensor calculation module to hide the time consumption of the vector calculation unit.

[0179] Figure 10 A schematic diagram of the process of the ping-pong strategy provided by an embodiment of the present disclosure.

[0180] Figure 10 The figure shows a producer thread bundle and two consumer thread bundles, the two consumer thread bundles are consumer thread bundle 1 and consumer thread bundle 2, consumer thread bundle 1 is an example of the first thread bundle, and consumer thread bundle 2 is an example of the second thread bundle. The back propagation weight update calculation includes Figure 5A The producer warp is used to execute the steps S502 and S503 described above. Figure 5A Step S501 in the backpropagation weight update calculation requires the use of the tensor computation module to obtain the target sub-tensor [1, n]. The computational processing is to further process the target sub-tensor [1, n], such as storing in registers, biasing, adding, and other non-matrix multiplication and accumulation operations. The computational processing requires the use of the vector computation unit to perform related operations.

[0181] In order to fully parallelize the computing resources of the tensor computing unit and the vector computing unit, two consumer warps are started and synchronized so that the two consumer warps use the tensor computing unit and the vector computing unit in a ping-pong strategy.

[0182] For example, Figure 10 As shown, in response to a TMA instruction from the producer warp, warp 1 loads the first elements of the input gradient tensor and the second elements of the activation tensor required to obtain a target sub-tensor (e.g., weight gradient sub-tensor 401) into the second cache. Then, warp 1 uses the tensor computation module to perform backpropagation weight update calculations. The producer warp then proceeds to execute a TMA instruction for warp 2, loading the first elements of the input gradient tensor and the second elements of the activation tensor required to obtain another target sub-tensor (e.g., weight gradient sub-tensor 402) into the second cache. Warp 2 then uses the tensor computation module to perform backpropagation weight update calculations. While warp 2 performs backpropagation weight update calculations, warp 1 performs computations. The subsequent process is similar. At a certain point in time, warp 1 and warp 2 each call the tensor computation module and the vector computation module to perform backpropagation weight update calculations and computations. This leverages the powerful computing power of the tensor computation unit to hide the time consumption of the vector computation unit, reducing time consumption and improving computational efficiency.

[0183] GPGPU can be divided into multiple computing units. Using the idea of block division, the weight gradient tensor is evenly divided into (TileM, TileN) granularity to each CU for computing tasks. For example, the granularity is TileM=32, TileN=64. Each CU is used to obtain the element [32,64] in the weight gradient tensor. Figure 9A and 9BAfter multiple iterations, after all pixels are accumulated, each CU obtains an output of size (1x3, TileM, TileN) = (1x3, 32, 64). At this point, each CU completes the backpropagation weight update calculation, then calls the vector calculation module to complete the corresponding calculation processing and writes the result to shared memory. After completion, each CU obtains a block of results of size (1x3, TileM, TileN). The final step here is to copy the calculation results from shared memory to global memory using the TMA instruction.

[0184] In the data processing method provided in at least one embodiment of the present disclosure, the weight gradient tensor is split into 1×n granularity, and the activation tensor can be reused n times in shared memory, thereby improving computational intensity, reducing memory access overhead for global memory, L2 cache, and main memory, and reducing and avoiding repeated loading of activation tensors. Furthermore, two second thread warps employ a ping-pong strategy to call the tensor computation module and the vector computation module in parallel, fully utilizing the parallel computing resources of the tensor computation module and the vector computation module. The powerful computing power of the tensor computation module hides the time consumption of the vector computation module, thereby improving computational efficiency, reducing computational time, and enhancing operator performance.

[0185] The embodiments of the present disclosure are applicable to the calculation of 1-dimensional, 2-dimensional, and 3-dimensional weight gradient tensors, and can be used for back-propagation weight updates with any feature map size, an S dimension in the convolution kernel that is an integer multiple of 1x3, any size of other dimensions, any stride length, and any padding.

[0186] In some embodiments of the present disclosure, a persistent kernel programming model can be used to optimize task partitioning and hardware load balancing. A persistent kernel is a GPU programming model that allows thread blocks to continue executing new tasks after completing their current workload, rather than being destroyed and restarted. This model significantly improves GPU resource utilization and application performance by reducing the overhead of thread block creation and destruction.

[0187] At least one embodiment of the present disclosure further provides a data processing device. Figure 11 A schematic block diagram of a data processing device provided in at least one embodiment of the present disclosure.

[0188] The data processing device is used to perform operations on the input gradient tensor and the activation tensor during the back propagation process to obtain the weight gradient tensor.

[0189] like Figure 11 As shown, the data processing device 100 includes an execution module 110 .

[0190] The execution module 110 is configured to perform the first operation multiple times to obtain multiple weight gradient sub-tensors, wherein the weight gradient tensor includes a height direction and a width direction, and each weight gradient sub-tensor is a part of the weight gradient tensor divided according to a granularity of 1×n within the two-dimensional matrix formed by the height direction and the width direction, where n is an integer greater than 1; and arrange the multiple weight gradient sub-tensors to obtain a weight gradient tensor.

[0191] For example, the execution module 110 includes codes and programs stored in a memory. The execution module 110 is implemented as a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities. The processing unit can be a general-purpose processor, and can also be a single-chip microcomputer, a microprocessor, a digital signal processor, a dedicated image processing chip, or a field programmable logic array, etc. The execution module 110 executes the codes and programs to implement some or all of the functions of the execution module 110 described above. For example, the execution module 110 can be a circuit board or a combination of multiple circuit boards for implementing the functions described above. In an embodiment of the present application, the circuit board or the combination of multiple circuit boards may include: (1) one or more processors; (2) one or more non-transitory memories connected to the processors; and (3) firmware stored in the memories that can be executed by the processors.

[0192] For example, the execution module can be implemented as Figure 3 In the computing units shown, different computing units perform different first operations on different data blocks to obtain different sub-output tensors.

[0193] For example, the execution module includes a loading module 1101 and a tensor calculation module 1102 .

[0194] The loading module 1101 is configured to load multiple first elements of the input gradient tensor and multiple second elements of the activation tensor from the first cache to the second cache, wherein the multiple second elements are multiple elements in the activation tensor corresponding to the target sub-tensor.

[0195] The tensor calculation module 1102 is configured to read the multiple first elements from the second cache to the tensor calculation module; and reuse the multiple second elements to perform n second operations to obtain the target sub-tensor, wherein each second operation is used to obtain the target element in the target sub-tensor.

[0196] For example, the tensor computing module 1102 can be a computing core dedicated to tensor operations, such as a tensor core, to accelerate machine learning training and inference tasks.

[0197] The tensor calculation module 1102 includes a reading unit 1112 and a calculation unit 1122. The reading unit 1112 is configured to read the plurality of second elements to obtain the plurality of third elements required for the second operation, and the calculation unit 1122 is configured to determine the target element based on the plurality of first elements and the plurality of third elements.

[0198] It should be noted that the execution module 110 can be used to implement Figure 4A The steps S10 and S20 shown in FIG. 1 are shown in FIG. 2 ; therefore, for a detailed description of the functions that the execution module 110 can implement, reference can be made to the description of steps S10 and S20 in the embodiment of the aforementioned data processing method. In addition, the data processing device 100 can achieve similar technical effects as the aforementioned data processing method, which will not be described in detail here.

[0199] It should be noted that in at least one embodiment of the present disclosure, the data processing device 100 may include more or fewer circuits or units, and the connection relationship between the various circuits or units is not limited and can be determined according to actual needs. The specific configuration of each circuit or unit is not limited and can be composed of analog devices according to circuit principles, or can be composed of digital chips, or constructed in other applicable ways.

[0200] For example, the data processing device 100 may be implemented in hardware, software, or a combination of hardware and software, and this disclosure does not impose any specific limitations on this.

[0201] The data processing method provided in at least one embodiment of the present disclosure can achieve technical effects similar to those of the aforementioned data processing method, and will not be described in detail here.

[0202] The data processing method and data processing device provided by at least one embodiment of the present disclosure can be applied to different systems or devices, such as Figure 13 The electronic device 300 shown. The electronic device 300 can be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an AR device, a VR device, a vehicle-mounted terminal, etc., and can also be a server, etc. The data processing method provided in at least one embodiment of the present disclosure can be applied to scenarios involving convolution operations such as CPU, high-performance computing (HPC) and artificial intelligence (AI) in the electronic device 300, such as tensor computing units, etc. Of course, the present disclosure is not limited to this, and any scenario, device, apparatus, etc. involving backpropagation calculations can adopt the data processing method or data processing apparatus provided in at least one embodiment of the present disclosure.

[0203] In some embodiments, the data processing device provided in at least one embodiment of the present disclosure may be a chip, for example, a system-on-a-chip (SoC). The SoC includes a processor, which may be a single-core processor or a multi-core processor, a memory, and an I / O interface. The processor may load data and applications from the memory and then process the data, for example, performing computational processing such as backpropagation calculations.

[0204] It should be noted that the specific physical meaning of the activation tensor in the data processing method or data processing device provided in at least one embodiment of the present disclosure may vary depending on the application scenario. For example, the data processing method provided in at least one embodiment of the present disclosure may be applied in fields such as speech processing, image processing, text processing, and video processing.

[0205] For example, in the field of speech processing, activation tensors can be the input and output parameters of convolution operations involved in tasks such as feature extraction, speech enhancement, and speech recognition.

[0206] For example, in the field of image processing, activation tensors can be the input and output parameters of convolution operations involved in tasks such as image recognition, feature extraction, image segmentation, target detection, image classification, and scene reconstruction.

[0207] For example, in the field of text processing, activation tensors can be the input and output parameters of convolution operations involved in tasks such as text classification, sentiment analysis, and text generation.

[0208] For example, in the field of video processing, the activation tensor can be the relevant parameters in the image processing field as mentioned above, or the input and output parameters of convolution operations specific to the video processing field, such as the optical flow operator (used to estimate motion between video frames) and the target tracking operator (used to track specific targets in the video).

[0209] Of course, the present disclosure is not limited to this. For other application scenarios or fields, as long as convolution operations are required, the data processing method described in at least one embodiment of the present disclosure can be applied, and they will not be described one by one here.

[0210] Figure 12 A schematic structural diagram of a processor provided in at least one embodiment of the present disclosure. Figure 12 As shown, the processor 200 includes an instruction parsing unit 201 , an execution unit 202 , a first cache 203 and a second cache 204 .

[0211] For example, the instruction parsing unit 201 is used to receive and parse data processing instructions.

[0212] For example, data processing instructions are used to perform operations on the input gradient tensor and activation tensor to obtain the weight gradient tensor during backpropagation.

[0213] For example, the data processing instructions are used to load the input gradient tensor and activation tensor from the first cache 203 to the second cache 204 during the back-propagation process.

[0214] For example, the first cache 203 includes a global cache, and the second cache 204 may include a shared memory. Figure 3 The graphics processor or general graphics processor architecture shown.

[0215] For example, the execution unit 202 executes the data processing instruction after the instruction parsing unit parses the data processing instruction.

[0216] For example, after receiving a data processing instruction, the processor parses the data processing instruction, such as decoding the data processing instruction, generating a microinstruction and sending the microinstruction to the instruction dispatch unit; the instruction dispatch unit sends the microinstruction to the corresponding scheduling queue according to the microinstruction category; the scheduler in the processor selects the appropriate microinstruction from the scheduling queue for execution based on the priority, dependency and other factors of the microinstruction; in response to the microinstruction, when all or the required input data (such as activation tensors and convolution kernel tensors) are prepared, the execution unit reads the data and executes the relevant operations of the data processing instruction.

[0217] For example, an execution unit may include hardware involved in executing data processing instructions, such as a multiplier, a register, a queue, an ALU (Arithmetic Logic Unit), etc.

[0218] For example, when the execution unit 202 executes a data processing instruction, it includes loading multiple first elements of the input gradient tensor and multiple second elements of the activation tensor from the first cache to the second cache, where the multiple second elements are multiple elements in the activation tensor corresponding to the target sub-tensor; reading the multiple first elements from the second cache to the tensor calculation module; and the tensor calculation module reusing the multiple second elements to perform n second operations to obtain the target sub-tensor, each second operation is used to obtain the target element in the target sub-tensor.

[0219] When processor-based upper-level software (for example, AI applications, HPC applications, and scientific computing applications, etc.) can send data processing instructions for convolution operations to the processor (such as the CPU or GPU) through a uniformly packaged function library, the data processing instructions can carry input parameters such as storage addresses; when the processor receives the data processing instructions, the instruction parsing unit 201 parses the data processing instructions, obtains the input parameters, loads the data indicated by the storage address from the memory to the register, and caches it in the cache component.

[0220] For example, the execution unit includes computing units, and different computing units are configured to perform different first operations.

[0221] Regarding the specific process of using the execution unit 202 to execute the data processing instruction, reference may be made to steps S10-S20 and other related contents in the aforementioned data processing method, and repeated details will be omitted.

[0222] The processor provided in at least one embodiment of the present disclosure can achieve technical effects similar to those of the aforementioned data processing method, and the repeated parts will not be repeated.

[0223] Figure 13 This is a schematic block diagram of an electronic device provided by an embodiment of the present disclosure. Figure 13 As shown, the electronic device 300 is suitable for implementing the data processing method provided by the embodiment of the present disclosure. It should be noted that Figure 13 The components of the electronic device 300 shown are merely exemplary and non-limiting. The electronic device 300 may also have other components according to actual application requirements.

[0224] like Figure 13 As shown, the electronic device 300 may include a processing device (eg, a central processing unit, a graphics processing unit, etc.) 301 , which may perform various appropriate actions and processes according to non-transitory computer-executable instructions stored in a memory to implement various functions.

[0225] For example, when the computer executable instructions are executed by the processing device 301, one or more steps of the data processing method according to any of the above embodiments may be executed. It should be noted that for a detailed description of the processing process of the data processing method, reference may be made to the relevant descriptions in the above embodiments of the data processing method.

[0226] For example, the memory may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) 303 and / or cache memory. For example, computer-executable instructions may be loaded from storage device 308 into RAM 303 to execute the computer-executable instructions. Non-volatile memory may include, for example, read-only memory (ROM) 302, a hard disk, an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, a flash memory, and the like. Various applications and various data, such as style images and various data used and / or generated by the applications, may also be stored in the computer-readable storage media.

[0227] For example, the processing device 301 , the ROM 302 , and the RAM 303 are connected to one another via a bus 304 . An input / output (I / O) interface 305 is also connected to the bus 304 .

[0228] Typically, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, a flash memory, etc.; and a communication device 309. The communication device 309 may allow the electronic device 300 to communicate with other electronic devices wirelessly or by wire to exchange data. Figure 13 While an electronic device 300 is shown with various devices, it should be understood that implementation or presence of all illustrated devices is not required, and the electronic device 300 may alternatively implement or possess more or fewer devices. For example, the processing device 301 may control other components in the electronic device 300 to perform desired functions. The processing device 301 may be a device with data processing and / or program execution capabilities, such as a central processing unit (CPU), a tensor processing unit (TPU), or a graphics processing unit (GPU). The central processing unit (CPU) may be of an X86, ARM, or RISC-V architecture. The GPU may be directly integrated into the SOC, directly integrated into the motherboard, or built into the motherboard's northbridge chip.

[0229] Figure 14 A schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure. Figure 14 As shown, the storage medium 400 may be a non-transitory computer-readable storage medium, and one or more computer-executable instructions 401 may be non-transitory stored on the storage medium 400. For example, when the computer-executable instructions 401 are executed by a processor, one or more steps in the data processing method described above may be performed.

[0230] For example, the storage medium 400 may be applied to the electronic device 300 . For example, the storage medium 400 may include the storage device 308 in the electronic device 300 .

[0231] For example, the storage device may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), a hard disk, an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, a flash memory, etc. One or more computer-executable instructions may be stored on the computer-readable storage medium, and the processor may execute the computer-executable instructions to implement various functions of the processor. The storage medium may also store various application programs and various data.

[0232] For example, the storage medium may include a memory card of a smart phone, a cache component of a tablet computer, a hard disk of a personal computer, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other applicable storage media.

[0233] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0234] The units involved in the embodiments described in this disclosure may be implemented in software or hardware, wherein the name of a unit does not necessarily limit the unit itself.

[0235] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0236] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the present disclosure is not limited to technical solutions formed by specific combinations of the aforementioned technical features. It also encompasses other technical solutions formed by any combination of the aforementioned technical features or their equivalents, without departing from the scope of the above disclosure. For example, a technical solution formed by replacing the aforementioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0237] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0238] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

[0239] Regarding this disclosure, the following points need to be explained:

[0240] (1) The drawings of the embodiments of the present disclosure only relate to the structures related to the embodiments of the present disclosure. Other structures may refer to conventional designs.

[0241] (2) In the absence of conflict, the embodiments of the present disclosure and the features therein may be combined with each other to form new embodiments.

[0242] The above description is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. The protection scope of the present disclosure shall be based on the protection scope of the claims.

Claims

1. A data processing method for performing an operation on an input gradient tensor and an activation tensor during backpropagation to obtain a weight gradient tensor, the method comprising: Performing the first operation multiple times to obtain multiple weight gradient sub-tensors, wherein the weight gradient tensor includes a height direction and a width direction, and each weight gradient sub-tensor is a part of the weight gradient tensor divided according to a 1×n granularity within a two-dimensional matrix formed by the height direction and the width direction, where n is an integer greater than 1; and Arrange the multiple weight gradient sub-tensors to obtain the weight gradient tensor, Wherein, each time the first operation is performed to obtain a target sub-tensor in the multiple weight gradient sub-tensors, the first operation includes: Loading a plurality of first elements of the input gradient tensor and a plurality of second elements of the activation tensor from a first buffer to a second buffer, wherein the plurality of second elements are a plurality of elements in the activation tensor corresponding to the target sub-tensor; Read the plurality of first elements from the second cache to a tensor calculation module; and The tensor calculation module reuses the plurality of second elements to perform n second operations to obtain the target sub-tensor, wherein each second operation is used to obtain a target element in the target sub-tensor, The second operation includes: Reading the plurality of second elements to obtain a plurality of third elements required for the second operation; and The target element is determined based on the plurality of first elements and the plurality of third elements.

2. The method according to claim 1, wherein The multiple first elements are elements in one of the multiple input gradient sub-tensors obtained by dividing the input gradient tensor according to the division granularity, and the multiple first elements correspond to the multiple third elements one by one. Determining the target element based on the plurality of first elements and the plurality of third elements includes: calculating a sum of products of a plurality of elements corresponding to the plurality of first elements and the plurality of third elements; and The target element is obtained by adding the sums of the multiple products corresponding to the multiple input gradient sub-tensors.

3. The method according to claim 1, wherein Loading the plurality of first elements of the input gradient tensor and the plurality of second elements of the activation tensor from the first buffer to the second buffer comprises: Loading, according to a first shape size, a plurality of elements of the input gradient tensor from the first buffer to the second buffer as the plurality of first elements; determining a second shape size of the plurality of second elements based on the first shape size and n; determining a starting position of the plurality of second elements; and Based on the second shape size and the starting position, multiple elements in the activation tensor are loaded from the first buffer to the second buffer as the multiple second elements.

4. The method according to claim 3, wherein: The input gradient tensor is expressed as [N, P, Q, K], where P represents height, Q represents width, N represents batch size, and K represents output channel number. The first shape size is [K1, D], where K1 represents the size of the multiple first elements in the output channel number dimension, and D represents the size of the multiple first elements in the first channel dimension. The first channel dimension is obtained by expanding the input gradient tensor in the height dimension, width dimension, and batch number dimension into one dimension. K1 is less than or equal to K. The activation tensor is expressed as [N, H, W, C], where W represents width, H represents height, and C represents the number of input channels. The second shape size is [D+n-1, C1], where D+n-1 represents the size of the multiple second elements in the second channel dimension, and the second channel dimension is obtained by expanding the activation tensor into one dimension in the height dimension, width dimension, and batch number dimension. C1 represents the size of the activation tensor in the input channel number dimension, and C1 is less than or equal to C.

5. The method according to claim 4, wherein Reading the plurality of second elements to obtain a plurality of third elements required for the second operation includes: Determine positions of the plurality of third elements in the plurality of second elements, wherein the plurality of third elements have a shape size of [D, C1], where D is a size in the second channel dimension; and The plurality of third elements are obtained based on a data block with a shape size of [D, C1] at the position in the plurality of second elements.

6. The method according to claim 5, wherein: Determining the starting positions of the plurality of second elements comprises: Get the fill parameters of the activation tensor; Determining the minimum index of the weight gradient sub-tensor in the height direction and the width direction; and The starting position is determined based on the minimum index value and the filling parameter.

7. The method according to claim 6, wherein: The weight gradient tensor is expressed as [R, S, K, C], where R represents height, S represents width, the index is [R1, S1, K2, C2], the filling parameter is (f, g), K2 is less than K, C2 is less than C, Determining the starting position based on the minimum index value and the filling parameter includes: Determine the starting coordinates of the plurality of second elements as [N1, R1-f, S1-g, C2], where N1 is the batch of the element corresponding to the minimum index, C2 is the input channel of the element corresponding to the minimum index, and N1, f, g, R1, S1, and C2 are all integers greater than or equal to 0; and The starting position is obtained by transforming the starting coordinates into the valid range of the activation tensor.

8. The method according to claim 7, wherein: Transforming the starting coordinates into a valid range of the activation tensor to obtain the starting position includes: In the order of width dimension and height dimension, the coordinate value of the starting coordinate in the width dimension and the coordinate value in the height dimension are converted into coordinate values greater than or equal to 0 in sequence.

9. The method according to claim 8, wherein In the order of width dimension and height dimension, the coordinate value of the starting coordinate in the width dimension and the coordinate value in the height dimension are converted into coordinate values greater than or equal to 0, including: Determine whether S1-g is less than 0; In response to S1-g<0, the starting coordinates [N1, R1-f, S1-g, C2] are transformed into [N1, R1-f-t1, S1-g+t1×W, C2], so that S1-g+t1×W>=0, wherein t1=|S1-g| / W is rounded up, and in response to R1-f-t1<0, [N1, R1-f-t1, S1-g+t1×W, C2] are further transformed into [N1-t2, R1-f-t1+t2×H, S1-g+t1×W, C2], so that R1-f-t1+t2×H>=0, wherein t2=|R1-f-t1| / H is rounded up; In response to S1-g>=0, determine whether R1-f is less than 0; In response to S1-g>=0 and R1-f<0, the starting coordinates [N1, R1-f, S1-g, C2] are transformed to [N1-t3, R1-f+t3×H, S1-g, C2] so that R1-f+t3×H>=0, where t3=|R1-f| / H is rounded up.

10. The method according to claim 7 or 8, wherein The plurality of third elements are obtained based on a data block with a shape size of [D, C1] at the position in the plurality of second elements, comprising: Determining a filling position in the data block for filling according to the minimum index value and the filling parameter; and The elements at the filling positions are masked to obtain the multiple third elements.

11. The method according to claim 1, wherein loading the plurality of first elements of the input gradient tensor and the plurality of second elements of the activation tensor from the first cache to the second cache using a producer warp; and The plurality of first elements are read from the second cache to the tensor calculation module using a consumer warp, and the tensor calculation module reuses the plurality of second elements to perform the second operation n times to obtain the target sub-tensor.

12. The method according to claim 11, wherein The consumer thread warp includes a first thread warp and a second thread warp, and the first thread warp and the second thread warp use the tensor calculation module in a ping-pong strategy, wherein the first thread warp and the second thread warp are used to obtain different weight gradient sub-tensors.

13. The method according to claim 12, further comprising: Calling a vector calculation module to perform further calculation processing on the weight gradient sub-tensor; In the process of the first warp executing the first operation, the second warp executes the computing process; in the process of the second warp executing the first operation, the first warp executes the computing process.

14. The method according to any one of claims 1 to 7, wherein: The plurality of first elements of the input gradient tensor and the plurality of second elements of the activation tensor are loaded from the first cache to the second cache using a tensor memory access instruction.

15. The method according to any one of claims 1 to 7, wherein: n is the size of the weight gradient tensor in the width direction. The weight gradient sub-tensors are arranged according to the 1×n granularity division method to obtain the weight gradient tensor, including: Arrange the n weight gradient sub-tensors in order of height to obtain the n×n weight gradient tensor.

16. A processor comprising an instruction parsing unit and an execution unit, wherein: The instruction parsing unit is used to receive and parse data processing instructions, wherein the data processing instructions are used to perform operations on the input gradient tensor and the activation tensor to obtain the weight gradient tensor during the back propagation process, The execution unit executes the data processing method according to any one of claims 1 to 15 after the instruction parsing unit parses the data processing instruction.

17. An electronic device comprising: a memory that non-transitorily stores computer-executable instructions; a processor configured to execute the computer-executable instructions, Wherein, when the computer executable instructions are executed by the processor, the data processing method according to any one of claims 1 to 15 is implemented.

18. A non-transitory computer-readable storage medium, wherein: The non-transitory computer-readable storage medium stores computer-executable instructions, When the computer-executable instructions are executed by a processor, the data processing method according to any one of claims 1 to 15 is implemented.

Images