Tensor computing device, data processor, tensor computing method, storage medium

Abstract

A tensor calculation device, a data processor, a tensor calculation method and a storage medium. The tensor calculation device comprises a receiving unit, a tensor processing unit and an arithmetic processing unit; the tensor processing unit is configured to perform a first tensor calculation on a first tensor to obtain a first intermediate tensor; the tensor processing unit is further configured to obtain a virtual tensor and perform a second tensor calculation on a second tensor according to the virtual tensor to obtain a second intermediate tensor, wherein the second intermediate tensor and the second tensor have the same shape, and the numerical values of the elements at all corresponding positions of the second intermediate tensor and the second tensor have the same preset proportional relationship; and the arithmetic processing unit is configured to perform arithmetic operation processing on the second intermediate tensor and the first intermediate tensor to obtain a result tensor. The tensor calculation device does not need to load data for performing the arithmetic operation, and there is no bandwidth competition phenomenon, thereby greatly improving the overall efficiency of the operator and effectively utilizing the powerful computing power of the tensor calculation device itself.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to a tensor computing apparatus, a tensor computing method, a data processor and a non-transitory computer readable storage medium. BACKGROUND

[0002] A tensor is a multi-linear mapping defined on the Cartesian product of some vector spaces and some dual spaces, for example, a scalar can be regarded as a 0-dimensional tensor, a vector can be regarded as a 1-dimensional tensor, and a matrix can be regarded as a 2-dimensional tensor. Tensor operations are widely used in processors such as parallel processors.

[0003] With the development of artificial intelligence and machine learning, new requirements are put forward for many parallel processor devices represented by parallel processors (for example, multi-core processors, graphics processors, digital signal processors, etc.). In neural network processing often used in the field of artificial intelligence, for example, convolutional neural network, matrix multiply and accumulation (MACC) calculation is often required, which is also a kind of tensor operation, for example, matrix multiply and accumulation calculation includes multiplying the elements at corresponding positions in two matrices, and then accumulating the multiplication results to obtain a calculation result. SUMMARY

[0004] At least one embodiment of the present disclosure provides a tensor computing apparatus, comprising a receiving unit, a tensor processing unit and an arithmetic processing unit, wherein the receiving unit is configured to receive a first tensor and a second tensor as computation input parameters; the tensor processing unit is configured to perform a first tensor computation on the first tensor to obtain a first intermediate tensor, wherein the first intermediate tensor and the second tensor have the same shape; the tensor processing unit is further configured to obtain a virtual tensor and perform a second tensor computation on the second tensor according to the virtual tensor to obtain a second intermediate tensor, wherein the second intermediate tensor and the second tensor have the same shape, and the numerical values of all corresponding positions of the second intermediate tensor and the second tensor are in the same preset proportional relationship; the arithmetic processing unit is configured to perform arithmetic operation processing on the second intermediate tensor and the first intermediate tensor to obtain a result tensor.

[0005] For example, in at least one embodiment of the present disclosure, a tensor computing apparatus is provided, wherein the receiving unit is further configured to receive a third tensor as a computation input parameter, and when the tensor processing unit performs a first tensor computation on the first tensor to obtain a first intermediate tensor, the following operation is included: performing the first tensor computation on the first tensor according to the third tensor to obtain the first intermediate tensor.

[0006] For example, in at least one embodiment of the present disclosure, in a tensor computing device, in response to the second tensor being a two-dimensional tensor, when the tensor processing unit performs obtaining the virtual tensor, the tensor processing unit comprises: constructing an identity matrix as the virtual tensor.

[0007] For example, in at least one embodiment of the present disclosure, in a tensor computing device, in response to the second tensor being a two-dimensional tensor, when the tensor processing unit performs obtaining the virtual tensor, the tensor processing unit comprises: obtaining a preset scale factor; constructing an identity matrix; and calculating a product of the preset scale factor and the identity matrix as the virtual tensor.

[0008] For example, in at least one embodiment of the present disclosure, in a tensor computing device, in response to the second tensor being an N-dimensional tensor, N being a positive integer and greater than 2, when the tensor processing unit performs obtaining the virtual tensor, the tensor processing unit comprises: constructing an N+1-dimensional tensor, wherein the N+1-dimensional tensor corresponds to the same number of input channels and output channels; and taking the N+1-dimensional tensor as the virtual tensor.

[0009] For example, in at least one embodiment of the present disclosure, in a tensor computing device, the number of output channels corresponding to the result tensor is an integer multiple of the number of output channels corresponding to the N+1-dimensional tensor.

[0010] For example, in at least one embodiment of the present disclosure, in a tensor computing device, the length of each dimension of the N+1-dimensional tensor, except the dimensions corresponding to the number of input channels and the number of output channels, is 1.

[0011] For example, in at least one embodiment of the present disclosure, in a tensor computing device, in response to N=3, the virtual tensor is a four-dimensional tensor, and the shape of the four-dimensional tensor is C*C*1*1, where C is a positive integer and represents the number of input channels or output channels corresponding to the four-dimensional tensor, the four-dimensional tensor is arranged as C output channels, each output channel includes a convolution kernel with a size of C*1*1, there is and only one element 1 in each convolution kernel, and other elements in each convolution kernel are 0, and the elements of the C convolution kernels at different depth positions are 1.

[0012] For example, in at least one embodiment of the present disclosure, in a tensor calculation device, in response to the second tensor being a two-dimensional tensor, when the tensor processing unit performs second tensor calculation on the second tensor according to the virtual tensor to obtain a second intermediate tensor, the following operations are included: dividing the second tensor into a plurality of first sub-matrices according to the size of the two-dimensional tensor; performing matrix multiplication processing on the plurality of first sub-matrices and the two-dimensional tensor respectively to obtain a plurality of second sub-matrices; and splicing the plurality of second sub-matrices according to the positional relationship of the plurality of first sub-matrices to obtain the second intermediate tensor.

[0013] For example, in at least one embodiment of the present disclosure, in a tensor calculation device, in response to the second tensor being an N-dimensional tensor, when the tensor processing unit performs second tensor calculation on the second tensor according to the virtual tensor to obtain a second intermediate tensor, the following operations are included: dividing the second tensor according to the number of input channels corresponding to the virtual tensor to obtain a plurality of first sub-tensors; performing convolution multiplication processing on the plurality of first sub-tensors and the virtual tensor respectively to obtain a plurality of second sub-tensors; and splicing the plurality of second sub-tensors according to the positional relationship of the plurality of first sub-tensors to obtain the second intermediate tensor.

[0014] For example, in at least one embodiment of the present disclosure, in a tensor calculation device, when the arithmetic processing unit performs arithmetic operation processing on the second intermediate tensor and the first intermediate tensor to obtain a result tensor, the following operations are included: performing addition processing on the second intermediate tensor and the first intermediate tensor to obtain the result tensor.

[0015] For example, in at least one embodiment of the present disclosure, in a tensor calculation device, the first tensor calculation includes matrix multiplication processing or convolution multiplication processing.

[0016] For example, in at least one embodiment of the present disclosure, in a tensor calculation device, the receiving unit includes an input buffer, the tensor processing unit includes a calculation core, and the output unit includes an output buffer. The input buffer and the output buffer are communicatively connected through the calculation core.

[0017] At least one embodiment of the present disclosure provides a data processor including the tensor calculation device according to any one of the embodiments of the present disclosure.

[0018] For example, in at least one embodiment of the present disclosure, in a data processor, the data processor further includes a storage device configured to input the first tensor and the second tensor into the receiving unit.

[0019] For example, in the data processor provided in at least one embodiment of the present disclosure, the data processor further comprises a vector calculation device, and the storage device and the vector calculation device are configured to receive the result tensor.

[0020] The tensor calculation method provided in at least one embodiment of the present disclosure is applied to a tensor calculation unit, and the tensor calculation method comprises the following steps: receiving a first tensor and a second tensor as calculation input parameters; performing first tensor calculation on the first tensor to obtain a first intermediate tensor, wherein the first intermediate tensor and the second tensor have the same shape; obtaining a virtual tensor, and performing second tensor calculation on the second tensor according to the virtual tensor to obtain a second intermediate tensor, wherein the second intermediate tensor is the same as the second tensor; and performing arithmetic operation processing on the second intermediate tensor and the first intermediate tensor to obtain a result tensor.

[0021] The tensor calculation method provided in at least one embodiment of the present disclosure comprises the following steps: receiving a tensor calculation instruction, wherein the tensor calculation instruction comprises a first tensor and a second tensor as calculation input parameters; and performing the tensor calculation instruction using a tensor calculation unit after analyzing the tensor calculation instruction, wherein performing the tensor calculation instruction using the tensor calculation unit comprises the following steps: performing first tensor calculation on the first tensor to obtain a first intermediate tensor, wherein the first intermediate tensor and the second tensor have the same shape; obtaining a virtual tensor, and performing second tensor calculation on the second tensor according to the virtual tensor to obtain a second intermediate tensor, wherein the second intermediate tensor and the second tensor have the same shape, and all elements of the second intermediate tensor and the second tensor at corresponding positions have the same preset proportional relationship; and performing arithmetic operation processing on the second intermediate tensor and the first intermediate tensor to obtain a result tensor.

[0022] For example, in the tensor calculation method provided in at least one embodiment of the present disclosure, the tensor calculation instruction further comprises a third tensor as a calculation input parameter; and performing first tensor calculation on the first tensor to obtain a first intermediate tensor comprises: performing the first tensor calculation on the first tensor by using the third tensor to obtain the first intermediate tensor.

[0023] The non-transitory computer-readable storage medium provided in at least one embodiment of the present disclosure stores computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the tensor calculation method according to any embodiment of the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present disclosure, not all embodiments of the present disclosure.

[0025] Figure 1A A data transmission structure diagram of an accelerator is shown;

[0026] Figure 1A A processing flow diagram of an accelerator is shown;

[0027] Figure 2 A schematic block diagram of a tensor computing device provided for at least one embodiment of the present disclosure is shown;

[0028] Figure 3 A processing flow diagram of a tensor computing device provided for at least one embodiment of the present disclosure is shown;

[0029] Figure 4 A processing flow diagram of a tensor computing device provided for at least one embodiment of the present disclosure is shown;

[0030] Figure 5 A processing flow diagram of a tensor computing device provided for at least one embodiment of the present disclosure is shown;

[0031] Figure 6 A processing flow diagram of a tensor processing device provided for an embodiment of the present disclosure is shown;

[0032] Figure 7 A schematic block diagram of a data processor provided for at least one embodiment of the present disclosure is shown;

[0033] Figure 8 A schematic flow diagram of a tensor computing method provided for at least one embodiment of the present disclosure is shown;

[0034] Figure 9 A schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0035] In order to make the objects, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions of the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all the embodiments of the present disclosure. Based on the described embodiments of the present disclosure, all other embodiments obtained by those of ordinary skill in the art without any creative effort fall within the scope of protection of the present disclosure.

[0036] Unless otherwise defined, technical terms or scientific terms used in the present disclosure shall have the meanings as generally understood by one of ordinary skill in the art to which the present disclosure belongs. The terms "first", "second", and similar terms do not denote any order, quantity, or importance, but are used to distinguish different components. The terms "include", "contain", and similar terms mean that the elements or objects before the terms encompass the elements or objects listed after the terms and equivalents thereof, and do not exclude other elements or objects. The terms "connect" and "connected" and similar terms do not limit to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The terms "upper", "lower", "left", "right", and the like are used only to indicate relative positional relationships, and when the absolute positions of the described objects are changed, the relative positional relationships can also be changed accordingly.

[0037] In order to keep the following description of the embodiments of the present disclosure clear and concise, the detailed description of some known functions and known components is omitted.

[0038] GPU (Graphic Process Unit) has been widely used in deep learning model training. For the tensor operation commonly used in deep learning model, the GPU manufacturers have made special optimization in the design of software and hardware to accelerate the calculation. For example, some GPU manufacturers provide special tensor calculation devices to optimize tensor calculation. For example, the tensor calculation device can include Tensor Core, and the use of Tensor Core greatly increases the data throughput and improves the calculation efficiency.

[0039] At present, most of the accelerators used for artificial intelligence usually adopt the combination of tensor calculation device and vector calculation device (for example, Vector Engine, also known as vector processing engine) to optimize the acceleration calculation. The tensor calculation device has high computing power, but can only calculate matrix multiplication or convolution multiplication and the like, and many operators used for artificial intelligence calculation not only contain matrix multiplication or convolution multiplication, but also usually need additional calculation, such as matrix addition, which is usually performed by the vector calculation device.

[0040] Figure 1A A data transmission structure diagram of an accelerator is shown.

[0041] As shown in Figure 1A The tensor calculation device includes an input buffer, an output buffer, and a calculation core. The input data is transmitted from the memory to the input buffer along at least one data transmission channel, and then transmitted from the input buffer to the calculation core. The matrix multiplication or convolution multiplication and the like are completed in the calculation core, and then the multiplication result is transmitted to the output buffer. The accumulation of the multiplication result is completed in the output buffer, and the accumulation result is transmitted to the memory or the vector calculation device.

[0042] Figure 1B A processing flowchart of an accelerator is shown.

[0043] As Figure 1B shown, a possible tensor calculation includes: W1 x X1 + X2, X1 represents a first tensor, X2 represents a second tensor, W1 represents a third tensor, and the tensor calculation device is used to calculate W1 x X1, and the addition operation is responsible for calculation by the vector calculation device. For example, the second tensor X2 is stored in the memory, and the vector calculation device needs to load data from the memory, for example, adding the second tensor X2 loaded in the memory to the intermediate tensor W1 x X1 calculated by the tensor calculation device. However, the vector calculation device generally shares a data channel with the data transmitted from the output buffer when loading data from the memory. Although the calculation amount of such addition operation is small, the data amount may be relatively large, and the second tensor X2 loaded from the memory and the intermediate tensor W1 x X1 transmitted from the output buffer share the same data channel, and the intermediate tensor W1 x X1 transmitted from the tensor calculation device will occupy the entire bandwidth of the data channel, resulting in no bandwidth available for the vector calculation device to load the second tensor X2 from the memory, or occupying the entire bandwidth of the data channel when loading the second tensor X2 from the memory, which causes the tensor calculation device to wait, thereby greatly wasting the computing power of the tensor calculation device, which makes the entire operator inefficient and unable to exert strong hardware performance.

[0044] As Figure 1A shown, since the input buffer and the output buffer of the tensor calculation device are respectively arranged at two different positions, the input buffer and the output buffer need to establish a communication connection through the computing core, and the tensor calculation device generally does not provide a transmission channel directly from the input buffer to the output buffer. Moreover, the tensor calculation device stipulates that the calculation result is stored in the output buffer to wait for output or further accumulation operation, and the input data is stored in the input buffer; if the data loaded from the memory is forced to be transmitted from the input buffer to the output buffer, the previous matrix multiplication result originally stored in the output buffer will be flushed.

[0045] Therefore, if the addition operation is directly performed by the tensor calculation device, an additional hardware transmission channel from the input buffer to the output buffer needs to be set, and the matrix data (for example, the second tensor X2) to be added loaded from the memory is transmitted from the input buffer to the output buffer through the hardware transmission channel. This way will increase a lot of hardware overhead, and also re-design the hardware, which increases the hardware development cost.

[0046] At least one embodiment of the present disclosure provides a tensor computing device, a data processor, a tensor computing method and a non-transitory computer readable storage medium. The tensor computing device comprises a receiving unit, a tensor processing unit and an arithmetic processing unit, wherein the receiving unit is configured to receive a first tensor and a second tensor as a calculation input parameter; the tensor processing unit is configured to perform a first tensor calculation on the first tensor to obtain a first intermediate tensor, wherein the first intermediate tensor and the second tensor have the same shape; the tensor processing unit is further configured to obtain a virtual tensor and perform a second tensor calculation on the second tensor according to the virtual tensor to obtain a second intermediate tensor, wherein the second intermediate tensor and the second tensor have the same shape, and all elements of the second intermediate tensor and the second tensor at corresponding positions have the same preset proportional relationship; and the arithmetic processing unit is configured to perform an arithmetic operation on the second intermediate tensor and the first intermediate tensor to obtain a result tensor.

[0047] In the above embodiment, the tensor computing device does not need to change the original design of the tensor computing device in terms of hardware structure, thereby reducing the hardware design cost and hardware overhead. By setting the virtual tensor, the arithmetic operation that needs to be completed in other hardware devices (such as a vector computing device) is converted to be completed directly in the tensor computing device, so that data loading for performing the arithmetic operation is not needed, and there is no bandwidth competition phenomenon, thereby greatly improving the overall efficiency of the operator and effectively utilizing the powerful computing power of the tensor computing device itself.

[0048] In addition, since the arithmetic operation usually has a relatively small amount of calculation compared with matrix multiplication or convolution multiplication, the present disclosure can be used in a scenario with a relatively small amount of calculation and a relatively large amount of data, such as addition calculation on two large amounts of data tensors, and the like, so that the related processing time of the virtual tensor added in the present disclosure is hidden behind the data transmission time, thereby not increasing the overall calculation time of the tensor computing device. Therefore, the tensor computing device provided by at least one embodiment of the present disclosure not only does not increase the overall calculation time of the tensor calculation, but also greatly improves the overall efficiency of the operator.

[0049] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings, but the present disclosure is not limited to these specific embodiments.

[0050] Figure 2 A schematic block diagram of a tensor computing device provided by at least one embodiment of the present disclosure is shown.

[0051] As shown in Figure 2 The tensor computing device 100 comprises a receiving unit 101, a tensor processing unit 102 and an arithmetic processing unit 103.

[0052] For example, also with reference to Figure 1AAs shown, the receiving unit 101 includes an input buffer, the tensor processing unit 102 includes a computing core, and the arithmetic processing unit 103 includes an output buffer.

[0053] Of course, it should be noted that the present disclosure is not limited thereto, and the structures of the receiving unit 101, the tensor processing unit 102, and the arithmetic processing unit 103 can be set as needed as long as the corresponding functions can be implemented. For example, in some embodiments, the arithmetic processing unit 103 can only complete the arithmetic operation processing, and the tensor computing device can additionally have an output unit for buffering, outputting, etc. the result tensor obtained by the arithmetic processing unit 103, and the present disclosure does not make specific limitations thereon. For example, in some other embodiments, the arithmetic processing unit 103 can not only complete the arithmetic operation processing, but also output the result tensor to the tensor computing device.

[0054] For example, as described previously, the input buffer and the output buffer establish a communication connection through the computing core, but there is no direct hardware data transmission path between the input buffer and the output buffer.

[0055] For example, in some embodiments, the tensor computing device 100 can be, for example, a tensor core.

[0056] For example, the receiving unit 101 is configured to receive a first tensor and a second tensor as computation input parameters.

[0057] For example, the receiving unit 101 is further configured to receive a third tensor as a computation input parameter.

[0058] For example, when the tensor processing unit 102 performs the first tensor computation on the first tensor to obtain a first intermediate tensor, the operations include performing the first tensor computation on the first tensor according to the third tensor to obtain the first intermediate tensor.

[0059] For example, the first tensor computation includes a matrix multiplication operation and a convolution multiplication operation.

[0060] For example, the third tensor can also be stored in the tensor computing device 100, and the third tensor is read to perform the first tensor computation when the first tensor computation is performed.

[0061] For example, the tensor processing unit 102 is configured to perform the first tensor computation on the first tensor to obtain a first intermediate tensor. For example, the first intermediate tensor and the second tensor have the same shape, and thus the first intermediate tensor can perform the arithmetic operation processing with the second tensor.

[0062] The tensor processing unit 102 is further configured to obtain a virtual tensor, and perform a second tensor calculation on the second tensor according to the virtual tensor to obtain a second intermediate tensor. For example, the second intermediate tensor and the second tensor have the same shape, so that they can be operated without shape transformation, and the numerical values of the elements at all corresponding positions of the second intermediate tensor and the second tensor have the same preset proportional relationship.

[0063] Here, the numerical values of the elements at all corresponding positions of the second intermediate tensor and the second tensor have the same preset proportional relationship, which means that the second intermediate tensor and the second tensor satisfy: the second intermediate tensor = preset proportional coefficient * the second tensor, and the preset proportional coefficient is in numerical form and can be set as needed.

[0064] For example, in some embodiments, the second intermediate tensor and the second tensor are the same, where “the same” means that the tensor shape and the elements at each corresponding position in the tensor are the same, and the preset proportional coefficient is 1.

[0065] For example, in other embodiments, the second intermediate tensor and the second tensor have the same shape, but the numerical values of the elements at all corresponding positions in the second intermediate tensor and the second tensor are not completely the same but have the same preset proportional relationship, for example, the preset proportional coefficient is 0.5, etc.

[0066] For example, the arithmetic processing unit 103 is configured to perform arithmetic operation processing on the second intermediate tensor and the first intermediate tensor to obtain a result tensor. For example, the arithmetic operation processing includes addition operation, subtraction operation, etc.

[0067] For example, when the arithmetic processing unit 103 performs arithmetic operation processing on the second intermediate tensor and the first intermediate tensor to obtain a result tensor, it includes performing the following operation: performing addition processing on the second intermediate tensor and the first intermediate tensor to obtain a result tensor.

[0068] For example, the result tensor can be directly output from the tensor calculation device 100 and transmitted to other components that need to use the result tensor, such as a storage device or a vector calculation device, etc.

[0069] For example, the receiving unit 101, the tensor processing unit 102, and the arithmetic processing unit 103 include codes and programs stored in a memory, and are implemented as a central processing unit (CPU) or other forms of processing units having data processing capability and / or instruction execution capability, for example. The processing units can be general-purpose processors, and also single-chip microcomputers, microprocessors, digital signal processors, dedicated image processing chips, or field programmable logic arrays, etc. The receiving unit 101, the tensor processing unit 102, and the arithmetic processing unit 103 execute the codes and programs to implement some or all of the functions of the receiving unit 101, the tensor processing unit 102, and the arithmetic processing unit 103 as described above.

[0070] Figure 3 A processing flowchart of the tensor computing device provided for at least one embodiment of the present disclosure.

[0071] As shown in Figure 3 , the tensor computation to be completed by the tensor computing device can be represented as: W1 x X1 + X2, where X1 represents a first tensor, W1 represents a third tensor, W1 x X1 represents a first intermediate tensor, and the first tensor computation (e.g., “x”) can be a matrix multiplication operation or a convolution multiplication operation according to the shapes of the first tensor X1 and the third tensor W1.

[0072] For example, X2 represents a second tensor, a virtual tensor is represented as W2, and a second intermediate tensor is represented as W2 x X2. The second intermediate tensor is the same as X2, i.e., W2 x X2 = X2.

[0073] For example, the arithmetic operation processing is an addition processing, the second intermediate tensor and the first intermediate tensor are added to obtain a result tensor, so that the result tensor = W2 x X2 + W1 x X1 = X2 + W1 x X1.

[0074] Therefore, the tensor computation using the virtual tensor obtains the same second intermediate tensor as the second tensor, and the result obtained by the result tensor is the same as the result obtained by the arithmetic operation processing of the second tensor and the first intermediate tensor by the vector computing device, so that this way does not affect the accuracy of the final result.

[0075] Referring to Figure 3 and Figure 1BWhen the virtual tensor is set, the second tensor is also transmitted to the output buffer in the form of the second intermediate tensor through tensor calculation of the virtual tensor, without flushing the original first intermediate tensor in the output buffer, so that the arithmetic operation processing can be directly performed in the tensor calculation device, without loading data to the vector calculation device for performing the arithmetic operation, without bandwidth competition phenomenon, and without setting an additional hardware path, thereby greatly improving the overall efficiency of the operator, effectively utilizing the powerful computing power of the tensor calculation device itself, and reducing hardware overhead.

[0076] The specific content of the tensor calculation device provided by at least one embodiment of the present disclosure will be described in detail below with reference to the drawings.

[0077] According to different shapes of the second tensor, the dimensions of the virtual tensor and the specific process of the second tensor calculation are slightly different. The following will be introduced respectively.

[0078] For example, in response to the second tensor being a two-dimensional tensor, when the tensor processing unit 102 performs the operation of obtaining the virtual tensor, the following operations are performed: constructing a unit matrix as the virtual tensor.

[0079] The unit matrix is a square matrix, and the elements on the diagonal line (i.e., the main diagonal line) from the upper left corner to the lower right corner are all 1, and the elements are all 0. The unit matrix is like 1 in numerical multiplication, so that the second intermediate tensor obtained by multiplying the second tensor is exactly the same as the second tensor.

[0080] For example, the size of the unit matrix can be determined according to at least the hardware resource consumption of the tensor calculation device in the process of performing the second tensor calculation, and the size of the optimal unit matrix is determined by balancing the actual hardware overhead.

[0081] For example, the virtual tensor is not limited to the unit matrix. If the tensor calculation to be completed by the tensor calculation device is W1xX1+scalexX2, where scale represents a preset scale coefficient, and the preset scale coefficient can be any numerical value, such as 0.5.

[0082] At this time, when the tensor processing unit performs the operation of obtaining the virtual tensor, the following operations are performed: obtaining the preset scale coefficient; constructing a unit matrix; and calculating the product of the preset scale coefficient and the unit matrix as the virtual tensor.

[0083] That is, at this time, the elements on the main diagonal line in the virtual tensor are all the preset scale coefficient, and the other elements are all 0.

[0084] At this time, the second intermediate tensor and the second tensor have the same shape, but the numerical values of the elements at all corresponding positions in the second intermediate tensor and the second tensor are not exactly the same but have the same preset scale relationship, and the preset scale relationship is determined according to the numerical value of the preset scale coefficient.

[0085] For example, the preset proportion coefficient can be preset, and a virtual tensor can be calculated, and the virtual tensor can be directly used in the tensor calculation. Similarly, the virtual tensor in the form of a unit matrix can be preset, and the virtual tensor can be stored in a storage area in the tensor calculation device. When the tensor calculation is performed, the virtual tensor is read to complete the second tensor calculation. Alternatively, the virtual tensor can be obtained as a calculation input parameter from the receiving unit. Alternatively, the virtual tensor can be constructed in real time before the tensor calculation is performed, and the present disclosure does not limit this.

[0086] Therefore, the virtual tensor in the present disclosure can be constructed in real time during the calculation process, or can be obtained from the tensor calculation device or other storage devices, and the present disclosure does not limit this.

[0087] For example, when the tensor processing unit 102 performs the second tensor calculation on the second tensor according to the virtual tensor to obtain the second intermediate tensor, the following operations are included: dividing the second tensor into a plurality of first sub-matrices according to the size of the two-dimensional tensor; performing matrix multiplication processing on the plurality of first sub-matrices and the two-dimensional tensor respectively to obtain a plurality of second sub-matrices; and splicing the plurality of second sub-matrices according to the positional relationship of the plurality of first sub-matrices to obtain the second intermediate tensor.

[0088] Figure 4 The processing flowchart of the tensor calculation device provided by at least one embodiment of the present disclosure is provided.

[0089] As shown in Figure 4 The first tensor is a two-dimensional tensor, and is a two-dimensional matrix of K rows and N columns. The third tensor is also a two-dimensional tensor, and is a two-dimensional matrix of M rows and K columns. The first tensor and the third tensor perform the first tensor calculation, for example, the first tensor calculation is a matrix multiplication operation, and the first intermediate tensor is obtained. The first intermediate tensor is a two-dimensional tensor of M rows and N columns. Here, M, N, and K are positive integers, which can be set arbitrarily as needed, and are not limited to the number of rows and the number of columns shown in Figure 4 .

[0090] As shown in Figure 4 The second tensor is a two-dimensional tensor, and is a two-dimensional matrix of M rows and N columns. The virtual tensor is a square matrix of P rows and P columns. For example, the virtual tensor can be a unit matrix of P rows and P columns, or the virtual tensor can have preset proportion coefficients on the main diagonal, and the other elements are all 0, and the present disclosure does not limit this. Here, P is a positive integer, and P can be determined at least according to the hardware resource consumption of the tensor calculation device.

[0091] For example, the second tensor can be divided into multiple first submatrices according to the size of the virtual tensor. Each first submatrix has the same size as the virtual tensor, and these first submatrices do not overlap. Here, "non-overlapping" means that any element of the second tensor exists in only one first submatrix. Figure 4 As shown, the second tensor, starting from the first row and first column, is uniformly divided into first submatrix 1, first submatrix 2, first submatrix 3, etc., each with a size of P rows and P columns. To improve computational efficiency, M and N can both be integer multiples of P, thus dividing the second tensor into an integer number of first submatrices for easier computation. In other words, the specific setting of P can be determined based on the hardware resource consumption of the tensor computing device and the specific values ​​of M and N.

[0092] For example, multiple first submatrices can be multiplied with a two-dimensional tensor to obtain multiple second submatrices. For instance, multiplying first submatrice 1 with a virtual tensor yields second submatrice 1; multiplying first submatrice 2 with a virtual tensor yields second submatrice 2; multiplying first submatrice 3 with a virtual tensor yields second submatrice 3, and so on.

[0093] Next, the multiple second submatrices are concatenated according to the positional relationships of the multiple first submatrices to obtain the second intermediate tensor, as shown in the figure. Figure 4 As shown. The resulting second intermediate tensor is a two-dimensional matrix with M rows and N columns, and the values ​​of all corresponding elements in the second tensor and the second intermediate tensor are in the same preset ratio, for example, all corresponding elements in the second intermediate tensor and the second tensor are the same.

[0094] Then, arithmetic operations are performed on the first and second intermediate tensors, such as adding or subtracting elements at corresponding positions in the first and second intermediate tensors, to obtain the result tensor.

[0095] Referring to the above processing procedure, if the size of the identity matrix is ​​set too large, the computational load of a single matrix multiplication operation may be large. However, if the size of the identity matrix is ​​set too small, the second tensor will be divided into too many first submatrices, which may also affect the final computational load. Therefore, the size of the identity matrix can be determined by balancing the above factors, based on the actual hardware conditions and hardware overhead.

[0096] For example, in the processing described above, arithmetic operations should originally be performed in the vector computing device. However, by setting up virtual tensors, the arithmetic operations are switched to the tensor computing device, avoiding unnecessary bandwidth consumption and greatly improving the operator's computing power.

[0097] For example, in response to the second tensor being an N-dimensional tensor, N being a positive integer and greater than 2, when the tensor processing unit performs the obtaining of the virtual tensor, the following operations are performed: constructing an N+1-dimensional tensor, wherein the N+1-dimensional tensor has the same number of input channels and output channels; and taking the N+1-dimensional tensor as the virtual tensor.

[0098] For example, the N+1-dimensional tensor constructed is taken as a convolution kernel to perform a convolution multiplication operation, that is, the virtual tensor is taken as a convolution kernel to perform a second tensor calculation, and the second tensor calculation includes a convolution multiplication operation. A multi-dimensional convolution operation usually includes single-channel multi-convolution, multi-channel single-convolution, and multi-channel multi-convolution. The number of input channels of the convolution kernel represented by the N+1-dimensional tensor is determined by the number of input channels of the second tensor, and usually the number of input channels of the N+1-dimensional tensor should be the same as the number of input channels of the second tensor. The number of output channels of the N+1-dimensional tensor, that is, the number of channels of the convolution kernel, determines how many channels of the second intermediate tensor are finally output.

[0099] For example, the number of output channels of the result tensor is an integer multiple of the number of output channels of the N+1-dimensional tensor. In addition, the number of output channels of the second intermediate tensor, the first intermediate tensor, and the second tensor having the same shape as the result tensor is an integer multiple of the number of output channels of the N+1-dimensional tensor. Therefore, for different second tensors, an N+1-dimensional tensor with a different number of output channels can be set to perform a convolution operation as a convolution kernel. At this time, because the number of output channels of the result tensor is an integer multiple, the specific element parameter setting of the N+1-dimensional tensor is the simplest, and the operation efficiency is the highest, and no additional processing is required for the extra part of elements in the second tensor that is not an integer multiple.

[0100] Of course, the present disclosure is not limited to this. If the number of output channels of the result tensor is not an integer multiple of the number of output channels of the N+1-dimensional tensor, additional processing of the extra part of elements is required.

[0101] For example, the number of input channels and the number of output channels of the virtual tensor can be determined comprehensively according to the hardware resource consumption when the tensor calculation device performs the second tensor calculation and the number of output channels of the result tensor.

[0102] For example, if N=3 and the virtual tensor is a 4-dimensional tensor, and the data shape of the second tensor is [W, H, C], W represents the width of the second tensor, H represents the height of the second tensor, and C represents the depth of the second tensor, then the number of input channels and the number of output channels of the 4-dimensional tensor constructed are both C, where W, H, and C are positive integers.

[0103] For example, the shape of the 4-dimensional tensor is C*C*1*1, the 4-dimensional tensor is arranged as C output channels, that is, C convolution kernels of output channels are represented, each output channel includes a convolution kernel with a size of C*1*1, the width of each convolution kernel is 1, the height is 1, and the depth is C. At this time, the total amount of calculation of the second tensor calculation is the smallest, and the specific element parameter setting of the virtual tensor is the easiest.

[0104] For example, in some embodiments, only 1 element in each convolution kernel is 1, and other elements in each convolution kernel are 0, and the elements of the C convolution kernels at different depth positions are 1.

[0105] For example, each convolution kernel includes C elements in depth order, which are defined as depth 1, depth 2, …, and depth C in turn. For example, in one example, the element at depth 1 in the convolution kernel of the first output channel is 1, and the remaining elements are 0, the element at depth 2 in the convolution kernel of the second output channel is 1, and the remaining elements are 0, …, the element at depth C in the convolution kernel of the C output channel is 1, and the remaining elements are 0.

[0106] At this time, the second intermediate tensor obtained by multiplying the second tensor is exactly the same as the second tensor.

[0107] For example, in other embodiments, similar to the two-dimensional tensor, a preset proportion coefficient can be obtained, and the product of the preset proportion coefficient and the aforementioned C*C*1*1 channel, four-dimensional tensor with only 1 element in each convolution kernel is 1, is calculated as a virtual tensor. That is, at this time, the virtual tensor includes C convolution kernels of output channels, only 1 element in each convolution kernel is a preset proportion coefficient, and other elements in each convolution kernel are 0, and the elements of the C convolution kernels at different depth positions are the preset proportion coefficient.

[0108] At this time, the second intermediate tensor and the second tensor have the same shape, but the element values of all corresponding positions in the second intermediate tensor and the second tensor are not exactly the same but have the same preset proportion relationship, which is determined according to the preset proportion coefficient value.

[0109] For example, for higher-dimensional vectors, for example, when N is greater than 3, the length of each dimension of the N+1-dimensional tensor except the input channel number and the output channel number corresponding dimensions is 1, that is, the N+1-dimensional tensor can be set as: C*C*1*…*1*…*1.

[0110] For example, when the tensor processing unit performs a second tensor calculation on the second tensor according to the virtual tensor to obtain a second intermediate tensor, the following operations are performed: the second tensor is divided according to the input channel number corresponding to the virtual tensor to obtain a plurality of first sub-tensors; the plurality of first sub-tensors are respectively subjected to convolution multiplication processing with the virtual tensor to obtain a plurality of second sub-tensors; and the plurality of second sub-tensors are spliced according to the positional relationship of the plurality of first sub-tensors to obtain the second intermediate tensor.

[0111] Figure 5 A processing flowchart of the tensor calculation device provided for at least one embodiment of the present disclosure is shown.

[0112] As shown in Figure 5 , the first tensor is a three-dimensional tensor, which can be represented as C*W*H, the width is W, the height is H, and the depth is C; and the third tensor is a four-dimensional tensor, which is represented as OC*C*1*1, and includes OC convolution kernels of the output channel, and the size of each convolution kernel is C*1*1, that is, the width of each convolution kernel is 1, the height is 1, and the depth is C. Therefore, the input channel number corresponding to the third tensor is C, and the output channel number is OC.

[0113] For example, the first tensor and the third tensor perform a first tensor calculation, for example, the first tensor calculation is a convolution multiplication operation, to obtain a first intermediate tensor. The first intermediate tensor is a three-dimensional tensor, which is represented as OC*W*H, the width is W, the height is H, and the depth is OC. Here, W, H, C, and OC are all positive integers, which can be set arbitrarily as needed.

[0114] As shown in Figure 5 , the second tensor is a three-dimensional tensor, the width is W, the height is H, and the depth is L. The virtual tensor is a four-dimensional tensor, which is represented as C*C*1*1, and includes C convolution kernels of the output channel, and the size of each convolution kernel is C*1*1, that is, the width of each convolution kernel is 1, the height is 1, and the depth is C. Therefore, the input channel number (i.e., the depth) and the output channel number of the virtual tensor are both C.

[0115] For example, C can be determined according to the hardware resource consumption of the tensor calculation device, for example, C can be 64.

[0116] For example, the second tensor is divided according to the input channel number corresponding to the virtual tensor to obtain a plurality of first sub-tensors, the input channel number of each first sub-tensor is the same as the input channel number of the virtual tensor, and the plurality of first sub-tensors do not overlap with each other. Here, the non-overlapping means that each element in the second tensor X2 is only divided into one first sub-tensor.

[0117] As shown in Figure 5 , the second tensor is divided into m first sub-tensors with the shape of C*W*H along the depth direction, respectively as Figure 5The first subtensor 1, first subtensor 2, ..., first subtensor m are shown in the diagram. Here, L is a positive integer and L = OC, and m is a positive integer and m * C = L.

[0118] Then, the multiple first subtensors are convolved and multiplied with the virtual tensor to obtain multiple second subtensors.

[0119] like Figure 5 As shown, the first sub-tensor 1 is convolved and multiplied with the virtual tensor to obtain the second sub-tensor 1, which has a size of C*W*H. The specific process is detailed in the multi-channel convolution operation process and will not be repeated here. The first sub-tensor 2 is convolved and multiplied with the virtual tensor to obtain the second sub-tensor 2, which also has a size of C*W*H. The specific process is not repeated here. ... The first sub-tensor m is convolved and multiplied with the virtual tensor to obtain the second sub-tensor m, which also has a size of C*W*H.

[0120] Next, multiple second sub-tensors are concatenated according to the positional relationships of multiple first sub-tensors to obtain a second intermediate tensor. For example, m second sub-tensors are reconcatenated according to the positional relationships between their corresponding m first sub-tensors to form a second intermediate tensor of shape L*W*H. The second intermediate tensor has the same shape as the second tensor, and the element values ​​at all corresponding positions of the second tensor and the second intermediate tensor have the same preset ratio relationship, for example, all corresponding elements of the second intermediate tensor and the second tensor are the same.

[0121] Then, arithmetic operations are performed on the first and second intermediate tensors, such as adding the elements at corresponding positions, to obtain the result tensor.

[0122] For example, in some embodiments, the second subtensor 1 to the second subtensor m can be directly subjected to arithmetic operations with the elements at the corresponding positions of the first intermediate tensor according to the positional relationship of the first subtensor 1 to the first subtensor m. At this time, the second subtensor 1 to the second subtensor m also constitute the second intermediate tensor in form and perform arithmetic operations. However, the second intermediate tensor at this time is a tensor in a virtual state.

[0123] Referring to the above processing procedure, if the number of input channels (or output channels) of the virtual tensor is set too large, the computational load of a single convolution multiplication operation may be high. However, if the number of input channels (or output channels) of the virtual tensor is set too small, the second tensor may be divided into too many first subtensors, which may also affect the final computational load. Therefore, the number of input channels (or output channels) of the virtual tensor can be determined by balancing the above factors, based on the actual hardware conditions and hardware overhead.

[0124] Figure 6 A flowchart of a tensor processing apparatus provided in an embodiment of this disclosure.

[0125] The processing procedure of the tensor processing device provided in at least one embodiment of the present disclosure will be specifically explained below in combination with Figure 6

[0126] Figure 6 is a specific processing procedure diagram of the tensor calculation procedure shown in Figure 3 For example, as shown in Figure 3 , the tensor calculation to be completed by the tensor calculation device can be represented as: W1 x X1 + X2. For specific meanings of the parameters, refer to the related description in Figure 3 , which will not be repeated here.

[0127] For example, the receiving unit 101 is configured to receive a first tensor X1, a second tensor X2 and a third tensor W1 as calculation input parameters. As shown in Figure 3 , the shape of the first tensor X1 is 64*56*56, that is, the width is 56, the height is 56, and the depth is 64; the shape of the second tensor X2 is 256*56*56, that is, the width is 56, the height is 56, and the depth is 256; the shape of the third tensor W1 is 256*64*1*1, including 256 convolution kernels of output channels, and the size of each convolution kernel is 64*1*1, that is, the width of each convolution kernel is 1, the height is 1, and the depth is 64, that is, the input channel number corresponding to the third tensor W1 is 64, and the output channel number is 256.

[0128] The tensor processing unit 102 is configured to perform a convolution multiplication operation on the first tensor X1 and the third tensor W1 to obtain a first intermediate tensor W1 x X1, as shown in Figure 6 , the shape of the first intermediate tensor W1 x X1 is 256*56*56, that is, the width is 56, the height is 56, and the depth is 256.

[0129] The tensor processing unit 102 is also configured to obtain a virtual tensor X2. As shown in Figure 6 , the virtual tensor W2 is a 4-dimensional tensor, and the corresponding input channel number and output channel number are both 64, which are determined according to the hardware overhead when the tensor calculation device performs the second tensor calculation, and the output channel number corresponding to the result tensor (or the second tensor).

[0130] For example, since the output channel number corresponding to the second tensor X2 is 256, the input channel number or the output channel number corresponding to the virtual tensor W2 can be set to a divisor of 256, for example, 32, 64, 128, etc., and the input channel number with the smallest hardware overhead is selected from them according to the hardware overhead, for example, the total calculation amount is the smallest when the input channel number is 64, thereby constructing the virtual tensor W2.

[0131] ​For example, the virtual tensor W2 can be constructed in real time according to the number of output channels corresponding to the second tensor and the empirical value given by the reference hardware overhead.

[0132] The tensor processing unit 102 is further configured to divide the second tensor X2 into a plurality of first sub-tensors according to the number of input channels corresponding to the virtual tensor W2. Since the depth of the second tensor X2 is 256 and the number of input channels (i.e., the depth) corresponding to the virtual tensor W2 is 64, the second tensor X2 can be divided into 4 first sub-tensors according to the depth direction, as shown in the first sub-tensor 1 to the first sub-tensor 4 in Figure 6 , the size of each first sub-tensor is 64*56*56, that is, the width is 56, the height is 56, and the depth is 64, and each first sub-tensor does not overlap.

[0133] Then, the tensor processing unit 102 is further configured to perform convolution multiplication processing on the plurality of first sub-tensors and the virtual tensor respectively to obtain a plurality of second sub-tensors. For example, the first sub-tensor 1 and the virtual tensor W2 perform matrix multiplication processing to obtain the second sub-tensor 1, the shape of the second sub-tensor 1 is 64*56*56, that is, the width is 56, the height is 56, and the depth is 64; the first sub-tensor 2 and the virtual tensor W2 perform matrix multiplication processing to obtain the second sub-tensor 2, the shape of the second sub-tensor 2 is 64*56*56, that is, the width is 56, the height is 56, and the depth is 64; the first sub-tensor 3 and the virtual tensor W2 perform matrix multiplication processing to obtain the second sub-tensor 3, the shape of the second sub-tensor 3 is 64*56*56, that is, the width is 56, the height is 56, and the depth is 64; the first sub-tensor 4 and the virtual tensor W2 perform matrix multiplication processing to obtain the second sub-tensor 4, the shape of the second sub-tensor 4 is 64*56*56, that is, the width is 56, the height is 56, and the depth is 64.

[0134] Then, the plurality of second sub-tensors are spliced according to the positional relationship of the plurality of first sub-tensors to obtain a second intermediate tensor W2*X2, as shown in Figure 6 , the shape of the obtained second intermediate tensor W2*X2 is 256*56*56, which is the same as the second tensor X2, and the elements at the corresponding positions are also the same as the second tensor X2.

[0135] Then, the first intermediate tensor W1*X1 and the second intermediate tensor W2*X2 are subjected to arithmetic operation processing, for example, the elements at the corresponding positions are added to obtain a result tensor.

[0136] In the tensor calculation device provided in at least one embodiment of the present disclosure, for tensor calculation of any dimension, such as high-dimensional convolution multiplication processing, the arithmetic operation processing can be converted to be executed in the tensor calculation device by setting a virtual tensor, unnecessary bandwidth occupation and data transmission waiting time are avoided, the operator computing power is greatly improved, the powerful computing efficiency of the tensor calculation device itself is fully utilized, and the hardware design does not need to be modified, thereby reducing the hardware overhead.

[0137] The present disclosure also provides a data processor in at least one embodiment. Figure 7 A schematic block diagram of the data processor provided in at least one embodiment of the present disclosure is shown.

[0138] As Figure 7 shown, the data processor 200 includes the tensor calculation device 201, the vector calculation device 202, and the storage device 203 as described in any embodiment of the present disclosure. For the structure, function, and technical effects of the tensor calculation device 201, reference can be made to the tensor calculation device 100 as described above, and details are not repeated here.

[0139] For example, the storage device 203 is configured to input the first tensor and the second tensor to the receiving unit. Of course, the storage device 203 can also input the third tensor as an input parameter to the receiving unit.

[0140] For example, the storage device 203 and the vector calculation device 202 are configured to receive the result tensor.

[0141] With reference to the related description of Figure 1A , at present, when the data processor 200 needs to complete the arithmetic operation processing of the result of the convolution multiplication or the matrix multiplication with another tensor, if the other tensor is located in the storage device, the other tensor is loaded into the vector calculation device, the result of the convolution multiplication or the matrix multiplication is also transmitted to the vector calculation device, and the arithmetic operation processing is completed in the vector calculation device. However, in the data processor provided in at least one embodiment of the present disclosure, the tensor calculation device 201 directly completes the arithmetic operation processing and the matrix / convolution multiplication operation, and outputs the result tensor to the storage device 203 or the vector calculation device 202, thereby avoiding the bandwidth occupation when transmitting data to the vector calculation device 202 and the waiting time caused by the bandwidth occupation, greatly improving the operator computing power, fully utilizing the powerful computing power of the tensor calculation device itself, and improving the processing capability of the data processor.

[0142] For example, the storage device 203 can include any structure of storage device capable of storing data, such as memory, cache, etc. For example, in the accelerator as Figure 1A shown, the storage device 203 can be a memory.

[0143] For example, the data transmission relationship and structural relationship among the tensor computing device 201, the vector computing device 202, and the storage device 203 can refer to the related description of Figure 1A , which will not be repeated here.

[0144] For example, the data processor 200 can be implemented as a GPU, a CPU (Central Processing Unit), an NPU (Neural Network Processing Unit), a DSP (Digital Signal Processor), etc. according to actual needs, and the present disclosure does not make any limitation in this regard.

[0145] For example, in an embodiment, the data processor 200 can be a GPU, the vector computing device 202 can be a vector computing engine, and the tensor computing device 201 can be a tensor core.

[0146] Corresponding to the above-mentioned tensor computing device, the present disclosure at least one embodiment also provides a tensor computing method. Figure 8 The schematic flowchart of the tensor computing method provided by at least one embodiment of the present disclosure is shown in FIG. 10.

[0147] For example, as shown in Figure 8 , the tensor computing method provided by the embodiment of the present disclosure includes steps S10 to S40.

[0148] In step S10, a first tensor and a second tensor are received as computation input parameters.

[0149] In step S20, a first tensor calculation is performed on the first tensor to obtain a first intermediate tensor, wherein the first intermediate tensor and the second tensor have the same shape.

[0150] In step S30, a virtual tensor is obtained, and a second tensor calculation is performed on the second tensor according to the virtual tensor to obtain a second intermediate tensor, wherein the second intermediate tensor and the second tensor have the same shape, and the numerical values of the elements at all corresponding positions of the second intermediate tensor and the second tensor have the same preset proportional relationship.

[0151] In step S40, an arithmetic operation is performed on the second intermediate tensor and the first intermediate tensor to obtain a result tensor.

[0152] For example, the tensor computing method provided by at least one embodiment of the present disclosure further includes receiving a third tensor as a computation input parameter.

[0153] For example, step S20 includes performing a first tensor calculation on the first tensor according to the third tensor to obtain a first intermediate tensor.

[0154] For example, the first tensor calculation can include a matrix multiplication calculation or a convolution multiplication calculation according to the different dimensions of the second tensor.

[0155] For example, in response to the second tensor being a two-dimensional tensor, in some embodiments, the obtaining the virtual tensor in step S30 can include: constructing an identity matrix as the virtual tensor.

[0156] For example, in some other embodiments, the obtaining the virtual tensor in step S30 can include: obtaining a preset scale coefficient; constructing an identity matrix; and calculating a product of the preset scale coefficient and the identity matrix as the virtual tensor.

[0157] For example, in step S30, when performing the second tensor calculation on the second tensor according to the virtual tensor to obtain the second intermediate tensor, the following operations are included: dividing the second tensor into a plurality of first sub-matrices according to the size of the two-dimensional tensor; performing matrix multiplication processing on the plurality of first sub-matrices and the two-dimensional tensor respectively to obtain a plurality of second sub-matrices; and splicing the plurality of second sub-matrices according to the positional relationship of the plurality of first sub-matrices to obtain the second intermediate tensor.

[0158] For example, in response to the second tensor being an N-dimensional tensor, N being a positive integer and greater than 2, the obtaining the virtual tensor in step S30 can include: constructing an N+1-dimensional tensor, wherein the N+1-dimensional tensor has the same number of input channels and output channels; and taking the N+1-dimensional tensor as the virtual tensor.

[0159] For example, the number of output channels corresponding to the result tensor and the number of output channels corresponding to the N+1-dimensional tensor have an integer multiple relationship.

[0160] For example, the length of each dimension of the N+1-dimensional tensor, except for the dimensions corresponding to the number of input channels and the number of output channels, is 1.

[0161] For example, in response to N=3 and the virtual tensor being a 4-dimensional tensor, the shape of the 4-dimensional tensor is C*C*1*1, where C is a positive integer and represents the number of input channels or the number of output channels corresponding to the 4-dimensional tensor, the 4-dimensional tensor is arranged as C output channels, each output channel includes a convolution kernel with a size of C*1*1, each convolution kernel has and only has one element being 1, and each convolution kernel has other elements being 0 except for the one element, and the elements of the C convolution kernels at different depth positions are 1.

[0162] For example, in step S30, when performing the second tensor calculation on the second tensor according to the virtual tensor to obtain the second intermediate tensor, the following operations are included: dividing the second tensor into a plurality of first sub-tensors according to the number of input channels corresponding to the virtual tensor; performing convolution multiplication processing on the plurality of first sub-tensors and the virtual tensor respectively to obtain a plurality of second sub-tensors; and splicing the plurality of second sub-tensors according to the positional relationship of the plurality of first sub-tensors to obtain the second intermediate tensor.

[0163] For example, the step S40 can include: performing addition processing on the second intermediate tensor and the first intermediate tensor to obtain a result tensor.

[0164] It should be noted that the specific description and related introduction of steps S10-S40 can refer to the related content of the aforementioned tensor calculation device. For example, the related description of step S10 can refer to the related content of the receiving unit 101 shown in Figure 2 ; the related description of steps S20 and S30 can refer to the related content of the tensor processing unit 102 shown in Figure 2 ; and the related description of step S40 can refer to the related content of the arithmetic processing unit 103 shown in Figure 2 .

[0165] In addition, the tensor calculation method can achieve similar technical effects as the aforementioned tensor calculation device, which will not be repeated here.

[0166] The disclosure at least one embodiment further provides a tensor calculation method. For example, the tensor calculation method includes: receiving a tensor calculation instruction, wherein the tensor calculation instruction includes a first tensor and a second tensor as calculation input parameters; and executing the tensor calculation instruction using a tensor calculation unit after analyzing the tensor calculation instruction.

[0167] For example, executing the tensor calculation instruction using the tensor calculation unit includes: performing a first tensor calculation on the first tensor to obtain a first intermediate tensor, wherein the first intermediate tensor and the second tensor have the same shape; obtaining a virtual tensor, and performing a second tensor calculation on the second tensor according to the virtual tensor to obtain a second intermediate tensor, wherein the second intermediate tensor is the same as the second tensor; and performing arithmetic operation processing on the second intermediate tensor and the first intermediate tensor to obtain a result tensor.

[0168] For example, the tensor calculation instruction further includes a third tensor as a calculation input parameter; and performing the first tensor calculation on the first tensor to obtain the first intermediate tensor includes: performing the first tensor calculation on the first tensor using the third tensor to obtain the first intermediate tensor.

[0169] For example, the tensor calculation instruction corresponds to the Y=W1xX1+X2 operation, wherein W1 is the first tensor, X1 is the third tensor, and X2 is the second tensor.

[0170] For example, the tensor calculation method provided by at least one embodiment of the disclosure can be applied to Figure 7 the data processor 200 shown in

[0171] For example, in the tensor calculation method provided by at least one embodiment of the present disclosure, a tensor calculation instruction is provided, and the tensor calculation instruction includes a first tensor, a second tensor, and a third tensor as calculation input parameters. For example, after the data processor receives the tensor calculation instruction, the data processor parses the tensor calculation instruction, for example, decodes the tensor calculation instruction, generates micro-instructions, and sends the micro-instructions to an instruction distribution unit; the instruction distribution unit sends the micro-instructions to corresponding scheduling queues according to the categories of the micro-instructions; and in response to the micro-instructions, after the first tensor, the second tensor, and the third tensor (all or the required part) are prepared, the tensor calculation unit reads data and executes the tensor calculation instruction operation.

[0172] For the specific process of executing the tensor calculation instruction using the tensor calculation unit, reference can be made to steps S10-S40 in the tensor calculation method as described above, and repeated parts will not be described herein.

[0173] Figure 9 A schematic diagram of a non-transitory computer-readable storage medium provided by at least one embodiment of the present disclosure is shown in FIG. 3. For example, as shown in FIG. 3, the storage medium 300 can be a non-transitory computer-readable storage medium, and one or more computer-readable instructions 301 can be non-transitorily stored on the storage medium 300. For example, when the computer-readable instructions 301 are executed by a processor, one or more steps in the tensor calculation method described above can be performed. Figure 9

[0174] For example, the storage medium 300 can be applied to the data processor described above, for example, the storage medium 300 can include a storage device in the data processor.

[0175] For example, the storage device can include any combination of one or more computer program products, and the computer program product can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. For example, the volatile memory can include random access memory (RAM), cache memory, and the like. The non-volatile memory can include read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), compact disc read-only memory (CD-ROM), USB memory, flash memory, and the like. One or more computer-readable instructions can be stored on the computer-readable storage medium, and the data processor can run the computer-readable instructions to implement various functions of the data processor. Various application programs and various data and the like can also be stored in the storage medium.

[0176] ​For example, the storage medium can include a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM), a compact disc read only memory (CD-ROM), a flash memory, or any combination of the above storage mediums, or other applicable storage medium.

[0177] For the present disclosure, the following points also need to be explained:

[0178] (1) The drawings of the embodiments of the present disclosure only relate to the structures involved in the embodiments of the present disclosure, and other structures can refer to the general design.

[0179] (2) In the case of no conflict, the embodiments of the present disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0180] The above is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto, and the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. A tensor computing apparatus, comprising an input buffer, a tensor processing unit, and an output buffer, wherein, The input buffer is configured to receive and buffer the first and second tensors as computation input parameters. The tensor processing unit includes a computation kernel configured to perform a first tensor computation on the first tensor to obtain a first intermediate tensor and cache it in the output cache, wherein the first intermediate tensor and the second tensor have the same shape; The tensor processing unit is further configured to acquire a virtual tensor and read the second tensor from the input buffer. The computation core is further configured to perform a second tensor calculation on the second tensor based on the virtual tensor to obtain a second intermediate tensor and output it to the output buffer. The second intermediate tensor and the second tensor have the same shape, and the values ​​of all corresponding elements of the second intermediate tensor and the second tensor are in the same preset ratio. The output buffer is configured to perform arithmetic operations on the second intermediate tensor and the first intermediate tensor in the buffer to obtain a result tensor; The input buffer and the output buffer establish a communication connection through the computing core. In response to the input data to the tensor computing device, the data is directly transmitted from the input buffer to the output buffer via the computing core, and the data originally stored in the output buffer is flushed by the input data. The result tensor is the same as the result of directly performing the arithmetic operation on the first intermediate tensor and the second tensor. The calculation of the first tensor includes matrix multiplication or convolution multiplication, the calculation of the second tensor includes matrix multiplication or convolution multiplication, and the arithmetic operation includes addition or subtraction.

2. The tensor computing device according to claim 1, wherein, The input buffer is also configured to receive a third tensor as a computation input parameter. When the tensor processing unit performs a first tensor calculation on the first tensor to obtain a first intermediate tensor, it includes performing the following operations: The first tensor computation is performed on the first tensor based on the third tensor to obtain the first intermediate tensor.

3. The tensor computing device according to claim 1, wherein, In response to the second tensor being a two-dimensional tensor, When the tensor processing unit acquires the virtual tensor, it includes performing the following operations: Construct an identity matrix as the virtual tensor.

4. The tensor computing device according to claim 1, wherein, In response to the second tensor being a two-dimensional tensor, When the tensor processing unit acquires the virtual tensor, it includes performing the following operations: Obtain the preset scaling factor; Construct the identity matrix; The product of the preset scaling factor and the identity matrix is ​​calculated to form the virtual tensor.

5. The tensor computing apparatus according to claim 1, wherein, In response to the second tensor being an N-dimensional tensor, where N is a positive integer greater than 2, When the tensor processing unit acquires the virtual tensor, it includes performing the following operations: Construct an N+1 dimensional tensor, wherein the number of input channels and the number of output channels corresponding to the N+1 dimensional tensor are the same; The N+1 dimensional tensor is used as the virtual tensor.

6. The tensor computing apparatus according to claim 5, wherein, The number of output channels corresponding to the result tensor is an integer multiple of the number of output channels corresponding to the N+1 dimensional tensor.

7. The tensor computing apparatus according to claim 5, wherein, In the N+1 dimensional tensor, the length of each dimension other than the dimensions corresponding to the number of input channels and the number of output channels is 1.

8. The tensor computing apparatus according to claim 5, wherein, In response to N=3, the virtual tensor is a 4-dimensional tensor. The 4D tensor has a shape of C*C*1*1, where C is a positive integer representing the number of input channels or output channels corresponding to the 4D tensor. The 4D tensor is arranged into C output channels. Each output channel includes a convolutional kernel of size C*1*1. Each convolutional kernel has exactly one element that is 1, and all other elements in each convolutional kernel except for the one element are 0. The elements of the C convolutional kernels at different depth positions are 1.

9. The tensor computing apparatus according to claim 1, wherein, In response to the second tensor being a two-dimensional tensor, When the tensor processing unit performs a second tensor calculation on the second tensor based on the virtual tensor to obtain a second intermediate tensor, it includes performing the following operations: The second tensor is divided into multiple first sub-matrices according to the size of the two-dimensional tensor; The plurality of first sub-matrices are multiplied by the two-dimensional tensor to obtain a plurality of second sub-matrices; The plurality of second submatrices are concatenated according to the positional relationship of the plurality of first submatrices to obtain the second intermediate tensor.

10. The tensor computing apparatus according to claim 1, wherein, In response to the second tensor being an N-dimensional tensor, When the tensor processing unit performs a second tensor calculation on the second tensor based on the virtual tensor to obtain a second intermediate tensor, it includes performing the following operations: The second tensor is divided according to the number of input channels corresponding to the virtual tensor to obtain multiple first sub-tensors; The plurality of first sub-tensors are convolved and multiplied with the virtual tensor respectively to obtain a plurality of second sub-tensors; The plurality of second sub-tensors are concatenated according to the positional relationship of the plurality of first sub-tensors to obtain the second intermediate tensor.

11. The tensor computing apparatus according to any one of claims 1-10, wherein, When the output buffer performs arithmetic operations on the second intermediate tensor and the first intermediate tensor to obtain the result tensor, it includes the following operations: The second intermediate tensor and the first intermediate tensor are added together to obtain the result tensor.

12. A data processor comprising the tensor computing apparatus as described in any one of claims 1-11.

13. The data processor according to claim 12, wherein, The data processor further includes a storage device configured to input the first tensor and the second tensor into the input buffer.

14. The data processor according to claim 13, wherein, The data processor also includes a vector computing device. The storage device and the vector computation device are configured to receive the result tensor.

15. A tensor computation method, applied to a tensor computation unit, the tensor computation unit comprising an input buffer, an output buffer, and a computation kernel, wherein the input buffer and the output buffer establish a communication connection through the computation kernel. The tensor calculation method includes: Receive the first and second tensors as input parameters for computation and cache them in the input buffer; The calculation kernel performs a first tensor calculation on the first tensor to obtain a first intermediate tensor, which is then cached in the output buffer. The first intermediate tensor and the second tensor have the same shape. A virtual tensor is obtained and the second tensor is read from the input buffer. The computation kernel performs a second tensor calculation on the second tensor based on the virtual tensor to obtain a second intermediate tensor and output it to the output buffer. The second intermediate tensor and the second tensor have the same shape, and the values ​​of all corresponding elements of the second intermediate tensor and the second tensor are in the same preset ratio. The output buffer performs arithmetic operations on the second intermediate tensor and the first intermediate tensor cached in the output buffer to obtain the result tensor; The result tensor is the same as the result of directly performing the arithmetic operation on the first intermediate tensor and the second tensor. In response to the data input to the tensor calculation unit, the data is directly transferred from the input buffer to the output buffer via the calculation kernel. The data originally stored in the output buffer is flushed by the input data. The first tensor calculation includes matrix multiplication or convolution multiplication. The second tensor calculation includes matrix multiplication or convolution multiplication. The arithmetic operation includes addition or subtraction.

16. A tensor computation method, comprising: Receive tensor calculation instructions, wherein the tensor calculation instructions include a first tensor and a second tensor as calculation input parameters; After parsing the tensor computation instruction, the tensor computation unit executes the instruction. The tensor computation unit includes an input buffer, an output buffer, and a computation kernel. The input buffer and the output buffer establish a communication connection through the computation kernel. The input buffer is configured to cache the first tensor and the second tensor. The execution of the tensor calculation instructions using the tensor calculation unit includes: The first tensor is read from the input buffer and the first tensor computation is performed on the first tensor by the computation kernel to obtain a first intermediate tensor and cache it in the output buffer, wherein the first intermediate tensor and the second tensor have the same shape; A virtual tensor is obtained and the second tensor is read from the input buffer. The computation kernel performs a second tensor calculation on the second tensor based on the virtual tensor to obtain a second intermediate tensor and output it to the output buffer. The second intermediate tensor and the second tensor have the same shape, and the values ​​of all elements at corresponding positions of the second intermediate tensor and the second tensor are in the same preset ratio. The output buffer performs arithmetic operations on the second intermediate tensor and the first intermediate tensor cached in the output buffer to obtain the result tensor; The result tensor is the same as the result of directly performing the arithmetic operation on the first intermediate tensor and the second tensor. In response to the data input to the tensor calculation unit, the data is directly transferred from the input buffer to the output buffer via the calculation kernel. The data originally stored in the output buffer is flushed by the input data. The first tensor calculation includes matrix multiplication or convolution multiplication. The second tensor calculation includes matrix multiplication or convolution multiplication. The arithmetic operation includes addition or subtraction.

17. The tensor calculation method according to claim 16, wherein, The tensor calculation instruction also includes a third tensor as a calculation input parameter; Performing the first tensor computation on the first tensor yields the first intermediate tensor, which includes: The first tensor is calculated using the third tensor to obtain the first intermediate tensor.

18. A non-transitory computer-readable storage medium, wherein, The non-transitory computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the tensor computation method according to claim 15 or 16.

Images