Winograd convolution operation method, device, equipment and storage medium
The Winograd convolution operation method addresses inefficiencies in convolutional neural network hardware by decomposing data into sub-tensors and using sub-devices for summation, reducing multiplication overhead and improving processing efficiency.
Patent Information
- Application Number
- CN201911061091.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-11-01
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2039-12-20
AI Technical Summary
The high computational overhead and inefficiency in hardware implementation of convolution operations in convolutional neural networks due to the need for numerous multipliers, which increases timing, power consumption, and area requirements.
The Winograd convolution operation method decomposes data into multiple sub-tensors, performs transformation operations on these sub-tensors, and sums the results using multiple operation sub-devices according to a set strategy, converting matrix multiplication into matrix summation operations.
This approach reduces the overhead caused by multiplication and improves processing efficiency by processing summation operations simultaneously across multiple sub-devices, enhancing the performance of convolutional neural network chips.
Smart Images

Figure CN112784206B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a winograd convolution operation method, device, equipment and storage medium. Background Art
[0002] With the development of artificial intelligence technology, convolutional neural network technology has emerged. Convolutional neural network technology is a type of feedforward neural network with deep structure that includes convolution calculations. It is one of the representative algorithms of deep learning. In convolutional neural network technology, neurons and convolution kernels are required to perform convolution operations.
[0003] In the prior art, when implementing convolution operations in hardware, a large number of multipliers are required on the chip. Since the cost of multiplication, such as timing, power consumption, and area, is much higher than that of addition in the hardware implementation process, the processing efficiency of the chip is greatly reduced when implementing convolution operations in hardware in the prior art. Summary of the invention
[0004] Based on this, it is necessary to provide a winograd convolution operation method, device, equipment and storage medium to address the above technical problems.
[0005] In a first aspect, an embodiment of the present application provides a winograd convolution operation method, which is applied to winograd convolution transformation, and the method includes:
[0006] In the transformation process of the winograd convolution operation, the data in the winograd convolution operation is disassembled into multiple sub-tensors;
[0007] Performing transformation operations on the multiple sub-tensors to obtain winograd transformation results of the multiple sub-tensors, and summing the winograd transformation results of the multiple sub-tensors to obtain a transformation result of the data, wherein the summing operation of the winograd transformation results of the multiple sub-tensors is completed by multiple operation sub-devices according to a set strategy;
[0008] A winograd convolution operation is performed according to the transformation result of the data.
[0009] In a second aspect, an embodiment of the present application provides a winograd convolution operation device, including:
[0010] The disassembly module is used to disassemble the data in the winograd convolution operation into multiple sub-tensors during the transformation process of the winograd convolution operation;
[0011] a transformation and summation operation module, configured to perform transformation operations on the multiple sub-tensors to obtain winograd transformation results of the multiple sub-tensors, and perform summation operations on the winograd transformation results of the multiple sub-tensors to obtain a transformation result of the data, wherein the summation operation of the winograd transformation results of the multiple sub-tensors is completed by multiple operation sub-devices according to a set strategy;
[0012] The convolution operation module is used to complete the winograd convolution operation according to the transformation result of the data.
[0013] In a third aspect, an embodiment of the present application provides a winograd convolution operation device, including: a processor and a memory; wherein,
[0014] The memory is used to store program code;
[0015] The processor is used to call the program code stored in the memory to execute the method described in the first aspect.
[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, and when the instructions are executed on a computer, the computer executes the method described in the first aspect.
[0017] In a fifth aspect, an embodiment of the present application provides an artificial intelligence chip, which includes the device described in the second aspect or the third aspect above.
[0018] In a sixth aspect, an embodiment of the present application provides an electronic device, comprising the artificial intelligence chip described in the fifth aspect.
[0019] In a seventh aspect, an embodiment of the present application provides a board card, which includes: a storage device, an interface device, a control device, and an artificial intelligence chip as described in the fifth aspect.
[0020] Wherein, the artificial intelligence chip is connected to the storage device, the control device and the interface device respectively;
[0021] The storage device is used to store data;
[0022] The interface device is used to realize data transmission between the artificial intelligence chip and external equipment;
[0023] The control device is used to monitor the status of the artificial intelligence chip.
[0024] Further, in the board as described above, the storage device comprises: a plurality of groups of storage units, each group of the storage units is connected to the artificial intelligence chip via a bus, and the storage units are: DDR SDRAM;
[0025] The chip includes: a DDR controller for controlling data transmission and data storage of each storage unit;
[0026] The interface device is: a standard PCIE interface.
[0027] The winograd convolution operation method, device, equipment and storage medium provided in the embodiment of the present application, in which the data in the winograd convolution operation is decomposed into multiple sub-tensors; the multiple sub-tensors are transformed to obtain the winograd transformation results of the multiple sub-tensors, and the winograd transformation results of the multiple sub-tensors are summed to obtain the transformation result of the data, wherein the summation operation of the winograd transformation results of the multiple sub-tensors is completed by multiple operation sub-devices according to the set strategy; the winograd convolution operation is completed according to the transformation result of the data. Since the transformation process of the original winograd convolution operation is converted from matrix multiplication to matrix summation operation, the overhead caused by multiplication is reduced. And when performing the summation operation, it is processed by multiple operation sub-devices at the same time, which improves the processing efficiency of the chip. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art.
[0029] Figure 1 An application scenario diagram of the winograd convolution operation method provided in an embodiment of the present application;
[0030] Figure 2 A flowchart of a winograd convolution operation method provided for one embodiment of the present application;
[0031] Figure 3 A flowchart of a winograd convolution operation method provided by another embodiment of the present application;
[0032] Figure 4 A flowchart of a winograd convolution operation method provided in yet another embodiment of the present application;
[0033] Figure 5 A schematic diagram of the structure of a winograd convolution operation device provided in one embodiment of the present application;
[0034] Figure 6A schematic diagram of the structure of a winograd convolution operation device provided in another embodiment of the present application;
[0035] Figure 7 A schematic diagram of a master-slave architecture of a winograd convolution operation device provided in another embodiment of the present application;
[0036] Figure 8 A schematic diagram of a processing system for the winograd convolution operation method of an embodiment of the present application;
[0037] Fig. 9 This is a structural block diagram of the board card according to an embodiment of the present application. DETAILED DESCRIPTION
[0038] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.
[0039] It should be understood that the terms "first", "second", "third", "fourth", etc. in the claims, specifications and drawings of the present disclosure are used to distinguish different objects rather than to describe a specific order. The terms "include" and "comprise" used in the specifications and claims of the present disclosure indicate the presence of the described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their collections.
[0040] It should also be understood that the terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the present disclosure. As used in this disclosure and claims, the singular forms of "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should also be further understood that the term "and / or" used in this disclosure and claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations.
[0041] As used in this specification and claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.
[0042] In order to clearly understand the technical solution of the present application, the technical terms involved in the prior art and the embodiments of the present application are explained below:
[0043] Convolution operation: Convolution operation means starting from the upper left corner of the image, opening an active window of the same size as the template. The active window corresponds to a window image, which is the convolution kernel. The window image corresponds to the pixels in the image, multiply and add them, and use the calculation result as the first pixel value of the new image after the convolution operation. Then, the active window moves one column to the right and performs the same operation. And so on, from left to right, from top to bottom, a new image can be obtained.
[0044] Winograd convolution operation: Winograd convolution operation is a convolution acceleration implementation method based on polynomial interpolation algorithm. It performs Winograd convolution forward transformation on the two inputs of the convolution operation: the first target matrix and the second target matrix, respectively, and then performs positional multiplication on the first target matrix and the second target matrix after the forward transformation. Finally, Winograd convolution inverse transformation is performed on the positional multiplication result again to obtain a convolution result equivalent to the original convolution operation.
[0045] Convolutional neural network: Convolutional neural network technology is a type of feedforward neural network with deep structure and convolution calculation. It is one of the representative algorithms of deep learning. In the convolutional layer and fully connected layer of convolutional neural network technology, neurons and convolution kernels are required to perform convolution operations to obtain feature data. It is widely used in image classification, image recognition, etc.
[0046] Since matrix multiplication is still performed in the winograd convolution forward transformation and the winograd convolution inverse transformation, there is still a lot of overhead in the hardware implementation process. Therefore, in order to further improve the processing efficiency, the embodiment of the present application also proposes a winograd convolution operation method. The winograd convolution operation method can be applied in the hardware implementation process of the convolutional neural network.
[0047] Specifically, the Winograd convolution operation method provided in the embodiment of the present application can be applied to various scenarios where convolutional neural networks are used for identification, classification, and prediction. For example, it can be applied in the fields of autonomous driving, robotics, smart home, office management, etc. Figure 1An application scenario of the winograd convolution operation method provided in the present application is shown in the figure. The electronic device is equipped with a visual sensor and is provided with a convolutional neural network chip. The visual sensor collects the user's body feature data, such as face data, iris data, fingerprint data, etc., and sends the body feature data to the electronic device. The electronic device recognizes the body feature data through the convolutional neural network chip. If the user is recognized as a legitimate user, the access control door is opened, otherwise the access control door is not opened. Since the winograd convolution operation method provided in the embodiment of the present application is used in the convolutional neural network chip, the processing efficiency of the convolutional neural network chip for data can be accelerated, and recognition, classification and prediction can be quickly performed.
[0048] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.
[0049] Figure 2 A flowchart of a Winograd convolution operation method provided by an embodiment of the present application. Figure 2 As shown, the execution subject of this embodiment is a winograd convolution operation device, and the winograd convolution operation method provided by this embodiment includes the following steps.
[0050] Step 101, during the transformation process of the winograd convolution operation, the data in the winograd convolution operation is decomposed into multiple sub-tensors.
[0051] Among them, the transformation type of the winograd convolution operation includes a winograd transformation type of a forward transformation and a winograd transformation type of an inverse transformation. If it is a winograd transformation type of a forward transformation, the data in the winograd convolution operation is neuron data or weight data, the neuron data can be a neuron matrix, and the weight data can be a weight matrix. If it is a winograd transformation type of an inverse transformation, the data in the winograd convolution operation is the transformation bitwise multiplication data, the transformation bitwise multiplication data is the transformation bitwise multiplication matrix, and the transformation bitwise multiplication matrix is the matrix obtained by bitwise multiplication of the result matrix after the forward transformation of the neuron matrix and the result matrix after the forward transformation of the weight matrix.
[0052] The resultant matrix after the neuron matrix is positively transformed can be called the neuron transformed matrix. The resultant matrix after the weight matrix is positively transformed can be called the weight transformed matrix.
[0053] For the case where the data dimension is one-dimensional, the winograd convolution operation can be expressed as: F = A T ((Gg)⊙(B T d)), when the data dimension is two-dimensional, the winograd convolution operation can be expressed as F = A T ((GgGT )⊙(B T dB))A.
[0054] In the case where the data dimension is one-dimensional, F is the convolution output matrix, d is the first data, B T is the transformation matrix of the winograd convolution positive transformation of d. g is the second data, and G is the transformation matrix of the winograd convolution positive transformation of g. (Gg)⊙(B T d) is the data obtained by performing bitwise multiplication of the first data and the second data after the positive transformation, which can be expressed as a transformation bitwise multiplication matrix. T It is the transformation matrix that performs the inverse winograd convolution on the transform bitwise multiplication matrix.
[0055] In the case where the data dimension is two-dimensional, F is the convolution output matrix, d is the first data, which can be a neuron matrix, and B T is the left multiplication matrix of the winograd convolution positive transformation of d. B is the right multiplication matrix of the winograd convolution positive transformation of d. g is the second data, which can be a weight matrix. G is the left multiplication matrix of the winograd convolution positive transformation of g, G T is the right multiplication matrix of the winograd convolution positive transformation of g. (GgG T )⊙(B T dB) is a matrix for performing bitwise multiplication of the first data and the second data after the positive transformation, which can be called a transformation bitwise multiplication matrix. T It is the left multiplication matrix of the inverse winograd convolution transformation of the transformation matrix, and A is the right multiplication matrix of the inverse winograd convolution transformation of the transformation matrix.
[0056] In this embodiment, the winograd convolution operation may include a winograd transformation type of a positive transformation, and may also include a winograd transformation type of an inverse transformation. And if the winograd transformation type of the present embodiment is a positive transformation, it may be a winograd transformation type for a positive transformation of a neuron matrix, and may also be a winograd transformation type for a positive transformation of a weight matrix. And in this embodiment, the scale of the data in the winograd convolution operation is not limited.
[0057] In this embodiment, when the data in the winograd convolution operation is decomposed into multiple sub-tensors, the data can be parsed as the sum of the multiple sub-tensors.
[0058] For example, the data can be represented as Then the multiple sub-tensors obtained after disassembling the data can be expressed as:
[0059]
[0060]
[0061] Step 102, transform the multiple sub-tensors to obtain the winograd transform results of the multiple sub-tensors, sum the winograd transform results of the multiple sub-tensors to obtain the transformation result of the data, wherein the summation of the winograd transform results of the multiple sub-tensors is completed by multiple operation sub-devices according to a set strategy.
[0062] Specifically, in this embodiment, the meta-sub-tensor corresponding to the multiple sub-tensors can be first obtained, and then the meta-sub-tensor is subjected to a winograd transformation, and the winograd transformation result of the sub-tensor is determined according to the winograd transformation results of the sub-tensor and the meta-sub-tensor. Then, according to the set strategy, the summation operation of the winograd transformation results of the multiple sub-tensors is completed by multiple operation sub-devices to obtain the transformation result of the data.
[0063] Among them, the set strategy can be a load balancing strategy. When multiple operating sub-devices perform the sum operation of the winograd transformation results of multiple sub-tensors, the addition execution tasks of each element in the result matrix undertaken by each operating sub-device are balanced.
[0064] In this embodiment, since the operator performs a sum operation on the winograd transformation results of multiple sub-tensors, the operator may be an adder.
[0065] Step 103, completing the winograd convolution operation according to the transformation result of the data.
[0066] In this embodiment, if the dimension of the data is two-dimensional, the transformation result of the neuron matrix is first determined, and then the transformation result of the weight matrix is determined. Then, the transformation result of the neuron matrix and the transformation result of the weight matrix are subjected to a positional multiplication to obtain a transformation positional multiplication matrix, and then the transformation result of the positional multiplication matrix is determined to complete the winograd convolution operation.
[0067] The winograd convolution operation method provided in this embodiment decomposes the data in the winograd convolution operation into multiple sub-tensors; performs transformation operations on the multiple sub-tensors to obtain the winograd transformation results of the multiple sub-tensors, and performs sum operations on the winograd transformation results of the multiple sub-tensors to obtain the transformation results of the data, wherein the sum operation of the winograd transformation results of the multiple sub-tensors is completed by multiple operation sub-devices according to the set strategy; and the winograd convolution operation is completed according to the transformation results of the data. Since the transformation process of the original winograd convolution operation is converted from matrix multiplication to matrix summation operation, the overhead caused by multiplication is reduced. And when performing the summation operation, it is processed by multiple operation sub-devices at the same time, which improves the processing efficiency of the chip.
[0068] Figure 3 This is a flowchart of a winograd convolution operation method provided by another embodiment of the present application. Figure 3 As shown, the winograd convolution operation method provided in this embodiment is Figure 2 On the basis of the illustrated embodiment, steps 101 to 103 are further refined, and the winograd convolution operation method provided in this embodiment includes the following steps.
[0069] Step 201, during the transformation process of the winograd convolution operation, decompose the data in the winograd convolution operation into multiple sub-tensors.
[0070] Furthermore, in this embodiment, the data in the winograd convolution operation is decomposed into multiple sub-tensors, including:
[0071] Multiple sub-tensors are obtained from data analysis, where the data is the sum of multiple sub-tensors, the number of multiple sub-tensors is the same as the number of non-zero elements in the data, each sub-tensor has a single non-zero element, and the non-zero element in the sub-tensor is the same as the non-zero element in the corresponding position in the data.
[0072] Specifically, the data is set as If each element in the data is non-zero, the multiple sub-tensors obtained after decomposing the data can be expressed as:
[0073]
[0074] Among them, the number of multiple sub-tensors is X*Y, and the number of non-zero elements in the data is also X*Y. That is, the number of multiple sub-tensors is the same as the number of non-zero elements in the data. And from the multiple sub-tensors, it can be known that there is a single non-zero element in each sub-tensor, and the non-zero element in the sub-tensor is the same as the non-zero element in the corresponding position in the data.
[0075] Step 202, transform the multiple sub-tensors to obtain winograd transform results of the multiple sub-tensors.
[0076] Furthermore, in this embodiment, step 202 includes the following steps:
[0077] Step 2021, obtain the winograd transformation result of the meta-sub-tensor corresponding to each sub-tensor.
[0078] Among them, the meta-sub-tensor is a tensor in which the non-zero elements of the sub-tensor are set to 1.
[0079] Specifically, corresponding to the multiple sub-tensors illustrated in step 201, the corresponding elementary sub-tensors are respectively expressed as:
[0080]
[0081] As an optional implementation, in step 2021, the Winograd transformation result of the elementary sub-tensor corresponding to each sub-tensor is obtained, specifically:
[0082] For each sub-tensor, multiply the left side of the meta-sub-tensor corresponding to the sub-tensor by the left multiplication matrix and the right side by the right multiplication matrix to obtain the winograd transformation result of the meta-sub-tensor.
[0083] Specifically, if the left multiplication matrix is expressed as The right multiplication matrix is represented as B Y×Y , after multiplying the meta-sub-tensor corresponding to each of the above sub-tensors by the left multiplication matrix and by the right multiplication matrix, the winograd transformation results of the meta-sub-tensors are:
[0084]
[0085]
[0086] The left multiplication matrix and the right multiplication matrix are determined by the size of the subtensor and the winograd transformation type, where the winograd transformation type includes the winograd transformation type of the forward transformation and the winograd transformation type of the inverse transformation. That is, if the size of the subtensor and the winograd transformation type are fixed, then each element in the left multiplication matrix and the right multiplication matrix is also fixed.
[0087] Step 2022, use the non-zero element values in the sub-tensor as coefficients to multiply the winograd transformation result of the corresponding element sub-tensor to obtain the winograd transformation result of the sub-tensor.
[0088] Furthermore, in this embodiment, each sub-tensor has a non-zero element value. The corresponding non-zero element value is obtained from each sub-tensor, and the non-zero element value is multiplied by the winograd transformation result of the corresponding meta-sub-tensor to obtain the winograd transformation result of the sub-tensor.
[0089] The winograd transformation results of each sub-tensor listed above can be expressed as follows:
[0090]
[0091] Step 2023, adding the winograd transformation results of multiple sub-tensors to obtain the winograd transformation result of the data.
[0092] The winograd transformation result of the data can be expressed as: S = Wino{d} = B T dB. Then the Winograd transformation result of the data obtained by adding the Winograd transformation results of multiple sub-tensors can be expressed as shown in formula (1):
[0093]
[0094] Step 203, summing up the winograd transformation results of the multiple sub-tensors to obtain the transformation result of the data. During the summing up of the winograd transformation results of the multiple sub-tensors, the summing up of the winograd transformation results of the multiple sub-tensors is distributed to multiple operation sub-devices for execution according to the principle of load balancing of multiple operation sub-devices.
[0095] Furthermore, in this embodiment, in step 203, according to the principle of load balancing of multiple operator devices, the summation operation of the winograd transformation results of multiple sub-tensors is distributed to multiple operator devices for execution, including the following steps:
[0096] Step 2031, determine the number of non-zero elements in the transformation matrix corresponding to each sub-tensor.
[0097] Furthermore, in this embodiment, the matrix formed by the winograd transformation results of each element sub-tensor can be represented as the transformation matrix of the corresponding sub-tensor. Therefore, the number of non-zero elements in the matrix corresponding to the winograd transformation result of the element sub-tensor is determined as the number of non-zero elements in the transformation matrix of the corresponding sub-tensor.
[0098] Step 2032, determining the order of summing the winograd transformation results of the multiple sub-tensors according to the number of non-zero elements.
[0099] Furthermore, in this embodiment, in order to enable multiple operator devices to perform the sum operation of the winograd transform results of multiple sub-tensors in a load-balanced manner, when determining the order of summing the winograd transform results of multiple sub-tensors, the winograd transform results of sub-tensors with a larger number of non-zero elements are summed up with the winograd transform results of sub-tensors with a smaller number of non-zero elements as much as possible, so that multiple operator units can participate in the summation operation at the same clock.
[0100] Step 2033, according to the order of summing the winograd transformation results of the determined multiple sub-tensors, the mapping relationship between the element positions in the preset result matrix and the operator devices will be used to calculate the addition execution tasks of each element in the result matrix and distribute them to multiple operator devices.
[0101] Among them, the number of additions in the addition execution task of each element in the result matrix is composed of non-zero element values in the data formed after the winograd transformation results of multiple sub-tensors.
[0102] The summation operation at the same element position in the result matrix is performed by the same operator device.
[0103] Furthermore, in this embodiment, in order to enable multiple operator devices to complete the addition calculation of each element in the result matrix in a load-balanced manner, a mapping relationship between the element position in the result matrix and the operator device can be pre-set based on the element position of the non-zero element in the transformation matrix in all sub-tensors and the number of operator devices. In this mapping relationship, it is indicated which operator device is responsible for processing the sum operation at each element position in the result matrix, that is, the same operator device performs the sum operation at the same element position in the result matrix. Then, according to the mapping relationship between the element position in the result matrix and the operator device, and in accordance with the determined order of sum operation of the winograd transformation results of the multiple sub-tensors, the addition execution tasks for calculating each element in the result matrix are correspondingly allocated to each operator device, so that multiple operator devices can complete the addition calculation of each element in the result matrix in a load-balanced manner.
[0104] Step 204, completing the winograd convolution operation according to the transformation result of the data.
[0105] In this embodiment, the implementation of step 204 is similar to the implementation of step 103 in the first embodiment, and will not be described in detail here.
[0106] The winograd convolution operation method provided in this embodiment distributes the summation operation of the winograd transformation results of multiple sub-tensors to multiple operation sub-devices for execution according to the principle of load balancing of multiple operation sub-devices during the summation operation of the winograd transformation results of multiple sub-tensors. Not only is the summation operation processed by multiple operation sub-devices at the same time, thereby improving the processing efficiency of the chip, but it can also achieve load balancing when performing summation operations among multiple operation sub-devices.
[0107] Figure 4 A flowchart of a winograd convolution operation method provided by another embodiment of the present application. Figure 4 As shown, the winograd convolution operation method provided in this embodiment is Figure 3 On the basis of the illustrated embodiment, step 203 is further refined, and the winograd convolution operation method provided by this embodiment includes the following steps.
[0108] Step 301, during the transformation process of the winograd convolution operation, decompose the data in the winograd convolution operation into multiple sub-tensors.
[0109] Step 302, performing transformation operations on the multiple sub-tensors to obtain Winograd transformation results of the multiple sub-tensors.
[0110] In this embodiment, the implementation of step 301 and step 302 is the same as Figure 3 The implementation methods of step 201 and step 202 in the illustrated embodiment are similar and will not be described in detail here.
[0111] Step 303: Determine the number of addition executions required for performing a summation operation according to the winograd transformation results of the multiple sub-tensors.
[0112] Furthermore, in this embodiment, when the winograd convolution operation is implemented in hardware, each operator can perform one operation in one clock. After the forward transformation or inverse transformation of the data in the winograd convolution operation is decomposed by addition as shown in formula (1), the number of addition executions is much larger than the number of multiplication executions. In order to match the computing power of the addition execution task and the multiplication execution task during the winograd convolution operation, when the operator performing the addition task completes the addition task, the operator performing the multiplication task can also perform the matching multiplication task. Therefore, in this embodiment, it is necessary to determine the number of clocks required for the calculation of the summation operation, and determine the number of addition executions required for the calculation of the summation operation, so that the addition execution task can be completed by the operator within the number of clocks.
[0113] Specifically, in this embodiment, the transformation matrix corresponding to the winograd transformation results of the multiple sub-tensors includes elements with zero element values and elements with non-zero element values. When the winograd transformation results of the sub-tensors are summed, it is related to the non-zero element values in each transformation matrix. Therefore, the number of addition executions required for calculating the summation operation can be determined according to the number of non-zero element values in the transformation matrix corresponding to the winograd transformation results of the multiple sub-tensors and the scale of the result matrix.
[0114] As an optional implementation, step 303 includes the following steps:
[0115] Step 3031, determine the non-zero element value at the same element position in the data formed after the winograd transformation results of each sub-tensor as the addition number of the element position in the result matrix.
[0116] Step 3032, determining the number of summation operations at each element position according to the number of additions at each element position in the result matrix.
[0117] Step 3033, the sum of the number of sum operations at each element position is determined as the number of addition executions required to calculate the sum operation.
[0118] Among them, the data formed after the winograd transformation result of each sub-tensor can be in the form of a corresponding matrix.
[0119] Further, in this embodiment, when summing the winograd transformation results of multiple sub-tensors, the element values at the same position in the matrix formed after the winograd transformation results of each sub-tensor are added as the value of the element position in the result matrix. In the matrix formed after the winograd transformation results of the sub-tensors, the element values at multiple element positions are zero. Therefore, the non-zero element values at the same element position in the matrix formed after the winograd transformation results of each sub-tensor can be determined as the number of additions at the element position in the result matrix. The number of additions at each element position differs from the number of summation operations performed at the element position by 1, so the difference between the number of additions at each element position in the result matrix and 1 is determined as the number of summation operations at each element position. The sum of the number of summation operations at all element positions is determined as the number of addition executions required to calculate the summation operation.
[0120] For ease of understanding, the data size in the winograd convolution operation is 4×4 as an example. For the data d of size 4×4, the winograd transformation results of the corresponding multiple sub-tensors are expressed as shown in formula (2), and the matrix formed after the winograd transformation results of the multiple sub-tensors is expressed as shown in (3), and the corresponding result matrix S is also 4×4. That is, the result matrix includes 16 elements, which is expressed as shown in formula (4).
[0121] The corresponding winograd transformation results of multiple sub-tensors are expressed as:
[0122]
[0123]
[0124] The matrix formed by the winograd transformation results of the corresponding multiple sub-tensors is expressed as:
[0125]
[0126] The resulting matrix S is expressed as:
[0127]
[0128] First, the non-zero element value at the same element position in the matrix formed by the winograd transformation results of multiple sub-tensors is determined as the addition number of the element position in the result matrix. 00 The number of additions at is 4, coming from d 00 , d 02 , d 20 , d 22 The winograd transformation results of multiple sub-tensors of S. 01 The number of additions at position is 4, coming from d 01 , d 02 , d 21 , d 22 The winograd transformation results of multiple sub-tensors of . Similarly, the number of additions at other element positions of the result matrix is also 4.
[0129] Secondly, according to the number of additions at each element position in the result matrix, the number of summation operations at each element position is determined. Since the number of additions at each element position in the result matrix is 4, the number of summation operations at each element position is 3.
[0130] Finally, the sum of the number of sum operations at each element position is determined as the number of addition executions required to calculate the sum operation. Through calculation, it is determined that the number of sum operations at each element position in the result matrix is 3 times, and there are a total of 16 element positions in the result matrix, so the sum of the number of sum operations at each element position is 48 times, that is, the number of addition executions required to calculate the sum operation is determined to be 48 times.
[0131] Step 304, determining the number of clocks according to the size of the sub-tensor or the result matrix.
[0132] Among them, the result matrix is the matrix after the winograd convolution positive transformation or the winograd convolution inverse transformation. For example, in formula (4), the result matrix after the winograd convolution positive transformation is S.
[0133] When determining the number of clocks used for calculating the summation operation, in order to match the computing power of the addition execution task and the multiplication execution task, the number of clocks required for the multiplication execution task is determined as the number of clocks required for the addition execution task.
[0134] Optionally, if the winograd convolution is transformed into a winograd forward convolution, the number of clocks required to perform the multiplication task is the same as the size of the result matrix. If the winograd convolution is transformed into an inverse winograd transformation, the number of clocks required to perform the multiplication task is the same as the size of the transformation-to-bit multiplication matrix, and the size of the transformation-to-bit multiplication matrix is the same as the size of the corresponding sub-tensor, so if the winograd convolution is transformed into an inverse winograd transformation, the number of clocks required to perform the multiplication task is the same as the size of the sub-tensor.
[0135] That is, if the winograd convolution transformation is the winograd convolution positive transformation, the size of the result matrix is X × Y, then the number of clocks used for the summation operation is determined to be X × Y. If the winograd convolution transformation is the winograd inverse transformation, the size of the sub-tensor is X × Y, then the number of clocks used for the summation operation is determined to be X × Y.
[0136] As an optional implementation, the number of clocks is determined according to the size of the sub-tensor or the result matrix, including:
[0137] Determine the maximum size of the sub-tensors and result matrices, and determine the number of clocks based on the maximum size.
[0138] Among them, if the winograd convolution transformation is a winograd convolution forward transformation, the number of clocks required to perform the multiplication task is equal to the size of the result matrix corresponding to the neuron matrix or the weight matrix in the winograd convolution operation. If the winograd convolution transformation is an inverse winograd convolution transformation, the number of clocks required to perform the multiplication task is equal to the size of the matrix corresponding to the data of the transformation positional multiplication matrix, that is, the inverse winograd convolution transformation. Since the size of the matrix corresponding to the data of the inverse winograd convolution transformation is equal to the size of the sub-tensor of the data, if the winograd convolution transformation is an inverse winograd convolution transformation, the number of clocks required to perform the multiplication task is equal to the size of the sub-tensor of the data. Since the winograd convolution forward transformation and the winograd convolution inverse transformation are corresponding, the maximum size of the sub-tensor and the corresponding result matrix in a winograd convolution transformation can be determined as the number of clocks used to calculate the summation operation.
[0139] Step 305, determining the number of operator devices according to the number of clocks and the number of addition executions required for the summation operation.
[0140] Furthermore, in this embodiment, step 305 includes the following steps:
[0141] Step 3051, calculate the ratio of the number of addition executions required to calculate the sum operation to the number of clocks used.
[0142] Step 3052: Determine the number of operator devices according to the ratio, wherein the number of operator devices is greater than the ratio.
[0143] Further, if the number of addition executions required for the sum operation is expressed as NumAdd, the number of clocks used for the sum operation is expressed as NumClk, and the number of operator devices is expressed as NumUnit, the determined number of operator devices can be expressed as: NumUnit>NumAdd / NumClk.
[0144] For example, if the size of the result matrix is 4 × 4, the number of addition executions is 48 times, the number of clocks used for the summation operation is 16 times, and the number of operator devices is greater than 3, such as 4.
[0145] In this embodiment, the number of operator devices is set to a value greater than the ratio of the number of addition executions required for the sum operation to the number of clocks used, so that the addition calculation of each element in the result matrix can be completed within the number of clocks. If the number of operator devices is set to a value slightly greater than the ratio, all operator devices can be controlled to be in a processing state at each clock, and finally the addition calculation of each element in the result matrix can be completed within the number of clocks.
[0146] Step 306, according to the order of summing the winograd transformation results of the determined multiple sub-tensors, the mapping relationship between the element positions in the preset result matrix and the operator devices will be used to calculate the addition execution tasks of each element in the result matrix and allocated to multiple operator devices to complete the addition calculation of each element in the result matrix within the number of clocks.
[0147] Furthermore, in this embodiment, after determining the number of addition executions required for the summation operation, the number of clocks, and the number of operator devices, the order of summation operation is performed according to the determined winograd transformation results of the multiple sub-tensors, and the mapping relationship between the element positions in the preset result matrix and the operator devices is used to calculate the addition execution tasks of each element in the result matrix and distribute them to multiple operator devices, which can not only achieve load balancing when performing summation operations between multiple operator devices, but also ensure that the addition calculation of each element in the result matrix is completed within the number of clocks. If the number of operator devices is set to a value slightly larger than the ratio, all operator devices can also be controlled to be in a processing state at each clock.
[0148] Therefore, as an optional implementation, in step 306, the addition execution tasks for calculating each element in the result matrix are correspondingly distributed to multiple operator devices to complete the addition calculation of each element in the result matrix within a predetermined number of clocks, including the following steps:
[0149] Step 3061: In the first N consecutive clocks, one of the addition numbers used to calculate the position of each element in the result matrix is input to the corresponding operator device to perform initial assignment on each element in the result matrix.
[0150] Step 3062: In the remaining clocks, the remaining addition numbers used to calculate the positions of each element in the result matrix are input to the corresponding operator devices for addition calculation until the values of each element in the result matrix are obtained.
[0151] Among them, all the operation sub-devices are in the processing state at each clock.
[0152] Furthermore, in this embodiment, the number of preceding continuous clocks required for inputting one of the addition numbers for calculating the position of each element in the result matrix into the corresponding operator device is first determined according to the number of elements in the result matrix and the number of operator devices. The number of preceding continuous clocks N is the ratio of the number of elements in the result matrix to the number of operator devices.
[0153] If the size of the resulting matrix is X × Y, then the number of elements in the resulting matrix is also X × Y, the number of operator devices is t, then the number of previous continuous clocks N = (X× Y) / t.
[0154] Secondly, in each of the first N consecutive clocks, one of the addition numbers used to calculate the n element positions in the result matrix is input to the corresponding operator device to perform initial assignments on the elements of the n element positions in the result matrix. The number of n is the same as the number of operator devices. That is, each operator performs initial assignments on the corresponding elements in the result matrix in the same clock. After the first N consecutive clocks, multiple operator devices perform initial assignments on all elements in the result matrix.
[0155] It is understandable that each operator device may include registers, and the number of registers may be the same as the number of element positions corresponding to the operator device, so that each register stores the addition number of the corresponding element position or the result of each addition operation. After the first N consecutive clocks, the register corresponding to each operator device stores one of the addition numbers of the corresponding element position.
[0156] It is worth noting that one of the addends input into the operator device for calculating the position of each element in the result matrix comes from the non-zero elements in the matrix formed after the winograd transformation results of the multiple sub-tensors in formula (3). When one of the addends used to calculate the position of each element in the result matrix is input into the corresponding operator device, the input order of the matrix formed after the winograd transformation results of the multiple sub-tensors needs to ensure that each operator device can be in the initial assignment state for the elements at the corresponding element positions in the result matrix in each clock.
[0157] Finally, in each clock of the remaining clock, each operator receives a remaining addend for calculating the corresponding element position in the result matrix, and the operator performs an addition operation on the remaining addend received this time and the addend stored in the register, and stores the addition result in the corresponding register again. Until the operator receives the last remaining addend for calculating the corresponding element position in the result matrix, the last remaining addend is added to the addition result in the register, and the element value of each element position in the result matrix is obtained.
[0158] It is worth noting that the remaining addition numbers input into the operator device for calculating the positions of each element in the result matrix also come from the non-zero elements in the matrix formed after the winograd transformation results of multiple sub-tensors. Therefore, when the remaining addition numbers for calculating the positions of each element in the result matrix are input into the corresponding operator device, the input order of the matrix formed after the winograd transformation results of multiple sub-tensors also needs to ensure that each operator device can be in a state of adding elements at the corresponding element positions in the result matrix in each clock.
[0159] In this optional implementation, at each clock of the first N consecutive clocks, all the operator devices are in a state of performing initial assignment on the elements at the corresponding element positions in the result matrix. At each clock of the remaining clocks, all the operator devices are in a state of performing addition operations on the elements at the corresponding element positions in the result matrix.
[0160] In this embodiment, when the addition execution tasks for calculating each element in the result matrix are allocated to a plurality of operator devices to complete the addition calculation of each element in the result matrix within a predetermined number of clocks, one of the addition numbers for calculating the position of each element in the result matrix is input to the corresponding operator device in the first N consecutive clocks to perform initial assignment of each element in the result matrix; in the remaining clocks, the remaining addition numbers for calculating the position of each element in the result matrix are input to the corresponding operator device for addition calculation until the value of each element in the result matrix is obtained. This provides a first method for controlling each operator device to be in a processing state at each clock, and for the same operator device to perform addition operations at the same element position in the result matrix, which can ensure that multiple operator devices are in a processing state in all clocks, further improve the execution speed of the chip when performing winograd transformation, and further improve the processing efficiency of the chip.
[0161] As another optional implementation, step 306 includes the following steps:
[0162] Step 306a, divide the number of clocks into a plurality of clock segments, divide the positions of each element in the result matrix into a plurality of element position sets, and the clock segments correspond to the element position sets one by one.
[0163] Optionally, in this embodiment, the number of clocks is evenly divided into multiple clock segments according to the number of operator devices. × The result matrix of 4 has 16 clocks and 4 operator devices, so the divided clock segments are 4.
[0164] Optionally, in this embodiment, multiple element positions in the result matrix are evenly divided into multiple element position sets according to the number of elements in the result matrix and the number of operator devices. In each of the divided element position sets, the operator devices corresponding to each element position need to be different to ensure that all operator devices are in a processing state at each clock.
[0165] Since the number of elements in the result matrix is the same as the number of clocks, the number of element position sets is the same as the number of clock segments. And the clock segments correspond to element position sets one by one, that is, multiple operator devices can complete the addition operation of each element position in the corresponding element position set within one clock segment.
[0166] Step 306b, for each element position set, within the first clock of the corresponding clock segment, one of the addends used to calculate all the element positions in the element position set is input into the corresponding operator device to perform initial assignment of values to each element position in the element position set; within the remaining clocks of the clock segment, the remaining addends used to calculate all the element positions in each element position set are input into the corresponding operator device for addition calculation until the values of each element in the element position set are obtained.
[0167] Among them, all the operation sub-devices are in the processing state at each clock.
[0168] Furthermore, in this embodiment, when the addition execution tasks for calculating each element in the result matrix are allocated to each operator device, the allocation is performed according to the clock segment and the element position set corresponding to the clock segment.
[0169] It should be noted that each operator device includes registers, and the number of registers is the same as the number of element positions corresponding to the operator device.
[0170] Specifically, in the first clock segment, the calculation tasks of the first element position set are correspondingly assigned to each operator device. In the first clock of the first clock segment, one of the addends used to calculate all the element positions in the element position set is input to the corresponding operator device, and after the operator device performs initial assignment of each element position in the element position set, the addend is stored in the first register. In each clock of the remaining clocks of the clock segment, a remaining addend used to calculate all the element positions in each element position set is input to the corresponding operator device, and the stored addend is obtained from the first register, and the addition calculation is performed with the remaining addend, and so on, until each operator device completes the addition operation of the corresponding element position in the element position set. Then, the addition operation result of the corresponding element position in the first element position set is stored in the first register of each operator device.
[0171] Similarly, in the Mth clock segment, the calculation tasks of the Mth element position set are correspondingly assigned to each operator device. In the first clock of the Mth clock segment, one of the addends used to calculate all the element positions in the element position set is input to the corresponding operator device, and after the initial assignment of each element position in the element position set, the addend is stored in the corresponding Mth register. In each clock of the remaining clocks of the clock segment, a remaining addend used to calculate all the element positions in each element position set is input to the corresponding operator device, and the stored addend is obtained from the Mth register, and the addition calculation is performed with the remaining addend, and so on, until each operator device completes the addition operation of the corresponding element position in the element position set. Then, the addition operation result of the corresponding element position in the Mth element position set is stored in the Mth register of each operator device.
[0172] Finally, each element value in each element position set is taken out from the corresponding register, and the result matrix is obtained according to the mapping relationship between the element position in the preset result matrix and the operator device.
[0173] The winograd convolution operation method provided in this embodiment, when the addition execution task for calculating each element in the result matrix is allocated to a plurality of operation sub-devices to complete the addition calculation of each element in the result matrix within a predetermined number of clocks, the number of clocks is divided into a plurality of clock segments, and the position of each element in the result matrix is divided into a plurality of element position sets, the clock segments correspond to the element position sets one by one, and for each element position set, within the first clock of the corresponding clock segment, one of the addition numbers for calculating all the element positions in the element position set is input to the corresponding operation sub-device to calculate the addition of each element position in the element position set. The initial value is assigned to the position; in the remaining clocks of the clock segment, the remaining addition number used to calculate all element positions in each element position set is input into the corresponding operator device for addition calculation until the value of each element in the element position set is obtained. A second method for controlling each operator device to be in a processing state in each clock, and the same operator device to perform addition operations at the same element position in the result matrix is provided, which can ensure that multiple operator devices are in a processing state in all clocks, further improve the execution speed of the chip in the winograd transformation, and further improve the processing efficiency of the chip.
[0174] The following is a description of the process of summing the winograd transformation results of multiple sub-tensors by multiple operator devices, using the neuron matrix, weight matrix and transformation matrix in the winograd convolution operation. × 4,4 × 3,3 × 4,3× 3, 4 × 1,1 × 4,3 × 1,1 × 3, the scale of the weight matrix is 3 × 3,3 × 2,2 × 3,2 × 2,3 × 1,1 × 3,2 × 1,1 × 2, and the size of the transformation matrix is 2 × 2,2 × 2,2 × 2,2 × 2,2 × 1,1 × 2,2 × 1,1 × 2.
[0175] For a neuron matrix size of 4 × 4, the neuron matrix d and the winograd convolution positive transformation matrix B T The matrix multiplication between and B is shown in formula (5):
[0176]
[0177] The Winograd transformation results of multiple sub-tensors are obtained by transforming them, which can be expressed as shown in the above formula (2):
[0178] Since it is a winograd convolution forward transformation of the neuron matrix, the number of clocks used for the summation operation is determined by the size of the result matrix, which is 4. × 4=16. When determining the number of addition executions required for calculating the sum operation based on the winograd transformation results of multiple sub-tensors, since the number of additions at each element position in the result matrix is 4, the number of summation operations at each element position in the result matrix is 3, and the total number of summation operations at each element position is 48, so the number of addition executions required for calculating the sum operation is 48.
[0179] The above method of determining the number of additions required to compute the sum is essentially the same as the difference between the number of non-zero elements in the matrix formed by the winograd transformation of all multiple sub-tensors and the size of the result matrix. So for a result matrix size of 4 × For a matrix of size 4, the number of addition operations required to calculate the sum is determined to be: 1*4+3*8+9*4-4*4=48.
[0180] Among them, 1*4 means that the number of matrices formed by the winograd transformation results of a sub-tensor with only one non-zero element is 4, 3*8 means that the number of matrices formed by the winograd transformation results of a sub-tensor with three non-zero elements is 8, 9*4 means that the number of matrices formed by the winograd transformation results of a sub-tensor with 9 non-zero elements is 4, and 4*4 is the size of the result matrix.
[0181] According to the number of clocks and the number of additions required, it is determined that the number of operator devices is greater than 3. × For a matrix of 4, 4 operator devices can complete the addition calculation of each element in the result matrix within 16 clocks, so the number of operator devices is 4. The four operator devices are numbered 0-3 respectively, and the operator device corresponding to each element position in the result matrix is shown in Table 1.
[0182] Table 1: The size of the neural matrix is 4×4, and the corresponding waybill units of each element in the result matrix of size 4×4
[0183] 0 2 1 3 2 2 1 1 3 3 0 0 1 3 0 2
[0184] For a neuron matrix size of 4 × 4, according to the mapping relationship between the element position in the result matrix and the operator device in Table 1, the same operator device can perform the sum operation at the same element position in the result matrix, and when the addition execution task for calculating each element in the result matrix is assigned to each operator device, the order of the matrix formed by the winograd transformation result of the sub-tensor is read as d 11 →d 01 →d 10 →d 00 →d 12 →d 03 →d 02 →d 13 →d 22 →d 30 →d 21 →d 31 →d 23 →d 33 →d 23 →d 32 , to ensure that all operating sub-devices are in processing state at each clock.
[0185] Then, in the 16 clocks, the first 4 consecutive clocks of the 4 operators will input one of the addition numbers used to calculate the position of each element in the result matrix into the corresponding operator device to perform initial assignment of each element in the result matrix. These 4 consecutive clock operator devices are in the initial assignment state. In the remaining 12 clocks, the remaining addition numbers used to calculate the position of each element in the result matrix are input into the corresponding operator device for addition calculation until the value of each element in the result matrix is obtained. The operator device is in the summation operation state in the remaining 12 clocks.
[0186] It is understandable that after obtaining the values of each element in the result matrix, the values of each element may be sequentially output to other components, such as a main processing memory, so that other components can store the result matrix.
[0187] For a neuron matrix size of 4 × For example, the matrix d of 3 is the matrix B of the neuron matrix and the winograd convolution positive transformation T The matrix multiplication between and B is shown in formula (6):
[0188]
[0189] The Winograd transformation results of multiple sub-tensors are obtained by transforming them, which can be expressed as shown in the above formula (7):
[0190]
[0191] Since it is a winograd convolution forward transformation of the neuron matrix, the number of clocks used for the summation operation is determined by the size of the result matrix, which is 4. × 3 = 12. The number of additions required to calculate the sum is determined to be: 1*4+3*6+9*2-4*3=28.
[0192] Among them, 1*4 means that the number of matrices formed by the winograd transformation results of a sub-tensor with only one non-zero element is 4, 3*6 means that the number of matrices formed by the winograd transformation results of a sub-tensor with three non-zero elements is 6, 9*2 means that the number of matrices formed by the winograd transformation results of a sub-tensor with 9 non-zero elements is 2, and 4*3 is the size of the result matrix.
[0193] According to the number of clocks and the required number of addition executions, the number of operator devices is determined to be 4. The four operator devices are numbered 0-3 respectively, and the operator device corresponding to each element position in the result matrix is shown in Table 2.
[0194] Table 2: Neuron matrix size is 4 ×3. Operator device corresponding to each element position in the result matrix of size 4×3
[0195] 0 0 0 1 1 1 2 2 2 3 3 3
[0196] For a neuron matrix size of 4 × 3, according to the mapping relationship between the element position in the result matrix and the operator device in Table 2, the same operator device can perform the sum operation at the same element position in the result matrix, and when the addition execution task for calculating each element in the result matrix is assigned to each operator device, the order of the matrix formed by the winograd transformation result of the sub-tensor is read as d 11 →d 01 →d 10 →d 00 →d 12 →d 02 →d 21 →d 31 →d 20 →d 30 →d 22 →d 33 , to ensure that all operating sub-devices are in processing state at each clock.
[0197] Then, in the 12 clocks, the first 3 consecutive clocks of the 4 operators input one of the addition numbers used to calculate the position of each element in the result matrix into the corresponding operator device to perform initial assignment of each element in the result matrix. The operator device is in the initial assignment state for these 3 consecutive clocks. In the remaining 9 clocks, the remaining addition numbers used to calculate the position of each element in the result matrix are input into the corresponding operator device for addition calculation until the value of each element in the result matrix is obtained. The operator device is in the summation operation state for 9 clocks.
[0198] It is understandable that after obtaining the values of each element in the result matrix, the element values may be sequentially output to other components, such as a main processing memory, so that the result matrix can be stored by other components.
[0199] For a neuron matrix size of 3 × For example, the matrix d of 4 is the matrix B of the neuron matrix and the winograd convolution positive transformation T The matrix multiplication between and B is shown in formula (8):
[0200]
[0201] The Winograd transformation results of multiple sub-tensors are obtained by transforming them, which can be expressed as shown in the above formula (9):
[0202]
[0203]
[0204] Since it is a winograd convolution forward transformation of the neuron matrix, the number of clocks used for the summation operation is determined by the size of the result matrix, which is 3. × 4 = 12. The number of additions required to calculate the sum is determined to be: 1*4+3*6+9*2-3*4=28.
[0205] Among them, 1*4 means that the number of matrices formed by the winograd transformation results of a sub-tensor with only one non-zero element is 4, 3*6 means that the number of matrices formed by the winograd transformation results of a sub-tensor with three non-zero elements is 6, 9*2 means that the number of matrices formed by the winograd transformation results of a sub-tensor with 9 non-zero elements is 2, and 3*4 is the size of the result matrix.
[0206] According to the number of clocks and the required number of addition executions, the number of operator devices is determined to be 4. The four operator devices are numbered 0-3 respectively, and the operator device corresponding to each element position in the result matrix is shown in Table 3.
[0207] Table 3: Neuron matrix size is 3 × 4, scale 3 × The operator device corresponding to each element position in the result matrix of 4
[0208] 0 1 2 3 0 1 2 3 0 1 2 3
[0209] For a neuron matrix size of 3 × 4, according to the mapping relationship between the element position in the result matrix and the operator device in Table 3, the same operator device can perform the sum operation at the same element position in the result matrix, and when the addition execution task for calculating each element in the result matrix is assigned to each operator device, the order of the matrix formed by the winograd transformation result of the sub-tensor is read as d 11 →d 10 →d 01 →d 00 →d 21 →d 20 →d 12 →d 13 →d 02 →d 03 →d 22 →d 23 , to ensure that all operator devices are in the processing state at each clock, and the same operator device performs the summation operation at the same element position in the result matrix.
[0210] Then, in the 12 clocks, the first 3 consecutive clocks of the 4 operators input one of the addition numbers used to calculate the position of each element in the result matrix into the corresponding operator device to perform initial assignment of each element in the result matrix. The operator device is in the initial assignment state for these 3 consecutive clocks. In the remaining 9 clocks, the remaining addition numbers used to calculate the position of each element in the result matrix are input into the corresponding operator device for addition calculation until the value of each element in the result matrix is obtained. The operator device is in the summation operation state for 9 clocks.
[0211] It is understandable that after obtaining the values of each element in the result matrix, the element values may be sequentially output to other components, such as a main processing memory, so that the result matrix can be stored by other components.
[0212] For a neuron matrix size of 3 × For example, the matrix d of 3 is the matrix B of the neuron matrix and the winograd convolution positive transformation T The matrix multiplication between and B is shown in formula (10):
[0213]
[0214] The Winograd transformation results of multiple sub-tensors are obtained by transforming them, which can be expressed as shown in the above formula (11):
[0215]
[0216]
[0217] Since it is a winograd convolution forward transformation of the neuron matrix, the number of clocks used for the summation operation is determined by the size of the result matrix, which is 3. × 3 = 9. The number of additions required to calculate the sum is determined to be: 1*4+3*4+9*1-3*3=16.
[0218] Among them, 1*4 means that the number of matrices formed by the winograd transformation results of a sub-tensor with only one non-zero element is 4, 3*4 means that the number of matrices formed by the winograd transformation results of a sub-tensor with three non-zero elements is 4, 9*1 means that the number of matrices formed by the winograd transformation results of a sub-tensor with 9 non-zero elements is 1, and 3*3 is the size of the result matrix.
[0219] According to the number of clocks and the required number of addition executions, the number of operator devices is determined to be 4. The four operator devices are numbered 0-3 respectively, and the operator device corresponding to each element position in the result matrix is shown in Table 4.
[0220] Table 4: Neuron matrix size is 3 × 3, scale 3 × The operator device corresponding to each element position in the result matrix of 3
[0221] 0 2 3 3 0 1 1 0 2
[0222] For a neuron matrix size of 3 × 3, according to the mapping relationship between the element position in the result matrix and the operator device in Table 4, the same operator device can perform the sum operation at the same element position in the result matrix, and when the addition execution task for calculating each element in the result matrix is assigned to each operator device, the order of the matrix formed by the winograd transformation result of the sub-tensor is read as d 11 →d 01 →d 10 →d 00 →d 12 →d 02 →d 21 →d 31 →d 20 .
[0223] In this neuron matrix the size is 3 × 3 in the winograd convolution forward transform, compared to the neuron matrix size of 4 × 4's winograd convolution forward transform, the number of operator devices is equal, but the number of addition executions is significantly reduced, so it is not necessary for all operator devices to be in the processing state at each clock to complete the addition calculation of each element in the result matrix within 9 clocks. Therefore, in the mapping relationship of Table 4, the operator device numbered 0 corresponds to 3 element positions in the result matrix, and the operator devices numbered 1, 2, and 3 correspond to 2 element positions in the result matrix. However, load balancing can also be achieved when the four operator devices perform summation operations.
[0224] It is understandable that after obtaining the values of each element in the result matrix, the element values may be sequentially output to other components, such as a main processing memory, so that the result matrix can be stored by other components.
[0225] For a neuron matrix size of 4 × 1, the matrix d is the matrix B of the neuron matrix and the winograd convolution positive transformation T The matrix multiplication between and B is shown in formula (12):
[0226]
[0227] The Winograd transformation results of multiple sub-tensors are obtained by transforming them, which can be expressed as shown in the above formula (13):
[0228]
[0229] Since it is a winograd convolution forward transformation of the neuron matrix, the number of clocks used for the summation operation is determined by the size of the result matrix, which is 4. × 1 = 4. The number of additions required to calculate the sum is determined to be: 1*2+3*2-4*1=4.
[0230] Among them, 1*2 means that the number of matrices formed by the winograd transformation results of a sub-tensor with only one non-zero element is 2, 3*2 means that the number of matrices formed by the winograd transformation results of a sub-tensor with three non-zero elements is 2, and 4*1 is the size of the result matrix.
[0231] According to the number of clocks and the required number of addition executions, the number of operator devices is determined to be 2. The two operator devices are numbered 0-1 respectively, and the operator device corresponding to each element position in the result matrix is shown in Table 5.
[0232] Table 5: Neuron matrix size is 4 × 1, scale 4 × 1 The operator device corresponding to each element position in the result matrix
[0233]
[0234]
[0235] For a neuron matrix size of 4 × 1, according to the mapping relationship between the element position in the result matrix and the operator device in Table 5, the sum operation at the same element position in the result matrix can be performed by the same operator device, and when the addition execution tasks for calculating each element in the result matrix are correspondingly assigned to each operator device, the order of the matrix formed by the winograd transformation result of the read sub-tensor is d0→d1→d2→d3, so as to ensure that all the operator devices are in the processing state at each clock, and the sum operation at the same element position in the result matrix is performed by the same operator device.
[0236] Then, in the first two consecutive clocks of the two operators, one of the addition numbers used to calculate the position of each element in the result matrix is input into the corresponding operator device within four clocks, so as to perform initial assignment on each element in the result matrix. The operator device is in the initial assignment state during these two consecutive clocks. In the remaining two clocks, the remaining addition numbers used to calculate the position of each element in the result matrix are input into the corresponding operator device for addition calculation until the value of each element in the result matrix is obtained. The operator device is in the summation operation state during these two clocks.
[0237] For a neuron matrix with a size of 1 × For example, the matrix d of 4 is 4, and the neuron matrix size is 4 × The winograd convolution operation of the winograd convolution transformation of the matrix d of 1 is similar and will not be repeated here.
[0238] For a neuron matrix size of 3 × 1, the matrix d is the matrix B of the neuron matrix and the winograd convolution positive transformation T The matrix multiplication between and B is shown in formula (14):
[0239]
[0240] The Winograd transformation results of multiple sub-tensors are obtained by transforming them, which can be expressed as shown in the above formula (15):
[0241]
[0242] Since it is a winograd convolution forward transformation of the neuron matrix, the number of clocks used for the summation operation is determined by the size of the result matrix, which is 3. × 1 = 3. The number of additions required to calculate the sum is determined to be: 1*2+3*1-3*1=2.
[0243] Among them, 1*2 means that the number of matrices formed by the winograd transformation results of a sub-tensor with only one non-zero element is 2, 3*1 means that the number of matrices formed by the winograd transformation results of a sub-tensor with three non-zero elements is 1, and 3*1 is the scale of the result matrix.
[0244] According to the number of clocks and the required number of addition executions, the number of operator devices is determined to be 3, and the 3 operator devices are numbered 0-2 respectively. The operator device corresponding to each element position in the result matrix is shown in Table 6.
[0245] Table 6: Neuron matrix size is 3 × 1, scale 3 ×1 The operator device corresponding to each element position in the result matrix
[0246] 0 1 2
[0247] For a neuron matrix size of 3 × 1, according to the mapping relationship between the element position in the result matrix and the operator device in Table 6, the sum operation at the same element position in the result matrix can be performed by the same operator device, and when the addition execution task for calculating each element in the result matrix is correspondingly assigned to each operator device, the order of the matrix formed by the winograd transformation result of the sub-tensor is read as d0→d1→d2.
[0248] In this neuron matrix the size is 3 × In the winograd convolution operation method of the winograd convolution forward transform of 1, since the number of operator devices is 3, there are 3 clocks, and the number of summation operations is only 2 times, it is not necessary for all operator devices to be in the processing state in each clock. In the last clock, the three operator devices can be in the idle state.
[0249] For a neuron matrix with a size of 1 × For example, the matrix d of 3 is 3, and the neuron matrix size is 3 × The process of performing winograd convolution transformation on the matrix d of 1 is similar and will not be described here.
[0250] For a 3×3 weight matrix, the weight matrix g is the matrix G of the winograd convolution positive transformation T The matrix multiplication between and G is shown in formula (16):
[0251]
[0252] The Winograd transformation results of multiple sub-tensors are obtained by transforming them, which can be expressed as shown in the above formula (17):
[0253]
[0254] Since it is a winograd convolution forward transformation of the weight matrix, the number of clocks used for the calculation and summation operation is determined by the size of the result matrix to be 4×4=16. The number of addition executions required for the calculation and summation operation is determined to be: 4*1+6*4+9*4-4*4=48.
[0255] Among them, 4*1 means that the number of matrices formed by the winograd transformation results of sub-tensors with 4 non-zero elements is 1, 6*4 means that the number of matrices formed by the winograd transformation results of sub-tensors with 6 non-zero elements is 4, 9*4 means that the number of matrices formed by the winograd transformation results of sub-tensors with 9 non-zero elements is 4, and 4*4 is the scale of the result matrix.
[0256] According to the number of clocks and the required number of addition executions, the number of operator devices is determined to be 4, and the 4 operator devices are numbered 0-3 respectively. The operator device corresponding to each element position in the result matrix is shown in Table 7.
[0257] Table 7: Weight matrix size is 3 × 3, scale 4 × The operator device corresponding to each element position in the result matrix of 4
[0258]
[0259]
[0260] For a weight matrix size of 3 × 3, according to the mapping relationship between the element position in the result matrix and the operator device in Table 7, the same operator device can perform the sum operation at the same element position in the result matrix, and when the addition execution task for calculating each element in the result matrix is assigned to each operator device, the order of the matrix formed by the winograd transformation result of the sub-tensor is read as g 11 → 01 → 10 → 12 → 21 → 00 → 02 → 02 → 20 → 20 → 22 → 22 .
[0261] It is understandable that after obtaining the values of each element in the result matrix, the element values may be sequentially output to other components, such as a main processing memory, so that the result matrix can be stored by other components.
[0262] For a weight matrix of size 3×2, the weight matrix g and the winograd convolution positive transformation matrix G T The matrix multiplication between and G is shown in formula (18):
[0263]
[0264] The Winograd transformation results of multiple sub-tensors are obtained by transforming them, which can be expressed as shown in the above formula (19):
[0265]
[0266] Since it is a winograd convolution forward transformation of the weight matrix, the number of clocks used for the calculation and summation operation is determined by the size of the result matrix to be 4×3=12. The number of addition executions required for the calculation and summation operation is determined to be: 4*1+6*4-4*3=20.
[0267] Among them, 4*1 means that the number of matrices formed by the winograd transformation results of sub-tensors with four non-zero elements is 1, 6*4 means that the number of matrices formed by the winograd transformation results of sub-tensors with six non-zero elements is 4, and 4*3 is the size of the result matrix.
[0268] According to the number of clocks and the required number of addition executions, the number of operator devices is determined to be 4, and the 4 operator devices are numbered 0-3 respectively. The operator device corresponding to each element position in the result matrix is shown in Table 8.
[0269] Table 8: Operator devices corresponding to the positions of the elements in the weight matrix of size 3×2 and size 4×3
[0270] 0 2 1 1 0 1 2 3 2 3 1 2
[0271] For a weight matrix size of 3 × 2, according to the mapping relationship between the element position in the result matrix and the operator device in Table 8, the same operator device can perform the sum operation at the same element position in the result matrix, and when the addition execution task for calculating each element in the result matrix is assigned to each operator device, the order of the matrix formed by the winograd transformation result of the sub-tensor is read as g 00 → 01 → 20 → 21 → 10 → 01 .
[0272] In this weight matrix, the size is 3 × In the winograd convolution forward transform of 2, it is not necessary for all operator devices to be in the processing state at each clock, so in the mapping relationship of Table 8, the operator devices numbered 0 and 3 correspond to 2 element positions in the result matrix, and the operator devices numbered 1 and 2 correspond to 4 element positions in the result matrix.
[0273] It is understandable that after obtaining the values of each element in the result matrix, the element values may be sequentially output to other components, such as a main processing memory, so that the result matrix can be stored by other components.
[0274] For a weight matrix of size 2×3, the weight matrix g and the winograd convolution positive transformation matrix G T The matrix multiplication between and G is shown in formula (20):
[0275]
[0276] The Winograd transformation results of multiple sub-tensors are obtained by transforming them, which can be expressed as shown in the above formula (21):
[0277]
[0278] Since it is a winograd convolution forward transformation of the weight matrix, the number of clocks used for the calculation and summation operation is determined by the size of the result matrix to be 3×4=12. The number of addition executions required for the calculation and summation operation is determined to be: 4*1+6*4-3*4=20.
[0279] Among them, 4*1 means that the number of matrices formed by the winograd transformation results of sub-tensors with four non-zero elements is 1, 6*4 means that the number of matrices formed by the winograd transformation results of sub-tensors with six non-zero elements is 4, and 3*4 is the size of the result matrix.
[0280] According to the number of clocks and the required number of addition executions, the number of operator devices is determined to be 4, and the 4 operator devices are numbered 0-3 respectively. The operator device corresponding to each element position in the result matrix is shown in Table 9.
[0281] Table 9: Operator devices corresponding to the positions of the elements in the weight matrix of size 2×3 and size 3×4
[0282] 1 1 2 2 2 0 3 1 0 1 2 3
[0283] For the winograd convolution forward transformation with a weight matrix size of 2×3, according to the mapping relationship between the element position in the result matrix and the operator device in Table 9, the same operator device can perform the sum operation at the same element position in the result matrix. When the addition execution task for calculating each element in the result matrix is assigned to each operator device, the order of reading the matrix formed by the winograd transformation result of the sub-tensor is g 00 → 02 → 10 → 12 →01 → 11 .
[0284] In the winograd convolution operation method of the winograd convolution forward transform with a weight matrix size of 3×2, it is not necessary for all operator devices to be in the processing state at each clock, so in the mapping relationship of Table 9, the operator devices numbered 0 and 3 correspond to 2 element positions in the result matrix, and the operator devices numbered 1 and 2 correspond to 4 element positions in the result matrix.
[0285] It is understandable that after obtaining the values of each element in the result matrix, the element values may be sequentially output to other components, such as a main processing memory, so that the result matrix can be stored by other components.
[0286] For a 2×2 weight matrix, the weight matrix g is the matrix G of the winograd convolution positive transformation T The matrix multiplication between and G is shown in formula (22):
[0287]
[0288] The Winograd transformation results of multiple sub-tensors are obtained by transforming them, which can be expressed as shown in the above formula (23):
[0289]
[0290] Since it is a winograd convolution forward transformation of the weight matrix, the number of clocks used for the calculation and summation operation is determined by the size of the result matrix to be 3×3=9. The number of addition executions required for the calculation and summation operation is determined to be: 4*4-3*3=7.
[0291] Among them, 4*4 means that the number of matrices formed by the winograd transformation results of the sub-tensors with four non-zero elements is 4, and 3*3 is the size of the result matrix.
[0292] According to the number of clocks and the required number of addition executions, the number of operator devices is determined to be 4, and the 4 operator devices are numbered 0-3 respectively. The operator device corresponding to each element position in the result matrix is shown in Table 10.
[0293] Table 10: Operator devices corresponding to the positions of the elements in the weight matrix of size 2×2 and size 3×3
[0294]
[0295]
[0296] For the winograd convolution forward transformation with a weight matrix size of 2×2, according to the mapping relationship between the element position in the result matrix and the operator device in Table 10, the sum operation at the same element position in the result matrix can be performed by the same operator device. When the addition execution task for calculating each element in the result matrix is assigned to each operator device, the order of reading the matrix formed by the winograd transformation result of the sub-tensor is g 00 → 01 → 10 → 11 .
[0297] In the winograd convolution operation method of the winograd convolution forward transform with a weight matrix size of 2×2, it is not necessary for all operator devices to be in the processing state at each clock, so in the mapping relationship of Table 10, the operator device numbered 0 corresponds to 1 element position in the result matrix, the operator devices numbered 1 and 2 correspond to 2 element positions in the result matrix, and the operator device numbered 3 corresponds to 4 element positions in the result matrix.
[0298] It is understandable that after obtaining the values of each element in the result matrix, the element values may be sequentially output to other components, such as a main processing memory, so that the result matrix can be stored by other components.
[0299] For a weight matrix of size 3×1, the weight matrix g and the winograd convolution positive transformation matrix G T The matrix multiplication between and G is shown in formula (24):
[0300]
[0301] The Winograd transformation results of multiple sub-tensors are obtained by transforming them, which can be expressed as shown in the above formula (25):
[0302]
[0303] Since it is a winograd convolution forward transformation of the weight matrix, the number of clocks used for the calculation and summation operation is determined by the size of the result matrix to be 4×1=4. The number of addition executions required for the calculation and summation operation is determined to be: 2*1+3*2-4*1=4.
[0304] Among them, 2*1 means that the number of matrices formed by the winograd transformation results of sub-tensors with 2 non-zero elements is 1, 3*2 means that the number of matrices formed by the winograd transformation results of sub-tensors with 3 non-zero elements is 2, and 4*1 is the scale of the result matrix.
[0305] According to the number of clocks and the required number of addition executions, the number of operator devices is determined to be 2, and the two operator devices are numbered 0-1 respectively. The operator device corresponding to each element position in the result matrix is shown in Table 11.
[0306] Table 11: Operator devices corresponding to the positions of each element in the weight matrix of size 3×1 and size 4×1
[0307] 0 0 1 1
[0308] For the winograd convolution forward transformation with a weight matrix size of 3×1, according to the mapping relationship between the element position in the result matrix and the operator device in Table 11, the same operator device can perform the sum operation at the same element position in the result matrix, and when the addition execution task for calculating each element in the result matrix is assigned to each operator device, the order of the matrix formed by the winograd transformation result of the sub-tensor is read as g0→g1→g2. This ensures that all operator devices are in the processing state at each clock.
[0309] It is understandable that after obtaining the values of each element in the result matrix, the element values may be sequentially output to other components, such as a main processing memory, so that the result matrix can be stored by other components.
[0310] For a matrix g with a weight matrix size of 1×3, the winograd convolution operation for the winograd convolution transformation is similar to that for a matrix g with a weight matrix size of 3×1, and will not be repeated here.
[0311] For a weight matrix of size 2×1, the weight matrix g and the winograd convolution positive transformation matrix G T The matrix multiplication between and G is shown in formula (26):
[0312]
[0313] The Winograd transformation results of multiple sub-tensors are obtained by transforming them, which can be expressed as shown in the above formula (27):
[0314]
[0315] Since it is a winograd convolution forward transformation of the weight matrix, the number of clocks used for the calculation and summation operation is determined by the size of the result matrix to be 3×1=3. The number of addition executions required for the calculation and summation operation is determined to be: 2*2-3*1=1.
[0316] Among them, 2*2 means that the number of matrices formed by the winograd transformation results of the sub-tensors with 2 non-zero elements is 2, and 3*1 is the size of the result matrix.
[0317] According to the number of clocks and the required number of addition executions, the number of operator devices is determined to be 2, and the two operator devices are numbered 0-1 respectively. The operator device corresponding to each element position in the result matrix is represented as shown in Table 12.
[0318] Table 12: Operator devices corresponding to the positions of the elements in the weight matrix of size 2×1 and size 3×1
[0319] 1 0 1
[0320] For the winograd convolution forward transform with a weight matrix size of 2×1, according to the mapping relationship between the element position in the result matrix and the operator device in Table 12, the same operator device can perform the summation operation at the same element position in the result matrix, and when the addition execution tasks for calculating each element in the result matrix are correspondingly assigned to each operator device, the order of the matrix formed by the winograd transformation results of the sub-tensors is read as g0→g1.
[0321] In the winograd convolution forward transform with a weight matrix size of 2×1, it is not necessary for all operator devices to be in the processing state at each clock, so in the mapping relationship of Table 12, the operator device numbered 0 corresponds to 1 element position in the result matrix, and the operator device numbered 1 corresponds to 2 element positions in the result matrix.
[0322] It is understandable that after obtaining the values of each element in the result matrix, the element values may be sequentially output to other components, such as a main processing memory, so that the result matrix can be stored by other components.
[0323] For a weight matrix g with a scale of 1×2, the winograd convolution transformation is similar to that for a weight matrix g with a scale of 2×1, and will not be repeated here.
[0324] For the matrix f with a size of 4×4, the matrix A of the transformation matrix and the inverse transform of the winograd convolution T The matrix multiplication between and A is shown in formula (28):
[0325]
[0326] The Winograd transformation results of multiple sub-tensors are obtained by transforming them, which can be expressed as shown in the above formula (29):
[0327]
[0328] Since it is the inverse transform of the winograd convolution of the transform bitwise multiplication matrix, the number of clocks used for the calculation and summation operation is determined by the size of the sub-tensor to be 4×4=16. The number of additions required is: 1*4+2*8+4*4-2*2=32.
[0329] Among them, 1*4 means that the number of matrices formed by the winograd transformation results of a sub-tensor with only one non-zero element is 4, 2*8 means that the number of matrices formed by the winograd transformation results of a sub-tensor with 2 non-zero elements is 8, 4*4 means that the number of matrices formed by the winograd transformation results of a sub-tensor with 4 non-zero elements is 4, and 2*2 is the size of the result matrix.
[0330] According to the number of clocks and the required number of addition executions, the number of operator devices is determined to be 4, and the 4 operator devices are numbered 0-3 respectively.
[0331] Since there are 4 element positions in the result matrix, they can be mapped one-to-one with the four operator devices. Therefore, according to this one-to-one mapping relationship, the addition execution tasks for calculating each element in the result matrix are assigned to the four operator devices, and the four operator devices complete the addition calculation of each element in the result matrix within 16 clocks. There is no need to plan the input order of the matrix formed by the winograd transformation results of the sub-tensors. The input order of the matrix formed by the winograd transformation results of the sub-tensors is sequential input: 00 →f 01 →f 02 →f 03 →f 10 →f 11 →f 12 →f 13 →f 20 →f 21 →f 22 →f 23 →f 30 →f 31 →f 32 →f 33 .
[0332] For the matrix f with a transformation matrix size of 4×3, the transformation matrix is the matrix A of the inverse transform of the winograd convolution T The matrix multiplication between and A is shown in formula (30):
[0333]
[0334] The Winograd transformation results of multiple sub-tensors are obtained by transforming them, which can be expressed as shown in the above formula (31):
[0335]
[0336] Since it is the inverse transform of the Winograd convolution of the bitwise multiplication matrix, the number of clocks used to calculate the summation operation is determined by the size of the sub-tensor to be 4×3=12, and the number of additions required is:
[0337] 1*4+2*6+4*2-2*2=20.
[0338] Among them, 1*4 means that the number of matrices formed by the winograd transformation results of a sub-tensor with only one non-zero element is 4, 2*6 means that the number of matrices formed by the winograd transformation results of a sub-tensor with 2 non-zero elements is 6, 4*2 means that the number of matrices formed by the winograd transformation results of a sub-tensor with 4 non-zero elements is 2, and 2*2 is the size of the result matrix.
[0339] According to the number of clocks and the required number of addition executions, the number of operator devices is determined to be 4, and the 4 operator devices are numbered 0-3 respectively.
[0340] Similarly, since there are 4 element positions in the result matrix, they can be mapped one-to-one to the four operator devices. Therefore, according to this one-to-one mapping relationship, the addition execution tasks for calculating each element in the result matrix are correspondingly assigned to the four operator devices, and the four operator devices complete the addition calculation of each element in the result matrix within 16 clocks. There is no need to plan the input order of the matrix formed by the winograd transformation results of the sub-tensors. The input order of the matrix formed by the winograd transformation results of the sub-tensors is sequential input: f 00 →f 01 →f 02 →f 10 →f 11 →f 12 →f 20 →f 21 →f 22 →f 30 →f 31 →f 32 .
[0341] For the matrix f with a transformation matrix size of 3×4, the transformation matrix is the matrix A of the inverse transform of the winograd convolution T The matrix multiplication between and A is shown in formula (32):
[0342]
[0343] The Winograd transformation results of multiple sub-tensors are obtained by transforming them, which can be expressed as shown in the above formula (33):
[0344]
[0345] Since it is the inverse transform of the winograd convolution of the transform bitwise multiplication matrix, the number of clocks used for the calculation and summation operation is determined by the size of the sub-tensor to be 3×4=12. The number of additions required is: 1*4+2*6+4*2-2*2=20.
[0346] Among them, 1*4 means that the number of matrices formed by the winograd transformation results of a sub-tensor with only one non-zero element is 4, 2*6 means that the number of matrices formed by the winograd transformation results of a sub-tensor with 2 non-zero elements is 6, 4*2 means that the number of matrices formed by the winograd transformation results of a sub-tensor with 4 non-zero elements is 2, and 2*2 is the size of the result matrix.
[0347] According to the number of clocks and the required number of addition executions, the number of operator devices is determined to be 4, and the 4 operator devices are numbered 0-3 respectively.
[0348] Since there are 4 element positions in the result matrix, they can be mapped one-to-one with the four operator devices. Therefore, according to this one-to-one mapping relationship, the addition execution tasks for calculating each element in the result matrix are assigned to the four operator devices, and the four operator devices complete the addition calculation of each element in the result matrix within 12 clocks. There is no need to plan the input order of the matrix formed by the winograd transformation results of the sub-tensors. The input order of the matrix formed by the winograd transformation results of the sub-tensors is sequential input: 00 →f 01 →f 02 →f 03 →f 10 →f 11 →f 12 →f 13 →f 20 →f 21 →f 22 →f 23 .
[0349] For the matrix f with a size of 3×3, the matrix A of the transformation matrix and the inverse transform of the winograd convolution T The matrix multiplication between and A is shown in formula (34):
[0350]
[0351] The Winograd transformation results of multiple sub-tensors are obtained by transforming them, which can be expressed as shown in the above formula (35):
[0352]
[0353] Since it is the inverse transform of the Winograd convolution of the bitwise multiplication matrix, the number of clocks used for the calculation and summation operation is determined by the size of the sub-tensor to be 3×3=9. The number of additions required is: 1*4+2*4+4*1-2*2=12.
[0354] Among them, 1*4 means that the number of matrices formed by the winograd transformation results of a sub-tensor with only one non-zero element is 4, 2*4 means that the number of matrices formed by the winograd transformation results of a sub-tensor with 2 non-zero elements is 4, 4*1 means that the number of matrices formed by the winograd transformation results of a sub-tensor with 4 non-zero elements is 1, and 2*2 is the scale of the result matrix.
[0355] According to the number of clocks and the required number of addition executions, the number of operator devices is determined to be 4, and the 4 operator devices are numbered 0-3 respectively.
[0356] Since there are 4 element positions in the result matrix, they can be mapped one-to-one with the four operator devices. Therefore, according to this one-to-one mapping relationship, the addition execution tasks for calculating each element in the result matrix are correspondingly assigned to the four operator devices, and the four operator devices complete the addition calculation of each element in the result matrix within 9 clocks. There is no need to plan the input order of the matrix formed by the winograd transformation results of the sub-tensors. The input order of the matrix formed by the winograd transformation results of the sub-tensors is sequential input: 00 →f 01 →f 02 →f 10 →f 11 →f 12 →f 20 →f 21 →f 22 .
[0357] For the matrix f with a transformation matrix size of 4×1, the transformation matrix is the matrix A of the inverse transform of the winograd convolution T The matrix multiplication between and A is shown in formula (36):
[0358]
[0359] The Winograd transformation results of multiple sub-tensors are obtained by transforming them, which can be expressed as shown in the above formula (37):
[0360]
[0361] Since it is the inverse transform of the Winograd convolution of the transform bitwise multiplication matrix, the number of clocks used for the calculation and summation operation is determined by the size of the sub-tensor to be 4×1=4. The number of additions required is: 1*2+2*2-2*2=4.
[0362] Among them, 1*2 means that the number of matrices formed by the winograd transformation results of a sub-tensor with only one non-zero element is 2, 2*2 means that the number of matrices formed by the winograd transformation results of a sub-tensor with two non-zero elements is 2, and 2*2 is the scale of the result matrix.
[0363] According to the number of clocks and the required number of addition executions, the number of operator devices is determined to be 4, and the 4 operator devices are numbered 0-3 respectively.
[0364] Since there are 4 element positions in the result matrix, they can be mapped one-to-one with the four operator devices. Therefore, according to this one-to-one mapping relationship, the addition execution tasks for calculating each element in the result matrix are assigned to the four operator devices, and the four operator devices complete the addition calculation of each element in the result matrix within 4 clocks. There is no need to plan the input order of the matrix formed by the winograd transformation results of the sub-tensors. The input order of the matrix formed by the winograd transformation results of the sub-tensors is sequential input: 00 →f 01 →f 02 →f 03 .
[0365] For the matrix f with a transformation matrix size of 3×1, the transformation matrix is the matrix A of the inverse transform of the winograd convolution T The matrix multiplication between and A is shown in formula (38):
[0366]
[0367] The Winograd transformation results of multiple sub-tensors are obtained by transforming them, which can be expressed as shown in the above formula (39):
[0368]
[0369] Since it is the inverse transform of the Winograd convolution of the transform bitwise multiplication matrix, the number of clocks used for the calculation and summation operation is determined by the size of the sub-tensor to be 3×1=3. The number of additions required is: 1*2+2*1-2*1=2.
[0370] Among them, 1*2 means that the number of matrices formed by the winograd transformation results of a sub-tensor with only one non-zero element is 2, 2*1 means that the number of matrices formed by the winograd transformation results of a sub-tensor with two non-zero elements is 1, and 2*1 is the scale of the result matrix.
[0371] According to the number of clocks and the required number of addition executions, the number of operator devices is determined to be 4, and the 4 operator devices are numbered 0-3 respectively.
[0372] Since there are two element positions in the result matrix, they can be mapped one-to-one with the two operator devices. Therefore, according to this one-to-one mapping relationship, the addition execution tasks for calculating each element in the result matrix are assigned to two operator devices, and the two operator devices complete the addition calculation of each element in the result matrix within 3 clocks. There is no need to plan the input order of the matrix formed by the winograd transformation results of the sub-tensors. The input order of the matrix formed by the winograd transformation results of the sub-tensors is sequential input: 00 →f 01 →f 02
[0373] Figure 5 A schematic diagram of the structure of a winograd convolution operation device provided in one embodiment of the present application is shown in FIG. Figure 5 As shown, the winograd convolution operation device 40 provided in this embodiment includes: a disassembly module 41, a transformation and summation operation module 42 and a convolution operation module 43.
[0374] Among them, the disassembly module 41 is used to disassemble the data in the winograd convolution operation into multiple sub-tensors during the transformation process of the winograd convolution operation.
[0375] The transformation and summation operation module 42 is used to transform multiple sub-tensors to obtain the winograd transformation results of the multiple sub-tensors, and sum the winograd transformation results of the multiple sub-tensors to obtain the transformation result of the data, wherein the summation operation of the winograd transformation results of the multiple sub-tensors is completed by multiple operation sub-devices according to a set strategy.
[0376] The convolution operation module 43 is used to complete the winograd convolution operation according to the transformation result of the data.
[0377] The Winograd convolution operation device provided in this embodiment can perform Figure 2 The technical solution of the method embodiment shown has similar implementation principles and technical effects, which will not be repeated here.
[0378] Figure 6A schematic diagram of the structure of a winograd convolution operation device provided in another embodiment of the present application is shown in FIG. Figure 6 As shown, the winograd convolution operation device 50 provided in this embodiment is Figure 5 Based on the winograd convolution operation device 40 provided in the illustrated embodiment, the following technical solutions are also included.
[0379] Furthermore, the transformation and summation operation module 42 is specifically used for:
[0380] In the process of performing summation operation on the winograd transformation results of multiple sub-tensors, the summation operation on the winograd transformation results of multiple sub-tensors is distributed to multiple operation sub-devices for execution according to the load balancing principle of multiple operation sub-devices.
[0381] Furthermore, the transformation summation operation module 42, when allocating the summation operation of the winograd transformation results of the plurality of sub-tensors to the plurality of operation sub-devices for execution according to the principle of load balancing of the plurality of operation sub-devices, is specifically used for:
[0382] Determine the number of non-zero elements in the transformation matrix corresponding to each sub-tensor. Determine the order of summing the winograd transformation results of multiple sub-tensors according to the number of non-zero elements. According to the determined order of summing the winograd transformation results of multiple sub-tensors, the mapping relationship between the element position in the preset result matrix and the operator device will be used to calculate the addition execution task of each element in the result matrix and distribute it to multiple operator devices. Among them, the number of additions in the addition execution task of each element in the result matrix is composed of the non-zero element values in the data formed after the winograd transformation results of multiple sub-tensors. Among them, the summation operation at the same element position in the result matrix is performed by the same operator device.
[0383] Furthermore, it also includes: an addition execution times determination module 51, a clock number determination module 52 and an operator device number determination module 53.
[0384] The addition execution number determination module 51 is used to determine the number of addition executions required for the sum operation according to the winograd transformation results of the multiple sub-tensors. The clock number determination module 52 is used to determine the number of clocks according to the size of the sub-tensors or the result matrix. The operation sub-device number determination module 53 is used to determine the number of operation sub-devices according to the number of clocks and the number of addition executions required for the sum operation.
[0385] Furthermore, the transformation summation operation module 42, when allocating the summation operation of the winograd transformation results of the plurality of sub-tensors to the plurality of operation sub-devices for execution according to the principle of load balancing of the plurality of operation sub-devices, is specifically used for:
[0386] According to the order of summing the winograd transformation results of the determined multiple sub-tensors, the mapping relationship between the element positions in the preset result matrix and the operator devices will be used to calculate the addition execution tasks of each element in the result matrix and distribute them to multiple operator devices to complete the addition calculation of each element in the result matrix within the number of clocks.
[0387] Optionally, the transformation summation operation module 42, when allocating the addition execution tasks for calculating each element in the result matrix to a plurality of operation sub-devices so as to complete the addition calculation of each element in the result matrix within a predetermined number of clocks, is specifically used to:
[0388] In the first N consecutive clocks, one of the addends used to calculate the position of each element in the result matrix is input to the corresponding operator device to perform initial assignment of each element in the result matrix; in the remaining clocks, the remaining addends used to calculate the position of each element in the result matrix are input to the corresponding operator device for addition calculation until the value of each element in the result matrix is obtained; wherein all operator devices are in a processing state at each clock.
[0389] Optionally, the transformation summation operation module 42, when allocating the addition execution tasks for calculating each element in the result matrix to a plurality of operation sub-devices so as to complete the addition calculation of each element in the result matrix within a predetermined number of clocks, is specifically used to:
[0390] The number of clocks is divided into multiple clock segments, and each element position in the result matrix is divided into multiple element position sets, with the clock segments corresponding to the element position sets one by one; for each element position set, within the first clock of the corresponding clock segment, one of the addition numbers used to calculate all element positions in the element position set is input into the corresponding operator device to perform initial assignment of each element position in the element position set; within the remaining clocks of the clock segment, the remaining addition numbers used to calculate all element positions in each element position set are input into the corresponding operator device for addition calculation until the values of each element in the element position set are obtained; wherein all operator devices are in a processing state at each clock.
[0391] Furthermore, the disassembly module 41 is specifically used for:
[0392] Multiple sub-tensors are obtained from data analysis, where the data is the sum of multiple sub-tensors, the number of multiple sub-tensors is the same as the number of non-zero elements in the data, each sub-tensor has a single non-zero element, and the non-zero element in the sub-tensor is the same as the non-zero element in the corresponding position in the data.
[0393] Furthermore, the transformation and summation operation module 42, when performing transformation operations on multiple sub-tensors to obtain the winograd transformation results of the multiple sub-tensors, is specifically used to:
[0394] Get the winograd transform result of the meta-sub-tensor corresponding to each sub-tensor; where the meta-sub-tensor is a tensor in which the non-0 elements of the sub-tensor are set to 1; use the non-0 element values in the sub-tensor as coefficients to multiply the winograd transform result of the corresponding meta-sub-tensor to obtain the winograd transform result of the sub-tensor; add the winograd transform results of multiple sub-tensors to obtain the winograd transform result of the data.
[0395] Furthermore, the transformation and summation operation module 42, when obtaining the winograd transformation result of the elementary sub-tensor corresponding to each sub-tensor, is specifically used to:
[0396] For each sub-tensor, the left side of the corresponding meta-sub-tensor of the sub-tensor is multiplied by the left multiplication matrix and the right side is multiplied by the right multiplication matrix to obtain the winograd transformation result of the meta-sub-tensor; wherein the left multiplication matrix and the right multiplication matrix are both determined by the scale of the sub-tensor and the winograd transformation type, wherein the winograd transformation type includes the winograd transformation type of the forward transformation and the winograd transformation type of the inverse transformation.
[0397] Furthermore, the clock number determination module 52 is specifically configured to:
[0398] Determine the maximum size of the sub-tensors and result matrices; based on the maximum size, determine the number of clocks.
[0399] Furthermore, the operator device number determination module 53 is specifically used to:
[0400] The ratio of the number of addition executions required for calculating the sum operation to the number of clocks used is obtained; and the number of operator devices is determined according to the ratio, wherein the number of operator devices is greater than the ratio.
[0401] Furthermore, the addition execution times determination module 51 is specifically used for:
[0402] The non-zero element value at the same element position in the data formed after the winograd transformation results of each sub-tensor is determined as the number of additions at the element position in the result matrix; the number of summation operations at each element position is determined according to the number of additions at each element position in the result matrix; the sum of the number of summation operations at each element position is determined as the number of addition executions required to calculate the summation operation.
[0403] Further, in this embodiment, the winograd transformation type is a winograd transformation type of a positive transformation, and the data is a neuron matrix or a weight matrix; or,
[0404] The winograd transformation type is the winograd transformation type of the inverse transformation, the data is the transformation bitwise multiplication matrix, and the transformation bitwise multiplication matrix is the matrix obtained by bitwise multiplication of the neuron transformation matrix and the weight transformation matrix.
[0405] The Winograd convolution operation device provided in this embodiment can perform Figure 3 to Figure 4 The technical solution of the method embodiment shown has similar implementation principles and technical effects, which will not be repeated here.
[0406] Optionally, the winograd convolution operation device in the embodiment of the present application is applied to a master processor and a slave processor of a master-slave architecture.
[0407] Specifically, Figure 7 As shown, when performing winograd convolution forward transformation, a plurality of operator devices are included in the main processing module, and winograd convolution forward transformation of the neuron matrix and the weight matrix can be performed. When performing winograd convolution inverse transformation, a plurality of operator devices are included in the inverse transformation module in the slave processor.
[0408] An embodiment of the present application also provides a winograd convolution operation device, which includes: a processor and a memory.
[0409] And a memory communicatively connected to at least one processor.
[0410] The memory is used to store program codes, and the processor is used to call the program codes stored in the memory and execute them to implement Figure 2 The winograd convolution operation method provided by the embodiment shown or Figure 3 to Figure 4 The illustrated embodiment provides a winograd convolution operation method.
[0411] For related instructions, please refer to Figure 3 to Figure 4 The relevant descriptions and effects corresponding to the steps are understood, and no further elaboration is given here.
[0412] In this embodiment, the memory and the processor are connected via a bus.
[0413] The embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored, and when the instructions are executed on a computer, the computer is executed to implement the present invention. Figure 2 The winograd convolution operation method provided by the embodiment shown or Figure 3 to Figure 4 The illustrated embodiment provides a winograd convolution operation method.
[0414] The winograd convolution operation method according to the embodiment of the present disclosure can be applied to any processor of a processing system (such as an artificial intelligence chip) including multiple processors (multi-core). The processor can be a general-purpose processor, such as a CPU (Central Processing Unit), or an artificial intelligence processor (IPU) for performing artificial intelligence operations. Artificial intelligence operations may include machine learning operations, brain-like operations, etc. Among them, machine learning operations include neural network operations, k-means operations, support vector machine operations, etc. The artificial intelligence processor may, for example, include one or a combination of GPU (Graphics Processing Unit), NPU (Neural-Network Processing Unit), DSP (Digital Signal Process), and Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA) chips. The present disclosure does not limit the specific type of the processor. In addition, the types of multiple processors in the processing system may be the same or different, and the present disclosure does not limit this.
[0415] In a possible implementation, the processor mentioned in the present disclosure may include multiple processing units, each of which may independently run various assigned tasks, such as convolution operation tasks, pooling tasks, or fully connected tasks, etc. The present disclosure does not limit the processing units and the tasks run by the processing units.
[0416] Figure 8 Schematic diagram of a processing system of the winograd convolution operation method of an embodiment of the present application. Figure 8 As shown, the processing system 60 includes multiple processors 61 and a memory 62. The multiple processors 61 are used to execute instruction sequences, and the memory 62 is used to store data, which may include a random access memory (RAM) and a register file. The multiple processors 61 in the processing system 60 can share part of the storage space, such as sharing part of the RAM storage space and the register file, or have their own storage space at the same time.
[0417] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the order of the actions described, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present disclosure.
[0418] It should be further noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
[0419] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present disclosure can also be implemented in other ways. For example, the division of units / modules described in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.
[0420] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present disclosure may be integrated into one unit / module, or each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.
[0421] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. Unless otherwise specified, the artificial intelligence processor may be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. Unless otherwise specified, the storage unit may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory RRAM (Resistive Random Access Memory), a dynamic random access memory DRAM (Dynamic Random Access Memory), a static random access memory SRAM (Static Random-Access Memory), an enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), a high-bandwidth memory HBM (High-Bandwidth Memory), a hybrid memory cube HMC (Hybrid Memory Cube), etc.
[0422] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.
[0423] In a possible implementation, an artificial intelligence chip is also disclosed, which includes the above-mentioned winograd convolution operation method device.
[0424] In a possible implementation, a board is also disclosed, which includes a storage device, an interface device, a control device and the above-mentioned artificial intelligence chip; wherein the artificial intelligence chip is connected to the storage device, the control device and the interface device respectively; the storage device is used to store data; the interface device is used to realize data transmission between the artificial intelligence chip and an external device; the control device is used to monitor the status of the artificial intelligence chip.
[0425] Fig. 9 For a structural block diagram of the board in the embodiment of the present application, see Fig. 9 In addition to the chip 389, the board may also include other supporting components, including but not limited to: a storage device 390, an interface device 391 and a control device 392;
[0426] The storage device 390 is connected to the artificial intelligence chip via a bus for storing data. The storage device may include multiple groups of storage units 393. Each group of storage units is connected to the artificial intelligence chip via a bus. It is understood that each group of storage units may be DDR SDRAM (English: Double Data Rate SDRAM, double rate synchronous dynamic random access memory).
[0427] DDR can double the speed of SDRAM without increasing the clock frequency. DDR allows data to be read out on the rising and falling edges of the clock pulse. The speed of DDR is twice that of standard SDRAM. In one embodiment, the storage device may include 4 groups of storage units. Each group of storage units may include multiple DDR4 particles (chips). In one embodiment, the artificial intelligence chip may include 4 72-bit DDR4 controllers, 64 bits of the above 72-bit DDR4 controllers are used to transmit data, and 8 bits are used for ECC verification. It can be understood that when DDR4-3200 particles are used in each group of storage units, the theoretical bandwidth of data transmission can reach 25600MB / s.
[0428] In one embodiment, each group of the storage units includes a plurality of double rate synchronous dynamic random access memories arranged in parallel. DDR can transmit data twice in one clock cycle. A controller for controlling DDR is arranged in the chip to control the data transmission and data storage of each of the storage units.
[0429] The interface device is electrically connected to the artificial intelligence chip. The interface device is used to realize data transmission between the artificial intelligence chip and an external device (such as a server or a computer). For example, in one embodiment, the interface device may be a standard PCIE interface. For example, the data to be processed is transmitted to the chip by the server through a standard PCIE interface to realize data transfer. Preferably, when the PCIE 3.0X 16 interface is used for transmission, the theoretical bandwidth can reach 16000MB / s. In another embodiment, the interface device may also be other interfaces. The present disclosure does not limit the specific forms of expression of the above-mentioned other interfaces, and the interface unit can realize the switching function. In addition, the calculation results of the artificial intelligence chip are still transmitted back to the external device (such as a server) by the interface device.
[0430] The control device is electrically connected to the artificial intelligence chip. The control device is used to monitor the state of the artificial intelligence chip. Specifically, the artificial intelligence chip and the control device can be electrically connected through an SPI interface. The control device may include a single-chip microcomputer (Micro Controller Unit, MCU). For example, the artificial intelligence chip may include multiple processing chips, multiple processing cores or multiple processing circuits, which can drive multiple loads. Therefore, the artificial intelligence chip can be in different working states such as multi-load and light load. The control device can realize the regulation of the working states of multiple processing chips, multiple processing and / or multiple processing circuits in the artificial intelligence chip.
[0431] In a possible implementation, an electronic device is disclosed, which includes the above-mentioned artificial intelligence chip. The electronic device includes a data processing device, a robot, a computer, a printer, a scanner, a tablet computer, a smart terminal, a mobile phone, a driving recorder, a navigator, a sensor, a camera, a server, a cloud server, a camera, a video camera, a projector, a watch, a headset, a mobile storage, a wearable device, a vehicle, a household appliance, and / or a medical device.
[0432] The transportation means include airplanes, ships and / or vehicles; the household appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, electric lights, gas stoves, and range hoods; the medical equipment includes magnetic resonance imaging, ultrasound machines and / or electrocardiographs.
[0433] In the above embodiments, the description of each embodiment has its own emphasis. For the part not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0434] The foregoing content can be better understood in accordance with the following terms:
[0435] Clause A1. A winograd convolution operation method, the method comprising:
[0436] In the transformation process of the winograd convolution operation, the data in the winograd convolution operation is disassembled into multiple sub-tensors;
[0437] Performing transformation operations on the multiple sub-tensors to obtain winograd transformation results of the multiple sub-tensors, and summing the winograd transformation results of the multiple sub-tensors to obtain a transformation result of the data, wherein the summing operation of the winograd transformation results of the multiple sub-tensors is completed by multiple operation sub-devices according to a set strategy;
[0438] A winograd convolution operation is performed according to the transformation result of the data.
[0439] Clause A2. The method according to clause A1, wherein the summation operation of the winograd transformation results of the plurality of sub-tensors is performed by a plurality of operation sub-devices according to a set strategy, including:
[0440] During the summing operation of the winograd transformation results of the plurality of sub-tensors, the summing operation of the winograd transformation results of the plurality of sub-tensors is distributed to the plurality of operation sub-devices for execution according to the load balancing principle of the plurality of operation sub-devices.
[0441] Clause A3. The method according to clause A2, wherein the summing operation of the winograd transformation results of the plurality of sub-tensors is distributed to the plurality of operation sub-devices for execution according to the principle of load balancing of the plurality of operation sub-devices, comprising:
[0442] Determine the number of non-zero elements in the transformation matrix corresponding to each sub-tensor;
[0443] Determine the order of summing the winograd transformation results of multiple sub-tensors according to the number of non-zero elements;
[0444] According to the determined order of summing the winograd transformation results of the plurality of sub-tensors, the mapping relationship between the element positions in the result matrix and the operator devices is preset to allocate the addition execution tasks for calculating the elements in the result matrix to the plurality of operator devices;
[0445] The number of additions in the addition execution task of each element in the result matrix is composed of non-zero element values in the data formed after the winograd transformation results of the multiple sub-tensors;
[0446] The summation operation at the same element position in the result matrix is performed by the same operator device.
[0447] Clause A4. The method according to clause A3, wherein before the summing operation of the winograd transformation results of the plurality of sub-tensors is distributed to the plurality of operation sub-devices for execution according to the principle of load balancing of the plurality of operation sub-devices, the method further comprises:
[0448] Determining the number of addition executions required for performing the summation operation according to the winograd transformation results of the multiple sub-tensors;
[0449] Determining the number of clocks according to the size of the sub-tensor or the result matrix;
[0450] The number of the operation sub-devices is determined according to the number of clocks and the number of addition executions required for the sum operation.
[0451] Clause A5. The method according to clause A4, wherein the summing operation of the winograd transformation results of the plurality of sub-tensors is distributed to the plurality of operation sub-devices for execution according to the principle of load balancing of the plurality of operation sub-devices, comprising:
[0452] According to the order of summing the winograd transformation results of the determined multiple sub-tensors, the mapping relationship between the element positions in the preset result matrix and the operator devices will be used to calculate the addition execution tasks of each element in the result matrix and distributed to multiple operator devices to complete the addition calculation of each element in the result matrix within the number of clocks.
[0453] Clause A6. The method according to clause A5, wherein the summing operation of the winograd transformation results of the plurality of sub-tensors is distributed to the plurality of operation sub-devices for execution according to the principle of load balancing of the plurality of operation sub-devices, comprising:
[0454] According to the order of summing the winograd transformation results of the determined multiple sub-tensors, the mapping relationship between the element positions in the preset result matrix and the operator devices will be used to calculate the addition execution tasks of each element in the result matrix and distributed to multiple operator devices to complete the addition calculation of each element in the result matrix within the number of clocks.
[0455] Clause A7. The method according to clause A5, wherein the addition execution task for calculating each element in the result matrix is distributed to a plurality of operator devices to complete the addition calculation of each element in the result matrix within a predetermined number of clocks, comprising:
[0456] Dividing the number of clocks into a plurality of clock segments, and dividing each element position in the result matrix into a plurality of element position sets, wherein the clock segments correspond to the element position sets in a one-to-one manner;
[0457] For each element position set, in the first clock of the corresponding clock segment, one of the addition numbers used to calculate all the element positions in the element position set is input to the corresponding operator device to perform initial assignment of each element position in the element position set; in the remaining clocks of the clock segment, the remaining addition numbers used to calculate all the element positions in each element position set are input to the corresponding operator device for addition calculation until the values of each element in the element position set are obtained;
[0458] Among them, all the operation sub-devices are in the processing state at each clock.
[0459] Clause A8. A method according to any one of clauses A1 to A7, wherein the decomposing of data in a winograd convolution operation into a plurality of sub-tensors comprises:
[0460] Multiple sub-tensors are obtained by parsing the data, wherein the data is the sum of the multiple sub-tensors, the number of the multiple sub-tensors is the same as the number of non-0 elements in the data, each of the sub-tensors has a single non-0 element, and the non-0 elements in the sub-tensors are the same as the non-0 elements in the corresponding positions in the data.
[0461] Clause A9. A method according to any one of clauses A1 to A8, wherein the step of performing a transformation operation on the plurality of sub-tensors to obtain winograd transformation results of the plurality of sub-tensors comprises:
[0462] Obtaining the winograd transformation result of the meta-sub-tensor corresponding to each sub-tensor; wherein the meta-sub-tensor is a tensor in which the non-zero elements of the sub-tensor are set to 1;
[0463] Using the non-zero element value in the sub-tensor as a coefficient to multiply the winograd transformation result of the corresponding element sub-tensor to obtain the winograd transformation result of the sub-tensor;
[0464] The winograd transformation results of multiple sub-tensors are added together to obtain the winograd transformation result of the data.
[0465] Clause A10. The method according to clause A9, wherein obtaining the Winograd transformation result of the meta-sub-tensor corresponding to each sub-tensor comprises:
[0466] For each of the sub-tensors, multiply the left side of the meta-sub-tensor corresponding to the sub-tensor by the left multiplication matrix and the right side by the right multiplication matrix to obtain the Winograd transformation result of the meta-sub-tensor;
[0467] The left multiplication matrix and the right multiplication matrix are both determined by the scale of the sub-tensor and the winograd transformation type, wherein the winograd transformation type includes a winograd transformation type of a forward transformation and a winograd transformation type of an inverse transformation.
[0468] Clause A11. The method according to any one of clauses A4-A10, wherein determining the number of clocks according to the size of the sub-tensor or the result matrix comprises:
[0469] determining a maximum size of the subtensors and the result matrix;
[0470] The number of clocks is determined according to the maximum scale.
[0471] Clause A12. A method according to any one of clauses A4 to A10, wherein determining the number of operator devices according to the number of clocks and the number of addition executions required to calculate the summation operation comprises:
[0472] Calculating the ratio of the number of addition executions required to calculate the sum operation to the number of clocks used;
[0473] The number of the operator devices is determined according to the ratio, wherein the number of the operator devices is greater than the ratio.
[0474] Clause A13. A method according to any one of clauses A4 to A10, wherein determining the number of addition executions required to perform the summation operation based on the winograd transformation results of the multiple sub-tensors comprises:
[0475] Determine the non-zero element value at the same element position in the data formed after the winograd transformation results of each of the sub-tensors as the addition number of the element position in the result matrix;
[0476] Determining the number of summation operations at each element position according to the number of additions at each element position in the result matrix;
[0477] The sum of the number of sum operations at the positions of the elements is determined as the number of addition executions required to calculate the sum operation.
[0478] Clause A14. A method according to any one of clauses A10-A13, wherein the winograd transform type is a winograd transform type of a positive transform, and the data is a neuron matrix or a weight matrix; or
[0479] The winograd transformation type is an inverse transformation winograd transformation type, the data is a transformation multiplication matrix, and the transformation multiplication matrix is a matrix obtained by multiplying a neuron transformation matrix and a weight transformation matrix.
[0480] Item A15. A winograd convolution operation device, comprising:
[0481] The disassembly module is used to disassemble the data in the winograd convolution operation into multiple sub-tensors during the transformation process of the winograd convolution operation;
[0482] a transformation and summation operation module, configured to perform transformation operations on the multiple sub-tensors to obtain winograd transformation results of the multiple sub-tensors, and perform summation operations on the winograd transformation results of the multiple sub-tensors to obtain a transformation result of the data, wherein the summation operation of the winograd transformation results of the multiple sub-tensors is completed by multiple operation sub-devices according to a set strategy;
[0483] The convolution operation module is used to complete the winograd convolution operation according to the transformation result of the data.
[0484] Item A16. According to the device described in Item A15, the winograd convolution operation device is applied to a master processor or a slave processor of a master-slave architecture.
[0485] Clause A17. A winograd convolution operation device, comprising: a processor and a memory; wherein:
[0486] The memory is used to store program code;
[0487] The processor is used to call the program code stored in the memory to execute any method described in any one of clauses A1 to A14.
[0488] Clause A18. A computer-readable storage medium storing instructions, which, when executed on a computer, enables the computer to execute the method described in any one of clauses A1 to A14.
[0489] Item A19. An artificial intelligence chip, comprising a winograd convolution operation device as described in any one of Items A1A17.
[0490] Item A20. An electronic device comprising an artificial intelligence chip as described in Item A19.
[0491] Clause A21. A board comprising: a storage device, an interface device, a control device, and an artificial intelligence chip as described in Clause A19;
[0492] Wherein, the artificial intelligence chip is connected to the storage device, the control device and the interface device respectively;
[0493] The storage device is used to store data;
[0494] The interface device is used to realize data transmission between the artificial intelligence chip and external equipment;
[0495] The control device is used to monitor the status of the artificial intelligence chip.
[0496] Clause A22. The board according to clause A21, wherein the storage device comprises: a plurality of groups of storage units, each group of the storage units is connected to the artificial intelligence chip via a bus, and the storage units are: DDR SDRAM;
[0497] The chip includes: a DDR controller for controlling data transmission and data storage of each storage unit;
[0498] The interface device is: a standard PCIE interface.
[0499] The embodiments of the present disclosure are described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present disclosure. The description of the above embodiments is only used to help understand the method and its core idea of the present disclosure. At the same time, changes or deformations made by those skilled in the art based on the ideas of the present disclosure, the specific implementation methods and the scope of application of the present disclosure, all belong to the scope of protection of the present disclosure. In summary, the content of this specification should not be understood as a limitation on the present disclosure.
Claims
1. A winograd convolution operation method, characterized in that: The method comprises: In the transformation process of the winograd convolution operation, a plurality of sub-tensors are obtained from the data by parsing, wherein the data is the sum of the plurality of sub-tensors, the number of the plurality of sub-tensors is the same as the number of non-zero elements in the data, each of the sub-tensors has a single non-zero element, and the non-zero element in the sub-tensor is the same as the non-zero element at a corresponding position in the data; the data is image data; Performing transformation operations on the multiple sub-tensors to obtain winograd transformation results of the multiple sub-tensors, and summing the winograd transformation results of the multiple sub-tensors to obtain a transformation result of the data, wherein the summing operation of the winograd transformation results of the multiple sub-tensors is completed by multiple operation sub-devices according to a set strategy; A winograd convolution operation is performed according to the transformation result of the data.
2. The method according to claim 1, characterized in that The summation operation of the winograd transformation results of the multiple sub-tensors is performed by multiple operation sub-devices according to a set strategy, including: During the summing operation of the winograd transformation results of the plurality of sub-tensors, the summing operation of the winograd transformation results of the plurality of sub-tensors is distributed to the plurality of operation sub-devices for execution according to the load balancing principle of the plurality of operation sub-devices.
3. The method according to claim 2, characterized in that The method of allocating the summation operation of the winograd transformation results of the plurality of sub-tensors to the plurality of operation sub-devices for execution according to the load balancing principle of the plurality of operation sub-devices includes: Determine the number of non-zero elements in the transformation matrix corresponding to each sub-tensor; Determine the order of summing the winograd transformation results of multiple sub-tensors according to the number of non-zero elements; According to the determined order of summing the winograd transformation results of the plurality of sub-tensors, the mapping relationship between the element positions in the result matrix and the operator devices is preset to allocate the addition execution tasks for calculating the elements in the result matrix to the plurality of operator devices; The number of additions in the addition execution task of each element in the result matrix is composed of non-zero element values in the data formed after the winograd transformation results of the multiple sub-tensors; The summation operation at the same element position in the result matrix is performed by the same operator device.
4. The method according to claim 3, characterized in that According to the load balancing principle of the multiple operator devices, before the summation operation of the winograd transformation results of the multiple sub-tensors is distributed to the multiple operator devices for execution, it also includes: Determining the number of addition executions required for performing the summation operation according to the winograd transformation results of the multiple sub-tensors; Determining the number of clocks according to the size of the sub-tensor or the result matrix; The number of the operation sub-devices is determined according to the number of clocks and the number of addition executions required for the sum operation.
5. The method according to claim 4, characterized in that The method of allocating the summation operation of the winograd transformation results of the plurality of sub-tensors to the plurality of operation sub-devices for execution according to the load balancing principle of the plurality of operation sub-devices includes: According to the order of summing the winograd transformation results of the determined multiple sub-tensors, the mapping relationship between the element positions in the preset result matrix and the operator devices will be used to calculate the addition execution tasks of each element in the result matrix and distributed to multiple operator devices to complete the addition calculation of each element in the result matrix within the number of clocks.
6. The method according to claim 5, characterized in that The adding execution task for calculating each element in the result matrix is distributed to a plurality of operator devices to complete the adding calculation of each element in the result matrix within a predetermined number of clocks, including: In the first N consecutive clocks, one of the addition numbers used to calculate the position of each element in the result matrix is input to the corresponding operator device to perform initial assignment on each element in the result matrix; In the remaining clocks, the remaining addition numbers used to calculate the positions of each element in the result matrix are input to the corresponding operator devices for addition calculation until the values of each element in the result matrix are obtained; Among them, all the operation sub-devices are in the processing state at each clock.
7. The method according to claim 5, characterized in that The adding execution task for calculating each element in the result matrix is distributed to a plurality of operator devices to complete the adding calculation of each element in the result matrix within a predetermined number of clocks, including: Dividing the number of clocks into a plurality of clock segments, and dividing each element position in the result matrix into a plurality of element position sets, wherein the clock segments correspond to the element position sets in a one-to-one manner; For each element position set, in the first clock of the corresponding clock segment, one of the addition numbers used to calculate all the element positions in the element position set is input to the corresponding operator device to perform initial assignment of each element position in the element position set; in the remaining clocks of the clock segment, the remaining addition numbers used to calculate all the element positions in each element position set are input to the corresponding operator device for addition calculation until the values of each element in the element position set are obtained; Among them, all the operation sub-devices are in the processing state at each clock.
8. The method according to any one of claims 1 to 7, characterized in that: The step of performing a transformation operation on the multiple sub-tensors to obtain winograd transformation results of the multiple sub-tensors includes: Obtaining the winograd transformation result of the meta-sub-tensor corresponding to each sub-tensor; wherein the meta-sub-tensor is a tensor in which the non-zero elements of the sub-tensor are set to 1; Using the non-zero element value in the sub-tensor as a coefficient to multiply the winograd transformation result of the corresponding element sub-tensor to obtain the winograd transformation result of the sub-tensor; The winograd transformation results of multiple sub-tensors are added together to obtain the winograd transformation result of the data.
9. The method according to claim 8, characterized in that The step of obtaining the winograd transformation result of the elementary sub-tensor corresponding to each sub-tensor includes: For each of the sub-tensors, multiply the left side of the meta-sub-tensor corresponding to the sub-tensor by the left multiplication matrix and the right side by the right multiplication matrix to obtain the Winograd transformation result of the meta-sub-tensor; The left multiplication matrix and the right multiplication matrix are both determined by the scale of the sub-tensor and the winograd transformation type, wherein the winograd transformation type includes a winograd transformation type of a forward transformation and a winograd transformation type of an inverse transformation.
10. The method according to any one of claims 4 to 9, characterized in that: The determining the number of clocks according to the size of the sub-tensor or the result matrix includes: determining a maximum size of the subtensors and the result matrix; The number of clocks is determined according to the maximum scale.
11. The method according to any one of claims 4 to 7 or claim 10, characterized in that: The determining the number of operator devices according to the number of clocks and the number of addition executions required for calculating the sum operation includes: Calculating the ratio of the number of addition executions required to calculate the sum operation to the number of clocks used; The number of the operator devices is determined according to the ratio, wherein the number of the operator devices is greater than the ratio.
12. The method according to any one of claims 4 to 11, characterized in that: The step of determining the number of addition executions required for performing the sum operation according to the winograd transformation results of the multiple sub-tensors includes: Determine the non-zero element value at the same element position in the data formed after the winograd transformation results of each of the sub-tensors as the addition number of the element position in the result matrix; Determining the number of summation operations at each element position according to the number of additions at each element position in the result matrix; The sum of the number of sum operations at the positions of the elements is determined as the number of addition executions required to calculate the sum operation.
13. The method according to any one of claims 9 to 12, characterized in that: The winograd transformation type is a winograd transformation type of a positive transformation, and the data is a neuron matrix or a weight matrix; or, The winograd transformation type is an inverse transformation winograd transformation type, the data is a transformation multiplication matrix, and the transformation multiplication matrix is a matrix obtained by multiplying a neuron transformation matrix and a weight transformation matrix.
14. A winograd convolution operation device, characterized in that: include: A disassembly module, used for parsing data to obtain multiple sub-tensors during the transformation process of the winograd convolution operation, wherein the data is the sum of the multiple sub-tensors, the number of the multiple sub-tensors is the same as the number of non-0 elements in the data, each of the sub-tensors has a single non-0 element, and the non-0 element in the sub-tensor is the same as the non-0 element at a corresponding position in the data; the data is image data; a transformation and summation operation module, configured to perform transformation operations on the multiple sub-tensors to obtain winograd transformation results of the multiple sub-tensors, and perform summation operations on the winograd transformation results of the multiple sub-tensors to obtain a transformation result of the data, wherein the summation operation of the winograd transformation results of the multiple sub-tensors is completed by multiple operation sub-devices according to a set strategy; The convolution operation module is used to complete the winograd convolution operation according to the transformation result of the data.
15. The device according to claim 14, characterized in that The winograd convolution operation device is applied to a master processor or a slave processor of a master-slave architecture.
16. A winograd convolution operation device, characterized in that: include: processor and memory; wherein, The memory is used to store program codes; The processor is used to call the program code stored in the memory to execute the method described in any one of claims 1 to 13.
17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a computer, the computer executes the method according to any one of claims 1 to 13.
18. An artificial intelligence chip, characterized in that: The artificial intelligence chip includes the winograd convolution operation device as described in claim 16.
19. An electronic device, characterized in that: The electronic device comprises the artificial intelligence chip as claimed in claim 18.
20. A board, characterized in that: The board includes: a storage device, an interface device and a control device and the artificial intelligence chip as claimed in claim 18; Wherein, the artificial intelligence chip is connected to the storage device, the control device and the interface device respectively; The storage device is used to store data; The interface device is used to realize data transmission between the artificial intelligence chip and external equipment; The control device is used to monitor the status of the artificial intelligence chip.
21. The board according to claim 20, characterized in that: The storage device comprises: a plurality of groups of storage units, each group of the storage units is connected to the artificial intelligence chip via a bus, and the storage units are: DDR SDRAM; The chip includes: a DDR controller for controlling data transmission and data storage of each storage unit; The interface device is: a standard PCIE interface.
22. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.
