A convolution calculation method and device

Abstract

This patent discloses a convolution calculation method and device for optimizing the existing convolution calculation process. Due to the time mismatch between multiplication and addition, a large amount of data may be piled up at the input of the adder waiting for calculation, which wastes calculation time and thus causes energy consumption and efficiency problems. This method and device reuses weights and input feature data, and multiple computing units perform parallel calculations to improve computing speed and power consumption. At the same time, by optimizing the calculation order of data, when the data is compressed, there will be no data competition, and no additional storage hardware and time processing calculation blockage are required, thus avoiding the stagnation of the computing module caused by channel blockage. Moreover, this method and device are applicable not only to compressed weight data and feature data, but also to dense data.

Description

Technical Field

[0001] The present invention relates to the field of computer systems based on specific computing models, and in particular to a convolution computing method and device. Background Art

[0002] In deep learning calculations, the convolution operation between weights and input features requires a large number of multiplication-accumulation (MAC) operations. In traditional computing methods, regardless of whether the data is compressed or not, if the calculation process is not optimized, there will be the following problems: 1. Input feature data, weight data, and output data are repeatedly read and written, resulting in a waste of storage space and bandwidth, as well as a large amount of energy consumption caused by data movement; 2. The computational parallelism is low, resulting in slow computing speed and low computing power; 3. There is competition between data, which leads to computational blockage and reduced computing power.

[0003] In addition, in convolution operations, due to the time mismatch between multiplication and addition, a large amount of data may accumulate at the adder input waiting for calculation, wasting computing time. When the memory space is insufficient and the multiplication result cannot be stored, the multiplier stops calculating, resulting in a blockage. If this is solved by adding hardware, it will result in a huge waste of hardware and computing power. Summary of the Invention

[0004] The present invention provides a convolution calculation method and device for solving the problem of data competition and reduced computing efficiency in the calculation process of deep learning, in which the output results of multiplication operations are correlated with each other.

[0005] This method and device reuses weights and input feature data, and multiple computing units perform parallel calculations to improve computing speed and power consumption. At the same time, by optimizing the calculation order of data, there will be no data competition when the data is compressed, and no additional storage hardware and time are required to process computing blockages, thus avoiding computing module stagnation caused by channel blockage. Moreover, the method and device are not only applicable to compressed weight data and feature data, but also to dense data.

[0006] The present invention provides a convolution calculation method, comprising the following steps:

[0007] Step S1, broadcasting a feature data to the first input terminals of all multipliers, wherein the feature data is image feature data;

[0008] Step S2, selecting weight data no greater than the number of the multipliers from all weight data and distributing the weight data to the second input end of the multiplier, wherein the weight data is obtained after the image is trained by the neural network model;

[0009] Step S3, the multiplier adds the calculation result to the data at the specific location of the memory and stores the result in the specific location;

[0010] Step S4: updating the first input end data and the second input end data until the calculation is completed to obtain a weighted feature image.

[0011] Preferably, the step S4 specifically includes:

[0012] S401, fixing the weight data of the second input terminal, updating the feature data of the first input terminal, and repeating step S3 until all feature data are calculated;

[0013] S402: Update the weight data of the second input terminal and repeat step S401 until all calculations are completed.

[0014] Preferably, the feature data and / or weight data are compressed.

[0015] Preferably, the specific location of the memory is uniquely determined by the coordinate information and channel information of the first and second input end data.

[0016] Preferably, the compression processing of the weight data specifically includes: rearranging the weight data and compressing the data using a compression algorithm.

[0017] Preferably, the weight data is rearranged by expanding a single convolution kernel into a row vector and combining the row vectors to form a weight data matrix.

[0018] Preferably, the weight data is obtained after the speech and text are trained by a neural network model, the feature data corresponds to speech feature data and text feature data, and step S4 corresponds to obtaining weighted feature speech and feature text.

[0019] The present invention provides a convolution calculation device using the above method, comprising:

[0020] A storage module, including an external memory, an internal memory and an output memory;

[0021] A calculation module, comprising a multiplier unit and an adder unit, wherein the multiplier unit comprises a first input terminal and a second input terminal;

[0022] The control module includes an address finder for determining a unique specific position in the storage module according to the coordinates and channels of the data inputted from the first input terminal and the second input terminal of the multiplier;

[0023] The control module also includes a controller for controlling the input and output operations of data;

[0024] The external memory is used to store feature data and weight data;

[0025] The multiplier unit is used to calculate the product of feature data and weight data.

[0026] The present invention has the following beneficial effects:

[0027] 1. Optimize the computing process to completely avoid the probability of coherent output results of computing units at the same time, eliminate data competition, and eliminate the need to add additional hardware to process potential competing data.

[0028] 2. Multiplexing weight data and pulse multiplexing input feature data to reduce bandwidth requirements and data movement power consumption;

[0029] 3. Support parallel computing of multiple computing units;

[0030] 4. Support weight data compression, reduce bandwidth requirements and data movement power consumption, and skip zero value calculation. When weight data is sparse, it can significantly improve computing power and reduce power consumption;

[0031] 5. Compatible with convolution calculations of dense data. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0033] Figure 1 is a flow chart of a convolution calculation method in one embodiment of the present invention;

[0034] Figure 2 1 is a schematic diagram of an addition calculation process in one embodiment of the present invention;

[0035] Figure 3 1 is a schematic diagram of inputting feature data and weight data into a multiplier in one embodiment of the present invention;

[0036] Figure 4 It is a schematic diagram of a combination of a convolution calculation method and a device in one embodiment of the present invention.

[0037] In the above figures, MUL is a multiplier and MUX is a selector. DETAILED DESCRIPTION

[0038] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0039] Example 1

[0040] This embodiment provides a convolution calculation method, in which both feature data and weight data are sparse data. Before performing the convolution calculation, the sparse data is first compressed using a compression algorithm to reduce the storage space occupied by the data and the bandwidth, time, and power consumption required for data movement. The compression algorithm includes but is not limited to the COO compression algorithm and the CSF compression algorithm. In this embodiment, the COO compression algorithm is used as an example for explanation. The specific method for compressing the weight data obtained after the image is trained by the neural network model includes:

[0041] First, rearrange the weight data. In convolution calculations, weight data is represented by convolution kernels. In convolution calculations, there are often multiple convolution kernels involved in the calculation, so it is necessary to rearrange and combine the data in multiple convolution kernels to form a new matrix to meet the format requirements of the subsequent compression algorithm for data. The rearrangement of weight data specifically includes:

[0042] Expand the weight data (K×K) in the first convolution kernel into a row vector (1×K 2 ), as the first row of the weight data matrix; expand the weight data in the second convolution kernel into a row vector, as the second row of the weight data matrix, and so on, until the weight data in all convolution kernels are completely rearranged.

[0043] Secondly, the COO compression coding algorithm is used to compress the weight data matrix formed after the rearrangement, and the compressed data is stored in the external memory of the computing unit. After the COO compression coding algorithm is used, the zeros in the sparse weight data matrix are omitted, and the non-zero data is stored in the form of row coordinates, column coordinates and values, which greatly reduces the storage space occupied by the data.

[0044] Similarly, the image feature data is also compressed using the COO compression coding algorithm and stored in the external memory of the computing unit.

[0045] The compression of weight data and feature data can be performed during the calculation process or in advance outside the calculation process.

[0046] For the convolution calculation of the compressed data, the specific method of the convolution calculation method is as follows. Figure 1:

[0047] Step S1, broadcasting a feature data to the first input terminals of all multipliers, wherein the feature data is image feature data;

[0048] The multiplier (MUL) hardware is a 2-input 1-output multiplier that inputs two values and outputs the product of the two values. The number of the multipliers is not limited in this method, but is generally multiple.

[0049] A single data in the image feature data, that is, a value in the compressed feature data, is broadcasted to the first input terminals of all multipliers. The broadcasting means that the data at the first input terminals of all multipliers are the same value.

[0050] Step S2, selecting weight data no greater than the number of the multipliers from all weight data and distributing the weight data to the second input end of the multiplier, wherein the weight data is obtained after the image is trained by the neural network model;

[0051] In this embodiment, the total weight data is the compressed feature data matrix. Generally speaking, to maximize the efficiency of all multipliers, values equal to the number of multipliers are sequentially selected from the compressed feature data matrix and distributed sequentially to the second input terminals of the multiplier units. This distribution means that the data at the second input terminals of all multipliers are values at different positions in the feature data matrix.

[0052] The schematic diagram of steps S1 and S2 is shown in Figure 3 .

[0053] Step S3: The multiplier adds the calculation result to the data at the specific location of the memory and stores the result in the specific location. Figure 2 ;

[0054] Step S3 is implemented by an adder and an addresser. The adder hardware is a 2-input 1-output hardware used to add two input data and calculate the sum and output it. The addresser is a hardware device that calculates a specific position in the memory based on coordinate information.

[0055] Because convolution is a multiplication-accumulation operation, the corresponding eigenvalues and weight values within the same convolution kernel range need to be multiplied and then added. In this embodiment, the addresser uniquely determines a specific location in the memory based on the coordinates of the first input data, the coordinates of the second input data, and the current channel information. The initial value of this location is 0. The multiplication results that need to be accumulated are added to the number in this specific location and then stored back in this specific location, thereby achieving the accumulation effect.

[0056] After each round of multiplication calculation, a round of addition calculation is performed to accumulate the multiplication result with the value at a specific location in the memory.

[0057] After a round of addition calculation is completed, the data input to the multiplier is updated, that is:

[0058] Step S4: Update the first input end data and the second input end data until the calculation is completed to obtain a weighted image.

[0059] In this embodiment, the data of the first input terminal and the second input terminal are updated as follows:

[0060] S401, fixing the weight data of the second input terminal, updating the feature data of the first input terminal, and repeating step S3 until all feature data are calculated;

[0061] Specifically, this embodiment first fixes the weight data distributed to the second input of each multiplier, and then sequentially updates the feature data broadcast to the first input until all feature data are multiplied by the same set of weight data. After each round of updates, step S3 is repeated, and the calculation result is added to the value at a specific memory location and stored back in the specific location.

[0062] After all feature data are multiplied with the same set of weight data, step S402 is performed to update the weight data of the second input end, and step S401 is repeated until all calculations are completed.

[0063] That is, a new set of feature data replaces the previously fixed set of feature data, and step S401 is repeated to fix the new set of feature data, update the feature data of the first input end, and traverse all feature data from the beginning until all feature data are fixed and calculated, and the accumulation operation is completed.

[0064] Finally, the feature data and the weight data complete a convolution operation to obtain weighted image data.

[0065] Similarly, since convolution calculations are not limited to processing image data, the weight data in the above method can also be obtained after the speech and text are trained by the neural network model, and the feature data can correspond to speech feature data and text feature data, and finally the weighted feature speech data and weighted feature text data are obtained.

[0066] Example 2

[0067] This embodiment provides a convolution calculation method, wherein one or both of the feature data and weight data are dense data and cannot be compressed. Compared with Example 1, Example 2 omits the corresponding rearrangement and data compression operations of the dense data, and directly inputs the dense feature data and / or weight data into the multiplier unit. The addresser uniquely determines the specific location where the result is accumulated and stored based on the position of the data in the matrix or its position in the external memory. The remaining steps remain consistent with Example 1.

[0068] Example 3

[0069] This embodiment provides a convolution computing device using the above-mentioned convolution computing method, including a storage module, a computing module, and a control module:

[0070] The storage module includes an external memory, an internal memory and an output memory;

[0071] The calculation module includes a multiplier unit and an adder unit; the multiplier unit includes a first input terminal and a second input terminal.

[0072] The control module includes an address finder for determining a unique specific position in the storage module according to the coordinates and channels of the data inputted from the first input terminal and the second input terminal of the multiplier;

[0073] The control module also includes a controller for controlling the input and output operations of data;

[0074] The external memory is used to store feature data and weight data;

[0075] The multiplier unit is used to calculate the product of feature data and weight data;

[0076] Example 3 is a hardware device for implementing Example 1 and / or Example 2. The functional definition, connection relationship definition and description of each module can refer to the definition and description of the method, which will not be described in detail here. The schematic diagram of the method and device can be found in Figure 4 .

[0077] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A convolution calculation method, characterized in that: include: Step S1, broadcasting a feature data to the first input terminals of all multipliers, wherein the feature data is image feature data, speech feature data, or text feature data; Step S2, selecting weight data no greater than the number of multipliers from all weight data and distributing the weight data to the second input end of the multiplier, wherein the weight data is obtained after the image, speech, or text is trained by the neural network model; The feature data and / or weight data are compressed, and the compression of the weight data specifically includes: rearranging the weight data and compressing the data using a compression algorithm; Step S3, the multiplier adds the calculation result to the data at a specific location in the memory and stores the result in the specific location; the specific location in the memory is uniquely determined by the coordinate information and channel information of the first and second input end data; Step S4: updating the first input end data and the second input end data until the calculation is completed to obtain a weighted feature image or feature speech or feature text.

2. The convolution calculation method according to claim 1, wherein: The step S4 specifically includes: S401, fixing the weight data of the second input terminal, updating the feature data of the first input terminal, and repeating step S3 until all feature data are calculated; S402: Update the weight data of the second input terminal and repeat step S401 until all calculations are completed.

3. The convolution calculation method according to claim 1, wherein: The weight data rearrangement is to expand a single convolution kernel into a row vector and combine the row vectors to form a weight data matrix.

4. A convolution computing device using the method according to any one of claims 1 to 3, comprising: A storage module, including an external memory, an internal memory and an output memory; A calculation module, comprising a multiplier unit and an adder unit, wherein the multiplier unit comprises a first input terminal and a second input terminal; The control module includes an address finder for determining a unique specific position in the storage module according to the coordinates and channels of the data inputted from the first input terminal and the second input terminal of the multiplier; The control module also includes a controller for controlling the input and output operations of data; The external memory is used to store feature data and weight data; The multiplier unit is used to calculate the product of feature data and weight data.

Images