A convolution calculation method and device

Abstract

This patent discloses a convolution calculation method and device for optimizing energy consumption and efficiency issues in existing convolution calculation processes. This patent makes special optimizations for convolution calculations with stride ≥ 2, while also being compatible with scenarios with stride = 1. This patent optimizes the calculation process, reduces the probability of coherence of the output results of the computing units at the same time, reduces data competition, and adds additional hardware to process potential competing data. It also reuses weight data and pulsating multiplexed input feature data to reduce bandwidth requirements and data movement power consumption.

Description

Technical Field

[0001] The present invention relates to the field of computer systems based on specific computing models, and in particular to a convolution computing method and device. Background Art

[0002] In deep learning calculations, the convolution operation between weights and input features requires a large number of multiplication-accumulation (MAC) operations. In traditional computing methods, if the calculation process is not optimized, there will be the following problems: 1. Input feature data, weight data, and output data are repeatedly read and written, resulting in a waste of storage space and bandwidth, as well as a large amount of energy consumption caused by data movement; 2. The computational parallelism is low, resulting in slow computing speed and low computing power; 3. There is competition between data, which leads to computational blockage and reduced computing power.

[0003] Existing optimization algorithms for convolution calculations usually only target the case of stride = 1, and are not optimized for stride ≥ 2. For convolutions with stride ≥ 2, traditional calculation methods are usually used, or the optimization algorithm for stride = 1 is continued to be used.

[0004] Using traditional computing methods will cause the input feature data, weight data, and output data to be read and written repeatedly, resulting in a waste of storage space and bandwidth, as well as a large amount of energy waste caused by data movement; the computing parallelism is low, resulting in slow computing speed and low computing power; and continuing to use the optimization algorithm for stride = 1 will lead to a large amount of redundant calculations, resulting in slow computing speed and a large amount of unnecessary power consumption.

[0005] In addition, when both the input weight data and feature data are sparse, there are a large number of zero-valued data in the numerical values. The MAC operation results of the zero-valued data in the convolution operation have no effect on the final output result. However, in actual operations, zero values will also be read and written, and MAC calculations will be performed, resulting in a waste of storage space, bandwidth, computing power and power consumption. Summary of the Invention

[0006] The present invention provides a convolution calculation method and device for solving the problems of slow calculation speed and high power consumption in the case of stride ≥ 2 in the calculation process of deep learning. This method is also compatible with the scenario of stride = 1.

[0007] The present invention provides a convolution calculation method, comprising the following steps:

[0008] S1, compressed weight data, the image weight data is obtained after the image is trained by the neural network model;

[0009] S2, adjust the order in which weight data enters the calculation unit;

[0010] S3, inputting the weight data and the image feature data into a multiplication calculation unit for calculation to obtain a first calculation result;

[0011] S4, performing addition calculation on the first calculation result and the data at a specific location in the memory to obtain a second calculation result, and storing the second calculation result in the specific location in the memory to obtain a weighted sum feature image;

[0012] Wherein: when there is competition between at least two of the first calculation results, one of the at least two first calculation results is temporarily stored and the calculation is performed again after the competition disappears.

[0013] Preferably, according to the coherence of the calculation results, the order in which the weight data having the same column coordinates enters the calculation unit is adjusted.

[0014] Preferably, before the characteristic data is input into the multiplication calculation unit and / or after the characteristic data is input into the multiplication calculation unit, characteristic data that needs to be repeatedly calculated is calculated by pulsating multiplexing.

[0015] Preferably, the specific position in the memory is uniquely determined based on the coordinate information of the weight data and feature data.

[0016] Preferably, the compression weight data includes:

[0017] Expand the weight data into row vectors and reconstruct the weight data matrix;

[0018] The weight data is compressed using a compression algorithm.

[0019] Preferably, the coherence of the calculation result is obtained through preliminary experiments.

[0020] Preferably, it is characterized in that the preliminary test specifically includes:

[0021] Select part or all of the weight data;

[0022] Exhaustively enumerate the arrangement order of the weight data, perform pre-calculation respectively, and obtain the coherence of the calculation result;

[0023] Preferably, the weight data is obtained after the speech and text are trained by the neural network model. Step S3 contains speech feature data and text feature data, and step S4 contains weighted feature speech and feature text.

[0024] The present invention also provides a convolution computing device, comprising:

[0025] A storage module, including an external memory, an internal memory and an output memory;

[0026] a computing module including a multiplier unit and an adder unit;

[0027] A control module including a selector unit for controlling a storage location of data in the memory;

[0028] The external memory is used to store feature data and compressed weight data;

[0029] The feature data and weight data are transmitted to the calculation module through the internal memory, and the order of the weight data entering the calculation module is adjusted in the internal memory;

[0030] The multiplier unit is used to calculate the product of feature data and weight data;

[0031] The selector unit is used to determine the storage location of the calculation result.

[0032] The present invention has the following beneficial effects:

[0033] 1. Optimize the computing process, reduce the probability of coherent output results of computing units at the same time, reduce data competition, and add additional hardware to handle potential competing data.

[0034] 2. Multiplexing weight data and pulse multiplexing input feature data to reduce bandwidth requirements and data movement power consumption;

[0035] 3. Support parallel computing of multiple computing units;

[0036] 4. Support weight data compression, reduce bandwidth requirements and data movement power consumption, and skip zero value calculation. When weight data is sparse, it can significantly improve computing power and reduce power consumption;

[0037] 5. For input data with stride ≥ 2, no additional calculation or bandwidth or data movement is required, which improves computing power and reduces power consumption;

[0038] 6. The calculation method is compatible with convolution calculation when stride=1. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0040] Figure 1 is a flow chart of a convolution calculation method in one embodiment of the present invention;

[0041] Figure 21 is a schematic diagram of adjusting the order in which weight data enters the calculation unit in one embodiment of the present invention;

[0042] Figure 3 1 is a schematic diagram of a calculation process combining a convolution calculation device and a method according to an embodiment of the present invention;

[0043] Figure 4 Schematic diagram of the addition operation process in one embodiment of the present invention.

[0044] In the above figures, MUL is a multiplier and MUX is a selector. DETAILED DESCRIPTION

[0045] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0046] In one embodiment of the present invention, a convolution calculation method is provided. Figure 1 , the specific method is as follows:

[0047] S1, compressed weight data, which is obtained after the image is trained by the neural network model, specifically including:

[0048] S101, expand the weight data into row vectors and reconstruct the weight data matrix;

[0049] The weight data is represented in the form of a convolution kernel. The weight data in each convolution kernel is expanded into a row vector as a row of the weight data matrix, until the weight data of all convolution kernels in all channels are expanded into row vectors to form a weight data matrix. The weight data includes the convolution kernel parameters of all positions in all channels of all convolution kernels.

[0050] S102, compressing the weight data matrix using a compression algorithm, wherein the compression algorithm includes but is not limited to a COO encoding algorithm and a CSF encoding algorithm.

[0051] The weight data matrix is compressed using a COO or CSF encoding algorithm (including but not limited to COO and CSF compression algorithms) and stored in an external memory.

[0052] S2, adjust the order in which weight data enters the calculation unit;

[0053] According to the register size and the size of each channel input feature data, all or part of the channel input feature data in the PE external memory is read in the matrix row order and loaded into the register unit.

[0054] Before calculation, the compressed weight data stored in the PE external memory is adjusted according to the coherence of the calculation results, and the order in which the weight data enters the calculation unit in the weight data with the same column coordinates is adjusted to ensure that the coherence of the calculation results is the lowest, and the weight data is stored in the corresponding FIFO in the PE unit.

[0055] The coherence of calculation results is a metric that describes conflicts in data calculation results. A conflicting calculation result occurs when the outputs of at least two multipliers within the same calculation cycle need to be added to the data at the same specific memory location. The more conflicts there are within a cycle, the greater the coherence of the calculation results. By adjusting the order in which weight data enters the calculation unit, the calculation results that need to be added to the data at the same specific memory location can be distributed across different calculation cycles, thereby reducing the coherence of the calculation results.

[0056] The coherence of the calculation results is obtained through preliminary experiments, and the specific method is as follows:

[0057] Select part or all of the weight data;

[0058] Exhaustively enumerate the arrangement order of the weight data, perform pre-calculation respectively, and obtain the coherence of the calculation result;

[0059] Since the computational cost of exhaustively testing the arrangement order of all weight data is significantly less than the convolution calculation of all feature data, and the arrangement order with the lowest or relatively low coherence can be effectively obtained, the

[0060] In one embodiment provided by the present invention, eight 4x4-sized weight data are stored in four FIFO memories (FIFO0, FIFO1, FIFO2, and FIFO3). According to the pre-calculated results, the order in which the weight data enters the calculation unit is adjusted in the following manner. Figure 2 :

[0061] 1) The weighted compressed data with column coordinate 0 is stored in FIFO0 in sequence;

[0062] 2) The weighted compressed data with column coordinate 1 is stored into FIFO1 starting from the second column data. After the data is stored, the first column data is stored into FIFO1;

[0063] 3) The weighted compressed data with column coordinate 2 is stored into FIFO2 starting from the 3rd column. After the data is stored, the 1st and 2nd column data are stored into FIFO2.

[0064] 4) The weighted compressed data with column coordinate 3 is stored into FIFO3 starting from the 4th column. After the data is stored, the 1st, 2nd and 3rd columns of data are stored into FIFO3.

[0065] Through the above sequence, the coherence of the calculation results can be reduced to the lowest level.

[0066] The steps S1 and S2 may be completed in real time in the computing device, or may be completed in advance outside the computing device.

[0067] S3, inputting the weight data and the feature data into a multiplication calculation unit for calculation to obtain a first calculation result;

[0068] The overall calculation framework is as follows Figure 3 As shown, the input of a single multiplier (MUL) is feature data and weight data, and the output result is the product of the two, that is, the first calculation result. The multiplier unit contains multiple multipliers. After loading a set of weight data, it continuously updates the feature data and performs calculations until all relevant calculations for the weight data are completed. The multiplier unit then loads the next set of weight data and continuously updates the feature data and performs calculations until all calculations are complete.

[0069] Before the characteristic data is input into the multiplication calculation unit and / or after the characteristic data is input into the multiplication calculation unit, the characteristic data that needs to be repeatedly calculated is calculated through pulsation multiplexing.

[0070] During calculation, the data of the register unit is stored in the register within the PE unit in sequence. The data can be pulsated between the registers inside the PE, and can also be pulsated between the register units. The data groups that need to be repeatedly calculated are moved within the memory and reused during calculation to reduce the repeated input of data into the memory.

[0071] S4, performing addition calculation on the first calculation result and the data at a specific location in the memory to obtain a second calculation result, and storing the second calculation result in the specific location in the memory;

[0072] Wherein: when there is competition between at least two of the first calculation results, one of the at least two first calculation results is temporarily stored and the calculation is performed again after the competition disappears.

[0073] See attached Figure 4The feature data and weight data input to the multiplier unit each correspond to a set of coordinate information. Based on the two sets of coordinate information for the feature data and weight data, a specific location in the output memory is determined. The specific location in the memory is uniquely determined based on the coordinate information for the weight data and feature data. The output result of a single multiplier and the data stored at a specific address in the output memory are added together in an adder unit, and the adder output, i.e., the second calculation result, is stored at that address.

[0074] Since all the multiplication results corresponding to the same convolution kernel area need to be added to each other, if more than two sets of weight data and feature data corresponding to the same convolution kernel are simultaneously input into different multipliers in the multiplier unit, the two output results will need to be stored in the same specific address in the output memory, causing an output address conflict. After adjusting the entry order of the weight data in step S2, the probability of conflict can be significantly reduced, but the occurrence of conflict cannot be avoided. Therefore, in order to solve the problem of output address conflicts that still exist in small quantities, the present invention provides the following solution:

[0075] In the event of a conflict between multiple first calculation results, one of them is retained for addition calculation, and the remaining first calculation results are transferred to the register for storage through the selector unit. When there is no conflict between the data in the register and the output result of the current multiplier unit, addition calculation and storage are performed again until all input data are calculated.

[0076] Similarly, when the weight data is obtained after the speech and text are trained by the neural network model, the corresponding data in step S3 are speech feature data and text feature data, and the corresponding data in step S4 are weighted feature speech and feature text. This method is still applicable to the convolution calculation of speech and text.

[0077] It should be understood that the order of execution of the steps in the above embodiments does not necessarily mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0078] In one embodiment, a convolution calculation device is provided. The convolution calculation device corresponds one-to-one to the convolution calculation method in the above embodiment, and a repeated description is not given here.

[0079] The device comprises:

[0080] A storage module, including an external memory, an internal memory and an output memory;

[0081] a computing module including a multiplier unit and an adder unit;

[0082] A control module including a selector unit for controlling a storage location of data in the memory;

[0083] The external memory is used to store feature data and compressed weight data;

[0084] The feature data and weight data are transmitted to the calculation module through the internal memory, and the order of the weight data entering the calculation module is adjusted in the internal memory;

[0085] The multiplier unit is used to calculate the product of feature data and weight data;

[0086] The selector unit is used to determine the storage location of the calculation result.

[0087] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0088] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A convolution calculation method, characterized in that: The following steps are involved: S1, compressing weight data, wherein the weight data is obtained after the image, voice or text is trained by the neural network model; S2, adjust the order in which weight data enters the calculation unit; S3, inputting the weight data and the image feature data or the voice feature data or the text feature data into a multiplication calculation unit for calculation to obtain a first calculation result; S4, adding the first calculation result to the data at a specific location in the memory to obtain a second calculation result, and storing the second calculation result in the specific location in the memory to obtain a weighted summed feature image, feature voice, or feature text; the specific location in the memory is uniquely determined based on the weight data and the coordinate information of the feature data; Wherein: when there is competition between at least two of the first calculation results, one of the at least two first calculation results is temporarily stored and the calculation is performed again after the competition disappears.

2. The convolution calculation method according to claim 1, wherein: According to the coherence of the calculation results, the order in which the weight data having the same column coordinates enters the calculation unit is adjusted.

3. The convolution calculation method according to claim 1, wherein: Before the characteristic data is input into the multiplication calculation unit and / or after the characteristic data is input into the multiplication calculation unit, the characteristic data that needs to be repeatedly calculated is calculated through pulsation multiplexing.

4. The convolution calculation method according to claim 1, wherein: The compression weight data includes: S101, expand the weight data into row vectors and reconstruct the weight data matrix; S102, compress the weight data using a compression algorithm.

5. The convolution calculation method according to claim 2, wherein: The coherence of the calculation results is obtained through preliminary experiments.

6. The convolution calculation method according to claim 5, characterized in that: The preliminary test specifically includes: Select part or all of the weight data; The arrangement order of the weight data is exhaustively enumerated, and pre-calculation is performed respectively to obtain the coherence of the calculation results.

7. A convolution computing device using the method according to any one of claims 1 to 6, comprising: A storage module, including an external memory, an internal memory and an output memory; a computing module including a multiplier unit and an adder unit; A control module including a selector unit for controlling a storage location of data in the memory; The external memory is used to store feature data and compressed weight data; The feature data and weight data are transmitted to the calculation module through the internal memory, and the order of the weight data entering the calculation module is adjusted in the internal memory; The multiplier unit is used to calculate the product of feature data and weight data; The selector unit is used to determine the storage location of the calculation result.

Images