A method and apparatus for convolution computation

Abstract

The patent discloses a convolution calculation method and device, which is used for optimizing the time mismatch of multiplication and addition in the existing convolution calculation process, so that a large amount of data accumulation at the input end of the adder is generated, the calculation time is wasted, and the energy consumption and efficiency problems are generated. The method and device improve the calculation speed and power consumption by multiplexing the weight and input feature data and parallel computing of multiple computing units. At the same time, by optimizing the calculation order of the data, when the data is compressed, there is no data competition, and no additional storage hardware and time processing calculation blocking are needed, so that the calculation module stagnation caused by channel blocking is avoided. Moreover, the method and device are not only suitable for compressed weight data and feature data, but also suitable for dense data. The patent is optimized for the case of stride >= 2, and is compatible with the convolution calculation of stride = 1.

Description

Technical Field

[0001] This invention relates to the field of computer systems based on specific computational models, and more particularly to a convolution computation method and apparatus. Background Technology

[0002] In deep learning computation, the convolution operation between weights and input features requires a large number of multiply-accumulate (MAC) operations. In traditional computation methods, regardless of whether the data is compressed or not, the following problems will occur if the computation process is not optimized: 1. Input feature data, weight data, and output data are repeatedly read and written, resulting in wasted storage space, bandwidth, and a large amount of energy wasted due to data movement; 2. Low computational parallelism leads to slow computation speed and low computing power; 3. Competition between data leads to computational blockage and reduced computing power.

[0003] Furthermore, in convolution operations, the timing mismatch between multiplication and addition can lead to a large backlog of data waiting to be calculated at the input of the adder, wasting computation time. When memory space is insufficient, the multiplication result cannot be stored, the multiplier stops calculating, and a blockage occurs. If this is solved by adding hardware, it will result in a huge waste of hardware and computing power. Summary of the Invention

[0004] This invention provides a convolution calculation method and apparatus to solve the problem of low computational efficiency when stride ≥ 2 in the deep learning computation process.

[0005] This method and apparatus improve computation speed and power consumption by reusing weights and input feature data and enabling multiple computing units to perform parallel computations. At the same time, by optimizing the computation order of the data, there is no data contention after the data is compressed, eliminating the need for additional storage hardware and time to handle computational blockages, and avoiding computational module stagnation caused by channel blockages. Furthermore, it is not only applicable to compressed weight data and feature data, but also to dense data.

[0006] This invention provides a convolution calculation method, specifically including:

[0007] Step S1: The weight data is rearranged and compressed using a compression algorithm. The weight data is obtained after the image is trained by a neural network model.

[0008] Step S2: According to the step size, classify the feature data and the weight data, where the feature data is image feature data;

[0009] Step S3: Distribute the same type of weight data to the first input of the multiplier;

[0010] Step S4: Select the feature data corresponding to the same type of weight data and broadcast it to the second input of the multiplier for multiplication calculation;

[0011] Step S5: The multiplier adds the calculation result to the data at a specific location in the memory and stores it in the specific location.

[0012] Step S6: Load the next type of weighted data, and repeat steps S4-S6 until the calculation is completed and the weighted image feature data is obtained.

[0013] Preferably, in step S4, starting from the starting position of the feature data, feature data corresponding to the same type of weight data are selected.

[0014] Preferably, the feature data corresponding to step S4 is non-zero data.

[0015] Preferably, in step S2, the number of categories for classifying the feature data and the weight data is the square of the step size value.

[0016] Preferably, in step S4, when multiplying feature data and weight data of the same class, the multiplication method of the feature data and weight data is determined by the step size.

[0017] Preferably, when the stride is 2 and the weight data is a 3*3 convolutional kernel, the feature data and the weight data are each divided into 4 categories.

[0018] Preferably, the feature data and weight data are multiplied in the following way:

[0019] The first type of feature data is multiplied by the four corner weight data;

[0020] The second type of feature data is multiplied by the weight data of the two horizontal sides;

[0021] The third type of feature data is multiplied by the weight data of the two vertical sides;

[0022] The fourth type of feature data is multiplied by one central weight data.

[0023] Preferably, in step S5, the specific location of the memory is uniquely determined by the coordinate information and channel information of the first and second input terminal data.

[0024] Preferably, the weight data is obtained after the speech and text are trained by a neural network model, and the feature data corresponds to speech feature data and text feature data. Step S6 corresponds to obtaining the weighted speech feature data and text feature data.

[0025] This invention provides a convolution calculation apparatus using the method described in claims 1-9, comprising:

[0026] The storage module includes external memory, internal memory, and output memory;

[0027] The calculation module includes a multiplier unit and an adder unit, wherein the multiplier unit includes a first input terminal and a second input terminal;

[0028] The control module includes an addresser for determining a unique specific location in the storage module based on the coordinates and channels of the input data from the first and second input terminals of the multiplier.

[0029] The control module also includes a controller for controlling the input and output of data;

[0030] The external memory is used to store feature data and weight data;

[0031] The multiplier unit is used to calculate the product of feature data and weight data.

[0032] The present invention has the following beneficial effects:

[0033] 1. Optimize the calculation process to completely avoid the coherence probability of the output results of the calculation units at the same time, eliminate data competition, and eliminate the need to add additional hardware to process potential competing data.

[0034] 2. Reuse weighted data and pulsed multiplexing of input feature data to reduce bandwidth requirements and data transfer power consumption;

[0035] 3. Supports parallel computing by multiple computing units;

[0036] 4. It supports weighted data compression, reducing bandwidth requirements and data migration power consumption, and skips zero-value calculations. When weighted data is sparse, it can significantly improve computing power and reduce power consumption.

[0037] 5. Optimize for stride ≥ 2, and be compatible with convolution calculations with stride = 1. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a flowchart of a convolution calculation method according to an embodiment of the present invention;

[0040] Figure 2 This is a schematic diagram of feature data and weight data input into a multiplier in one embodiment of the present invention;

[0041] Figure 3 This is a schematic diagram of the combination of a computing device and a method in one embodiment of the present invention.

[0042] In the above figure, MUL is a multiplier, column index is the column coordinate of the compressed data, row index is the row coordinate of the compressed data, and values ​​are the values ​​of the compressed data. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] Example 1

[0045] This embodiment provides a convolution calculation method where the weight data is sparse. The weight data is obtained after training an image using a neural network model; see [reference needed]. Figure 1 Before performing convolution calculations, the sparse data is first compressed using a compression algorithm (step S1) to reduce the storage space occupied by the data and decrease the bandwidth, time, and power consumption required for data movement. The compression algorithm includes, but is not limited to, the COO compression algorithm and the CSF compression algorithm. In this embodiment, the COO compression algorithm is used as an example to compress the weight data obtained after the image has been trained by the neural network model. In this embodiment, the weight data uses a 3×3 convolution kernel as an example. The calculation methods for other convolution kernel sizes can be obtained without creative effort and are all within the scope of this patent. The specific compression method includes:

[0046] First, the weight data is rearranged. In convolution calculations, weight data is represented in the form of convolution kernels. Multiple convolution kernels often participate in the calculation; therefore, the data from these kernels needs to be rearranged and combined to form a new matrix to meet the data format requirements of subsequent compression algorithms. This rearrangement of the weight data specifically includes:

[0047] The weight data (3×3) in the first convolution kernel is expanded into a row vector (1×9) and used as the first row of the weight data matrix; the weight data in the second convolution kernel is expanded into a row vector and used as the second row of the weight data matrix, and so on, until the weight data in all convolution kernels are rearranged.

[0048] Secondly, the rearranged weight data matrix is ​​compressed using the COO compression coding algorithm, and the compressed data is stored in the external memory of the computing unit. After compression using the COO compression coding algorithm, zeros in the sparse weight data matrix are omitted, and non-zero data is stored in the form of row coordinates, column coordinates, and values, which significantly reduces the storage space occupied by the data.

[0049] Compression of weighted data can be performed during the calculation process or beforehand outside of the calculation process.

[0050] The specific method for convolution calculation of the compressed data described above is as follows:

[0051] Step S2: Based on the step size, classify the feature data and the weight data. The feature data is image feature data; in this embodiment, the feature data is not compressed. The classification criteria for the feature data and weight data are as follows:

[0052] First, based on the stride, the number of feature data and weight data categories is determined, with the number of categories corresponding to the stride. In this embodiment, taking a stride of 2 as an example, the feature data is divided into four categories, the number of stride squared. Each category corresponds to a coordinate of the feature data. The weight data is also divided into four categories according to the multiplication relationship. Please refer to the following diagram for a schematic of the multiplication relationship. Taking a 3×3 convolution kernel with a stride of 2 as an example, the multiplication method of feature data and weight data is as follows:

[0053] The first type of feature data is multiplied by the first type of angle weight data (A);

[0054] The second type of feature data is multiplied by the second type of horizontal weight data (B);

[0055] The third type of feature data is multiplied by the third type of vertical edge weight data (C);

[0056] The fourth type of feature data is multiplied by the fourth type of center weight data (D).

[0057] The coordinates of the first type of feature data in the matrix are (2m-1, 2n-1) (m and n are positive integers);

[0058] The coordinates of the second type of feature data in the matrix are (2m-1, 2n) (m and n are positive integers);

[0059] The coordinates of the third type of feature data in the matrix are (2m, 2n-1) (m and n are positive integers);

[0060] The coordinates of the fourth type of feature data in the matrix are (2m, 2n) (m and n are positive integers);

[0061] A C A B D B A C A

[0062] 3×3" product $%&'()*+,

[0063] Step S3: Distribute the same type of weight data to the first input of the multiplier; the distribution means that different weight data are input to the first input of each multiplier in a many-to-many manner.

[0064] For step S4, please refer to [link / reference]. Figure 2 The feature data corresponding to the same type of weight data is selected and broadcast to the second input of the multiplier for multiplication. The selection of the weight data is implemented by the "addressing judgment unit". The addressing judgment unit extracts the feature data of the corresponding category from the starting address of the feature data matrix according to the type of weight data at the first input of the multiplier and the corresponding rules of the multiplication method. The feature data is broadcast to the second input of all multipliers. If the feature data is 0, it is skipped and the next data is searched.

[0065] In this embodiment, steps S3 and S4 are as follows: In the first round of calculation, the four corner weight data (A) of the first category are distributed to the first input of the parallel multipliers. The addressing unit retrieves the corresponding category weight data sequentially from the starting address of the feature data matrix according to the coordinates (2m-1, 2n-1) (m and n are positive integers), and broadcasts this data to the second input of all multipliers. If a feature data value of 0 is encountered, it is skipped, and the next data is searched. Multiplication is performed when both inputs of the multipliers have data.

[0066] Step S5: The adder adds the calculation result to the data at a specific location in the memory and stores it in the specific location;

[0067] The adder adds the calculation result to data at a specific location in the memory and then stores the result at that specific location.

[0068] Step S5 is implemented by an adder and an addresser. The adder is a 2-input, 1-output hardware device used to add two input data, calculate the sum, and output the result. The addresser is a hardware device that calculates a specific location in the memory based on coordinate information.

[0069] Since convolution is a multiplication-accumulation operation, it requires multiplying the corresponding feature values ​​and weight values ​​within the same convolution kernel range before adding them. In this embodiment, the addresser uniquely determines a specific location in memory based on the coordinates of the first input data, the coordinates of the second input data, and the current channel information. This location is initially set to 0. The multiplication results that need to be accumulated are added to the number at this specific location and then stored back at that location, thus achieving the accumulation effect.

[0070] After each round of multiplication, an addition is performed, and the result of the multiplication is added to the value at a specific location in memory.

[0071] After each round of multiplication and addition calculations is completed, if there are still remaining feature data that have not been multiplied with the weight data at the first input of the multiplier, then the weight data is fixed and the feature data at the second input is updated, until all feature data that need to be multiplied with the weight data at the first input has been calculated, then proceed to step S6.

[0072] Step S6: Load the next type of weighted data, and repeat steps S4-S6 until the calculation is completed and the weighted image feature data is obtained.

[0073] Example 2

[0074] This embodiment provides a convolution calculation device using the above-described convolution calculation method, including a storage module, a calculation module, and a control module. Please refer to [link to documentation]. Figure 3 :

[0075] The storage module includes an external memory, an internal memory, and an output memory; Figure 3 (The details of the storage module are not shown in the diagram.)

[0076] The calculation module includes a multiplier unit and an adder unit; the multiplier unit includes a first input terminal and a second input terminal.

[0077] The control module includes an addresser for determining a unique specific location in the storage module based on the coordinates and channels of the input data from the first and second input terminals of the multiplier.

[0078] The control module also includes a controller for controlling the input and output of data;

[0079] The external memory is used to store feature data and weight data;

[0080] The multiplier unit is used to calculate the product of the feature data and the weight data;

[0081] Example 2 is a hardware device for implementing Example 1. The functional limitations, connection limitations and descriptions of each module can be referred to the limitations and descriptions of the method, and will not be repeated here.

[0082] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A convolution calculation method, characterized in that, include: Step S1: Rearrange the weight data and compress the data using a compression algorithm. The weight data is obtained after the image has been trained by a neural network model. The compression includes: the weight data is represented in the form of convolution kernels, and the data in multiple convolution kernels are rearranged and combined to form a new matrix. Step S2: Determine the number of feature data and weight data categories based on the step size, and classify the feature data and the weight data. The feature data is image feature data. Step S3: Distribute the same type of weight data to the first input of the multiplier; Step S4: Select the feature data corresponding to the same type of weight data and broadcast it to the second input of the multiplier for multiplication calculation; Step S5: The multiplier adds the calculation result to the data at a specific location in the memory and stores it in the specific location. Step S6: Load the next type of weighted data, and repeat steps S4-S6 until the calculation is completed and the weighted image feature data is obtained.

2. The convolution calculation method according to claim 1, characterized in that, In step S4, starting from the beginning position of the feature data, feature data corresponding to the same type of weight data are selected.

3. The convolution calculation method according to claim 1 or 2, characterized in that, The feature data corresponding to step S4 is non-zero data.

4. The convolution calculation method according to claim 1, characterized in that, In step S2, the number of categories for classifying the feature data and the weight data is the square of the step size value.

5. The convolution calculation method according to claim 1, characterized in that, In step S4, when multiplying feature data and weight data of the same class, the multiplication method of the feature data and weight data is determined by the step size.

6. The convolution calculation method according to claim 4, characterized in that, When the stride is 2 and the weight data is a 3*3 convolutional kernel, the feature data and the weight data are each divided into 4 categories.

7. The convolution calculation method according to claim 6, characterized in that, The feature data and weight data are multiplied in the following way: The first type of feature data is multiplied by the first type of angle weight data; The second type of feature data is multiplied by the second type of horizontal weight data; The third type of feature data is multiplied by the third type of vertical edge weight data; The fourth type of feature data is multiplied by the fourth type of center weight data.

8. The convolution calculation method according to claim 1, characterized in that, In step S5, the specific location of the memory is uniquely determined by the coordinate information and channel information of the first and second input terminal data.

9. The convolution calculation method according to claim 1, wherein the weight data is obtained after the speech and text are trained by a neural network model, and the feature data corresponds to speech feature data and text feature data, and step S6 corresponds to obtaining weighted speech feature data and text feature data.

10. A convolution computation apparatus using the method of any one of claims 1-9, comprising: The storage module includes external memory, internal memory, and output memory; The calculation module includes a multiplier unit and an adder unit, wherein the multiplier unit includes a first input terminal and a second input terminal; The control module includes an addressing detector and an addresser; The addressing determiner is used to select the feature data corresponding to the weight data according to the corresponding rules of the multiplication method, and input it into the second input terminal of the multiplier unit; The addresser is used to determine a unique specific location in the storage module based on the coordinates and channels of the input data from the first and second input terminals of the multiplier; The control module also includes a controller for controlling the input and output of data; The external memory is used to store feature data and weight data; The multiplier unit is used to calculate the product of feature data and weight data.

Images