Low-power-consumption convolution calculation system for event stream data

Abstract

The invention discloses a low-power-consumption convolution calculation system for event stream data, which converts the convolution processing of the event stream data into local calculation by utilizing the dilution of the event stream, thereby avoiding redundant calculation and reducing energy consumption. The system comprises a mapping calculation module, a scattering addition module and a result reconstruction module, the mapping calculation module processes input event stream data according to convolution kernel bias to obtain a mapping table of input coordinates and output coordinates; the scattering addition module inputs the mapping table, extracts corresponding weight data from the storage device and adds the weight data to a corresponding position of the output matrix; and the result reconstruction module reconstructs the output matrix into a proper output tensor and then stores the output tensor. In the system, calculation only occurs at the place where an event occurs, redundant calculation is avoided, meanwhile, the sparsity of event stream data is utilized, unnecessary storage access overhead is reduced, and power consumption is greatly reduced.

Description

Technical Field

[0001] The present invention belongs to the fields of artificial intelligence, neural network technology, neuromorphic data and hardware acceleration, and in particular relates to a low-power convolution computing system for event stream data. Background Art

[0002] With the widespread popularity of smart devices and the Internet of Things, the demand for real-time data processing has increased dramatically, especially in low-power embedded devices. As an efficient and timely way to process continuous data streams, event stream data processing technology has been widely used in various scenarios, such as autonomous driving, intelligent monitoring, edge computing, etc. However, due to the high frequency, unstructured, and sparse characteristics of event stream data, there is a large energy consumption and computational burden when performing efficient convolution calculations. This is particularly challenging for resource-constrained embedded devices.

[0003] Traditional convolutional computing systems are mostly based on static data, and when directly applied to event stream data, there will be very large unnecessary calculations. Current research and technology have not yet found an effective solution for optimizing power consumption when processing event stream data. Therefore, there is an urgent need for a convolutional computing system that can be optimized for event stream data and has low power consumption characteristics to meet the growing real-time processing needs. Summary of the invention

[0004] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a low-power convolution computing system for event stream data, which optimizes operations such as mapping table generation and weight accumulation in the convolution calculation process, reduces resource consumption in the calculation process, and improves efficiency.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] A low-power convolution computing system for event stream data, comprising a mapping module, a scattering addition module, a result reconstruction module and a storage module; the storage module is used to store convolution kernel information, weight information and output tensors;

[0007] The mapping module receives event stream data as input, combines the convolution kernel information obtained from the storage module, and processes the event stream data to obtain a mapping table of input coordinates and output coordinates;

[0008] The scattering addition module inputs the mapping table, combines the weight information obtained from the storage module, adds it to the corresponding position of the output matrix, obtains the output matrix and stores it;

[0009] The result reconstruction module receives the output matrix, combines the convolution kernel information obtained from the storage module, reconstructs it into a suitable output tensor and stores it.

[0010] Furthermore, the mapping module first checks whether the convolution kernel information and the event stream data are aligned and whether the convolution calculation can be performed, and then calculates the output coordinates according to the bias of different positions of the convolution kernel and the input event stream coordinates, and finally combines the input coordinates, bias and output coordinates into a mapping table.

[0011] Furthermore, after receiving the mapping table, the scatter addition module first arranges the mapping table appropriately to reduce repeated storage access, then obtains the weight index according to the bias calculation in the mapping table, extracts the corresponding weight from the storage according to the index, obtains the output index according to the output coordinate calculation, and calls the adder on the hardware device to accumulate the corresponding weight and write it to the output matrix position corresponding to the output index.

[0012] Furthermore, during the accumulation process, if the event stream data itself has a numerical attribute, its input coordinates are taken out according to the mapping table, and a numerical value is obtained from the storage according to the input coordinates. The numerical value is then multiplied by the weight and then accumulated. If the numerical type is an integer and the range is small, the above multiplication can be replaced by multiple additions to obtain lower energy consumption.

[0013] Furthermore, the structure reconstruction module first calculates the output tensor size, and then reconstructs the output matrix into an output tensor according to the specific size for storage and output.

[0014] Furthermore, if the storage of the output matrix and the output tensor both satisfy the same linear continuous storage property, the two share the underlying storage space, and the operation of this module can be hidden, further reducing storage access.

[0015] Furthermore, the scatter-add module performs pipeline operation, and executes the three steps of extracting from storage, accumulating and writing in a pipeline, thereby concealing the time consumption of read and write operations and calculation operations.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] 1) Reduce power consumption: The convolution calculation method proposed in the present invention significantly reduces the amount of calculation and the number of data transmissions in the event stream data processing process through the optimization operations of mapping and scatter addition, thereby reducing power consumption. It is particularly suitable for embedded devices and Internet of Things application scenarios with low power consumption requirements.

[0018] 2) Improve computing efficiency: By constructing a mapping table of input and output coordinates, the convolution kernel calculation and data weight operations are simplified to coordinate mapping and weight accumulation, avoiding a large number of repeated calculation operations, greatly improving computing efficiency, and further improving computing efficiency by using the pipeline operation of reading, writing and computing. Especially when the event stream data is sparse, the present invention can more effectively reduce redundant calculations.

[0019] 3) The intermediate links of the system of the present invention only need to store data with event locations, thus avoiding the storage of a large amount of useless data and saving storage overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is an example of event stream data.

[0021] Figure 2 It is a schematic diagram of the system structure of the present invention.

[0022] Figure 3 It is a schematic diagram of the mapping module of the present invention and its input and output.

[0023] Figure 4 It is a schematic diagram of the scattering-adding module of the present invention.

[0024] Figure 5 It is a schematic diagram of the reconstruction of the results of the present invention.

[0025] Figure 6 It is a schematic diagram of the pipeline operation of the scattering-adding module of the present invention. DETAILED DESCRIPTION

[0026] The embodiments of the present invention are described in detail below with reference to the accompanying drawings and examples.

[0027] The present invention is a low-power convolution computing system for event stream data. By utilizing the dilution property of event streams, the system converts the convolution processing of event stream data into local computing, thus avoiding redundant computing and reducing energy consumption.

[0028] The event stream data of the present invention is as follows Figure 1 As shown, in XY space, event i can be expressed as (x i ,y i ,t i ), where t i The time dimension is taken as the convolution calculation of 1-dimensional event stream data. For the convenience of description, in this embodiment, the input channel of the 1-dimensional event stream data is the second dimension. In this embodiment, the convolution kernel information includes the convolution kernel size, the number of input channels (C in ), number of output channels (t out), stride, padding, dilation, etc. The convolution kernel weight is used as weight information and is stored in the storage module together with the convolution kernel information. The storage module of the present invention can be any device, unit or device that can realize the storage function.

[0029] like Figure 2 As shown, the low-power convolution computing system for event stream data of the present invention includes three main modules, namely, a mapping module, a scattering addition module, and a result reconstruction module, and can obtain storage convolution kernel information and weight information from the storage module and store them in the output tensor. Obviously, the storage module can be a part of the system of the present invention or a separate external part. After receiving the event stream data input, the system of the present invention is processed by the mapping module, the scattering addition module, and the result reconstruction module in turn. The main functions of the above three main modules are as follows:

[0030] The mapping module obtains the convolution kernel information from the storage module, inputs the event stream data for processing, and obtains a mapping table of the input coordinates and output coordinates of the event stream data.

[0031] The mapping table obtained by the mapping module is input into the scattering addition module, and the corresponding weight information is extracted from the storage module, added to the corresponding position of the output matrix, and the output matrix is ​​obtained and stored.

[0032] The result reconstruction module receives the output matrix, obtains the convolution kernel information from the storage module, reconstructs the output matrix into a suitable output tensor and stores it.

[0033] The mapping module is mainly used to generate mapping tables, such as Figure 3 As shown, in the mapping module, it is necessary to first check whether the convolution kernel information and the event stream data are aligned to determine whether the convolution calculation can be performed. If the convolution information and the correct stream data are not aligned, the system reports an error and needs to re-enter the aligned convolution information and event stream data. In this embodiment, the module will determine whether the coordinate length and the convolution kernel size match and whether the number of input channels of the two are consistent.

[0034] After confirming the alignment, the output coordinates are calculated according to the offsets of different positions of the convolution kernel and the input event stream data coordinates, and it is determined whether the output coordinates exceed the output tensor size. If yes, they are discarded, otherwise they are added to the mapping table. In this embodiment, the module uses the following formula to calculate the output coordinate O for each event input coordinate I and each position offset of the convolution kernel. In this embodiment, the coordinate I has only one dimension.

[0035]

[0036] Where P is the convolution kernel padding size, D is the convolution kernel expansion size, S is the convolution kernel step size, and K offsetis the coordinate of the current position offset of the convolution kernel. So for each input coordinate I, there will be the same number of output coordinates as the convolution kernel size.

[0037] Finally, the input coordinates, bias and output coordinates are combined into a mapping table. The convolution kernel information, event stream coordinates and mapping table format are as follows: Figure 3 shown.

[0038] The scatter addition module mainly performs addition operations to obtain the output matrix, such as Figure 4 As shown, after receiving the mapping table, the scatter-add module first arranges the mapping table appropriately to reduce repeated storage access. In this embodiment, the module arranges the items with the same output index in the mapping table together. Then, the weight index is calculated according to the offset in the mapping table. Specifically, the starting index corresponding to the weight W is WI and the offset offset, and the current weight index is defined as WI+offset.

[0039] The corresponding weight is extracted from the storage module according to the weight index, and the output index is calculated according to the output coordinates. Specifically, the starting index corresponding to the output matrix O is OI, the output coordinates are Coords, and the output index is OI+Linear(Coords). The Linear() function converts the coordinates into an integer value according to the output matrix storage method. In this example, the output matrix is ​​a two-dimensional matrix stored in row priority, the matrix size is X*Y, the output coordinates are [x, y], then Linear(Coords) is x*Y+y, and the output index is OI+x*Y+y.

[0040] Call the accumulator or adder on the hardware device to accumulate the corresponding weights. The formula is as follows:

[0041]

[0042] Among them, offset i Represents the bias of the entry in the mapping table whose output coordinates are Coords.

[0043] Finally, the result is written into the output matrix position corresponding to the output index. In this embodiment, because the mapping table is arranged, the same output index only needs to be written and stored once.

[0044] In the accumulation process, if the event stream data itself has a numerical attribute, its input coordinates are taken out according to the mapping table, and the numerical value is obtained from the storage module according to the input coordinates. Then, the numerical value is multiplied by the weight and then accumulated. The formula is expressed as follows:

[0045]

[0046] If the value type is an integer and the range is less than the preset threshold, the multiplication is replaced by multiple additions to obtain lower energy consumption, which can be expressed as:

[0047]

[0048] Where M is a numerical value, that is, the numerical characteristics of the event stream data event itself, such as the event amplitude.

[0049] In this embodiment, the event stream data is binary and does not require additional multiplication operations. It only needs to call the adder to perform the addition operation.

[0050] The structure reconstruction module is mainly used to realize the tensor representation of the output matrix, such as Figure 5 As shown, the structure reconstruction module first calculates the size of the output tensor. In this embodiment, the size of the output tensor is calculated according to the following formula:

[0051]

[0052] Where O_len is the output tensor length, I_len is the input length, K is the convolution kernel size, P is the convolution kernel padding size, D is the convolution kernel expansion size, and S is the convolution kernel step size.

[0053] Then, according to the specific size, the corresponding values ​​in the output matrix are copied to the output tensor according to the storage position and storage order of the output tensor in the storage space, so as to reconstruct the output matrix into the output tensor, store it and output it. In this embodiment, [C out ,O_len] is rearranged, where C out is the number of output channels.

[0054] In the present invention, in order to reduce storage access, if the storage of the output matrix and output tensor of the introduced weights both satisfy the property of linear continuous storage, the output matrix and output tensor of the introduced weights share the storage space in the storage module, and the operation of the result reconstruction module is hidden, that is, there is no need to execute this module.

[0055] like Figure 6 As shown, the scatter-add module of the present invention can be executed in a pipeline, that is, the three steps of extracting from storage, accumulating and writing are executed in a pipeline, thereby covering up the time consumption of read and write operations and calculation operations.

[0056] In the embodiment of the present invention, the above method is used to obtain the convolution result, which can ensure the correctness of the convolution result. At the same time, assuming that the event stream data sparsity is α (α<1), the input dimension is [N, Cin, L], the kernel size is [k, Cin, Cout], and the output is [N, Cout, L]. Let N s=N*Cin*L, and compared with the common convolution calculation system, img2Col is as follows,

[0057]

[0058] In hardware, the cost of multiplication and addition is significantly higher than that of addition. From the above table, it can be seen that the present invention uses addition instead of multiplication and addition, which can effectively reduce energy consumption and have better storage space overhead and time complexity. In practice, α is usually around 0.2, so the present invention reduces storage overhead by 5 times and running time by 50%.

[0059] The method or system of the present invention can be implemented in an electronic device, and the method of the present invention can be executed in the electronic device, and the system obtained by the training of the present invention can also be implanted in the electronic device. Further, the electronic device can include a picture acquisition unit and a final display unit, and the pictures collected by the acquisition unit can be directly or after necessary preprocessing sent to the system of the present invention, and the final recognition result can be output and displayed on the display unit.

[0060] Take the DVS128-Gesture gesture recognition dataset as an example. This dataset is shot with an event camera and is event stream data. Its length and width are 128*128, and the input coordinate I is in the form of [t, x, y], where t represents the time coordinate and x, y represent the space coordinate. The convolution kernel information used is that the convolution kernel size is 3*3, the number of input channels is 1, the number of output channels is 32, the step size is 1, the padding size is 0, and the expansion size is 1.

[0061] like Figure 3 As shown, in this embodiment, the coordinate space dimension matches the convolution kernel size, and the number of input channels of the two is the same, so they are aligned and can perform convolution operations. Then the output coordinates are calculated based on the offsets of different positions of the convolution kernel and the input event stream coordinates. The mapping module offsets each event input coordinate I and each position of the convolution kernel. In this embodiment, the convolution kernel has a total of 9 position offsets, and the output coordinate O is calculated using the following formula:

[0062]

[0063] As mentioned above, here P width =P Heig =0,D width =D Heigh =1, S width =S Hei =3. Finally, the output coordinates are obtained Determine whether this output coordinate exceeds the output tensor size, if yes, discard it, otherwise add it to the mapping table. So the final mapping table format is [I, O, W], where W is the weight corresponding to the offset.

[0064] like Figure 4 As shown, after receiving the mapping table, the scatter addition module first arranges the mapping table appropriately. In this embodiment, the module will arrange the items with the same output coordinate O in the mapping table together. Then, the weight index is calculated based on the offset in the mapping table. In this embodiment, the initial weight index is WI, then the weight index is WI+offset, and the corresponding weight is extracted from the storage according to the index. The output index is calculated based on the output coordinate. The implementation method is as follows: The starting index corresponding to the output matrix O is OI. In this example, the output matrix is ​​a two-dimensional matrix stored in row priority, the matrix size is X*Y, and the output coordinate is [x, y], then the output index is OI+x*Y+y. The accumulator on the hardware device is called to accumulate the corresponding weights and write them to the output matrix position corresponding to the output index. The implementation method is as follows: O[OI+x*Y+y]=∑ i W[WI+offset i ], in this embodiment, the number of output channels is Cout=32, so the weight read is a 32-dimensional vector, and the addition is a 32-dimensional vector. In this embodiment, because the mapping table is arranged, only one write is required for the same output index. In this embodiment, the event stream data is binary, has no additional numerical features, and does not require additional multiplication operations. It only needs to call the adder to perform the addition operation.

[0065] like Figure 5 As shown, the structure reconstruction module first calculates the size of the output tensor. In this embodiment, the size of the output tensor is calculated according to the following formula:

[0066]

[0067]

[0068] Where O_Hlen and O_Wlen are the output tensor lengths, I_Wlen and I_Hlen are the input tensor lengths of 128, K width and K Height The convolution kernel size is 3, P width , P Height , D width , D Height , S width and S Height Same as above. Then, according to the specific size, the output matrix is ​​reconstructed into an output tensor for storage and output. In this embodiment, [t, C out, O_Wlen, O_Hlen] are rearranged, where C out is the number of output channels. The implementation method is as follows: according to the storage position and storage order of the output tensor in the storage space, the corresponding value in the output matrix is ​​copied to the output tensor, that is, for the value of the [t, c, x, y] coordinate in the output tensor, the index t*C in the output matrix is ​​copied out *O_Wlen*O_Hlen+c*O_Wlen*O_Hlen+x*O_Hlen+y value.

Claims

1. A low-power convolution computing system for event stream data, characterized in that: It includes a mapping module, a scattering addition module, a result reconstruction module and a storage module; the storage module is used to store convolution kernel information, weight information and output tensor; The mapping module receives event stream data as input, combines the convolution kernel information obtained from the storage module, and processes the event stream data to obtain a mapping table of input coordinates and output coordinates; The scattering addition module inputs the mapping table, combines the weight information obtained from the storage module, adds it to the corresponding position of the output matrix, obtains the output matrix and stores it; The result reconstruction module receives the output matrix, combines the convolution kernel information obtained from the storage module, reconstructs it into a suitable output tensor and stores it.

2. A low-power convolution computing system for event stream data according to claim 1, characterized in that: In the mapping module, first check whether the convolution kernel information and the event stream data are aligned. After confirming the alignment, calculate the output coordinates according to the offsets of different positions of the convolution kernel and the input event stream data coordinates, and determine whether the output coordinates exceed the output tensor size. If so, discard them; otherwise, add them to the mapping table. Finally, combine the input coordinates, offsets and output coordinates into a mapping table.

3. A low-power convolution computing system for event stream data according to claim 2, characterized in that: The output coordinates are calculated according to the offsets of different positions of the convolution kernel and the input event stream data coordinates. The implementation method is as follows: According to the input coordinates I of each event and the offset of each position of the convolution kernel, the output coordinates O are calculated using the following formula: Where P is the convolution kernel padding size, D is the convolution kernel expansion size, S is the convolution kernel step size, and K offset The coordinates of the current position offset of the convolution kernel. For each event input coordinate I, there will be the same number of output coordinates as the convolution kernel size.

4. A low-power convolution computing system for event stream data according to claim 1, characterized in that: The scattering addition module obtains a weight index according to the bias calculation in the mapping table, extracts the corresponding weight from the storage module according to the weight index, obtains the output index according to the output coordinate calculation, and calls the adder on the hardware device to accumulate the corresponding weight and write it to the output matrix position corresponding to the output index.

5. A low-power convolution computing system for event stream data according to claim 4, characterized in that: The weight index is obtained according to the offset calculation in the mapping table, and the implementation method is as follows: for the starting index corresponding to the weight W is WI and the offset offset, the current weight index is WI+offset; The output index is calculated based on the output coordinates, and the implementation method is as follows: For the output matrix O, the starting index corresponding to it is OI, and the output coordinates are Coords, then the output index is OI+Linear(Coords). The Linear() function converts the coordinates into an integer value according to the output matrix storage method. For a two-dimensional output matrix with a matrix size of X*Y, the output coordinates are [x, y], then Linear(Coords) is x*Y+y, and the output index is OI+x*Y+y. The corresponding weights are accumulated, and the implementation method is as follows: Among them, offset i Represents the bias of the entry in the mapping table whose output coordinates are Coords.

6. A low-power convolution computing system for event stream data according to claim 5, characterized in that: In the accumulation process, if the event stream data itself has a numerical attribute, its input coordinates are taken out according to the mapping table, and a numerical value is obtained from the storage module according to the input coordinates. Then, the numerical value is multiplied by the weight and then accumulated, which is expressed as follows: If the value type is an integer and the range is less than the preset threshold, the multiplication is replaced by multiple additions to obtain lower energy consumption, which can be expressed as: Where M is a numerical value.

7. A low-power convolution computing system for event stream data according to claim 4, 5 or 6, characterized in that: After receiving the mapping table, the scatter-add module first arranges the mapping table to reduce repeated storage accesses. The arrangement rule is: putting items in the mapping table corresponding to the same output index together.

8. A low-power convolution computing system for event stream data according to claim 1, characterized in that: The structure reconstruction module first calculates the output tensor size, and the implementation method is as follows: Where O_len is the output tensor length, I_len is the input length, K is the convolution kernel size, P is the convolution kernel padding size, D is the convolution kernel expansion size, and S is the convolution kernel step size; Then, according to the specific size, the corresponding values ​​in the output matrix are copied to the output tensor according to the storage location and storage order of the output tensor in the storage space, so as to reconstruct the output matrix into the output tensor.

9. A low-power convolution computing system for event stream data according to claim 8, characterized in that: If the storage of the output matrix and output tensor of the introduced weights both satisfy the property of linear continuous storage, the output matrix and output tensor of the introduced weights share the storage space in the storage module, and the operation of the result reconstruction module is hidden, that is, there is no need to execute this module, further reducing storage access.

10. A low-power convolution computing system for event stream data according to claim 9, characterized in that: The scatter-add module performs pipeline operation, and executes the three steps of extracting, accumulating and writing from the storage module in a pipeline, thereby covering up the time consumption of the reading and writing operations and the calculation operations.

Images