A low-power convolution computing system for event stream data

Abstract

A low-power convolution computing system for event stream data, which utilizes event stream sparsity, converts convolution processing of event stream data into local computation, avoids redundant computation, and reduces energy consumption. The system includes a mapping computation module, a scattering addition module, and a result reconstruction module. The mapping computation module processes input event stream data according to a convolution kernel offset to obtain a mapping table of input coordinates and output coordinates. The scattering addition module inputs the mapping table, extracts corresponding weight data from a storage device, and adds the weight data to a corresponding position of an output matrix. The result reconstruction module reconstructs the output matrix into a suitable output tensor and then stores the output tensor. In the system, computation only occurs at event occurrence locations, redundant computation is avoided, and unnecessary storage access overhead is reduced by utilizing event stream data sparsity, thereby greatly reducing power consumption.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of artificial intelligence, neural network technology, neural morphological data and hardware acceleration, and particularly relates to a low-power convolution calculation system for event stream data. BACKGROUND

[0002] With the wide popularity of intelligent devices and the Internet of Things, the demand for real-time data processing has increased dramatically, especially in low-power embedded devices. Event stream data processing technology, as an efficient and timely way to process continuous data streams, has been widely used in various scenarios such as autonomous driving, intelligent monitoring, edge computing, etc. However, due to the characteristics of high frequency, unstructured, and sparsification of event stream data, there is a great energy consumption and computational burden in efficient convolution calculation. This is particularly challenging for resource-constrained embedded devices.

[0003] Traditional convolution calculation systems are mostly based on static data, and directly applying them to event stream data will result in a lot of unnecessary calculations. Current research and technology have not yet found an effective solution to the power consumption optimization of event stream data processing. Therefore, there is an urgent need for a convolution calculation system that can optimize event stream data and has low power consumption to meet the growing demand for real-time processing. SUMMARY

[0004] In order to overcome the above-mentioned shortcomings of the prior art, the purpose of the present application is to provide a low-power convolution calculation system for event stream data, which optimizes the mapping table generation and weight accumulation operations in the convolution calculation process, reduces the consumption of resources in the calculation process, and improves the efficiency.

[0005] In order to achieve the above-mentioned purpose, the technical solution adopted by the present application is:

[0006] A low-power convolution calculation system for event stream data, comprising a mapping module, a scattering addition module, a result reconstruction module and a storage module; the storage module is used to store convolution kernel information, weight information and output tensor;

[0007] The mapping module inputs event stream data, and processes the mapping table of event stream data input coordinates and output coordinates in combination with the convolution kernel information obtained from the storage module;

[0008] The scattering addition module inputs the mapping table, and adds to the corresponding position of the output matrix in combination with the weight information obtained from the storage module to obtain the output matrix and store it;

[0009] The result reconstruction module receives the output matrix, combines the convolution kernel information obtained from the storage module, reconstructs it into a suitable output tensor and stores it.

[0010] Further, the mapping module first checks whether the convolution kernel information and the event stream data are aligned and whether convolution calculation can be performed, then calculates the output coordinates according to the bias of different positions of the convolution kernel and the input event stream coordinates, and finally combines the input coordinates, the bias and the output coordinates into a mapping table.

[0011] Further, the scattering addition module, after receiving the mapping table, first arranges the mapping table appropriately to reduce repeated storage access, then calculates the weight index according to the bias in the mapping table, extracts the corresponding weight from the storage according to the index, calculates the output index according to the output coordinates, and calls the adder on the hardware device to add the corresponding weight and write it to the output matrix position corresponding to the output index.

[0012] Further, in the accumulation process, if the event stream data itself has a numerical property, the input coordinates are taken according to the mapping table, the numerical value is obtained from the storage according to the input coordinates, and then the numerical value and the weight are multiplied and added, if the numerical type is an integer, the range is small, and the above multiplication can be replaced by multiple additions to obtain lower energy consumption.

[0013] Further, the structure reconstruction module first calculates the output tensor size, then reconstructs the output matrix into an output tensor according to the specific size, and stores and outputs.

[0014] Further, if the storage of the output matrix and the output tensor both satisfy the same linear continuous storage property, the two share the underlying storage space, and the operation of this module can be hidden, further reducing storage access.

[0015] Further, the scattering addition module performs pipelining operation, and the three steps of extraction, accumulation and writing from the storage are performed in pipelining, thereby masking the time consumption of read-write operation and calculation operation.

[0016] Compared with the prior art, the beneficial effects of the present application are:

[0017] 1) Reduce power consumption: the convolution calculation method proposed in the present application significantly reduces the calculation amount and data transmission times in the event stream data processing process through the optimization operation of mapping and scattering addition, thereby reducing the power consumption, and is particularly suitable for low-power demand embedded devices and Internet of Things application scenarios.

[0018] 2), improve the calculation efficiency: by constructing the mapping table of input and output coordinates, the convolution kernel calculation and data weight operation are simplified to coordinate mapping and weight accumulation, avoiding a large number of repeated calculation operations, greatly improving the calculation efficiency, and further improving the calculation efficiency by using the flow running of reading and writing and calculation. Especially in the case of sparse event stream data, the present application can more effectively reduce redundant calculation.

[0019] 3), the intermediate link of the system only needs to store data with event position, avoiding a large amount of useless data storage and saving storage overhead. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 is an event stream data example.

[0021] Figure 2 is a schematic diagram of the system structure of the present application.

[0022] Figure 3 is a schematic diagram of the mapping module and its input and output of the present application.

[0023] Figure 4 is a schematic diagram of the scattering addition module of the present application.

[0024] Figure 5 is a schematic diagram of the result reconstruction of the present application.

[0025] Figure 6 is a schematic diagram of the flow running of the scattering addition module of the present application. DETAILED DESCRIPTION

[0026] The embodiment of the present application will be described in detail below in combination with the drawings and examples.

[0027] The present application is a low-power convolution calculation system for event stream data, which utilizes the dilution of event stream, converts the convolution processing of event stream data into local calculation, avoids redundant calculation, and reduces energy consumption.

[0028] The event stream data of the present application is shown in Figure 1 In the XY space, event i can be represented as (x i , y i , t i ), wherein t i is the time dimension, taking the convolution calculation of 1-dimensional event stream data as an example, in order to facilitate the description, in this embodiment, the input channel of 1-dimensional event stream data is the second dimension, and the convolution kernel information in this embodiment includes the convolution kernel size, the input channel number (C in ), the output channel number (t outstride, padding, dilation, etc. The convolution kernel weight is stored in the storage module as weight information together with the convolution kernel information. The storage module can be any device or unit or element capable of storing information.

[0029] As shown in Figure 2 The low-power convolution calculation system for event stream data of the present application comprises three main modules, namely, a mapping module, a scattering addition module, and a result reconstruction module, and can obtain stored convolution kernel information and weight information from the storage module and store the output tensor. Obviously, the storage module can be part of the system of the present application or a separate external part. After receiving the event stream data input, the system of the present application is processed in turn by the mapping module, the scattering addition module, and the result reconstruction module. The main functions of the above three main modules are as follows:

[0030] The mapping module obtains the convolution kernel information from the storage module, processes the input event stream data, and obtains the mapping table of the event stream data input coordinates and output coordinates.

[0031] The scattering addition module inputs the mapping table obtained from the mapping module, extracts the corresponding weight information from the storage module, adds it to the corresponding position of the output matrix, obtains the output matrix, and stores it.

[0032] The result reconstruction module receives the output matrix, obtains the convolution kernel information from the storage module, reconstructs the output matrix into a suitable output tensor, and stores it.

[0033] The mapping module is mainly used to generate the mapping table, as shown in Figure 3 In the mapping module, it is necessary to check whether the convolution kernel information and the event stream data are aligned to determine whether the convolution calculation can be performed. If the convolution information and the correct stream data are not aligned, the system will report an error and the aligned convolution information and event stream data need to be re-input. In this embodiment, the module will determine whether the coordinate length and the convolution kernel size are matched and whether the number of input channels of the two is consistent.

[0034] After confirming the alignment, the output coordinates are calculated according to the offset of different positions of the convolution kernel and the input event stream data coordinates, and it is determined whether the output coordinates exceed the size of the output tensor. If yes, it is discarded, otherwise it is added to the mapping table. In this embodiment, for each event input coordinate I and the offset offset of each position of the convolution kernel, the output coordinate O is calculated using the following formula. In this embodiment, the coordinate I has only one dimension.

[0035]

[0036] where P is the padding size of the convolution kernel, D is the dilation size of the convolution kernel, S is the stride of the convolution kernel, K offsetThe coordinate of the current position of the convolution kernel is offset. So for each input coordinate I, there will be the same number of output coordinates as the size of the convolution kernel.

[0037] Finally, the input coordinate, offset and output coordinate are combined into a mapping table. The convolution kernel information, event stream coordinate and mapping table format are shown as Figure 3

[0038] The scatter addition module mainly performs addition operation to obtain the output matrix, as shown in Figure 4 After receiving the mapping table, the scatter addition module first arranges the mapping table properly to reduce repeated memory access. In this embodiment, the module arranges the items with the same output index in the mapping table together. Then the weight index is calculated according to the offset in the mapping table. Specifically, for the starting index WI corresponding to the weight W and the offset, the current weight index is defined as WI+offset.

[0039] The corresponding weight is extracted from the storage module according to the weight index, and the output index is calculated according to the output coordinate. Specifically, for the starting index OI corresponding to the output matrix O and the output coordinate Coords, the output index is OI+Linear(Coords). The Linear() function converts the coordinate into an integer value according to the storage method of the output matrix. In this example, the output matrix is a two-dimensional matrix stored in row-major order, the matrix size is X*Y, and the output coordinate is [x, y]. Then Linear(Coords) is x*Y+y, and the output index is OI+x*Y+y.

[0040] Call the accumulator or adder on the hardware device to accumulate the corresponding weight, and the formula is as follows:

[0041]

[0042] Where offset i represents the offset of the item with the i-th output coordinate Coords in the mapping table.

[0043] Finally, the result is written to the output matrix position corresponding to the output index. In this embodiment, because the mapping table is arranged, only one storage is needed for the same output index.

[0044] In the accumulation process, if the event stream data itself has a numerical property, then according to the mapping table, the input coordinate is taken out, the numerical value is obtained from the storage module according to the input coordinate, and then the numerical value and the weight are multiplied and accumulated, which can be expressed by the formula:

[0045]

[0046] ​If the numerical value is an integer and its range is less than a preset threshold, multiplication is replaced with multiple additions to achieve lower energy consumption, expressed by the formula:

[0047]

[0048] Where M is a numerical value, that is, the numerical characteristics of the event itself in the event stream data, such as the event magnitude.

[0049] In this embodiment, the event stream data is binary, so no additional multiplication operation is required; only the adder needs to be called to perform the addition operation.

[0050] The structure reconstruction module is mainly used to implement the tensor representation of the output matrix, such as... Figure 5 As shown, the structure reconstruction module first calculates the output tensor size. In this embodiment, the size of the output tensor is calculated according to the following formula:

[0051]

[0052] Where O_len is the output tensor length, I_len is the input length, K is the kernel size, P is the kernel padding size, D is the kernel expansion size, and S is the kernel stride.

[0053] Then, based on the specific dimensions, and according to the storage location and order of the output tensor in the storage space, the corresponding values ​​in the output matrix are copied into the output tensor, thereby reconstructing the output matrix into an output tensor, and then storing and outputting it. In this embodiment, this will be done according to [C out The shape of [,O_len] is rearranged, where C out This represents the number of output channels.

[0054] In this invention, to reduce storage access, if the storage of the weighted output matrix and the output tensor both satisfy the property of linear continuous storage, then the weighted output matrix and the output tensor share the storage space in the storage module, and the operation of the result reconstruction module is hidden, that is, this module does not need to be executed.

[0055] like Figure 6 As shown, the scattering addition module of the present invention can be executed in a pipelined manner, that is, the three steps of retrieving, accumulating and writing from storage are executed in a pipelined manner, thereby masking the time consumption of read and write operations and calculation operations.

[0056] In the embodiments of the present invention, the above method is used to obtain the convolution result, which can ensure the correctness of the convolution result. Simultaneously, it is assumed that the sparsity of the event stream data is α (α<1), the input dimension is [N,Cin,L], the kernel size is [k,Cin,Cout], and the output is [N,Cout,L]. Let N... s=N*Cin*L, and common convolution calculation system, compared with img2Col as follows,

[0057]

[0058] In hardware, the cost of multiply-add operation is significantly higher than that of addition operation, as can be seen from the above table, the present application uses addition operation instead of multiply-add operation, which can effectively reduce energy consumption, and has better storage space overhead and time complexity. In practice, alpha is usually about 0.2, so the present application reduces the storage overhead by 5 times and the running time by 50%.

[0059] The method or system of the present application can be implemented in an electronic device, the method of the present application is executed in the electronic device, and the system obtained by training of the present application can be implanted in the electronic device. Further, the electronic device can include a picture acquisition unit and a final display unit, the pictures acquired by the acquisition unit can be directly or after necessary preprocessing into the system of the present application, and the final recognition result is output and displayed on the display unit.

[0060] Taking the DVS128-Gesture gesture recognition dataset as an example, the dataset is captured by an event camera and is event stream data. Its length and width are 128*128, the input coordinate I is in the form of [t, x, y], wherein t represents the time coordinate, x and y represent the space coordinate. The convolution kernel information used is that the convolution kernel size is 3*3, the input channel number is 1, the output channel number is 32, the step is 1, the padding size is 0, and the expansion size is 1.

[0061] As shown in Figure 3 In the embodiment, the coordinate space dimension and the convolution kernel size are matched, the input channel numbers of the two are consistent, so they are aligned and can be convolved. Then the output coordinates are calculated according to the bias of different positions of the convolution kernel and the input event stream coordinates, and the mapping module calculates the output coordinates O for each event input coordinate I and the bias offset of each position of the convolution kernel. In the embodiment, the convolution kernel has 9 position biases offset, and the output coordinates O are calculated using the following formula:

[0062]

[0063] As described above, here P width =P Heig =0, D width =D Heigh =1, S width =S Hei =3. Finally, the output coordinates Determine if the output coordinates exceed the output tensor size; if so, discard them; otherwise, add them to the mapping table. Therefore, the final mapping table format is [I, O, W], where W is the weight corresponding to the offset.

[0064] like Figure 4 As shown, after receiving the mapping table, the scattering and adding module first arranges the mapping table appropriately. In this embodiment, the module arranges items with the same output coordinate O together. Then, it calculates the weight index based on the offset in the mapping table. In this embodiment, the initial weight index is WI, so the weight index is WI + offset. The corresponding weight is retrieved from storage according to the index, and the output index is calculated based on the output coordinates. The implementation method is as follows: For the initial index corresponding to the output matrix O, which is OI, in this example, the output matrix is ​​a row-major two-dimensional matrix with a size of X*Y and output coordinates [x, y]. Therefore, the output index is OI + x*Y + y. The accumulator on the hardware device is called to accumulate the corresponding weights and write them to the output matrix position corresponding to the output index. The implementation method is as follows: O[OI + x*Y + y] = ∑ i W[WI+offset i In this embodiment, the number of output channels is Cout = 32, so the weights read are 32-dimensional vectors, and the operation is a 32-dimensional vector addition. In this embodiment, because the mapping table is arranged, only one write operation is needed for the same output index. In this embodiment, the event stream data is binary, has no additional numerical features, and requires no additional multiplication operations; only the adder needs to be called to perform the addition operation.

[0065] like Figure 5 As shown, the structure reconstruction module first calculates the output tensor size. In this embodiment, the size of the output tensor is calculated according to the following formula:

[0066]

[0067]

[0068] Where O_Hlen and O_Wlen are the output tensor lengths, and I_Wlen and I_Hlen are the input tensor lengths of 128, K. width and K Height The convolution kernel size is 3, P width P Height D width D Height S width and S Height As described above. Then, based on the specific dimensions, the output matrix is ​​reconstructed into an output tensor for storage and output. In this embodiment, it will be processed according to [t, C]. out, O_Wlen, O_Hlen] is reshaped, where C out is the number of output channels, the implementation method is as follows: according to the storage position and storage order of the output tensor in the storage space, the corresponding value in the output matrix is copied into the output tensor, that is, the value of the [t, c, x, y] coordinates in the output tensor is copied The value of the index t*C out *O_Wlen*O_Hlen+c*O_Wlen*O_Hlen+x*O_Hlen+y in the output matrix.

Claims

1. A low-power convolutional computing system for event stream data, characterized by, The application comprises a mapping module, a scattering addition module, a result reconstruction module and a storage module; the storage module is used for storing convolution kernel information, weight information and an output tensor; The mapping module inputs event stream data, combines the convolution kernel information obtained from the storage module, and processes to obtain a mapping table of event stream data input coordinates and output coordinates; the event stream data is a DVS128-Gesture gesture recognition data set; The scattering addition module inputs the mapping table, combines the weight information obtained from the storage module, and adds to the corresponding position of the output matrix to obtain an output matrix and store it; The result reconstruction module receives the output matrix, combines the convolution kernel information obtained from the storage module, reconstructs it into a suitable output tensor and stores it; In the mapping module, first, check whether the convolution kernel information and the event stream data are aligned, and after confirming the alignment, calculate the output coordinates according to the bias of different positions of the convolution kernel and the input event stream data coordinates, judge whether the output coordinates exceed the size of the output tensor, if yes, discard, otherwise, add to the mapping table, and finally combine the input coordinates, the bias and the output coordinates into the mapping table; The method for calculating the output coordinates according to the bias of different positions of the convolution kernel and the input event stream data coordinates is as follows: According to each event input coordinate I and the bias offset of each position of the convolution kernel, the output coordinate O is calculated using the following formula: where P is a convolution kernel padding size, D is a convolution kernel expansion size, S is a convolution kernel step size, K offset is a coordinate of the current position of the convolution kernel, and for each event input coordinate I, there will be the same number of output coordinates as the size of the convolution kernel. The scattering addition module calculates the weight index according to the bias in the mapping table, extracts the corresponding weight from the storage module according to the weight index, calculates the output index according to the output coordinate, and calls the adder on the hardware device to add the corresponding weight and write it to the output matrix position corresponding to the output index; The method for calculating the weight index according to the bias in the mapping table is as follows: for the starting index WI corresponding to the weight W and the bias offset, the current weight index is WI+offset; The method for calculating the output index according to the output coordinate is as follows: For the starting index OI corresponding to the output matrix O and the output coordinate Coords, the output index is OI+Linear(Coords), and the Linear() function converts the coordinates into an integer value according to the storage method of the output matrix; for a two-dimensional output matrix with a size of X*Y, the output coordinate is [x, y], and Linear(Coords) is x*Y+y, and the output index is OI+x*Y+y; The method for adding the corresponding weight is as follows: wherein offset i offset represents the offset of the item in the mapping table whose i-th output coordinate is Coords. In the accumulation process, if the event stream data itself has a numerical value attribute, then according to the mapping table, the input coordinates are taken out, the numerical value is obtained from the storage module according to the input coordinates, and then the numerical value and the weight are multiplied and added, which is expressed by the formula as follows: If the numerical value type is an integer and the range is less than a preset threshold, then the multiplication is replaced by multiple additions to obtain lower energy consumption, which is expressed by the formula as follows: where M is the numerical value. The scattering addition module arranges the mapping table after receiving the mapping table to reduce repeated storage access, and the arrangement rule is that items corresponding to the same output index in the mapping table are placed together; The result reconstruction module first calculates the output tensor size, and the implementation method is as follows: Wherein, O_len is the output tensor length, I_len is the input length, K is the convolution kernel size, P is the convolution kernel padding size, D is the convolution kernel expansion size, and S is the convolution kernel step size; Then, according to the storage position and storage order of the output tensor in the storage space, the corresponding values in the output matrix are copied into the output tensor, so that the output matrix is reconstructed into the output tensor.

2. The low-power convolutional computing system for event stream data of claim 1, wherein, If the storage of the output matrix and the output tensor introduced weight both satisfy the linear continuous storage property, the output matrix and the output tensor introduced weight share the storage space in the storage module, the operation of the result reconstruction module is hidden, that is, this module does not need to be executed, further reducing the storage access.

3. The low-power convolutional computing system for event stream data of claim 2, wherein, The scattering addition module performs pipelining operation, and the extraction, accumulation and writing of the storage module are performed in pipelining, so as to cover the time consumption of read-write operation and calculation operation.

Images