Efficient sparse convolution method for event camera data and system thereof

Abstract

The application belongs to the technical field of artificial intelligence, and particularly relates to an efficient sparse convolution method for event camera data and a system thereof. Δt , and is preprocessed; step 2: effective operation detection is performed on the data preprocessed in step 1; step 3: the effective operation detected in step 2 is reorganized into a high-efficiency operation form of matrix multiplication; and step 4: post-processing is performed on the operation result obtained in step 3, and image convolution operation is completed. The application is used to solve the problem that the existing image convolution algorithm cannot effectively utilize the spatial sparsity of the event camera, thereby causing calculation redundancy and high inference delay.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence, and particularly relates to an efficient sparse convolution method for event camera data and a system thereof. BACKGROUND

[0002] Low-delay visual perception inference capability is of great significance to mobile robots, especially in scenarios such as autonomous driving and unmanned aerial vehicle delivery. In such scenarios, the robot itself moves at a high speed, and the environment is complex and changeable, which is prone to safety accidents. In such scenarios, fast and low-delay visual perception inference can effectively improve the mobility of the robot and make the movement process of the robot more secure. However, the commonly used robot visual sensors and processing algorithms are generally difficult to meet the needs of low-delay perception inference, and often cannot avoid sudden situations in time, causing loss of life and property.

[0003] An event camera is a visual sensor inspired by the working principle of the retina, mainly drawing on the working mechanism of the transient visual pathway in the human retina, and has great potential in fast perception and reaction of robots. Unlike conventional CMOS cameras, the event camera expresses visual information by recording "events" rather than recording image frames. Compared with conventional CMOS cameras, the event camera does not have a shutter device, and each pixel unit is always in an exposed state, and each pixel unit works independently and is sensitive to the change in light intensity. Therefore, when the light intensity at a certain pixel unit changes, the pixel unit immediately sends a signal, which is an "event". As the content entering the camera field of view continues to change, the event camera captures a series of "events" and records the visual information in this time period. The time difference between the "events" generated by the camera and the external light stimulus is extremely small, which makes the event camera have extremely high temporal resolution. At the same time, the independent working (i.e. asynchronous working) characteristics of the pixel units also make the visual data have low redundancy and high spatial sparsity.

[0004] Although the event camera has great potential to improve the fast perception and reaction capability of mobile robots due to its high temporal resolution and high spatial sparsity, the working principle of the event camera is different from that of the standard camera. Using traditional visual algorithms such as convolutional neural networks to process event camera information will discard its inherent sparsity, resulting in high computational complexity and high delay, which hinders the full play of its fast perception and reaction potential. How to reasonably utilize the data sparsity of the event camera and design an efficient sparse processing algorithm has become a problem to be solved. At present, some researches have paid attention to and utilized the sparsity of the event camera, reducing the computational complexity of the convolutional neural network, but in actual operation, they still face the problem of low efficiency, and have not brought effective improvement in time efficiency. SUMMARY

[0005] The application provides an efficient sparse convolution method for event camera data, which solves the problem that existing image convolution algorithms cannot effectively utilize the spatial sparsity of event cameras, thereby causing calculation redundancy and high inference delay.

[0006] The application provides an efficient sparse convolution system for event camera data, which implements an efficient sparse convolution method for event camera data.

[0007] The application is implemented by the following technical solutions:

[0008] An efficient sparse convolution method for event camera data, the efficient sparse convolution method comprising the following steps:

[0009] Step 1: obtaining event camera data S within a time period of Δt Δt , and preprocessing;

[0010] Step 2: detecting effective operation of the data preprocessed in step 1;

[0011] Step 3: reorganizing the effective operation detected in step 2 into a high-efficiency operation form of matrix multiplication;

[0012] Step 4: post-processing the operation result obtained in step 3 to complete image convolution operation.

[0013] Further, the step 1 is specifically: expressing the obtained event camera data S Δt within a time period of Δt as a set of events event i , that is, S Δt ={event i |t i ∈Δt}; the event event i contains a time dimension t i , a polarity dimension p i , a width dimension and a height dimension, that is, event i =(x i ,y i ,t i ,p i );

[0014] The time dimension t i is specifically: when Δt is small enough, events in the event camera data S Δt occur simultaneously, and the time axis information of visual events within the time period of Δt is compressed;

[0015] The polarity dimension p i is specifically: different polarities p ievents are considered as information on different channels of the image, i.e. taking polarity as the channel dimension;

[0016] The width x i The dimension and the height y i The dimension is specifically, one event event i is expressed as a multi-dimensional vector T i , i.e. T i = [N, p i , h i , w i ], where p i , h i , w i ∈event i , N represents the batch dimension, p i represents the channel dimension, h i and w i then represent the height and width dimensions;

[0017] T i converted from all events in Δt time, and the real value part of the image tensor is superimposed, and 0 is used to fill in the pixel position where no event occurs.

[0018] Further, the step 2 is specifically: using the position of the valid pixel to detect the position of the valid sub-convolution operation.

[0019] Further, the correspondence between the position of the valid sub-convolution operation and the position of the valid pixel Valid is specifically: all the positions of the valid pixels are recorded in Valid, and all the valid sub-convolution positions are obtained by traversing Valid and taking the union, and the positions of all the valid sub-convolutions are obtained, denoted as Solid.

[0020] Further, the step 3 is specifically:

[0021] Step 3.1: for each valid sub-convolution, the image data in one channel in the convolution window is unfolded into a column vector in row-major order, and the column vectors obtained in different channels are sequentially spliced by column to form a column vector corresponding to the current valid sub-convolution;

[0022] Step 3.2: traverse all valid sub-convolution operations in the order in Solid, and repeat step 3.1 to convert all image data required by the valid operations into column vectors;

[0023] Step 3.3: place all the column vectors corresponding to the above valid sub-convolutions in adjacent memory space, complete the row splicing, and obtain an image matrix.

[0024] Furthermore, in actual image convolution operations, the matrix size corresponding to the convolution kernel is:

[0025] [channel_out,kernel_size*kernel_size*channel_in]

[0026] The matrix size corresponding to the image data is:

[0027] [kernel_size*kernel_size*channel_in,Solid]

[0028] In general image convolution operations, the matrix size corresponding to the image data is:

[0029] [kernel_size*kernel_size*channel_in,N*h_out*w_out]

[0030] Therefore, the calculation ratio is,

[0031] η=Solid / (N*h_out*w_out).

[0032] Furthermore, the step 4 is specifically as follows:

[0033] Step 4.1: Fill the corresponding positions in the obtained product matrix Ps with 0 elements to expand it to the same size and shape as the matrix multiplication result Pd in ​​the general image convolution;

[0034] Step 4.2: According to the description in step 4.1, the relationship between Ps and Pd is as follows:

[0035] Ps={p i |p i ∈Pd;i∈Solid}

[0036] That is, the result obtained after taking the column combination of the elements in Solid in the Pd matrix is ​​Ps;

[0037] Step 4.3: Therefore, reverse the above relationship and remap the elements in Ps to new positions based on Solid, and fill the complement of Solid with 0 to obtain the desired result, which is recorded as Pds;

[0038] Step 4.4: Convert the Pds obtained in step 4.3 into the format of general image convolution. The result Pds obtained by general matrix multiplication has a data arrangement format of [C, N, H, W]. In modern neural networks, the data arrangement format generally used is [N, C, H, W]. Therefore, by exchanging the coordinate axis positions, the final required image convolution result can be obtained.

[0039] An efficient sparse convolution system for event camera data, using an efficient sparse convolution method for event camera data as described above, comprising:

[0040] a preprocessing module, obtaining event camera data S in a time period of Δt Δt , and performing preprocessing;

[0041] an effective operation module, performing effective operation detection on the preprocessed data;

[0042] based on the detected effective operation, reorganizing the effective operation;

[0043] a post-processing module, based on the effective operation reorganized by the effective operation module, performing post-processing of the operation result.

[0044] A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method as described above when executing the computer program.

[0045] A computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method as described above.

[0046] The beneficial effects of the present application are:

[0047] The present application takes advantage of the high spatial sparsity of event camera data, discards the redundant and invalid calculations, greatly reduces the computational complexity in the reasoning process, and greatly improves the reasoning efficiency in actual operation by reasonably organizing the remaining effective operation.

[0048] The present application takes advantage of the sparsity of event camera image data, discards the invalid 0 element multiplication and addition operation, and only retains the effective sub-convolution operation, thereby greatly reducing the computational complexity.

[0049] The present application adopts a more reasonable operation organization method, thereby successfully converting the reduction of operation amount into a significant reduction of reasoning time, and fully plays the low delay characteristics of the event camera.

[0050] The present application reduces the computational complexity of event camera image data processing and improves the time efficiency of reasoning. Related qualitative and quantitative experiments verify the advantages of the method of the present application. The present application will help to improve the perception and reaction ability of robots in high-speed and emergency situations, and provide safety guarantee for large-scale application of robots in future life scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1: This is a diagram showing the working principle of the event camera of the present invention, wherein (a) is a diagram showing the working principle of a single pixel of the event camera, and (b) is a diagram comparing the imaging principles of the event camera and an ordinary CMOS camera.

[0052] Figure 2 It is a display diagram of the spatial sparsity characteristics of the event camera of the present invention, wherein (a) is a three-dimensional display effect diagram of the event stream, (b) is a display diagram of the spatial sparsity characteristics of the event camera with an aggregated event stream of a 1ms time window and an information density of 0.2%, (c) is a display diagram of the spatial sparsity characteristics of the event camera with an aggregated information stream of a 10ms time window and an information density of 2%, and (d) is a display diagram of the spatial sparsity characteristics of the event camera with an aggregated event stream of a 100ms time window and an information density of 20%.

[0053] Figure 3 : is a schematic diagram of the method principle of the present invention, wherein (a) is a schematic diagram of a general image convolution operation which is also solved by general matrix multiplication, but a large number of invalid calculations will occur when processing event camera image data, and (b) is a schematic diagram of using the method of the present invention to discard invalid operations and reorganize valid operations into matrix multiplication form.

[0054] Figure 4 It is the pseudo code of the effective subconvolution detection part of the present invention.

[0055] Figure 5 1 is a schematic diagram comparing the inference time and computational complexity of the present invention and general image convolution, wherein (a) is a schematic diagram comparing computational complexity, and (b) is a schematic diagram comparing computational complexity.

[0056] Figure 6 It is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0057] In the following description, specific details such as specific system structures and technologies are provided for illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present application with unnecessary details.

[0058] It will be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0059] It should also be understood that the terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0060] The following is attached to this application specification Figures 1-6 , clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of this application.

[0061] In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.

[0062] Implementation Method 1

[0063] This embodiment provides an efficient sparse convolution method for event camera data, which includes the following steps:

[0064] Step 1: Obtain event camera data S within the Δt time period Δt , and preprocessing;

[0065] Step 2: Perform valid operation detection on the data preprocessed in step 1;

[0066] Step 3: Reorganize the efficient operations detected in step 2 into efficient matrix multiplication forms;

[0067] Step 4: Post-process the operation results obtained in step 3 to complete the image convolution operation.

[0068] Furthermore, the main difference between the working principle of event cameras and standard cameras is that each pixel unit in an event camera works independently and asynchronously. The process of generating data by each pixel unit is not regulated by a unified clock cycle, but is controlled by the changes in the light information in the environment. The basic perception result of each pixel unit is called an event. When the logarithmic intensity of the light stimulus received by a pixel unit exceeds a preset threshold c compared with the previous moment, that is, |Log(I)-Log(I last )|>c, it will generate and output an event i, which contains the pixel coordinates (x i ,yi ), the time t at which the event occurs i , and the polarity p representing whether the light intensity change exceeds the threshold upward or downward i .

[0069] event i =(x i ,y i ,t i ,p i As the light information in the environment changes, the event camera can output a continuous asynchronous event stream in the time domain. A single event contains very little information, and it is difficult to get the desired result by processing a single event. By recording all events that occur within a short period of time Δt, a set of events S can be obtained. Δt , that is, S Δt ={event i |t i ∈Δt}. When the ambient light source remains unchanged, the brightness change in the image is usually caused by the movement of the object, especially the movement at the edge of the object can cause a large change value. Therefore, the set S generally contains the edge information of the object, which is very important information for visual tasks such as target detection. By sliding the window, the content of S can be continuously updated to obtain new perception data. At the same time, by controlling Δt, the time receptive field and the amount of spatiotemporal information of a single processing can be controlled. Modern AI algorithms usually use tensor as the basic data structure. Considering the scalability and compatibility of the proposed method, the event camera data S is expressed here as an image tensor form similar to the traditional image. According to the above definition, an event event i Contains information in four dimensions. First, for the time t dimension, when Δt is small enough, we can approximately assume that the events in S occur simultaneously, and compress the time axis information of the visual events in the Δt time period. For the polarity dimension, there are only two polarity p values ​​for events, and events with the same p value are closely related in space and logic. Therefore, events with different polarity p can be regarded as information on different channels of the image, that is, polarity is used as the channel dimension. Finally, for the width and height dimensions, events event i The x and y in the event represent the pixel location where the event occurs, which is consistent with traditional images. i It can be represented as a multidimensional vector T like the pixels of a traditional image i , that is, T i =[N,p i ,h i ,w i ]. Where p i ,h i ,w i∈event i , N represents the batch dimension, p i represents the channel dimension, h i and w i Then represents the height and width dimensions. T is obtained by converting all events in Δt time. i The superposition can be used as the real-valued portion of the image tensor. However, within the time Δt, there are still a large number of pixel locations that do not correspond to visual events. For these locations where no events occurred, they are filled with 0. This results in a data representation tensor ET that is compatible with modern computer vision techniques. ET=0+∑T i . event i The xi and yi in the ET record the positions of valid pixels in the ET, and the pixel positions of all valid information in the ET are recorded as Valid. Then Valid={(x i ,y i )|x i ,y i ∈event i ,event i ∈S Δt}.

[0070] Since ET contains a large number of zero elements, directly performing operations such as convolution will result in a large amount of meaningless redundant calculations. How to use data sparsity to design an efficient sparse convolution algorithm will be introduced in detail in step 2.

[0071] Furthermore, step 2 is specifically as follows: Image convolution is an important information processing method in robot vision. According to the definition, image convolution is performed by convolving the convolution kernel with the data at a specific window on the input image (relative to image convolution, the convolution operation of a specific window is referred to as subconvolution below), and the operation is completed by traversing the input image through a sliding window.

[0072] Due to the high sparsity of event camera data, subconvolution operations at specific windows will include a certain amount of invalid zero calculations. When the local image data within the subconvolution window contains nonzero elements, it is considered a valid subconvolution. When the local image data corresponding to the subconvolution window is all zero, it is considered an invalid subconvolution. The main purpose of this section is to efficiently detect the location of valid subconvolution operations within the entire image convolution operation. Thanks to the efficient raw data representation of the event camera, the locations of valid pixels in the ET tensor can be easily determined.

[0073] There is a strong correspondence between the valid pixel position Valid and the valid sub-convolution position. The valid sub-convolution window must contain valid pixels, and the valid pixels must correspond to a valid sub-convolution. Therefore, the position of the valid sub-convolution can be directly calculated based on Valid.

[0074] Specifically, each valid pixel except those at the edge of the image will correspond to K sub-convolution operations, where K is the size of the sub-convolution window. By matching the pixel with the elements at different positions in the convolution kernel, the positions of the K sub-convolution operations can be obtained. The number of sub-convolution operations corresponding to the pixels at the edge of the image will be less than K, and additional judgment is required to detect whether the position of the sub-convolution is correct and valid. It is known that the positions of all valid pixels are recorded in Valid, so by traversing Valid and taking the union of all the valid sub-convolution positions obtained, the positions of all valid sub-convolutions can be obtained, which are recorded as Solid. Since event camera data is usually extremely sparse, the number of elements in Valid is much smaller than the actual image size. Therefore, this greatly improves the time efficiency compared to detecting valid sub-convolutions by traversing according to the definition of valid sub-convolutions.

[0075] Furthermore, step 3 is specifically as follows: Image convolution is essentially a combination of many logically parallel multiplication and addition operations, a very regular operation form. In practice, image convolution is often converted into a general matrix multiplication (GEMM) form, which is more suitable for computer parallel operations. This conversion to a general matrix multiplication form can greatly accelerate the operation speed. To facilitate the introduction of the operation organization involved in the present invention, the principle of the general image convolution operation is first introduced here. In a general image convolution operation, the convolution kernel is expanded into a row vector, and the convolution kernel vectors of different channels form the convolution kernel matrix. The image data in the image convolution operation is expanded into a column vector, and the image matrix is ​​obtained by combining data from different channels and batches. At this time, if the resulting convolution kernel matrix is ​​multiplied with the image matrix, a general matrix multiplication operation is completed, and the result of the image convolution operation can be obtained (the result is slightly different in the element ordering). Because the general image convolution operation does not take into account invalid calculations, that is, multiplication and addition operations involving zero elements, a large amount of computing resources will be wasted when processing event camera data, slowing down the operation time. The present invention innovates in the process of processing image matrices. On the basis of removing invalid operation data, the remaining valid operations are converted into general matrix multiplication operations to complete efficient operations. The specific steps of step 3 are:

[0076] Step 3.1: For each valid subconvolution, follow the general image convolution method, expand the image data of one channel in the convolution window into a column vector in a row-first manner, and concatenate the column vectors obtained in different channels in sequence to form the column vector corresponding to the current valid subconvolution;

[0077] Step 3.2: Traverse all valid sub-convolution operations in the order in Solid, and repeat step 3.1 to convert all image data required for all valid operations into several column vectors;

[0078] Step 3.3: Place the column vectors corresponding to all the above valid sub-convolutions into adjacent memory spaces and complete the row-wise concatenation to obtain an image matrix; this matrix is ​​the matrix used for general matrix multiplication operations; since in general image convolution operations, the process of converting an image into a matrix that participates in general matrix multiplication operations is called im2col, the above process is called sparse im2col in this invention.

[0079] Specifically, the processing of the convolution kernel matrix remains consistent with general image convolution algorithms. Compared to im2col in general image convolution, sparse im2col does not need to directly traverse the entire input tensor, but only the smaller effective convolution portion. This offers two advantages. First, the computational complexity of the conversion process is greatly reduced, making the transition from convolution to general matrix multiplication faster for the computer. Second, the resulting matrix is ​​smaller, reducing computer storage space consumption. A smaller matrix also means that the computational complexity of subsequent matrix multiplication operations is significantly reduced.

[0080] Furthermore, in actual image convolution operations, the matrix size corresponding to the convolution kernel is:

[0081] [channel_out,kernel_size*kernel_size*channel_in]

[0082] The matrix size corresponding to the image data is:

[0083] [kernel_size*kernel_size*channel_in,Solid]

[0084] In general image convolution operations, the matrix size corresponding to the image data is:

[0085] [kernel_size*kernel_size*channel_in,N*h_out*w_out]

[0086] Therefore, the ratio of the computational cost of the present invention to that of the general image convolution can be calculated as follows:

[0087] η=Solid / (N*h_out*w_out).

[0088] Specifically, the computational load (FLOPs) of the algorithm involved in this invention is proportional to the sparsity of Solid. This method observes computational sparsity at the sub-convolution level, retaining only the valid sub-convolutions and organizing the remaining valid sub-convolutions into general matrix multiplication operations. This not only significantly reduces the computational load but also ensures hardware friendliness.

[0089] Furthermore, the general image convolution operation completed by the general matrix multiplication, its matrix product result Pd is equivalent to the theoretical convolution result. However, the general matrix multiplication operation completed in step 3 only focuses on the operation of non-zero elements, so its product result Ps is different from the general image convolution operation. It is a subset of the complete convolution operation result that only contains valid information. In order to be compatible and adaptable to other computing components in modern neural networks, such as maximum pooling, batch normalization, etc., Ps needs to be further converted into the same form as the general image convolution. The specific step 4 is:

[0090] Step 4.1: Fill the corresponding positions in the obtained product matrix Ps with 0 elements to expand it to the same size and shape as the matrix multiplication result Pd in ​​the general image convolution. This can be done by reusing Solid.

[0091] Step 4.2: According to the description in step 4.1, the relationship between Ps and Pd is as follows:

[0092] Ps={p i |p i ∈Pd;i∈Solid}

[0093] That is, the result obtained after taking the column combination of the elements in Solid in the Pd matrix is ​​Ps; this is not difficult to explain, because in matrix multiplication, the columns of the product and the columns of the multiplicand matrix have a one-to-one correspondence;

[0094] Step 4.3: Therefore, reverse the above relationship and remap the elements in Ps to new positions based on Solid, and fill the complement of Solid with 0 to obtain the desired result, which is recorded as Pds;

[0095] Step 4.4: Convert the Pds obtained in step 4.3 into the format of general image convolution. The result Pds obtained by general matrix multiplication has a data arrangement format of [C, N, H, W]. In modern neural networks, the data arrangement format generally used is [N, C, H, W]. Therefore, by exchanging the coordinate axis positions, the final required image convolution result can be obtained.

[0096] from Figure 5As can be seen from the figure, (a) the inference time of the present invention is greatly reduced compared with general image convolution and other sparse schemes, and (b) the computational complexity is reduced by an order of magnitude compared with general image convolution.

[0097] Specifically, we selected a DAVIS346 camera as the event camera visual sensor and used the ROS driver package to record the event camera raw data, obtaining a raw event stream with a spatial resolution of 260 x 346. We recorded the raw data in a variety of different scenarios, including indoors, outdoors, static, and dynamic, to avoid the objectivity of the results being affected by a single scenario.

[0098] The camera raw data is segmented into a series of time windows of Δt, yielding a series of event groups with varying time lengths. The starting and ending Δt values ​​are set to 1ms, 100ms, and 1ms steps. A total of 100 groups of different Δt values ​​are sampled. Accordingly, the raw data is segmented into 100 different time windows, yielding 100 sets of data. Each of these 100 sets of data is then tensorized to yield 100 sets of tensorized visual data. The data is arranged in the format [N, C, H, W], where N is the batch size, set to 8, and C is the number of data channels. Since the event camera has both positive and negative event signals, C is 2. H and W represent the image height and width, respectively, set to 260 and 346 based on the camera parameters. The sparsity of these 100 sets of data decreases as Δt increases, testing the performance of the proposed sparse convolution method at varying sparsity levels.

[0099] Set the convolution kernel size to the commonly used 3*3, the convolution step to 1, the padding to 0, the input channel to 2, the output channel to 20, and use the Kaiming initialization method to assign the convolution kernel weights.

[0100] The sparse convolution method described in this invention is implemented in C++ and packaged as a PyTorch operation function in the form of a PyTorch extension operator, similar to general image convolution. This function is called in PyTorch to perform convolution operations on the above 100 sets of data to perform sparse convolution.

[0101] During the computational experiments, the system time was read before and after the sparse convolution step, and the difference between the two was taken to obtain the sparse convolution inference time. Multiple tests were performed to eliminate the influence of other factors such as the computer operating status on inference time. The ratio of the image matrix size involved in the general matrix multiplication operation during each inference process to the theoretical image matrix size in the general image convolution operation was calculated and used as the normalized computational effort.

[0102] After multiple tests of indoor and outdoor scenes, dynamic and static scenes, the results are as followsFigure 5 As shown in Fig. Figure 5 a is the comparison of operation time, it can be obviously seen that the reasoning time efficiency of the present application is much better than that of the SCN sparse convolution method. The reason is that the SCN uses the way of hash table to reorganize the effective operation, which destroys the continuity in the calculation, so in most cases its performance cannot even be better than that of the general image convolution. While the present application uses the general matrix multiplication to overcome the problem of discontinuous calculation, and its reasoning performance is better than that of the general image convolution in the normal working range. Figure 5 b is the comparison of operation amount, obviously the method described in the present application effectively reduces the calculation amount, and reduces the calculation amount by one order of magnitude. At the same time, since the present application does not introduce new calculation compared with the general image convolution in logic, therefore the sparse convolution result is exactly the same as that of the general image convolution, and the reasoning accuracy is not affected at all.

[0103] Embodiment two

[0104] The embodiment provides a high-efficiency sparse convolution system for event camera data, which uses the high-efficiency sparse convolution method for event camera data as described in embodiment one, and the high-efficiency sparse convolution system comprises:

[0105] a preprocessing module, which acquires event camera data S in a time period of Δt Δt and performs preprocessing;

[0106] an effective operation module, which performs effective operation detection on the preprocessed data;

[0107] based on the detected effective operation, the effective operation is reorganized again;

[0108] a post-processing module, which performs post-processing on the operation result based on the effective operation reorganized by the effective operation module.

[0109] Embodiment three

[0110] The electronic device provided by the embodiment of the present application comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the memory is used for storing software programs and modules, and the processor performs various functional applications and data processing by running the software programs and modules stored in the memory. The memory and the processor are connected through a bus. Specifically, the processor realizes any step in the above embodiment one by running the above computer program stored in the memory.

[0111] It should be appreciated that in embodiments of the present application, the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc. The general-purpose processor can be a microprocessor or can also be any conventional processor.

[0112] The memory can include read-only memory, flash memory, and random access memory, and provide instructions and data to the processor. Part or all of the memory can also include non-volatile random access memory.

[0113] As can be seen from the above, the electronic device provided by the embodiments of the present application can realize the efficient sparse convolution method for event camera data as described in Embodiment One by running a computer program. By utilizing the sparsity of event camera image data, the invalid 0 element multiplication and addition operation is discarded, and only the valid sub-convolution operation is retained, thereby greatly reducing the computational complexity. Secondly, by utilizing the form of dense matrix multiplication to complete the valid sub-convolution operation, compared with other sparse convolution methods, a more reasonable operation organization method is adopted, thereby successfully reducing the operation amount and greatly reducing the inference time, and fully exerting the low delay characteristics of the event camera.

[0114] It should be understood that if the above-mentioned integrated modules / units are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The above-mentioned computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the above-mentioned computer program includes computer program code, and the above-mentioned computer program code can be in source code form, object code form, executable file or some intermediate form. The above-mentioned computer-readable medium may include: any entity or device capable of carrying the above-mentioned computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the above-mentioned computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0115] The above description of the disclosed embodiments will enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein, but is to be construed in the widest manner consistent with the principles and novel features disclosed herein.

[0116] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the implementation method can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method implementation method, and will not be repeated here.

[0117] It should be noted that the methods and detailed examples provided in the above embodiments can be combined with the devices and equipment provided in the embodiments, and references can be made to each other, and no further details will be given.

[0118] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0119] In the embodiments provided by the present invention, it should be understood that the disclosed apparatus / terminal equipment and methods can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For example, the division of the modules or units described above is merely a logical functional division. In actual implementation, other division methods may be used. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not implemented.

[0120] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. An efficient sparse convolution method for event camera data, characterized by: The efficient sparse convolution method comprises the following steps: Step 1: Obtain event camera data S within the Δt time period Δt , and preprocessing; Step 2: Perform valid operation detection on the data preprocessed in step 1; Step 3: Reorganize the efficient operations detected in step 2 into an efficient operation form of matrix multiplication; Step 4: Post-process the operation results obtained in step 3 to complete the image convolution operation; The step 2 specifically comprises: using the position of the valid pixel to detect the position of the valid sub-convolution operation; The specific corresponding relationship between the position of the effective subconvolution operation and the position of the effective pixel Valid is: It is known that the positions of all valid pixels are recorded in Valid. Traverse Valid and take the union of all valid sub-convolution positions to obtain the positions of all valid sub-convolutions, which are recorded as Solid. The step 3 is specifically as follows: Step 3.1: For each valid subconvolution, expand the image data of one channel in the convolution window into a column vector in a row-first manner, and concatenate the column vectors obtained in different channels in sequence to form the column vector corresponding to the current valid subconvolution; Step 3.2: Traverse all valid sub-convolution operations in the order in Solid, and repeat step 3.1 to convert all image data required for all valid operations into several column vectors; Step 3.3: Place the column vectors corresponding to all the above valid sub-convolutions into adjacent memory spaces and complete row-wise splicing to obtain an image matrix; The step 4 is specifically as follows: Step 4.1: Fill the corresponding positions in the obtained product matrix Ps with 0 elements to expand it to the same size and shape as the matrix multiplication result Pd in ​​the general image convolution; Step 4.2: According to the description in step 4.1, the relationship between Ps and Pd is as follows: Ps={p i |p i ∈Pd;i∈Solid} That is, the result obtained after taking the column combination of the elements in Solid in the Pd matrix is ​​Ps; Step 4.3: Therefore, reverse the above relationship and remap the elements in Ps to new positions based on Solid, and fill the complement of Solid with 0 to obtain the desired result, which is recorded as Pds; Step 4.4: Convert the Pds obtained in step 4.3 into the format of general image convolution. The result Pds obtained by general matrix multiplication has a data arrangement format of [C, N, H, W]. In modern neural networks, the data arrangement format generally used is [N, C, H, W]. Therefore, by exchanging the coordinate axis positions, the final required image convolution result can be obtained.

2. The efficient sparse convolution method for event camera data according to claim 1, characterized in that: The step 1 is specifically as follows: acquiring the event camera data S of the Δt time period Δt Expressed as event i The set of S Δt ={event i |t i ∈Δt}; the event event i Including time t i Dimension, polarity p i Dimension, width dimension and height dimension, that is, event i =(x i ,y i ,t i ,p i ); The time t i Specifically, when Δt is small enough, the event camera data S Δt The events within the time period Δt occur simultaneously, and the time axis information of the visual events within the time period Δt is compressed; The polarity p i The specific dimension is to divide different polarities p i The events are regarded as information on different channels of the image, that is, polarity is used as the channel dimension; The width x i Dimension and height y i The specific dimension is an event event i Represented as a multidimensional vector T i , that is, T i =[N,p i ,h i ,w i ], where p i ,h i ,w i ∈event i , N represents the batch dimension, p i represents the channel dimension, h i and w i represents the height and width dimensions; T obtained by converting all events within Δt i The superposition is the real-valued part of the image tensor, and the pixel positions where no events occur are filled with 0.

3. The efficient sparse convolution method for event camera data according to claim 1, characterized in that: In the actual image convolution operation, the matrix size corresponding to the convolution kernel is: [channel_out,kernel_size*kernel_size*channel_in] The matrix size corresponding to the image data is: [kernel_size*kernel_size*channel_in,Solid] In general image convolution operations, the matrix size corresponding to the image data is: [kernel_size*kernel_size*channel_in,N*h_out*w_out] Therefore, the calculation ratio is, η=Solid / (N*h_out*w_out).

4. An efficient sparse convolution system for event camera data, characterized by: The efficient sparse convolution system uses an efficient sparse convolution method for event camera data according to any one of claims 1 to 3, and the efficient sparse convolution system includes: Preprocessing module, obtains event camera data S within the Δt time period Δt , and preprocessing; The effective operation module performs effective operation detection on the pre-processed data; Based on the detected valid operations, the valid operations are reorganized; The post-processing module performs post-processing of the operation results based on the effective operations reorganized by the effective operation module.

5. A computer device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 1 to 3 is implemented.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images