Efficient sparse convolution method and system for event camera data

Abstract

The invention belongs to the technical field of artificial intelligence, and particularly relates to an event camera data-oriented efficient sparse convolution method and system. The method comprises the following steps: step 1, acquiring event camera data S [delta] t in a [delta] t time period, and performing preprocessing; 2, effective operation detection is carried out on the data preprocessed in the step 1; 3, reorganizing the effective operation detected in the step 2 into an efficient operation form of matrix multiplication; and 4, performing post-processing on an operation result obtained in the step 3, and completing image convolution operation. The method is used for solving the problems of calculation redundancy and high reasoning delay caused by the fact that an existing image convolution algorithm is difficult to effectively utilize event camera space sparsity.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to an efficient sparse convolution method and system for event camera data. Background Art

[0002] Low-latency visual perception and reasoning capabilities are of great significance for mobile robots, especially in scenarios such as autonomous driving and drone delivery. The robot itself moves at a high speed and the environment is complex and changeable, making it very easy for safety accidents to occur. In such scenarios, fast and low-latency visual perception and reasoning can effectively improve the robot's maneuverability and make the robot's movement process safer. However, currently commonly used robot vision sensors and their processing algorithms are usually difficult to meet the needs of low-latency perception and reasoning. When faced with some emergencies, it is often difficult to avoid them in time, resulting in loss of life and property.

[0003] The event camera is a visual sensor developed based on the working principle of the retina. It mainly draws on the working mechanism of the transient visual pathway in the human retina and has great potential in the rapid perception and response of robots. The event camera does not express visual information by recording image frames like conventional CMOS cameras, but expresses visual information by recording "events". Compared with conventional CMOS cameras, there is no shutter device in the event camera. Its pixel units are always in an exposed state. At the same time, each pixel unit works independently and is sensitive to the change in light intensity. Therefore, when the light intensity at a certain pixel unit changes to a certain extent, the pixel unit immediately sends a signal, which is the "event". As the content entering the camera's field of view continues to change, the event camera captures a series of "events" and records the visual information within this period of time. The time difference between the "event" generated by the camera and the external light stimulus is extremely small, which makes the event camera have extremely high temporal resolution. At the same time, the characteristics of its pixel units working independently (i.e. asynchronous work) also make the redundancy of visual data low and the spatial sparsity high.

[0004] Although event cameras have great potential to improve the rapid perception and response capabilities of mobile robots due to their high temporal resolution and high spatial sparsity, event cameras work differently from standard cameras. Using traditional visual algorithms such as convolutional neural networks to process event camera information will discard its inherent sparsity, resulting in high computational complexity and high latency, which hinders its rapid perception and response potential. How to reasonably utilize the data sparsity of event cameras and design efficient sparse processing algorithms has become an urgent problem to be solved. At present, some studies have noticed and utilized the sparse properties of event cameras to reduce the computational complexity of convolutional neural networks, but in actual operations, they face the problem of inefficient operations and do not bring about effective improvement in time efficiency. Summary of the invention

[0005] The present invention provides an efficient sparse convolution method for event camera data, which is used to solve the problem that the existing image convolution algorithm is difficult to effectively utilize the spatial sparsity of event cameras, resulting in computational redundancy and high inference delay.

[0006] The present invention provides an efficient sparse convolution system for event camera data, which is used to implement an efficient sparse convolution method for event camera data.

[0007] The present invention is achieved through the following technical solutions:

[0008] An efficient sparse convolution method for event camera data, the efficient sparse convolution method comprising the following steps:

[0009] Step 1: Obtain event camera data S within a time period of Δt Δt , and preprocessing;

[0010] Step 2: Perform effective operation detection on the data preprocessed in step 1;

[0011] Step 3: Reorganize the efficient operations detected in step 2 into an efficient operation form of matrix multiplication;

[0012] Step 4: Post-process the operation result obtained in step 3 to complete the image convolution operation.

[0013] Furthermore, the step 1 is specifically as follows: acquiring the event camera data S of the time period Δt Δt Expressed as an event i The set of S Δt ={event i |t i ∈Δt}; the event event i Including time t i Dimension, polarity p i Dimension, width dimension and height dimension, that is, event i =(x i ,y i ,t i ,p i );

[0014] The time t i The dimension is specifically that when Δt is small enough, the event camera data S Δt The events within the time period Δt occur simultaneously, and the time axis information of the visual events within the time period Δt is compressed;

[0015] The polarity p i The specific dimension is to divide different polarities p iThe events are regarded as information on different channels of the image, that is, polarity is used as the channel dimension;

[0016] The width x i Dimension and height y i The specific dimension is an event event i Represented as a multidimensional vector T i , that is, T i =[N,p i ,h i ,w i ], where p i ,h i ,w i ∈event i , N represents the batch dimension, p i represents the channel dimension, h i and w i represents the height and width dimensions;

[0017] T obtained by converting all events within Δt i The real-valued part of the superposition image tensor is filled with 0 for the pixel positions where no events occurred.

[0018] Furthermore, the step 2 specifically includes: using the position of the valid pixel to detect the position of the valid sub-convolution operation.

[0019] Furthermore, the corresponding relationship between the position of the effective sub-convolution operation and the position of the effective pixel Valid is specifically as follows: the positions of all known effective pixels are all recorded in Valid, Valid is traversed and the union of all the obtained effective sub-convolution positions is taken to obtain the positions of all the effective sub-convolutions, which are recorded as Solid.

[0020] Furthermore, the step 3 is specifically as follows:

[0021] Step 3.1: For each valid sub-convolution, expand the image data of one channel in the convolution window into a column vector in a row-first manner, and concatenate the column vectors obtained in different channels in sequence by column to form a column vector corresponding to the current valid sub-convolution;

[0022] Step 3.2: Traverse all valid sub-convolution operations in the order in Solid, and repeat step 3.1 to convert all image data required for all valid operations into several column vectors;

[0023] Step 3.3: Put the column vectors corresponding to all the above valid sub-convolutions into adjacent memory spaces, complete the row-by-row splicing, and you can get an image matrix.

[0024] Furthermore, in the actual image convolution operation, the matrix size corresponding to the convolution kernel is:

[0025] [channel_out,kernel_size*kernel_size*channel_in]

[0026] The matrix size corresponding to the image data is:

[0027] [kernel_size*kernel_size*channel_in,Solid]

[0028] In general image convolution operations, the matrix size corresponding to the image data is:

[0029] [kernel_size*kernel_size*channel_in,N*h_out*w_out]

[0030] Therefore, the calculation ratio is,

[0031] η=Solid / (N*h_out*w_out).

[0032] Furthermore, the step 4 is specifically as follows:

[0033] Step 4.1: Fill the corresponding positions in the obtained product matrix Ps with 0 elements to expand it to the same size and shape as the matrix multiplication result Pd in ​​the general image convolution;

[0034] Step 4.2: According to the description in step 4.1, there is the following relationship between Ps and Pd:

[0035] Ps={p i |p i ∈Pd; i∈Solid}

[0036] That is, the result obtained after taking the column combination of the elements in Solid in the Pd matrix is ​​Ps;

[0037] Step 4.3: Therefore, by using the above relationship in reverse, remap the elements in Ps to new positions based on Solid, and fill the complement positions of Solid with 0 to get the desired result, which is recorded as Pds;

[0038] Step 4.4: Convert the Pds obtained in step 4.3 into the format of general image convolution. The result Pds obtained by general matrix multiplication has a data arrangement format of [C, N, H, W], while in modern neural networks, the generally used data arrangement format is [N, C, H, W]. Therefore, by exchanging the coordinate axis positions, the final image convolution result can be obtained.

[0039] An efficient sparse convolution system for event camera data, the efficient sparse convolution system uses the efficient sparse convolution method for event camera data as described above, and the efficient sparse convolution system includes:

[0040] Preprocessing module, obtains event camera data S within the Δt time period Δt , and preprocessing;

[0041] The effective operation module performs effective operation detection on the preprocessed data;

[0042] Based on the detected effective operations, reorganize the effective operations;

[0043] The post-processing module performs post-processing of the operation results based on the effective operations reorganized by the effective operation module.

[0044] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the above method is implemented.

[0045] A computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0046] The beneficial effects of the present invention are:

[0047] The present invention utilizes the high sparsity of event camera data space and discards redundant and invalid calculations therein, thereby greatly reducing the computational complexity in the reasoning process. At the same time, by reasonably organizing the remaining effective operations, the reasoning efficiency in actual operations is greatly improved.

[0048] The present invention utilizes the sparsity of event camera image data, discards invalid zero-element multiplication and addition operations, and only retains valid sub-convolution operations, thereby greatly reducing the computational complexity.

[0049] The present invention adopts a more reasonable operation organization method, thereby successfully converting the reduction in the amount of calculation into a significant reduction in reasoning time, giving full play to the low-latency characteristics of the event camera.

[0050] The present invention reduces the computational complexity of event camera image data processing and improves the time efficiency of reasoning. Relevant qualitative and quantitative experiments verify the advantages of the method described in the present invention. The present invention will help improve the perception and response capabilities of robots in high-speed and emergency situations, and provide safety guarantees for the large-scale application of robots in life scenarios in the future. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 11 is a diagram showing the working principle of the event camera of the present invention, wherein (a) is a diagram showing the working principle of a single pixel of the event camera, and (b) is a diagram showing the comparison of the imaging principles of the event camera and a common CMOS camera.

[0052] Figure 2 It is a display diagram of the spatial sparsity characteristics of the event camera of the present invention, wherein (a) is a three-dimensional display effect diagram of the event flow, (b) is an event camera spatial sparsity characteristics display diagram of the event flow aggregated in a 1ms time window, with an information density of 0.2%, (c) is an event camera spatial sparsity characteristics display diagram of the information flow aggregated in a 10ms time window, with an information density of 2%, and (d) is an event camera spatial sparsity characteristics display diagram of the event flow aggregated in a 100ms time window, with an information density of 20%.

[0053] Figure 3 1 is a schematic diagram of the method principle of the present invention, wherein (a) is a schematic diagram of a general image convolution operation which is also solved by general matrix multiplication, but a large number of invalid calculations will occur when processing event camera image data, and (b) is a schematic diagram of using the method of the present invention to discard invalid operations and reorganize valid operations into matrix multiplication form.

[0054] Figure 4 It is the pseudo code for detecting the effective sub-convolution part of the present invention.

[0055] Figure 5 Schematic diagram comparing the inference time and computational complexity of the present invention and general image convolution, wherein (a) is a schematic diagram comparing the computational complexity, and (b) is a schematic diagram comparing the computational complexity.

[0056] Figure 6 It is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0057] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0058] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0059] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, the singular forms "a", "an" and "the" are intended to include plural forms unless the context clearly indicates otherwise.

[0060] The following is attached to this application specification Figure 1-6 , the technical solutions in the embodiments of the present application are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0061] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.

[0062] Implementation Method 1

[0063] This embodiment provides an efficient sparse convolution method for event camera data, and the efficient sparse convolution method includes the following steps:

[0064] Step 1: Obtain event camera data S within a time period of Δt Δt , and preprocessing;

[0065] Step 2: Perform effective operation detection on the data preprocessed in step 1;

[0066] Step 3: Reorganize the efficient operations detected in step 2 into efficient operation forms of matrix multiplication;

[0067] Step 4: Post-process the operation result obtained in step 3 to complete the image convolution operation.

[0068] Furthermore, the main difference between the event camera and the standard camera in terms of working principle is that each pixel unit in the event camera works independently and asynchronously, and the process of each pixel unit generating data is not regulated by a unified clock cycle, but is controlled by the changes in the light information in the environment. The basic perception result of each pixel unit is called an event. When the logarithmic intensity of the light stimulus received by a pixel unit exceeds a preset threshold c compared with the previous moment, that is, |Log(I)-Log(I last )|>c, it will generate and output an event i, which contains the pixel coordinates (x i ,yi ), the time t at which the event occurs i , and the polarity p representing whether the light intensity change exceeds the threshold upward or downward i .

[0069] Event i =(x i ,y i ,t i ,p i ). As the light information in the environment changes, the event camera can output a continuous asynchronous event stream in the time domain. A single event contains very little information, and it is difficult to get the desired result by processing a single event. By recording all events that occur within a short period of time Δt, a set of events S can be obtained. Δt , that is, S Δt ={event i |t i ∈Δt}. When the ambient light source remains unchanged, the brightness change in the image is usually caused by the movement of the object, especially the movement at the edge of the object can cause a large change value. Therefore, the set S generally contains the edge information of the object, which is very important information for visual tasks such as target detection. By sliding the window, the content of S can be continuously updated to obtain new perception data. At the same time, by controlling Δt, the time receptive field and the amount of spatiotemporal information processed in a single time can be controlled. Modern AI algorithms usually use tensor as the basic data structure. Considering the scalability and compatibility of the proposed method, the event camera data S is expressed here as an image tensor similar to traditional images. According to the above definition, an event event i It contains information in four dimensions. First, for the time t dimension, when Δt is small enough, we can approximately assume that the events in S occur simultaneously, and compress the timeline information of the visual events in the Δt time period. For the polarity dimension, there are only two polarity p values ​​for events, and events with the same p value are closely related in space and logic. Therefore, events with different polarity p can be regarded as information on different channels of the image, that is, polarity is used as the channel dimension. Finally, for the width and height dimensions, the event event i The x and y in the graph represent the pixel location where the event occurs, which is consistent with traditional images. i It can be represented as a multidimensional vector T like the pixels of a traditional image i , that is, T i =[N,p i ,h i ,w i ]. i ,h i ,w i∈event i , N represents the batch dimension, p i represents the channel dimension, h i and w i Then represents the height and width dimensions. T obtained by converting all events within Δt time i The superposition can be used as the real-valued part of the image tensor, and there are still a large number of pixel positions that do not have corresponding visual events within the Δt time. For the positions where no events occurred, they are filled with 0. In this way, a data expression form tensor ET that is compatible with modern computer vision technology is obtained. ET=0+∑T i . event i The xi and yi in ET record the positions of valid pixels in ET, and the pixel positions of all valid information in ET are recorded as Valid. Then Valid={(x i ,y i )|x i ,y i ∈event i ,event i ∈S Δt}.

[0070] Since ET contains a large number of 0 elements, directly performing operations such as convolution will result in a large number of meaningless redundant calculations. How to use data sparsity to design an efficient sparse convolution algorithm will be introduced in detail in step 2.

[0071] Furthermore, step 2 is specifically as follows: Image convolution is an important information processing method in robot vision. According to the definition, image convolution is performed by convolving the convolution kernel with the data at a specific window on the input image (relative to image convolution, the convolution operation of a specific window is referred to as subconvolution below), and the operation is completed by traversing the input image through a sliding window.

[0072] Due to the large sparsity of event camera data, the subconvolution operation at a specific window will contain a certain amount of invalid zero calculations. When the local image data within the subconvolution window contains non-zero elements, it is called a valid subconvolution, and when the local image data corresponding to the subconvolution window is all zero data, it is called an invalid subconvolution. The main purpose of this part is to efficiently detect the location of the valid subconvolution operation in the entire image convolution operation. Thanks to the efficiency of the event camera raw data representation, the location of the valid pixel Valid in the ET tensor can be easily obtained.

[0073] There is a strong correspondence between the valid pixel position Valid and the valid sub-convolution position. The window of the valid sub-convolution must contain valid pixels, and the valid pixels must correspond to a valid sub-convolution, so the position of the valid sub-convolution can be directly calculated based on Valid.

[0074] Specifically, each valid pixel except the edge of the image will correspond to K sub-convolution operations, where K is the size of the sub-convolution window. The positions of the K sub-convolution operations can be obtained by matching the pixel with the elements at different positions in the convolution kernel. The number of sub-convolution operations corresponding to the pixels at the edge of the image will be less than K, and additional judgment is required to detect whether the position of the sub-convolution is correct and valid. It is known that the positions of all valid pixels are recorded in Valid, so by traversing Valid and taking the union of all the valid sub-convolution positions obtained, the positions of all valid sub-convolutions can be obtained, which are recorded as Solid. Since event camera data is usually extremely sparse, the number of elements in Valid is much smaller than the size of the actual image, so this greatly improves the time efficiency compared to detecting valid sub-convolutions by traversing according to the definition of valid sub-convolutions.

[0075] Further, the step 3 is specifically as follows: image convolution is essentially a combination of many logically parallel multiplication and addition operations, which is a very regular operation form. In fact, image convolution is often converted into a general matrix multiplication (GEMM) form that is more suitable for computer parallel operation during the actual operation process. The conversion of the general matrix multiplication form can greatly speed up the operation. In order to facilitate the introduction of the operation organization form involved in the present invention, the principle of general image convolution operation is first introduced here. In the general image convolution operation, the convolution kernel is expanded into a row vector, and the convolution kernel vectors of different channels constitute the convolution kernel matrix. The image data in the image convolution operation is expanded into a column vector, and the image matrix is ​​obtained by combining data from different channels and different batches. At this time, if the obtained convolution kernel matrix is ​​multiplied with the image matrix, a general matrix multiplication operation is completed, and the result of the image convolution operation can be obtained (the result is slightly different in the element order). Since invalid calculations, that is, multiplication and addition operations involving 0 elements, are not considered in the general image convolution operation, a large amount of computing resources will be wasted when processing event camera data, slowing down the operation time. The present invention makes innovations in the process of processing the image matrix. On the basis of removing invalid operation data, the remaining valid operations are converted into general matrix multiplication operations to complete efficient operations. The step 3 is specifically as follows:

[0076] Step 3.1: For each valid sub-convolution, follow the general image convolution method, expand the image data of one channel in the convolution window into a column vector in a row-first manner, and concatenate the column vectors obtained in different channels in sequence to form a column vector corresponding to the current valid sub-convolution;

[0077] Step 3.2: Traverse all valid sub-convolution operations in the order in Solid, and repeat step 3.1 to convert all image data required for all valid operations into several column vectors;

[0078] Step 3.3: Put the column vectors corresponding to all the above valid sub-convolutions into adjacent memory spaces, complete the row-by-row concatenation, and obtain an image matrix; this matrix is ​​the matrix used for general matrix multiplication operations; since in general image convolution operations, the process of converting an image into a matrix participating in general matrix multiplication operations is called im2col, the above process is called sparse im2col in this invention.

[0079] Specifically, the process of processing the convolution kernel matrix is ​​consistent with the general image convolution algorithm. Compared with im2col in general image convolution, sparse im2col does not need to directly traverse the entire input tensor, but only needs to traverse the effective convolution part which accounts for a smaller proportion, so it brings two benefits. On the one hand, the amount of calculation in the conversion process is greatly reduced, and for the computer, the conversion process from convolution to general matrix multiplication can be completed faster. On the other hand, the matrix obtained after the conversion is smaller, which can reduce the computer's storage space consumption, and the smaller matrix also means that the amount of calculation of subsequent matrix multiplication operations will be greatly reduced.

[0080] Furthermore, in the actual image convolution operation, the matrix size corresponding to the convolution kernel is:

[0081] [channel_out,kernel_size*kernel_size*channel_in]

[0082] The matrix size corresponding to the image data is:

[0083] [kernel_size*kernel_size*channel_in,Solid]

[0084] In general image convolution operations, the matrix size corresponding to the image data is:

[0085] [kernel_size*kernel_size*channel_in,N*h_out*w_out]

[0086] Therefore, the ratio of the computational amount of the present invention to the computational amount of general image convolution can be calculated as follows:

[0087] η=Solid / (N*h_out*w_out).

[0088] Specifically, the computational amount FLOPs of the algorithm involved in the present invention is proportional to the sparsity of Solid. The present invention observes the computational sparsity at the sub-convolution level, retains only the valid sub-convolutions, and organizes the remaining valid sub-convolutions into general matrix multiplication operations, which not only greatly reduces the amount of computation, but also ensures the hardware friendliness of the computation.

[0089] Furthermore, the matrix product result Pd of the general image convolution operation completed by the general matrix multiplication is equivalent to the theoretical convolution result. However, the general matrix multiplication operation completed in step 3 only focuses on the operation of non-zero elements, so its product result Ps is different from the general image convolution operation. It is a subset of the complete convolution operation result that only contains valid information. In order to be compatible and adapt to other computing components in modern neural networks, such as maximum pooling, batch normalization, etc., Ps needs to be further converted into the same form as the general image convolution. The specific step 4 is:

[0090] Step 4.1: Fill the corresponding positions in the obtained product matrix Ps with 0 elements to expand it to the same size and shape as the matrix multiplication result Pd in ​​the general image convolution; this can be done by reusing Solid;

[0091] Step 4.2: According to the description in step 4.1, there is the following relationship between Ps and Pd:

[0092] Ps={p i |p i ∈Pd; i∈Solid}

[0093] That is, the result obtained after taking the column combination of the elements in Solid in the Pd matrix is ​​Ps; this is not difficult to explain, because in matrix multiplication, the columns of the product and the columns of the multiplicand matrix have a one-to-one correspondence;

[0094] Step 4.3: Therefore, by using the above relationship in reverse, remap the elements in Ps to new positions based on Solid, and fill the complement positions of Solid with 0 to get the desired result, which is recorded as Pds;

[0095] Step 4.4: Convert the Pds obtained in step 4.3 into the format of general image convolution. The result Pds obtained by general matrix multiplication has a data arrangement format of [C, N, H, W], while in modern neural networks, the generally used data arrangement format is [N, C, H, W]. Therefore, by exchanging the coordinate axis positions, the final image convolution result can be obtained.

[0096] from Figure 5It can be seen that in (a), the inference time of the present invention is greatly reduced compared with general image convolution and other sparse schemes, and the amount of calculation in (b) is reduced by an order of magnitude compared with general image convolution.

[0097] Specifically, the DAVIS346 camera is selected as the specific event camera visual sensor, and the ros driver package is used to record the event camera raw data to obtain the original event stream with a spatial resolution of 260*346. The raw data is recorded in a variety of different scenes such as indoors, outdoors, static, and dynamic to avoid the influence of a single scene on the objectivity of the results.

[0098] Take a series of Δt time windows to segment the camera raw data and obtain a series of event groups with different time lengths. The starting value of Δt is set to 1ms, the ending value is set to 100ms, and the step size is set to 1ms. A total of 100 groups of different Δt values ​​are taken. Correspondingly, the raw data is segmented according to 100 time windows to obtain 100 groups of data. Then these 100 groups of data are tensorized to obtain 100 groups of tensorized visual data. The data is arranged in the format of [N, C, H, W], where N is the batch size, which is set to 8, and C is the number of data channels. Since the event camera has both positive and negative event signals, C is 2 here. H and W represent the height and width of the image, respectively, which are set to 260 and 346 according to the camera parameters. The sparsity of these 100 groups of data decreases with the increase of Δt, which is used to test the performance of the proposed sparse convolution method under different sparsities.

[0099] Set the convolution kernel size to the commonly used 3*3, the convolution step size to 1, the padding to 0, the input channel to 2, the output channel to 20, and use the kaiming initialization method to assign the convolution kernel weights.

[0100] The sparse convolution method of the present invention is implemented in C++ and packaged as a pytorch operation function in the form of a pytorch extension operator, similar to general image convolution. The function is called in pytorch to perform convolution operations on the above 100 sets of data to perform sparse convolution.

[0101] During the operation experiment, the system time is read before and after the sparse convolution step, and the difference between the two is used to obtain the sparse convolution inference time. Multiple experiments are performed to eliminate the influence of other factors such as the computer operation status on the inference time. At the same time, the ratio of the image matrix size involved in the general matrix multiplication operation in each inference process to the image matrix size in the general image convolution operation in theory is counted, and this is used as the normalized experimental result of the amount of operation.

[0102] After multiple tests in indoor and outdoor scenes, dynamic and static scenes, etc., the results are as follows Figure 5 As shown, Figure 5 a is the comparison of operation time, it can be clearly seen that the reasoning time efficiency of the present invention is much better than that of the SCN sparse convolution method. The reason is that SCN uses a hash table to reorganize effective operations, which destroys the continuity in the calculation. Therefore, in most cases, its performance is not even better than that of general image convolution. The present invention overcomes the problem of discontinuous calculation by using general matrix multiplication, and its reasoning performance is better than that of general image convolution within the normal working range. Figure 5 b shows a comparison of the amount of calculation. Obviously, the method described in the present invention effectively reduces the amount of calculation by an order of magnitude. At the same time, since the present invention does not introduce any new calculations compared to general image convolution, the sparse convolution result is exactly the same as the general image convolution, and the reasoning accuracy is not affected in any way.

[0103] Implementation Method 2

[0104] This embodiment provides an efficient sparse convolution system for event camera data. The efficient sparse convolution system uses an efficient sparse convolution method for event camera data as described in Embodiment 1. The efficient sparse convolution system includes:

[0105] Preprocessing module, obtains event camera data S within the Δt time period Δt , and preprocessing;

[0106] The effective operation module performs effective operation detection on the preprocessed data;

[0107] Based on the detected effective operations, reorganize the effective operations;

[0108] The post-processing module performs post-processing of the operation results based on the effective operations reorganized by the effective operation module.

[0109] Implementation Method 3

[0110] An embodiment of the present invention provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the memory is used to store software programs and modules, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory and the processor are connected via a bus. Specifically, the processor implements any step in the first embodiment above by running the computer program stored in the memory.

[0111] It should be understood that in the embodiments of the present invention, the processor referred to may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0112] The memory may include a read-only memory, a flash memory, and a random access memory, and provides instructions and data to the processor. A part or all of the memory may also include a nonvolatile random access memory.

[0113] As can be seen from the above, the electronic device provided by the embodiment of the present invention can implement the efficient sparse convolution method for event camera data as described in embodiment 1 by running a computer program. By utilizing the sparsity of the event camera image data, the invalid 0-element multiplication and addition operations are discarded, and only the effective sub-convolution operations are retained, thereby greatly reducing the computational complexity. Secondly, by using the form of dense matrix multiplication to complete the effective sub-convolution operation, compared with other sparse convolution methods, due to the use of a more reasonable operation organization method, the reduction in the amount of operation is successfully converted into a significant reduction in reasoning time, giving full play to the low latency characteristics of the event camera.

[0114] It should be understood that if the above-mentioned integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned implementation method, and can also be completed by instructing the relevant hardware through a computer program. The above-mentioned computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method implementations when executed by the processor. Among them, the above-mentioned computer program includes computer program code, and the above-mentioned computer program code can be in source code form, object code form, executable file or some intermediate form. The above-mentioned computer-readable medium may include: any entity or device capable of carrying the above-mentioned computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the above-mentioned computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0115] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest range consistent with the principles and novel features disclosed herein.

[0116] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the implementation method can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method implementation method, which will not be repeated here.

[0117] It should be noted that the methods and detailed examples provided in the above embodiments can be combined with the devices and equipment provided in the embodiments, and references can be made to each other, and no further details will be given.

[0118] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0119] In the embodiments provided by the present invention, it should be understood that the disclosed device / terminal equipment and method can be implemented in other ways. For example, the device / equipment implementation described above is only illustrative, for example, the division of the above modules or units is only a logical function division, and in actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0120] The above-mentioned embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above-mentioned embodiments, or replace some of the technical features therein by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. An efficient sparse convolution method for event camera data, characterized in that: The efficient sparse convolution method comprises the following steps: Step 1: Obtain event camera data S within a time period of Δt Δt , and preprocessing; Step 2: Perform effective operation detection on the data preprocessed in step 1; Step 3: Reorganize the efficient operations detected in step 2 into an efficient operation form of matrix multiplication; Step 4: Post-process the operation result obtained in step 3 to complete the image convolution operation.

2. The efficient sparse convolution method for event camera data according to claim 1, characterized in that: The step 1 is specifically as follows: acquiring the event camera data S of the Δt time period Δt Expressed as an event i The set of S Δt ={event i |t i ∈Δt}; the event event i Including time t i Dimension, polarity p i Dimension, width dimension and height dimension, that is, event i =(x i ,y i ,t i ,p i ); The time t i The dimension is specifically that when Δt is small enough, the event camera data S Δt The events within the time period Δt occur simultaneously, and the time axis information of the visual events within the time period Δt is compressed; The polarity p i The specific dimension is to divide different polarities p i The events are regarded as information on different channels of the image, that is, polarity is used as the channel dimension; The width x i Dimension and height y i The specific dimension is an event event i Represented as a multidimensional vector T i , that is, T i =[N,p i ,h i ,w i ], where p i ,h i ,w i ∈event i , N represents the batch dimension, p i represents the channel dimension, h i and w i represents the height and width dimensions; T obtained by converting all events within Δt i The real-valued part of the superposition image tensor is filled with 0 for the pixel positions where no events occurred.

3. The efficient sparse convolution method for event camera data according to claim 2, characterized in that: The step 2 specifically includes: using the position of the valid pixel to detect the position of the valid sub-convolution operation.

4. The efficient sparse convolution method for event camera data according to claim 3, characterized in that: The corresponding relationship between the position of the effective subconvolution operation and the position of the effective pixel Valid is specifically as follows: It is known that the positions of all valid pixels are recorded in Valid. Valid is traversed and the union of all valid sub-convolution positions is obtained to obtain the positions of all valid sub-convolutions, which are recorded as Solid.

5. The efficient sparse convolution method for event camera data according to claim 3, characterized in that: The step 3 is specifically as follows: Step 3.1: For each valid sub-convolution, expand the image data of one channel in the convolution window into a column vector in a row-first manner, and concatenate the column vectors obtained in different channels in sequence by column to form a column vector corresponding to the current valid sub-convolution; Step 3.2: Traverse all valid sub-convolution operations in the order in Solid, and repeat step 3.1 to convert all image data required for all valid operations into several column vectors; Step 3.3: Put the column vectors corresponding to all the above valid sub-convolutions into adjacent memory spaces, complete the row-by-row splicing, and you can get an image matrix.

6. The efficient sparse convolution method for event camera data according to claim 5, characterized in that: In the actual image convolution operation, the matrix size corresponding to the convolution kernel is: [channel_out,kernel_size*kernel_size*channel_in] The matrix size corresponding to the image data is: [kernel_size*kernel_size*channel_in,Solid] In general image convolution operations, the matrix size corresponding to the image data is: [kernel_size*kernel_size*channel_in,N*h_out*w_out] Therefore, the calculation ratio is, η=Solid / (N*h_out*w_out).

7. The efficient sparse convolution method for event camera data according to claim 6, characterized in that: The step 4 is specifically as follows: Step 4.1: Fill the corresponding positions in the obtained product matrix Ps with 0 elements to expand it to the same size and shape as the matrix multiplication result Pd in ​​the general image convolution; Step 4.2: According to the description in step 4.1, there is the following relationship between Ps and Pd: Ps={p i |p i ∈Pd;i∈Solid} That is, the result obtained after taking the column combination of the elements in Solid in the Pd matrix is ​​Ps; Step 4.3: Therefore, by using the above relationship in reverse, remap the elements in Ps to new positions based on Solid, and fill the complement positions of Solid with 0 to get the desired result, which is recorded as Pds; Step 4.4: Convert the Pds obtained in step 4.3 into the format of general image convolution. The result Pds obtained by general matrix multiplication has a data arrangement format of [C, N, H, W], while in modern neural networks, the generally used data arrangement format is [N, C, H, W]. Therefore, by exchanging the coordinate axis positions, the final image convolution result can be obtained.

8. An efficient sparse convolution system for event camera data, characterized in that: The efficient sparse convolution system uses an efficient sparse convolution method for event camera data according to any one of claims 1 to 7, and the efficient sparse convolution system includes: Preprocessing module, obtains event camera data S within the Δt time period Δt , and preprocessing; The effective operation module performs effective operation detection on the preprocessed data; Based on the detected effective operations, reorganize the effective operations; The post-processing module performs post-processing of the operation results based on the effective operations reorganized by the effective operation module.

9. A computer device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images