Data processing and motion estimation method and device based on event camera

Through the data processing method of event camera, encoding and binary self-supervised image generation are used to use the update time of event stream data, and combining pulse and convolutional neural networks, the problems of high acquisition costs and low efficiency in the existing technology are solved, and the application is achieved High-efficiency motion estimation and data volume reduction for different types of event cameras.

CN114627150BActive Publication Date: 2024-11-19CAPITAL NORMAL UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210249593.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-14
Publication Date
2024-11-19
Estimated Expiration
2042-03-14

AI Technical Summary

Technical Problem

The existing data processing methods based on event cameras have high acquisition costs and lack of universality, and have failed to fully utilize the advantages of asynchronous updates and space-time discreteness, resulting in low algorithm efficiency and large data computing volume.

Method used

A data processing and motion estimation method based on event camera is proposed. By obtaining the update time of event stream data, preprocessing and encoding, training data is generated, and it is restored to binary self-supervised images, using pulsed neural networks and convolutional nerves The network is trained, and the overall loss function is used for loss calculation to realize motion estimation.

Benefits of technology

Simplify the spatiotemporal encoding method of data, reduce the acquisition cost of self-supervised data, and make the method suitable for different types of event camera data formats, make full use of spatiotemporal data, and achieve effective reduction of data volume, and be used for motion estimation, optical flow estimation and Object tracking and other applications.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114627150B_ABST
    Figure CN114627150B_ABST
Patent Text Reader

Abstract

The present invention proposes a data processing and motion estimation method and device based on an event camera, wherein the method includes: obtaining event stream data, and obtaining the update time corresponding to the event stream data; preprocessing and encoding the event stream data according to the update time to obtain the encoded data corresponding to the event stream data; and processing the encoded data by a downsampling method to generate training data; restoring the event stream data to an event volume graph, processing the event volume graph by a downsampling method to generate a binary self-supervised image; inputting the training data into a neural network for training, and using the self-supervised image and the overall loss function to calculate the training loss to complete the training of the neural network; and performing motion estimation through the trained neural network. The method processes the encoded data through a pulse neural network, and can be used for applications such as motion estimation, optical flow estimation, and object tracking.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the field of computer vision and to a motion estimation method based on an event camera. Background Art

[0002] The current methods for processing event cameras are: (1) restoring event stream data into binary images, and then using traditional visual algorithms on the binary images to perform motion estimation on the data; (2) extracting spatiotemporal information of different polarities from event stream data and processing them separately to meet the requirements. For traditional image data, motion estimation is mainly performed through methods such as frame difference.

[0003] For existing data processing methods based on event cameras, grayscale images need to be provided during the learning process for self-supervised learning. For self-collected data, this method has a high collection cost and is therefore extremely dependent on the data set. In addition, for existing methods, a large number of algorithms require asynchronous event streams to provide polarity information, and for some models of event cameras, it is impossible to guarantee that both polarity information and asynchronous timestamp information are provided, resulting in the lack of versatility of some algorithms.

[0004] Moreover, for the existing neural network processing methods, most of them adopt the traditional neural network approach, which fails to maximize the advantages of asynchronous update and spatial and temporal discreteness of event cameras, while significantly reducing the amount of data calculation. Summary of the invention

[0005] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0006] To this end, the first object of the present invention is to propose a data processing and motion estimation method based on an event camera, which is used for applications such as motion estimation, optical flow estimation, and object tracking.

[0007] The second objective of the present invention is to provide a data processing and motion estimation device based on an event camera.

[0008] A third object of the present invention is to provide a computer device.

[0009] A fourth object of the present invention is to provide a computer-readable storage medium.

[0010] To achieve the above-mentioned purpose, a first embodiment of the present invention proposes a data processing and motion estimation method based on an event camera, comprising:

[0011] Obtaining event stream data and obtaining the update time corresponding to the event stream data;

[0012] Preprocess and encode the event stream data according to the update time to obtain encoded data corresponding to the event stream data; and process the encoded data by a downsampling method to generate training data;

[0013] Restoring the event stream data into an event volume graph, processing the event volume graph by the downsampling method, and generating a binary self-supervised image;

[0014] Inputting the training data into a neural network for training, and using the self-supervised image and the overall loss function to calculate the training loss, thereby completing the training of the neural network;

[0015] Motion estimation is performed using a trained neural network.

[0016] In addition, the data processing and motion estimation method based on the event camera according to the above embodiment of the present invention may also have the following additional technical features:

[0017] Further, in one embodiment of the present invention, pre-processing and encoding the event stream data according to the update time to obtain encoded data corresponding to the event stream data includes:

[0018] Event interception based on sampling time;

[0019] Encode the intercepted event loop through the "event-time" encoding method;

[0020] Output encoded data.

[0021] Furthermore, in one embodiment of the present invention, encoding the intercepted event loop by using an “event-time” encoding method includes:

[0022] Dividing the intercepted events into N parts;

[0023] For the nth part of the equal division, the event accumulation is calculated pixel by pixel on the "event" channel and normalized; the time data of the last event occurring at each pixel is normalized on the "time" channel; and the "event-time" dual-channel coded data is obtained by combining the "event" channel and the "time" channel;

[0024] Repeat the previous step until all N events are processed;

[0025] The two-channel coded data are combined to obtain "event-time" four-channel coded data.

[0026] Furthermore, in one embodiment of the present invention, the downsampling method comprises:

[0027] Setting a threshold value, the threshold value being used to compare with the number of events occurring in a specific space and time;

[0028] The downsampling method is used to process the data size. The formula is as follows:

[0029] W h (m,n)=max{V h (x+i,y+i)|i∈[1,0,-1]},

[0030] Where (x, y) is the coordinate in the original image, (m, n) is the coordinate in the output image, and the value range of h is [1, N];

[0031] If the event volume data is greater than a preset threshold, the data is valid; otherwise, the event generated at this point is a noise event and should be removed.

[0032] Furthermore, in one embodiment of the present invention, the event stream data is restored into an event volume graph, and the event volume graph is processed by the downsampling method to generate a binary self-supervised image, including:

[0033] Event interception based on sampling time;

[0034] Calculating the cumulative amount of events pixel by pixel for the intercepted time to obtain event amount data;

[0035] Using a downsampling method to process the event volume data size and filter out noise data in the event volume data;

[0036] The accumulated value of each pixel is mapped to the grayscale value of the binary image, and a single-channel binary self-supervised image with the same size as the encoded data is output.

[0037] Furthermore, in one embodiment of the present invention, the step of inputting the encoded data into a neural network for training comprises:

[0038] The neural network is composed of a pulse neural network and a traditional artificial neural network. The encoded data is first encoded by an SNN and then decoded by an ANN. The ANN part is composed of a convolutional neural network and a residual network.

[0039] Furthermore, in one embodiment of the present invention, the overall loss function includes:

[0040] The loss function includes prediction loss and smoothing loss. The calculation function of the overall loss is shown as follows:

[0041] L total =L patch +λL smooth ,

[0042] Among them, L patch To predict the loss, L smooth is the smoothing loss, and λ is the weight value.

[0043] To achieve the above-mentioned purpose, a second embodiment of the present invention proposes a data processing and motion estimation device based on an event camera, comprising:

[0044] An acquisition module, used to acquire event stream data and acquire the update time corresponding to the event stream data;

[0045] A preprocessing module, configured to preprocess and encode the event stream data according to the update time to obtain encoded data corresponding to the event stream data; and to process the encoded data by a downsampling method to generate training data;

[0046] A generation module, used for restoring the event stream data into an event volume graph, processing the event volume graph by the downsampling method, and generating a binary self-supervised image;

[0047] A training module, used for inputting the training data into a neural network for training, and using the self-supervised image and the overall loss function to calculate the training loss, so as to complete the training of the neural network;

[0048] The motion estimation module is used to perform motion estimation through the trained neural network.

[0049] To achieve the above-mentioned purpose, the third aspect of the present invention proposes a computer device, characterized in that it includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, it implements the data processing and motion estimation method based on the event camera as described above.

[0050] To achieve the above objectives, a fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the event camera-based data processing and motion estimation method as described above.

[0051] The data processing and motion estimation method based on event cameras proposed in the present invention simplifies the spatiotemporal encoding method of data and reduces the acquisition cost of self-supervised data, so that the method can be applied to event camera data formats of different models. The data processing and motion estimation method based on event cameras processes the encoded data through a pulse neural network and can be used for applications such as motion estimation, optical flow estimation, and object tracking. Compared with traditional image processing methods, it can make full use of spatiotemporal data and effectively reduce the amount of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0053] Figure 1 The present invention provides a flow chart of a method for data processing and motion estimation based on an event camera according to an embodiment of the present invention.

[0054] Figure 2 The present invention provides a flow chart of a data processing and motion estimation device based on an event camera according to an embodiment of the present invention.

[0055] Figure 3 The present invention provides an overall flow chart of the embodiment of the present invention.

[0056] Figure 4 A flowchart of encoding event data provided by an embodiment of the present invention.

[0057] FIG5( a ) is a schematic diagram of a single data encoding provided by an embodiment of the present invention.

[0058] FIG. 5( b ) is a schematic diagram of a single data encoding provided by an embodiment of the present invention

[0059] Figure 6 A schematic diagram of the structure of a single coded data provided by an embodiment of the present invention.

[0060] Figure 7 A schematic diagram of the encoding result when N is 10 provided in an embodiment of the present invention.

[0061] FIG8( a ) is a schematic diagram of a binary frame restored from the original collected event stream data provided by an embodiment of the present invention.

[0062] FIG8( b ) is a schematic diagram of a binary frame restored from event stream data after denoising provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0063] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0064] The following describes a method and apparatus for data processing and motion estimation based on an event camera according to an embodiment of the present invention with reference to the accompanying drawings.

[0065] Figure 1The present invention provides a flow chart of a method for data processing and motion estimation based on an event camera according to an embodiment of the present invention.

[0066] like Figure 1 As shown, the data processing and motion estimation method based on event camera includes the following steps:

[0067] S1: Obtain event stream data and the update time corresponding to the event stream data;

[0068] S2: pre-process and encode the event stream data according to the update time to obtain the encoded data corresponding to the event stream data; and process the encoded data by a downsampling method to generate training data;

[0069] S3: restoring the event stream data into an event volume graph, processing the event volume graph by the downsampling method, and generating a binary self-supervised image;

[0070] S4: inputting the training data into the neural network for training, and using the self-supervised image and the overall loss function to calculate the training loss, so as to complete the training of the neural network;

[0071] S5: Perform motion estimation using the trained neural network.

[0072] The overall data processing flow of this embodiment is as follows Figure 3 As shown, for the data output by the event camera, the encoding method of the present invention can be used for encoding, image size adjustment and denoising, and then self-supervised data is generated by the proposed event camera self-supervised data generation method, and then the training loss is calculated during the training process by the proposed improved loss function.

[0073] First, the data collected from the event camera is encoded. Before using the neural network to process the data, the event stream data needs to be pre-processed and encoded. In order to give full play to the asynchronous characteristics of the event stream data, we designed an "event-time" encoding method. This method can be applied to any model of event camera and fully utilizes the time and space discrete characteristics of the event camera data.

[0074] Because some types of event cameras do not output event stream data asynchronously, in order to ensure that they are suitable for correct encoding, the event stream data should first be sorted in ascending order of update time. Then the "event-time" encoding method proposed in this patent is adopted. This method can utilize the characteristics of the pulse neural network and make full use of the output data of the event camera. The specific encoding steps of this method are as follows. The encoding process is as follows Figure 4 .

[0075] Furthermore, in one embodiment of the present invention, the event stream data is pre-processed and encoded according to the update time to obtain the encoded data corresponding to the event stream data, including:

[0076] Event interception based on sampling time;

[0077] Encode the intercepted event loop through the "event-time" encoding method;

[0078] Output encoded data:

[0079] Furthermore, in one embodiment of the present invention, the intercepted event loop is encoded by an "event-time" encoding method, including:

[0080] Dividing the intercepted events into N parts;

[0081] For the nth part of the equal division, the event accumulation is calculated pixel by pixel on the "event" channel and normalized; the time data of the last event at each pixel is normalized on the "time" channel; and the "event-time" dual-channel coded data is obtained by combining the "event" channel and the "time" channel;

[0082] Repeat the previous step until all N events are processed;

[0083] The two-channel encoded data are combined to obtain "event-time" four-channel encoded data.

[0084] For the collected event stream data, a fixed time T is used sample ms sampling time window, intercept the event stream data within the sampling time window, and encode each intercepted part in turn.

[0085] First, the event stream data in the sampling time window is divided into N parts, and the event data in each part is encoded using the "event-time" dual-channel encoding process.

[0086] For the "event" channel, the events in the nth part need to be accumulated pixel by pixel. The calculation method is shown in formula (1), where δ is the Kronecker delta function, u is the current pixel, and k is the index of the event data. This formula can be used to obtain the original event volume data on the first channel "event" channel of the "event-time" encoding. In order to ensure that the data is consistent in scale, the event volume data is normalized to obtain the first channel "event" channel of the "event-time" encoding. The calculation formula is shown in formula (2), where is the value of the event quantity corresponding to the pixel with the least event quantity in the current nth (n∈[1,N]) part, is the event value corresponding to the pixel with the largest number of events, e n It is the value of the event quantity corresponding to the current pixel.

[0087]

[0088]

[0089] For the "time" channel, the time data of the last event at each pixel point needs to be normalized. In order to keep the scale consistent with the event volume data, the time data is normalized to obtain the second channel "time" channel of "event-time" encoding. The calculation formula is shown in formula (3), where is the minimum value of the timestamp in the nth part, taking the time corresponding to the first event in the time window, is the maximum value of the timestamp in the nth part, taking the time corresponding to the last event in the time window, t n The time when the current event occurs.

[0090]

[0091] Combine these two channels, with the "event" channel as the first channel and the "time" channel as the second channel, to get the "event-time" encoding of this part. Execute this encoding on N parts in sequence to get N "event-time" encodings within the sampling time window.

[0092] The result after encoding is shown in Figure 5(a) (schematic diagram, used for illustration only, not real data). The event information of the nth part constitutes a dual-channel data, where the first channel is the normalized value of the cumulative sum of the number of events generated at the pixel point in the event batch, ranging from [0, 255]; the second channel is the linearly normalized value of the time of the last event generated by the pixel point that generated the event in the first channel, ranging from [0, 255].

[0093] After the "event-time" encoding is completed, in order to facilitate neural network processing, the data directly output by the event camera needs to be scaled to the input size corresponding to the neural network, and denoising operations should be performed at the same time. Here, a downsampling method based on event camera data is proposed, which can also perform denoising operations at the same time.

[0094] Furthermore, in one embodiment of the present invention, the downsampling method comprises:

[0095] Setting a threshold value, the threshold value being used to compare with the number of events occurring in a specific space and time;

[0096] The downsampling method is used to process the data size. The formula is as follows:

[0097] W h (m,n)=max{V h (x+i,y+i)|i∈[1,0,-1]},

[0098] Where (x, y) is the coordinate in the original image, (m, n) is the coordinate in the output image, and the value range of h is [1, N];

[0099] If the event volume data is greater than a preset threshold, the data is valid; otherwise, the event generated at this point is a noise event and should be removed.

[0100] Specifically, a threshold Thr1 needs to be set in advance, and the data obtained in the previous step is subjected to a maximum pooling operation with a step length of 3. The operation is shown in formula (4), where (x, y) is the coordinate in the original image, (m, n) is the coordinate in the output image, and the value range of h is [1, N]. The values ​​of the pixel and its eight neighborhoods are compared to find the maximum event value.

[0101] At the same time, in order to filter noise, if the data is greater than the preset threshold Thr1, the value is considered valid, and the original normalized data of the pixel point is kept in the encoded dual-channel "event-time" encoded data; otherwise, the event generated by the point is considered to be a noise event and should be removed, and the event amount and time values ​​in the encoded data are both 0. After this step, the encoded data can be effectively resized and the noise event can be effectively removed.

[0102] W h (m,n)=max{V h (x+i,y+i)|i∈[1,0,-1]} (4)

[0103] After completing the "event-time" encoding, resizing, and denoising, in order to fully utilize the discrete characteristics of the event stream and the discrete nature of the spiking neural network in time, the N groups of encoded "event-time" data need to be combined. Divide it into two parts, front and back, and combine them into four-channel data in turn. The combination result is shown in Figure 5(b), that is, the combination method is the kth group of "event-time" encoding and the kth group of "event-time" encoding. The event-time codes are combined into a set of four-channel data, where k is [1, N] and finally generates The encoding result after taking N=10 in the time window is as follows: Figure 6 As shown, the total generated Four-channel encoding results (this figure is for illustration only and does not represent real data, and N can be 8, 10, 12, 14, etc.).

[0104] Compared with traditional frame-based cameras, existing event camera datasets with real supervised data are very scarce. In order to facilitate the self-supervised training of neural networks, this patent proposes a method for obtaining self-supervised images, which can greatly reduce the cost of obtaining the self-supervised data required in the training process.

[0105] Furthermore, in one embodiment of the present invention, the event stream data is restored into an event volume graph, and the event volume graph is processed by a downsampling method to generate a binary self-supervisory image, including:

[0106] Event interception based on sampling time;

[0107] Calculating the cumulative amount of events pixel by pixel for the intercepted time to obtain event amount data;

[0108] Using a downsampling method to process the event volume data size and filter out noise data in the event volume data;

[0109] The accumulated value of each pixel is mapped to the grayscale value of the binary image, and a single-channel binary self-supervised image with the same size as the encoded data is output.

[0110] This method uses the method of restoring asynchronous event stream data into event frames (i.e., images used for self-supervised learning). Because event cameras are extremely susceptible to noise (including environmental noise and hardware inherent noise), in order to reduce the impact of noise and enhance the learning effect, denoising operations are required when generating self-supervised images. For pixels that truly feel changes in brightness and darkness, multiple events are often generated continuously at this point, while sporadic event points that appear in space and time can be regarded as noise points. This feature can be used to perform better denoising operations. The specific steps for generating binary self-supervised images are as follows. The process is as follows Figure 7 shown.

[0111] First, a threshold Thr2 is set, which can be used to filter noise events and is used to compare with the number of events occurring in a specific space and time.

[0112] Formula (1) is used to accumulate the events at each pixel point within the sampling time, that is, to calculate the sample How many events occur at this pixel, and record the value as V(x,y), where (x,y) represents the coordinates of the pixel. The data obtained at this time is the event volume data with the same size as the original camera output size.

[0113] The event volume data output in the previous step is cropped and scaled. The downsampling method in the present invention is used to convert it to the same size as the encoded data. The data size is processed using the downsampling method, and the formula is as follows:

[0114] W h (m,n)=max{V h (x+i,y+i)|i∈[1,0,-1]},

[0115] Where (x, y) is the coordinate in the original image, (m, n) is the coordinate in the output image, and the value range of h is [1, N];

[0116] To filter noise, as shown in formula (5), if the data is greater than the preset threshold Thr2, the value is considered valid and is reflected as a white point in the self-supervised binary image, that is, the grayscale value is 255; otherwise, the event generated by the point is considered to be a noise event and should be removed, which is reflected as a black background in the self-supervised binary image, that is, the grayscale value is 0.

[0117]

[0118] After this step, a single-channel binary self-supervisory image with the same size as the encoded data is obtained. As shown in Figure 8(a) and Figure 8(b), the self-supervisory image is restored from the event stream data generated when a tennis ball moves quickly from the bottom of the picture to the top of the picture. Figure 8(a) is a binary image generated without denoising, and Figure 8(b) is a binary image generated after denoising. It can be clearly seen from the figure that this method can effectively remove noise while preserving the original object information as much as possible. The effectiveness of this method and the downsampling denoising method is verified.

[0119] Furthermore, in one embodiment of the present invention, the encoded data is input into a neural network for training, including:

[0120] The neural network consists of a pulse neural network and a traditional artificial neural network. The encoded data is first encoded by SNN and then decoded by ANN. The ANN part consists of a convolutional neural network and a residual network.

[0121] The neural network used to process the data is composed of a pulse neural network and a traditional artificial neural network. The data is first encoded by SNN to fully extract the spatiotemporal information, and then decoded by ANN to ensure the effectiveness of the calculation. The ANN part is composed of structures such as convolutional neural network and residual network.

[0122] Among them, when inputting the neural network, in order to fully reflect the effect of SNN, it is necessary to follow the following Figure 6 The coded data shown is input in sequence, one four-channel data at a time, for a total of For the data predicted during the training process, the binary self-supervised images generated in the patent and the proposed loss function are used for learning.

[0123] Furthermore, in one embodiment of the present invention, the overall loss function includes:

[0124] The loss function includes prediction loss and smoothing loss. The calculation function of the overall loss is shown as follows:

[0125] L total =L patch +λL smooth ,

[0126] Among them, L patch To predict the loss, L smooth is the smoothing loss, and λ is the weight value.

[0127] Since the present invention uses the generation of binary self-supervisory images from event stream data for loss calculation. This method reduces the cost of data acquisition but also brings certain problems. It can be clearly seen from Figure 8 that in order to reduce the impact of noise, the binary self-supervisory image restored using the event stream has obvious internal voids, which makes it discontinuous inside and at the edges. This situation will have a great impact on the pixel-by-pixel loss calculation method. In order to minimize the impact of this phenomenon in the loss calculation process, the loss calculation method described in formula (7) is subsequently improved. The grayscale difference of each point is obtained by taking the difference of the M×M size patch around the pixel as the center, and then averaging it. By calculating L patch The average grayscale difference between the point and its neighboring points is calculated by using the neighbor pixel loss method (i.e., neighbor pixel loss). As shown in formula (9), the grayscale values ​​of the neighboring pixels around a certain pixel are also calculated together for loss, which can minimize the impact of holes inside and at the edges of the binary self-supervised image.

[0128] The loss function in the present invention is mainly divided into two parts, namely prediction loss and smoothing loss. The prediction loss is to calculate the grayscale difference at the same pixel position before and after the prediction for optimization, and the smoothing loss is to make the predicted values ​​between adjacent pixels have spatial consistency. In order to properly evaluate the prediction loss within the sampling time, the binary self-supervised image construction method proposed by the present invention is adopted in the training network, and the self-supervised image constructed in the time window before the time window and the self-supervised image constructed in the current time window are used as the start and end states of the prediction, that is, the binary images corresponding to the start and end states are (I t ,I t +d t ), where d t is the step length of the time window between each calculation. The predicted value output by the neural network will be used to transform the pixel coordinates between the two images, that is, to reverse the predicted data and transform the pixel coordinate information from I t +dt Convert to I t ’. At this time, the converted image I t ′ and the original image I t The corresponding grayscale values ​​at the same position should be the same.

[0129] The purpose of prediction loss is to minimize the first binary image I t And the reconstructed image I restored by the second binary image and predicted data t The loss function uses the "constant brightness" assumption, which is shown in formula (6), that is, within the time δt, the horizontal coordinate is transformed by the distance δx, and the vertical coordinate is transformed by the distance δy. That is, the (x, y) coordinate on the first binary image is transformed to the (x+δx, y+δy) coordinate on the second image. The grayscale information corresponding to these two pixels should be the same, where (δx, δy) is the predicted value in the x, y direction.

[0130] I(x,y,t)=I(x+δx,y+δy,t+δt) (6)

[0131] Based on this assumption, the pixel-by-pixel prediction loss function can be designed as shown in formula (7).

[0132]

[0133] Among them I t and I t +d t represents the first binary image and the second binary image, u(x,y) and v(x,y) are the predicted values ​​of the point (x,y) in the horizontal and vertical directions (i.e., the x direction and the y direction), respectively, and u and v are the coordinates of the pixel points after coordinate transformation. ρ is Charbonnier Loss. A large number of literatures have verified that this method can be used to improve model performance, make the model more robust, better handle model outliers, and enable the model to converge faster. Its function is shown in formula (8). The ε value in this formula is an adjustable hyperparameter.

[0134]

[0135]

[0136] At the same time, in order to enhance the spatial consistency of the data predicted by the pixel and its neighboring pixels, a smoothing function L is used smooth To minimize the predicted error between neighboring pixels, as a regularization operation for prediction. The loss is expressed as shown in formula (10).

[0137]

[0138] Where H is the height of the predicted output, W is the width of the predicted output, and u and v are the horizontal and vertical coordinates of the predicted matrix, respectively. That is, the gradient of the prediction is calculated to obtain the smoothing loss L smooth .

[0139] Finally, a weight value λ is used to predict the loss L patch and smoothing loss L smooth The prediction loss and smoothing loss are calculated, and the calculation function of the overall loss is shown in formula (11).

[0140] L total =L patch +λL smooth (11)

[0141] The data processing and motion estimation method based on the event camera proposed in the embodiment of the present invention simplifies the spatiotemporal encoding method of data and reduces the acquisition cost of self-supervised data, so that the method can be applied to event camera data formats of different models. The data processing and motion estimation method based on the event camera processes the encoded data through a pulse neural network and can be used for applications such as motion estimation, optical flow estimation, and object tracking. Compared with the traditional image processing method, it can make full use of the data in spatiotemporal space and effectively reduce the amount of data.

[0142] In order to implement the above embodiment, the present invention also proposes a data processing and motion estimation device based on an event camera.

[0143] Figure 2 A schematic diagram of the structure of a data processing and motion estimation device based on an event camera provided by an embodiment of the present invention.

[0144] like Figure 2 As shown, the data processing and motion estimation device based on the event camera includes: an acquisition module 10, a preprocessing module 20, a generation module 30, a training module 40, and a motion estimation module 50, wherein:

[0145] The acquisition module is used to acquire event stream data and the update time corresponding to the event stream data;

[0146] A preprocessing module is used to preprocess and encode the event stream data according to the update time to obtain the encoded data corresponding to the event stream data; and to process the encoded data by a downsampling method to generate training data;

[0147] A generation module is used to restore the event stream data into an event volume graph, process the event volume graph through a downsampling method, and generate a binary self-supervised image;

[0148] The training module is used to input the training data into the neural network for training, and use the self-supervised image and the overall loss function to calculate the training loss to complete the training of the neural network;

[0149] The motion estimation module is used to perform motion estimation through the trained neural network.

[0150] To achieve the above-mentioned purpose, the third aspect of the present invention proposes a computer device, characterized in that it includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, it implements the data processing and motion estimation method based on the event camera as described above.

[0151] To achieve the above objectives, a fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the event camera-based data processing and motion estimation method as described above.

[0152] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0153] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0154] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0155] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0156] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0157] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0158] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0159] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A data processing and motion estimation method based on an event camera, characterized in that: The following steps are involved: Obtaining event stream data and obtaining the update time corresponding to the event stream data; Preprocess and encode the event stream data according to the update time to obtain encoded data corresponding to the event stream data; and process the encoded data by a downsampling method to generate training data; Restoring the event stream data into an event volume graph, processing the event volume graph by the downsampling method, and generating a binary self-supervised image; Inputting the training data into a neural network for training, and using the self-supervised image and the overall loss function to calculate the training loss, thereby completing the training of the neural network; Motion estimation through trained neural network; The preprocessing and encoding of the event stream data according to the update time to obtain the encoded data corresponding to the event stream data includes: Event interception based on sampling time; Encode the intercepted event loop using the "event-time" encoding method; Output encoded data; The intercepted event loop is encoded by the "event-time" encoding method, including: Dividing the intercepted events into N parts; For the nth part of the equal division, the event accumulation is calculated pixel by pixel on the "event" channel and normalized; the time data of the last event occurring at each pixel is normalized on the "time" channel; and the "event-time" dual-channel coded data is obtained by combining the "event" channel and the "time" channel; Repeat the previous step until all N events are processed; Combining the two-channel coded data to obtain "event-time" four-channel coded data; The step of restoring the event stream data into an event volume graph, processing the event volume graph by the downsampling method, and generating a binary self-supervised image includes: Event interception based on sampling time; Calculating the cumulative amount of events pixel by pixel for the intercepted time to obtain event amount data; Using a downsampling method to process the event volume data size and filter out noise data in the event volume data; The accumulated value of each pixel is mapped to the grayscale value of the binary image, and a single-channel binary self-supervised image with the same size as the encoded data is output.

2. The method according to claim 1, characterized in that The downsampling method comprises: Setting a threshold value, the threshold value being used to compare with the number of events occurring in a specific space and time; The downsampling method is used to process the data size. The formula is as follows: W h (m,n)=max{V h (x+i,y+i)|i∈[1,0,-1]}, Where (x, y) is the coordinate in the original image, (m, n) is the coordinate in the output image, and the value range of h is [1, N]; If the event volume data is greater than a preset threshold, the data is valid; otherwise, the event generated at this point is a noise event and should be removed.

3. The method according to claim 1, characterized in that The step of inputting the encoded data into a neural network for training comprises: The neural network is composed of a pulse neural network and a traditional artificial neural network. The encoded data is first encoded by an SNN and then decoded by an ANN. The ANN part is composed of a convolutional neural network and a residual network.

4. The method according to claim 1, characterized in that: The overall loss function includes: The loss function includes prediction loss and smoothing loss. The calculation function of the overall loss is shown as follows: THE total =L patch +λL smooth , Among them, L patch To predict the loss, L smooth is the smoothing loss, and λ is the weight value.

5. A data processing and motion estimation device based on an event camera, characterized in that: include: An acquisition module, used to acquire event stream data and acquire the update time corresponding to the event stream data; A preprocessing module, configured to preprocess and encode the event stream data according to the update time to obtain encoded data corresponding to the event stream data; and to process the encoded data by a downsampling method to generate training data; A generation module, used for restoring the event stream data into an event volume graph, processing the event volume graph by the downsampling method, and generating a binary self-supervised image; A training module, used for inputting the training data into a neural network for training, and using the self-supervised image and the overall loss function to calculate the training loss, so as to complete the training of the neural network; A motion estimation module, used for performing motion estimation through a trained neural network; The preprocessing and encoding of the event stream data according to the update time to obtain the encoded data corresponding to the event stream data includes: Event interception based on sampling time; Encode the intercepted event loop using the "event-time" encoding method; Output encoded data; The intercepted event loop is encoded by the "event-time" encoding method, including: Dividing the intercepted events into N parts; For the nth part of the equal division, the event accumulation is calculated pixel by pixel on the "event" channel and normalized; the time data of the last event occurring at each pixel is normalized on the "time" channel; and the "event-time" dual-channel coded data is obtained by combining the "event" channel and the "time" channel; Repeat the previous step until all N events are processed; Combining the two-channel coded data to obtain "event-time" four-channel coded data; The step of restoring the event stream data into an event volume graph, processing the event volume graph by the downsampling method, and generating a binary self-supervised image includes: Event interception based on sampling time; Calculating the cumulative amount of events pixel by pixel for the intercepted time to obtain event amount data; Using a downsampling method to process the event volume data size and filter out noise data in the event volume data; The accumulated value of each pixel is mapped to the grayscale value of the binary image, and a single-channel binary self-supervised image with the same size as the encoded data is output.

6. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for data processing and motion estimation based on an event camera as claimed in any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for data processing and motion estimation based on an event camera as described in any one of claims 1 to 4 is implemented.