Biomimetic visual sensor optical flow prediction method based on hybrid neural network

Abstract

The application relates to the field of image processing and discloses a hybrid neural network-based light flow prediction method for a biomimetic visual sensor, which solves the problems of existing light flow prediction methods, such as the existence of smearing or target loss, the influence on light flow estimation accuracy, the increase in calculation amount, the loss of the low data amount advantage of an event camera and the like, and the method comprises the following steps: firstly, acquiring a public data set on a network and adaptively slicing spatio-temporal data flow; then, constructing a hybrid neural network for light flow prediction of the biomimetic visual sensor, so that the network can perform light flow prediction; then, designing a loss function for overall network training, performing supervised training on the network, and obtaining a supervised light flow estimation model with faster speed and higher accuracy; and finally, using the trained model to perform light flow prediction on the spatio-temporal data flow of the biomimetic visual sensor. The light flow prediction method improves the light flow prediction accuracy and the accuracy of predicted light flow.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more specifically to a method for predicting optical flow in a biomimetic visual sensor based on a hybrid neural network. Background Technology

[0002] Due to the discrete nature of spatiotemporal data in bionic vision sensors, we cannot directly use traditional convolutional neural networks (CNNs) for optical flow prediction. Therefore, previous methods for optical flow prediction in bionic vision sensors mostly involved slicing the spatiotemporal data stream into images with a fixed number of events or fixed time intervals, then converting these images into frames, and finally feeding these frames into a traditional CNN for training. However, this method of slicing into images with a fixed number of events or fixed time intervals results in target blurring or information loss in the images. Furthermore, the network performs calculations on every pixel in the image frame, significantly increasing the computational load and negating the advantage of low data volume in bionic vision sensors.

[0003] Currently, the mainstream imaging devices are CCD and CMOS image sensors, which output images in a frame-by-frame imaging mode. The output images are relatively intuitive and pleasing to the human eye. However, the frame-based transmission method images every frame in a scene, resulting in a large amount of redundant data, which is not conducive to real-time image processing. Moreover, the frame rate limitation causes the loss of motion information of high-speed moving targets during frame intervals, making it unsuitable for the field of machine vision. Therefore, bionic vision sensors have emerged. Due to its special pixel structure, this sensor only images where the light intensity changes. It generates events (e = {x, y, t, p}) based on the light intensity changes of the pixels, and outputs an event stream. Each event (e = {x, y, t, p}) contains position information (x, y), polarity information (light intensity increases or decreases) (p), and time information (t). Therefore, the output data of this bionic vision sensor is a spatiotemporal data stream composed of discrete events.

[0004] Due to its unique data format, existing image processing algorithms are not suitable for bionic vision sensors. To promote the application of bionic vision sensors in machine vision, some researchers have proposed optical flow estimation based on bionic vision sensors.

[0005] Existing patent 202011408555.1 proposes an end-to-end unsupervised optical flow estimation method based on an event camera. Utilizing the event stream output from the event camera, the raw data is first preprocessed to convert four-dimensional data into three-dimensional data. Then, each sample is divided into multiple subsequences, and each subsequence is processed individually using ConvLSTM. After all subsequences are processed, they are concatenated by channel to form the final three-dimensional data fed into the optical flow prediction network. An optical flow prediction network similar to an encoder / decoder is employed. Photometric error loss is designed using the two consecutive grayscale frames of the event stream data output at a fixed frequency from the event camera, along with a smoothness loss, which together serve as the unsupervised loss, enabling the network to ultimately estimate the optical flow.

[0006] The methods mentioned above, while being end-to-end unsupervised optical flow estimation methods, involve segmenting the spatiotemporal data stream into fixed event segments during the data preprocessing stage. Improper segmentation can lead to motion blur or target loss, thus affecting the accuracy of optical flow prediction. Furthermore, these methods still construct image frames from events and feed them into the network for computation, significantly increasing the computational load and negating the advantage of low data volume from event cameras. Summary of the Invention

[0007] To address the problems of existing optical flow prediction methods, such as motion blur or target loss affecting the accuracy of optical flow prediction and increasing computational load, thus losing the advantage of low data volume of event cameras, this invention provides a biomimetic visual sensor optical flow prediction method based on hybrid neural networks.

[0008] A biomimetic vision sensor optical flow prediction method based on hybrid neural networks is implemented through the following steps:

[0009] Step 1: Obtain the spatiotemporal data stream and labeled optical flow images of the bionic vision sensor in the public dataset; and perform adaptive slicing on the spatiotemporal data stream;

[0010] Step 2: Construct a hybrid neural network for optical flow prediction in biomimetic vision sensors, enabling the network to perform optical flow prediction;

[0011] Step 3: Design the loss function for overall network training, perform supervised training on the network, and obtain a supervised optical flow estimation model;

[0012] Step 4: Use the supervised optical flow estimation model to predict the optical flow of the spatiotemporal data stream of the bionic vision sensor to obtain the predicted optical flow image.

[0013] The beneficial effects of this invention are:

[0014] The optical flow prediction method proposed in this invention performs adaptive segmentation based on the target motion state in the scene during the spatiotemporal data stream preprocessing stage, so that there is neither target trailing nor information loss in the spatiotemporal data stream slices entering the network, thereby improving the accuracy of optical flow prediction.

[0015] The network model proposed in this invention can directly receive discrete spatiotemporal data streams, eliminating the need to construct image frames from the spatiotemporal data streams. This greatly reduces the amount of data, improves the computation speed, and retains the advantage of low data volume of bionic vision sensors.

[0016] This invention utilizes the amplitude and angle differences between labeled and predicted optical flow to construct a loss function, and performs supervised training on the network to improve the accuracy of optical flow prediction. Attached Figure Description

[0017] Figure 1 This is a flowchart of the biomimetic visual sensor optical flow prediction method based on hybrid neural networks described in this invention;

[0018] Figure 2 This is a schematic diagram of the hybrid neural network model used in this invention. Detailed Implementation

[0019] Combination Figure 1 and Figure 2 This embodiment describes a biomimetic vision sensor optical flow prediction method based on a hybrid neural network, which is implemented through the following steps:

[0020] Step 1: Obtain the spatiotemporal data stream and labeled optical flow images of the bionic vision sensor in the public dataset; and perform adaptive slicing on the spatiotemporal data stream;

[0021] In this embodiment, a publicly available dataset is first acquired online, from which spatiotemporal data stream data and labeled optical flow images of a bionic vision sensor are obtained. An adaptive slicing method based on the spatiotemporal data stream of a bionic vision sensor (patent number 2021110973961.0) is used to adaptively slice the spatiotemporal data stream, limiting its spatial resolution to 256×256. The sliced ​​data is then fed discretely into a trained hybrid neural network for optical flow prediction. This ensures that the spatiotemporal data stream slices fed into the network have neither missing information nor target ghosting. Compared with existing bionic vision sensor optical flow prediction methods, this method has the advantages of high prediction accuracy and low computational cost.

[0022] Step 2: Construct a hybrid neural network for optical flow prediction in biomimetic vision sensors, enabling the network to perform optical flow prediction;

[0023] In this embodiment, the hybrid neural network structure adopts a multi-scale feature extraction approach. Through multi-scale feature extraction, more detailed information is obtained at high scales, and more contour information is obtained at low scales, thereby making the optical flow information predicted by the network more accurate. This hybrid neural network consists of an SNN downsampling pulse feature encoding module, a pulse feature synchronization module, a Transformer global feature extraction module, a CNN upsampling decoding module, and an optical flow information output layer. The overall network structure is as follows: Figure 2 As shown.

[0024] The SNN downsampling pulse feature encoding module is mainly used to extract pulse sequence feature maps at different scales. It consists of a three-layer SNN network, each layer containing several neurons based on the Leaky integrate-and-fire (LIF) model to perform downsampling and feature encoding. The three-layer SNN network outputs 64, 128, and 256 feature channels, respectively. The three-layer network is defined as follows:

[0025] The first layer defines the input as a single-channel spatial resolution of 256×256, the spatiotemporal data stream as discrete in the time domain, and the output as a 64-channel encoded pulse sequence feature map with a spatial resolution of 128×128.

[0026] The second layer is defined as a 64-channel pulse sequence feature map with a spatial resolution of 128×128, and an encoded 128-channel pulse sequence feature map with a spatial resolution of 64×64.

[0027] The third layer is defined as an input of 128 channels with a spatial resolution of 64×64 pulse sequence feature maps, and an output of 256 channels with a spatial resolution of 32×32 pulse sequence feature maps.

[0028] The pulse feature synchronization module is used to convert the pulse feature sequence output by the SNN downsampling pulse feature encoding module into a two-dimensional feature image, which is then fused with the two-dimensional feature map output by the CNN decoding module.

[0029] The Transformer global feature extraction module is a commonly used neural network Transformer module. Here, it is innovatively introduced into the optical flow prediction network of the biomimetic vision sensor. It mainly extracts global depth features to obtain the contour information of the spatiotemporal data stream.

[0030] The CNN upsampling module is mainly used for upsampling two-dimensional feature maps. It consists of three CNN layers, each containing a deconvolution module for upsampling. Each layer is defined as follows:

[0031] The first layer takes a 512-channel 2D feature map with a spatial resolution of 32x32 as input. After upsampling, the deconvolution module outputs a decoded 128-channel 2D feature map with a spatial resolution of 64x64.

[0032] The second layer defines a 256-channel 2D feature map with a spatial resolution of 64x64 as input. After upsampling, the deconvolution module outputs a decoded 64-channel 2D feature map with a spatial resolution of 128x128.

[0033] The third layer defines a 128-channel 2D feature map with a spatial resolution of 128x128. After upsampling, the deconvolution module produces a decoded 2D feature map with a spatial resolution of 256x256 for two channels.

[0034] The optical flow information output layer outputs the optical flow information predicted by the network, taking a two-dimensional feature map with a spatial resolution of 256x256 as input. This image is then passed through a convolutional layer with a kernel size of 3x3 and a stride of 1 to obtain the predicted optical flow information.

[0035] In this embodiment, the hybrid neural network employs skip connections. The output of the first layer of the SNN downsampling pulse feature encoding module is connected to the input of the third layer of the CNN upsampling decoding module after passing through the pulse feature synchronization module. The output of the second layer of the SNN downsampling pulse encoding module is connected to the input of the second layer of the CNN upsampling decoding module after passing through the pulse feature synchronization module. The output of the third layer of the SNN downsampling pulse encoding module is connected to the input of the first layer of the CNN upsampling decoding module after passing through the pulse feature synchronization module. This embodiment enables the network to receive discrete spatiotemporal data streams, reducing computational load while maintaining the advantage of low data volume from the event camera.

[0036] Step 3: Design the loss function for overall network training, perform supervised training on the network, and obtain a supervised optical flow estimation model;

[0037] The amplitude and angle differences between the labeled optical flow image and the predicted optical flow image from step one are used as loss functions to perform supervised training on the optical flow prediction network, thereby improving the accuracy of optical flow prediction. The calculation formula is shown below.

[0038]

[0039] Where x is the horizontal movement speed of the pixel, and y is the vertical movement speed of the pixel. The optical flow from the marked i-th event slice to the (i+1)-th event slice, Let λ be the optical flow predicted by the optical flow prediction network from the i-th event slice to the (i+1)-th event slice. λ is a weighting factor used to adjust the influence of amplitude difference and angle difference on the loss function.

[0040] Step 4: Use the supervised optical flow estimation model obtained in Step 3 to predict the optical flow of the spatiotemporal data stream of the bionic vision sensor and obtain the predicted optical flow image.

[0041] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0042] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A biomimetic vision sensor optical flow prediction method based on hybrid neural networks, characterized by: This method is implemented by the following steps: Step 1: Obtain the spatiotemporal data stream and labeled optical flow images of the bionic vision sensor in the public dataset; and perform adaptive slicing on the spatiotemporal data stream; Step 2: Construct a hybrid neural network for optical flow prediction in biomimetic vision sensors, enabling the network to perform optical flow prediction; The hybrid neural network consists of an SNN downsampling pulse feature encoding module, a pulse feature synchronization module, a global feature extraction module, a CNN upsampling decoding module, and an optical flow information output layer. The SNN downsampling pulse feature encoding module is used to extract pulse sequence feature maps at different scales. It consists of a three-layer SNN network, each layer containing several neurons based on LIF model, used to complete downsampling and feature encoding. The pulse feature synchronization module is used to convert the feature pulse sequence output by the SNN downsampling pulse feature encoding module into a two-dimensional feature image, which is then fused with the two-dimensional feature map output by the CNN upsampling decoding module. The global feature extraction module extracts the feature sequence output by the third-layer SNN network to perform deep global feature extraction and obtain the contour information of the spatiotemporal data stream. The CNN upsampling decoding module is used to upsample the two-dimensional feature map output by the pulse feature synchronization module and the global feature extraction module; It consists of a three-layer CNN network, each of which includes a deconvolution module for upsampling; The optical flow information output layer is used to output the optical flow information predicted by the hybrid neural network. It takes a two-dimensional feature map with a spatial resolution of 256×256 as input and then passes it through a convolutional layer with a kernel of 3×3 and a stride of 1 to obtain the predicted optical flow information. Step 3: Design the loss function for overall network training, perform supervised training on the network, and obtain a supervised optical flow estimation model; Step 4: Use the supervised optical flow estimation model to predict the optical flow of the spatiotemporal data stream of the bionic vision sensor to obtain the predicted optical flow image.

2. The method for predicting optical flow in a biomimetic visual sensor based on a hybrid neural network according to claim 1, characterized in that: The number of output feature channels for the three-layer SNN networks are 64, 128, and 256, respectively; the three-layer networks are as follows: The first layer input is a single-channel spatial resolution of 256×256, which discretizes the spatiotemporal data stream in the time domain and outputs a 64-channel encoded pulse sequence feature map with a spatial resolution of 128×128. The second layer inputs a 64-channel pulse sequence feature map with a spatial resolution of 128×128, and outputs a 128-channel encoded pulse sequence feature map with a spatial resolution of 64×64. The third layer input has a spatial resolution of 64×64 pulse sequence feature maps with 128 channels, and the output has a spatial resolution of 32×32 pulse sequence feature maps with 256 channels after encoding.

3. The method for predicting optical flow in a biomimetic visual sensor based on a hybrid neural network according to claim 1, characterized in that: The three-layer CNN network structure is as follows: The first layer input is a 2D feature map with 512 channels and a spatial resolution of 32×32. After upsampling, the deconvolution module outputs a decoded 2D feature map with 128 channels and a spatial resolution of 64×64. The second layer takes a 256-channel two-dimensional feature map with a spatial resolution of 64x64 as input, and after upsampling, the deconvolution module outputs a decoded 64-channel two-dimensional feature map with a spatial resolution of 128x128. The third layer takes a 128-channel two-dimensional feature map with a spatial resolution of 128×128 as input. After upsampling, the deconvolution module produces a decoded two-channel two-dimensional feature map with a spatial resolution of 256×256.

4. The method for predicting optical flow in a biomimetic vision sensor based on a hybrid neural network according to claim 1, characterized in that: The hybrid neural network employs skip connections. The output of the first layer of the SNN downsampling pulse feature encoding module is connected to the input of the third layer of the CNN upsampling decoding module after passing through the pulse feature synchronization module. The output of the second layer of the SNN downsampling pulse encoding module is connected to the input of the second layer of the CNN upsampling decoding module after passing through the pulse feature synchronization module. The output of the third layer of the SNN downsampling pulse encoding module is connected to the input of the first layer of the CNN upsampling decoding module after passing through the pulse feature synchronization module.

5. The method for predicting optical flow in a biomimetic vision sensor based on a hybrid neural network according to claim 1, characterized in that: In step three, the loss function for the overall network training is designed as follows: The amplitude and angle differences between the labeled optical flow image and the predicted optical flow image from step one are used as the loss function to train the optical flow prediction network in a supervised manner. The calculation formula is as follows: ； In the formula, x is the horizontal movement speed of the pixel, and y is the vertical movement speed of the pixel; The optical flow from the marked i-th event slice to the (i+1)-th event slice, For the optical flow from the i-th event slice to the (i+1)-th event slice predicted by the hybrid neural network, This is a weighting factor used to adjust the influence of amplitude difference and angle difference on the loss function.

Images