High-frequency optical flow estimation method and device, electronic equipment and storage medium

Abstract

The application relates to the technical field of computer vision and neuromorphic computing, in particular to a high-frequency optical flow estimation method and device, electronic equipment and a storage medium, wherein the method comprises the following steps: acquiring event stream data of high-frequency optical flow; generating an event tensor sequence according to the event stream data; inputting the event tensor sequence into a pre-constructed prediction model to output a prediction candidate result and a distribution parameter prediction result of the high-frequency optical flow; and aggregating the prediction candidate result of the high-frequency optical flow according to the distribution parameter prediction result to obtain an estimation result of the high-frequency optical flow. Therefore, the problems that in the related art, it is difficult to obtain the high-frequency optical flow estimation result from the event stream data, the accuracy of the uncertainty evaluation of the optical flow estimation result is low, the denoising effect of the event stream data is poor, the stability of the generated optical flow map video is poor, and the actual use needs cannot be met are solved.

Description

Technical Field

[0001] This application relates to the fields of computer vision and neuromorphic computing technology, and in particular to a high-frequency optical flow estimation method, apparatus, electronic device and storage medium. Background Technology

[0002] An event camera is a bio-inspired sensor that works by outputting data if and only if the intensity of the light branch changes; the output data is called an event stream. Optical flow estimation is a fundamental visual task of event cameras.

[0003] In related technologies, for optical flow estimation, a method based on a handcrafted model can be used, which establishes a matching relationship between event signals and pixel motion, and solves the problem using hyperparameters and optimization tools; another method based on neural networks, inspired by RGB (Red-Green-Blue color mode) visual algorithms, can solve some visual problems in the field of event cameras to a certain extent.

[0004] However, the application of event cameras in the field of vision is still immature, and the stability of event stream data is far inferior to that of RGB images. At the same time, the uncertainty estimation of deep neural network models is more complex in terms of uncertainty modeling and optimization, making it difficult to guarantee the diversity of building ensemble models. Summary of the Invention

[0005] This application provides a high-frequency optical flow estimation method, apparatus, electronic device, and storage medium to solve the problems in related technologies, such as the high difficulty in obtaining high-frequency optical flow estimation results from event stream data, the low accuracy in assessing the uncertainty of optical flow estimation results, the poor denoising effect on event stream data, resulting in poor stability of the generated optical flow map video and failing to meet practical needs.

[0006] The first aspect of this application provides a high-frequency optical flow estimation method, comprising the following steps: acquiring event stream data of high-frequency optical flow; generating an event tensor sequence based on the event stream data; inputting the event tensor sequence into a pre-built prediction model; and outputting prediction candidate results and distribution parameter prediction results of the high-frequency optical flow; and aggregating the prediction candidate results of the high-frequency optical flow based on the distribution parameter prediction results to obtain the estimation result of the high-frequency optical flow.

[0007] Optionally, generating an event tensor sequence based on the event stream data includes: dividing the event stream data into data packets with the same time interval; accumulating the data packets in the spatial domain and stacking them into the event tensor sequence.

[0008] Optionally, the expression for the data packet is:

[0009] E k ={e i |wheret k-1 ≤t i ≤t k},

[0010] Among them, E k For video frame group I k The corresponding event packet; e i This indicates that the sequence number is i and the timestamp is t in the event stream. i Each event e is a single event, and each event e has corresponding x, y, p, t attributes; the timestamp of the time interval is t. k-1 and t k ;

[0011] The expression for the event tensor in the event tensor sequence is:

[0012] T k = stack(R k ),

[0013] Among them, T k R represents the event tensors in the event tensor sequence, where stack represents the stacking operation; k Let R be an initial tensor of size (H, W, B), where H and W correspond to the spatial resolution of the event camera, B is a discretization hyperparameter, and the initial tensor R is... k The value at (x, y, t) is:

[0014]

[0015] k b (a) = max(0, 1 - |a|),

[0016]

[0017] Where N is the total number of video frames in the event stream data, x i and y i For the data packet E k Event e i The two-dimensional spatial coordinates of the frame, t i and p i The data packet E is respectively k Event e i Frame timestamps and polarity, k b (a) indicates bilinear sampling of the spatiotemporal neighborhood.

[0018] Optionally, the prediction model includes multiple encoders, an optical flow decoder, and a distributed parameter decoder. The event tensors in the event tensor sequence are grouped and input into multiple encoders for forward propagation to obtain multiple sets of feature vectors. The multiple sets of feature vectors are respectively sent to the corresponding optical flow decoder and distributed parameter decoder for decoding to generate multiple optical flow prediction candidate results and distributed parameter prediction results.

[0019] Optionally, the encoder includes multiple randomly initialized encoder networks, each encoder network including an input convolutional layer, a hidden convolutional layer, and an output convolutional layer; the optical flow decoder and the distributed parameter decoder both include an input layer, a hidden convolutional layer, an upsampling layer, and an output convolutional layer, wherein a global average pooling layer is provided before the output layer of the distributed parameter decoder.

[0020] Optionally, the aggregation formula for the estimation results is:

[0021]

[0022]

[0023]

[0024]

[0025]

[0026]

[0027] In neural network prediction models, we have:

[0028] u i =μ u,i ,

[0029] v i =μ v,i ,

[0030] Among them, u fin and v fin The results are for optical flow estimation, where N is a hyperparameter and w u,i ,w v,i u is the normalized weight for the candidate optical flow predictions in the x and y directions. i and v i These are the predicted optical flow values ​​for each candidate optical flow in the x and y directions, respectively. These are the predicted variances of the candidate optical flow in the x and y directions, respectively, μ. u,i μ v,i μ is the predicted standard deviation of the optical flow of the candidate. u and μ vThese are the mean predicted values ​​of the average distribution of all predicted candidate optical flows in the x and y directions, respectively, μ. u,i and μ v,i These are the predicted mean values ​​of the distribution of each predicted candidate optical flow in the x and y directions, respectively.

[0031] Optionally, after aggregating the prediction candidate results of the high-frequency optical flow based on the prediction results of the distribution parameters to obtain the estimation result of the high-frequency optical flow, the method further includes: generating an optical flow image within a corresponding time interval based on the estimation result; calculating the uncertainty assessment result of the optical flow image based on the estimation result; and constructing an optical flow image sequence based on the optical flow image and the uncertainty assessment result; wherein,

[0032] The formula for calculating the uncertainty assessment result is as follows:

[0033]

[0034]

[0035] Among them, g u and g v For the uncertainty assessment results, and These are the variance prediction values ​​for each predicted candidate optical flow in the two x and y directions, respectively.

[0036] A second aspect of this application provides a high-frequency optical flow estimation device, comprising: an acquisition module for acquiring event stream data of high-frequency optical flow; a calculation module for generating an event tensor sequence based on the event stream data, inputting the event tensor sequence into a pre-built prediction model, and outputting prediction candidate results and distribution parameter prediction results of the high-frequency optical flow; and an aggregation module for aggregating the prediction candidate results of the high-frequency optical flow based on the distribution parameter prediction results to obtain an estimation result of the high-frequency optical flow.

[0037] Optionally, the computing module is further configured to: divide the event stream data into data packets with the same time interval; accumulate the data packets in the spatial domain and stack them into the event tensor sequence.

[0038] Optionally, the expression for the data packet is:

[0039] E k ={e i |wheret k-1 ≤t i ≤t k},

[0040] Among them, E k For video frame group I k The corresponding event packet; ei This indicates that the sequence number is i and the timestamp is t in the event stream. i Each event e is a single event, and each event e has corresponding x, y, p, t attributes; the timestamp of the time interval is t. k-1 and t k ;

[0041] The expression for the event tensor in the event tensor sequence is:

[0042] T k = stack(R) k ),

[0043] Among them, T k R represents the event tensors in the event tensor sequence, where stack represents the stacking operation; k Let R be an initial tensor of size (H, W, B), where H and W correspond to the spatial resolution of the event camera, B is a discretization hyperparameter, and the initial tensor R is... k The value at (x, y, t) is:

[0044]

[0045] k b (a) = max(0, 1 - |a|),

[0046]

[0047] Where N is the total number of video frames in the event stream data, x i and y i For the data packet E k Event e i The two-dimensional spatial coordinates of the frame, t i and p i The data packet E is respectively k Event e i Frame timestamps and polarity, k b (a) indicates bilinear sampling of the spatiotemporal neighborhood.

[0048] Optionally, the prediction model includes multiple encoders, an optical flow decoder, and a distributed parameter decoder. The event tensors in the event tensor sequence are grouped and input into multiple encoders for forward propagation to obtain multiple sets of feature vectors. The multiple sets of feature vectors are respectively sent to the corresponding optical flow decoder and distributed parameter decoder for decoding to generate multiple optical flow prediction candidate results and distributed parameter prediction results.

[0049] Optionally, the encoder includes multiple randomly initialized encoder networks, each encoder network including an input convolutional layer, a hidden convolutional layer, and an output convolutional layer; the optical flow decoder and the distributed parameter decoder both include an input layer, a hidden convolutional layer, an upsampling layer, and an output convolutional layer, wherein a global average pooling layer is provided before the output layer of the distributed parameter decoder.

[0050] Optionally, the aggregation formula for the estimation results is:

[0051]

[0052]

[0053]

[0054]

[0055]

[0056]

[0057] In neural network prediction models, we have:

[0058] u i =μ u,i ,

[0059] v i =μ v,i ,

[0060] Among them, u fin and v fin The results are for optical flow estimation, where N is a hyperparameter and w u,i ,w v,i u is the normalized weight for the candidate optical flow predictions in the x and y directions. i and v i These are the predicted optical flow values ​​for each candidate optical flow in the x and y directions, respectively. These are the predicted variances of the candidate optical flow in the x and y directions, respectively, σ u,i , σ v,i μ is the predicted standard deviation of the optical flow of the candidate. u and μ v These are the mean predicted values ​​of the average distribution of all predicted candidate optical flows in the x and y directions, respectively, μ. u,i and μ v,i These are the predicted mean values ​​of the distribution of each predicted candidate optical flow in the x and y directions, respectively.

[0061] Optionally, the aggregation module is further configured to: generate optical flow images within the corresponding time interval based on the estimation results; calculate the uncertainty assessment results of the optical flow images based on the estimation results; and construct an optical flow image sequence based on the optical flow images and the uncertainty assessment results; wherein,

[0062] The formula for calculating the uncertainty assessment result is as follows:

[0063]

[0064]

[0065] Among them, g u and g v For the uncertainty assessment results, and These are the variance prediction values ​​for each predicted candidate optical flow in the two x and y directions, respectively.

[0066] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the high-frequency optical flow estimation method as described in the above embodiments.

[0067] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the high-frequency optical flow estimation method as described in the above embodiments.

[0068] Therefore, this application has at least the following beneficial effects:

[0069] This application embodiment can acquire high-frequency optical flow estimation results through model prediction and aggregation operations in the field of event cameras, and can perform uncertainty assessment on the optical flow estimation results. This can better denoise event stream data, improve the accuracy and reliability of high-frequency optical flow estimation results, improve the stability and robustness of optical flow map video generation effects, and meet practical needs.

[0070] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0071] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0072] Figure 1 This is a flowchart of the high-frequency optical flow estimation method according to an embodiment of this application;

[0073] Figure 2 This is a flowchart of a high-frequency optical flow estimation method according to an embodiment of this application;

[0074] Figure 3 This is a schematic diagram of streaming data visualization according to an embodiment of this application;

[0075] Figure 4 This is a schematic diagram of the optical flow image and uncertainty evaluation results according to an embodiment of this application;

[0076] Figure 5 This is an example diagram of a high-frequency optical flow estimation device according to an embodiment of this application;

[0077] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0078] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0079] Event cameras are bio-inspired sensors that operate on a significantly different principle from traditional cameras. Unlike traditional cameras that acquire absolute light intensity at a fixed frame rate, event cameras output data only when the light intensity of a branch changes. This output data is called an event stream. Therefore, event cameras offer advantages such as high dynamic range, high temporal resolution, and no dynamic blur. Event cameras can be applied to optical flow estimation, a fundamental visual task. Exploring methods for evaluating uncertainties in neural network-based algorithms is crucial for researching event camera-based optical flow estimation and other visual tasks, as well as for model reliability evaluation methods.

[0080] In the field of optical flow estimation based on event cameras, a method based on handcrafted models can be used to establish a matching relationship between event signals and pixel motion, and to solve the problem using hyperparameters and optimization tools. Inspired by traditional RGB vision algorithms, neural network-based methods can solve some vision problems in the field of event cameras in a similar way.

[0081] However, the two types of cameras mentioned above have significant differences in characteristics and main application branches. In the field of event cameras, neural network-based methods are still under exploration, and one of the unexplored problems is the uncertainty assessment problem.

[0082] The application of event cameras in the field of vision is still immature. In particular, some neural network algorithms borrowed from the field of RGB image vision have shown good performance in some scenarios. However, since the stability of event stream data is far less than that of RGB images, how to evaluate the uncertainty of the results is an important issue in terms of the reliability of its application. At the same time, the uncertainty estimation of deep neural network models in the field of RGB image vision often relies on model ensembles based on complex prior knowledge or neural networks. However, the complexity of uncertainty problem modeling and optimization makes it difficult to guarantee the diversity of ensemble model construction.

[0083] To address the aforementioned issues, in the field of event cameras, due to the modal differences in the input signals, combinations of events of different lengths within the same time window are naturally suitable for generating multiple ensemble models and for evaluating the uncertainty of prediction results.

[0084] Therefore, in view of the problems in the above-mentioned background technology, such as the high difficulty in obtaining high-frequency optical flow estimation results from event stream data, the low accuracy in assessing the uncertainty of optical flow estimation results, the poor effect of denoising event stream data, resulting in poor stability of the generated optical flow map video and failing to meet the needs of practical use, this application provides a high-frequency optical flow estimation method. The high-frequency optical flow estimation method, apparatus, electronic device and storage medium of this application embodiment are described below with reference to the accompanying drawings.

[0085] Specifically, Figure 1 This is a flowchart illustrating a high-frequency optical flow estimation method provided in an embodiment of this application.

[0086] like Figure 1 As shown, this high-frequency optical flow estimation method includes the following steps:

[0087] In step S101, event stream data of high-frequency optical flow is acquired.

[0088] It is understandable that, such as Figure 2 As shown, in this embodiment, high-frequency optical flow event stream data can be obtained first, so that subsequent embodiments can use the event stream data to estimate the high-frequency optical flow; wherein, in this embodiment, at least one method can be used to obtain the high-frequency optical flow event stream data, and no specific limitation is made thereto.

[0089] In step S102, an event tensor sequence is generated based on the event stream data. The event tensor sequence is then input into a pre-built prediction model, and the prediction candidate results for high-frequency optical flow and the prediction results for distribution parameters are output.

[0090] It is understood that, in the embodiments of this application, after obtaining the event stream data in step S101 above, the event tensor sequence can be generated using the data, and the prediction candidate results and distribution parameter prediction results of high-frequency optical flow can be output through a pre-built model.

[0091] In this embodiment of the application, generating an event tensor sequence based on event stream data includes: dividing the event stream data into data packets with the same time interval; accumulating the data packets in the spatial domain and stacking them into an event tensor sequence.

[0092] The time interval can be set according to the actual situation, and there are no specific limitations on it.

[0093] It is understood that the embodiments of this application can perform data processing on the obtained event stream data, divide the event stream data into data packets with the same time interval according to any time interval, and then accumulate the data packets in the spatial domain to obtain the event tensor sequence; the specific process can be as follows:

[0094] In this embodiment of the application, the expression for the data packet is:

[0095] E k ={e i |wheret k-1 ≤t i ≤t k},

[0096] Among them, E k For video frame group I k The corresponding event packet; e i This indicates that the sequence number is i and the timestamp is t in the event stream. i Each event e is a single event, and each event e has corresponding x, y, p, t attributes; the timestamp of the time interval is t. k-1 and t k The expression for the event tensor in the event tensor sequence is:

[0097] T k = stack(R) k ),

[0098] Among them, T k R represents the event tensors in the event tensor sequence, where stack represents the stacking operation; k Let R be an initial tensor of size (H, W, B), where H and W correspond to the spatial resolution of the event camera, B is a discretization hyperparameter, and the initial tensor R is... k The value at (x, y, t) is:

[0099]

[0100] kb (a) = max(0, 1 - |a|),

[0101]

[0102] Where N is the total number of video frames in the event stream data, x i and y i For data packet E k Event e i The two-dimensional spatial coordinates of the frame, t i and p i Data packet E k Event e i Frame timestamps and polarity, k b (a) indicates bilinear sampling of the spatiotemporal neighborhood.

[0103] The stack() operation can be used to stack packets. The stack() operation can stack R k Perform descending stacking along dimension B: For the tensor (H,W,1) when B=m, sum the (Bm) unstacked tensors (B>m) on top of it to obtain T. k ,Right now:

[0104]

[0105] This completes data processing and stacking, resulting in an event tensor sequence. In this embodiment, the optical flow estimation map generated from a video frame (for illustrative purposes only) and event stream packets at a certain moment can be as follows: Figure 3 As shown.

[0106] In this embodiment, the prediction model includes multiple encoders, an optical flow decoder, and a distributed parameter decoder. The event tensors in the event tensor sequence are grouped and input into multiple encoders for forward propagation to obtain multiple sets of feature vectors. The multiple sets of feature vectors are respectively sent to the corresponding optical flow decoder and distributed parameter decoder for decoding to generate multiple optical flow prediction candidate results and distributed parameter prediction results.

[0107] The encoder contains multiple randomly initialized encoder networks, each of which contains an input convolutional layer, a hidden convolutional layer, and an output convolutional layer. The optical flow decoder and the distributed parameter decoder both contain an input layer, a hidden convolutional layer, an upsampling layer, and an output convolutional layer. The distributed parameter decoder also has a global average pooling layer before its output layer.

[0108] It is understood that the embodiments of this application can construct an integrated convolutional neural network. The encoder part of the integrated convolutional neural network includes N randomly initialized encoder networks, wherein each network includes an input convolutional layer, a hidden convolutional layer, and an output convolutional layer. Then, the event tensor is grouped and input into the multiple encoders of the integrated convolutional neural network for forward propagation to obtain multiple sets of feature vectors. The feature vectors are respectively fed into the corresponding optical flow decoder and distributed parameter decoder for decoding to generate multiple optical flow prediction candidate results and distributed parameter prediction results.

[0109] Specifically, in this embodiment of the application, the number of input channels of the input convolutional layer can be set to B, and the corresponding event tensor T can be set. k The input convolutional layer has a kernel size of 3*3, a stride of 1, and 16 output channels. The hidden convolutional layers also have 16 input channels, a kernel size of 3*3, a stride of 1, and 16 output channels. The output convolutional layer has 16 input channels, a kernel size of 3*3, a stride of 1, and 64 output channels. N is a hyperparameter. Furthermore, the encoder's input convolutional layers include random drop-out layers along the channel dimension, where each of the B channels has a probability p of being set to 0, where p is a hyperparameter.

[0110] The network decoder also contains N identical structures. The optical flow decoder includes an input layer, a hidden convolutional layer, an upsampling layer, and an output convolutional layer. In this embodiment, the number of input channels in the input layer can be set to 64, and the number of output channels in the output layer can be set to 2. Compared to the optical flow decoder, the distributed parameter decoder adds an additional global average pooling layer before the output layer, and has 4 output channels, which correspond to the mean and variance in the x and y directions of each preset Gaussian distribution sub-model in the ensemble model.

[0111] Therefore, in this embodiment of the application, the result of the event tensor can be input into the corresponding encoder to generate multiple optical flow prediction candidate results and distribution parameter prediction results.

[0112] In step S103, the prediction candidate results of high-frequency optical flow are aggregated based on the distribution parameter prediction results to obtain the estimation result of high-frequency optical flow.

[0113] It is understood that, in this embodiment of the application, candidate optical flow prediction results can be aggregated based on the distribution parameter prediction results to generate optical flow image calculations within the corresponding time interval; wherein, the final optical flow estimation result u in this embodiment of the application is... fin ,v fin The results are obtained by aggregating candidate results from optical flow prediction.

[0114] In this embodiment of the application, the aggregation formula for the estimation results is:

[0115]

[0116]

[0117]

[0118]

[0119]

[0120]

[0121] In neural network prediction models, we have:

[0122] u i =μ u,i ,

[0123] v i =μ v,i ,

[0124] Among them, u fin and v fin The results are for optical flow estimation, where N is a hyperparameter and w u,i ,w v,i u is the normalized weight for the candidate optical flow predictions in the x and y directions. i and v i These are the predicted optical flow values ​​for each candidate optical flow in the x and y directions, respectively. These are the predicted variances of the candidate optical flow in the x and y directions, respectively, σ u,i , σ v,i μ is the predicted standard deviation of the optical flow of the candidate. u and μ v These are the mean predicted values ​​of the average distribution of all predicted candidate optical flows in the x and y directions, respectively, μ. u,i and μ v,i These are the predicted mean values ​​of the distribution of each predicted candidate optical flow in the x and y directions, respectively.

[0125] It should be noted that, in the embodiments of this application, after obtaining the high-frequency optical flow estimation result, the uncertainty assessment result can also be calculated, and the visualization diagram of the calculation result can be shown as follows. Figure 4 As shown.

[0126] Specifically, after aggregating the predicted candidate results of high-frequency optical flow based on the distribution parameter prediction results to obtain the estimated result of high-frequency optical flow, the method further includes: generating optical flow images within the corresponding time interval based on the estimation results; calculating the uncertainty assessment result of the optical flow images based on the estimation results; and constructing an optical flow image sequence based on the optical flow images and the uncertainty assessment result; wherein,

[0127] The formula for calculating the uncertainty assessment result is as follows:

[0128]

[0129]

[0130] Among them, g u and g v For the uncertainty assessment results, and These are the variance prediction values ​​for each predicted candidate optical flow in the two x and y directions, respectively.

[0131] According to the high-frequency optical flow estimation method proposed in the embodiments of this application, the estimation results of high-frequency optical flow can be obtained in the field of event cameras through model prediction and aggregation, and the uncertainty assessment of the optical flow estimation results can be performed. This can better denoise the event stream data, improve the accuracy and reliability of the high-frequency optical flow estimation results, improve the stability and robustness of the optical flow map video generation effect, and meet the needs of practical use.

[0132] Next, the high-frequency optical flow estimation apparatus according to the embodiments of this application is described with reference to the accompanying drawings.

[0133] Figure 5 This is a block diagram of a high-frequency optical flow estimation device according to an embodiment of this application.

[0134] like Figure 5 As shown, the high-frequency optical flow estimation device 10 includes: an acquisition module 100, a calculation module 200, and an aggregation module 300.

[0135] The module 100 is used to acquire event stream data of high-frequency optical flow; the calculation module 200 is used to generate event tensor sequences based on the event stream data, input the event tensor sequences into a pre-built prediction model, and output the prediction candidate results and distribution parameter prediction results of high-frequency optical flow; the aggregation module 300 is used to aggregate the prediction candidate results of high-frequency optical flow based on the distribution parameter prediction results to obtain the estimation results of high-frequency optical flow.

[0136] In this embodiment of the application, the calculation module 200 is further configured to: divide the event stream data into data packets with the same time interval; accumulate the data packets in the spatial domain and stack them into an event tensor sequence.

[0137] In this embodiment of the application, the expression for the data packet is:

[0138] E k ={e i |wheret k-1 ≤t i ≤t k},

[0139] Among them, E k For video frame group I k The corresponding event packet; e i This indicates that the sequence number is i and the timestamp is t in the event stream. i Each event e is a single event, and each event e has corresponding x, y, p, t attributes; the timestamp of the time interval is t. k-1 and t k The expression for the event tensor in the event tensor sequence is:

[0140] T k = stack(R) k ),

[0141] Among them, T k R represents the event tensors in the event tensor sequence, where stack represents the stacking operation; k Let R be an initial tensor of size (H, W, B), where H and W correspond to the spatial resolution of the event camera, B is a discretization hyperparameter, and the initial tensor R is... k The value at (x, y, t) is:

[0142]

[0143] k b (a) = max(0, 1 - |a|),

[0144]

[0145] Where N is the total number of video frames in the event stream data, x i and y i For data packet E k Event e i The two-dimensional spatial coordinates of the frame, t i and p i Data packet E k Event e i Frame timestamps and polarity, k b (a) indicates bilinear sampling of the spatiotemporal neighborhood.

[0146] In this embodiment, the prediction model includes multiple encoders, an optical flow decoder, and a distributed parameter decoder. The event tensors in the event tensor sequence are grouped and input into multiple encoders for forward propagation to obtain multiple sets of feature vectors. The multiple sets of feature vectors are respectively sent to the corresponding optical flow decoder and distributed parameter decoder for decoding to generate multiple optical flow prediction candidate results and distributed parameter prediction results.

[0147] In this embodiment, the encoder includes multiple randomly initialized encoder networks, each encoder network including an input convolutional layer, a hidden convolutional layer, and an output convolutional layer; the optical flow decoder and the distributed parameter decoder both include an input layer, a hidden convolutional layer, an upsampling layer, and an output convolutional layer, wherein a global average pooling layer is provided before the output layer of the distributed parameter decoder.

[0148] In this embodiment of the application, the aggregation formula for the estimation results is:

[0149]

[0150]

[0151]

[0152]

[0153]

[0154]

[0155] In neural network prediction models, we have:

[0156] u i =μ u,i ,

[0157] v i =μ v,i ,

[0158] Among them, u fin and v fin The results are for optical flow estimation, where N is a hyperparameter and w u,i ,w v,i u is the normalized weight for the candidate optical flow predictions in the x and y directions. i and v i These are the predicted optical flow values ​​for each candidate optical flow in the x and y directions, respectively. These are the predicted variances of the candidate optical flow in the x and y directions, respectively, σ u,i , σ v,i μ is the predicted standard deviation of the optical flow of the candidate. u and μ v These are the mean predicted values ​​of the average distribution of all predicted candidate optical flows in the x and y directions, respectively, μ. u,i and μ v,i These are the predicted mean values ​​of the distribution of each predicted candidate optical flow in the x and y directions, respectively.

[0159] In this embodiment, the aggregation module 300 is further configured to: generate optical flow images within the corresponding time interval based on the estimation results; calculate the uncertainty assessment results of the optical flow images based on the estimation results; and construct an optical flow image sequence based on the optical flow images and the uncertainty assessment results; wherein,

[0160] The formula for calculating the uncertainty assessment result is as follows:

[0161]

[0162]

[0163] Among them, g u and g v For the uncertainty assessment results, and These are the variance prediction values ​​for each predicted candidate optical flow in the two x and y directions, respectively.

[0164] It should be noted that the foregoing explanation of the high-frequency optical flow estimation method embodiment also applies to the high-frequency optical flow estimation device of this embodiment, and will not be repeated here.

[0165] The high-frequency optical flow estimation device proposed in the embodiments of this application can obtain the estimation results of high-frequency optical flow in the field of event cameras through model prediction and aggregation, and can perform uncertainty assessment on the optical flow estimation results. This can better denoise the event stream data, improve the accuracy and reliability of the high-frequency optical flow estimation results, improve the stability and robustness of the optical flow map video generation effect, and meet the needs of practical use.

[0166] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0167] The memory 601, the processor 602, and the computer program stored on the memory 601 and capable of running on the processor 602.

[0168] When the processor 602 executes the program, it implements the high-frequency optical flow estimation method provided in the above embodiments.

[0169] Furthermore, electronic devices also include:

[0170] Communication interface 603 is used for communication between memory 601 and processor 602.

[0171] The memory 601 is used to store computer programs that can run on the processor 602.

[0172] The memory 601 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage.

[0173] If the memory 601, processor 602, and communication interface 603 are implemented independently, then the communication interface 603, memory 601, and processor 602 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0174] Optionally, in a specific implementation, if the memory 601, processor 602, and communication interface 603 are integrated on a single chip, then the memory 601, processor 602, and communication interface 603 can communicate with each other through an internal interface.

[0175] The processor 602 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of this application.

[0176] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the high-frequency optical flow estimation method described above.

[0177] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0178] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0179] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0180] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (FPGAs), field-programmable gate arrays (FPGAs), etc.

[0181] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0182] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A high-frequency optical flow estimation method, characterized in that, Includes the following steps: Acquire event stream data of high-frequency optical flow; An event tensor sequence is generated based on the event stream data. This event tensor sequence is then input into a pre-built prediction model, which outputs candidate prediction results for the high-frequency optical flow and predicted distribution parameters. The prediction model includes multiple encoders, an optical flow decoder, and a distribution parameter decoder. The event tensors in the event tensor sequence are grouped and input into multiple encoders for forward propagation to obtain multiple sets of feature vectors; the multiple sets of feature vectors are respectively fed into the corresponding optical flow decoder and distributed parameter decoder for decoding to generate multiple optical flow prediction candidate results and distributed parameter prediction results; The estimated result of the high-frequency optical flow is obtained by aggregating the prediction candidate results of the distribution parameter prediction results; The encoder comprises multiple randomly initialized encoder networks, each of which includes an input convolutional layer, a hidden convolutional layer, and an output convolutional layer. Let the number of input channels of the input convolutional layer be B. The input convolutional layer of the encoder includes a random drop-out layer in the channel dimension. Each input of the B channels has a probability p of being set to 0, where p is a hyperparameter. The aggregation formula for the estimation results is: ， ， ， ， ， ， In neural network prediction models, we have: ， ， in, and The results are for optical flow estimation, where N is a hyperparameter. The normalized weights for the candidate optical flow predictions in the x and y directions are... and These are the predicted optical flow values ​​for each candidate optical flow in the x and y directions, respectively. These are the predicted variance values ​​of each candidate optical flow in the x and y directions, respectively. , The standard deviation of the candidate optical flow is the predicted value. and These are the mean predicted values ​​of the average distribution of all predicted candidate optical flows in the x and y directions, respectively. and These are the predicted mean values ​​of the distribution of each predicted candidate optical flow in the x and y directions, respectively.

2. The high-frequency optical flow estimation method according to claim 1, characterized in that, The step of generating an event tensor sequence based on the event stream data includes: The event stream data is divided into data packets with the same time interval; The data packets are accumulated in the spatial domain and stacked into the event tensor sequence.

3. The high-frequency optical flow estimation method according to claim 2, characterized in that, The expression for the data packet is: ， Among them, E k For video frame group I k The corresponding event packet; e i This indicates that the sequence number is i and the timestamp is t in the event stream. i Each event e is a single event, and each event e has corresponding x, y, p, t attributes; the timestamp of the time interval is r. k-1 and r k ; The expression for the event tensor in the event tensor sequence is: ， in, Let `stack` be the event tensor in the event tensor sequence, and `stack` be the stack operation. The size is The initial tensor, where H and W correspond to the spatial resolution of the event camera. B For discretization of hyperparameters, the initial tensor The value at (x, y, t) is: ， ， ， in, x i and y i For the data packet E k Event e i The two-dimensional spatial coordinates of the frame t i and p i The data packet E is respectively k Event e i Frame timestamps and polarity This indicates bilinear sampling of the spatiotemporal neighborhood.

4. The high-frequency optical flow estimation method according to claim 1, characterized in that, Both the optical flow decoder and the distributed parameter decoder include an input layer, a hidden convolutional layer, an upsampling layer, and an output convolutional layer. The distributed parameter decoder also has a global average pooling layer before its output layer.

5. The high-frequency optical flow estimation method according to claim 1, characterized in that, After aggregating the prediction candidate results of the high-frequency optical flow based on the prediction results of the distribution parameters to obtain the estimation result of the high-frequency optical flow, the method further includes: Generate optical flow images within the corresponding time intervals based on the estimation results; The uncertainty assessment result of the optical flow image is calculated based on the estimation result, and an optical flow image sequence is constructed based on the optical flow image and the uncertainty assessment result; wherein, The formula for calculating the uncertainty assessment result is as follows: ， ， in, and For the uncertainty assessment results, and These are the variance prediction values ​​for each predicted candidate optical flow in the two x and y directions, respectively.

6. A high-frequency optical flow estimation device, characterized in that, include: The acquisition module is used to acquire event stream data of high-frequency optical flow; The computation module is used to generate an event tensor sequence based on the event stream data, input the event tensor sequence into a pre-built prediction model, and output the prediction candidate results and distribution parameter prediction results of the high-frequency optical flow. The prediction model includes multiple encoders, an optical flow decoder, and a distribution parameter decoder. The event tensors in the event tensor sequence are grouped and input into multiple encoders for forward propagation to obtain multiple sets of feature vectors; the multiple sets of feature vectors are respectively fed into the corresponding optical flow decoder and distributed parameter decoder for decoding to generate multiple optical flow prediction candidate results and distributed parameter prediction results; An aggregation module is used to aggregate the prediction candidate results of the high-frequency optical flow based on the prediction results of the distribution parameters to obtain the estimation result of the high-frequency optical flow; The encoder comprises multiple randomly initialized encoder networks, each of which includes an input convolutional layer, a hidden convolutional layer, and an output convolutional layer. Let the number of input channels of the input convolutional layer be B. The input convolutional layer of the encoder includes a random drop-out layer in the channel dimension. Each input of the B channels has a probability p of being set to 0, where p is a hyperparameter. The aggregation formula for the estimation results is: ， ， ， ， ， ， In neural network prediction models, we have: ， ， in, and The results are for optical flow estimation, where N is a hyperparameter. The normalized weights for the candidate optical flow predictions in the x and y directions are... and These are the predicted optical flow values ​​for each candidate optical flow in the x and y directions, respectively. These are the predicted variance values ​​of each candidate optical flow in the x and y directions, respectively. , The standard deviation of the candidate optical flow is the predicted value. and These are the mean predicted values ​​of the average distribution of all predicted candidate optical flows in the x and y directions, respectively. and These are the predicted mean values ​​of the distribution of each predicted candidate optical flow in the x and y directions, respectively.

7. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the high-frequency optical flow estimation method as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the high-frequency optical flow estimation method as described in any one of claims 1-5.

Images