Method and system, device, medium for computing optical flow based on event camera

Abstract

The application relates to the technical field of image processing and computer vision, and discloses a method for calculating optical flow based on an event camera, which comprises the following steps: constructing a training data set according to data collected by the event camera and an optical flow label, and constructing an optical flow calculation network model; converting an event sequence into a discrete event cumulative image by using an image, inputting the discrete event cumulative image into the optical flow calculation network model, and running the optical flow calculation network model; collecting two continuous event flow data pieces in the same time interval by using the event camera, then converting the two continuous event flow data pieces into discrete time cumulative images, inputting the discrete time cumulative images into the trained optical flow calculation network model, calculating the displacement of all pixels in a spatial image plane in the time interval as optical flow, and storing the optical flow. The system comprises a data acquisition module, an optical flow calculation module and a result storage module. The application also discloses an electronic device and a computer readable storage medium. The application can efficiently and quickly estimate the motion of each pixel point and output dense optical flow.

Description

Technical Field

[0001] This invention relates to the fields of image processing and computer vision, and in particular to a method, system, device, and medium for calculating optical flow based on an event camera. It fully utilizes the high temporal resolution and spatial sparsity of the event camera's data characteristics to efficiently and quickly estimate the motion of each pixel and output dense optical flow. Background Technology

[0002] Event cameras are a novel type of biomimetic vision sensor, whose working mechanism is more similar to that of the human eye, making them widely known. Unlike traditional frame-based cameras that capture images at a fixed frame rate, event cameras can asynchronously detect logarithmic changes in illumination intensity for each pixel and output event information including location, time, and polarity when the change exceeds a certain threshold. Event cameras boast advantages such as high temporal resolution (microsecond level), high dynamic range (140dB to 60dB), high pixel bandwidth (kilohertz level), and low power consumption. With these unique output methods and operating characteristics, they are particularly suitable for challenging environments with high-speed motion, significant changes in lighting conditions, or low energy consumption.

[0003] In recent years, optical flow estimation based on event cameras has attracted widespread attention. Early deep learning-based methods (such as cited document 1) represented event streams as static frames and used the encoder-decoder structure of U-Net to predict optical flow, which could only handle small-scale motion. Current mainstream deep learning-based methods obtain motion information by calculating the feature correlation between two consecutive event streams, and then estimate the optical flow. This approach can even estimate optical flow in regions without triggered events. However, current methods (such as cited documents 3, 4, and 5) improve upon the recursive refinement structure used in cited document 2, without considering the inference speed issue of this structure. This recursive refinement structure uses a recurrent neural network to iteratively output predicted values ​​and then uses these predicted values ​​to capture more accurate motion information, thereby continuously refining the predicted values. Compared to cited document 1, this significantly improves the accuracy of optical flow estimation, but it greatly increases the network inference time, failing to meet the speed requirements of downstream tasks such as video stabilization and multi-exposure enhancement. i3, 4, and 5 add features such as a global attention module and a density adjustment module to the network structure of cited document 2. While this can estimate optical flow more accurately, it also increases the inference time of the network.

[0004] Previous methods did not take into account the spatial sparsity of event data and did not distinguish between regions with and without event triggers. As a result, a large amount of computational cost was wasted in regions without event triggers when calculating optical flow, and even overfitting problems occurred.

[0005] In summary, this invention fully utilizes the high temporal resolution and spatial sparsity of event camera data, and incorporates a multi-scale global optimization scheme based on the encoder-decoder structure, providing a method, system, device, and medium for calculating optical flow based on event cameras, capable of generating full-resolution optical flow with minimal computational overhead.

[0006] Citation document 1: Zhu AZ, Yuan L, Chaney K, et al. Unsupervised event-based learning of optical flow, depth, and egomotion[C] / / Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition.2019:989-997.

[0007] Citation 2: Gehrig M, M,Gehrig D,et al.E-raft:Dense optical flow from event cameras[C] / / 2021International Conference on 3D Vision(3DV).IEEE,2021:197-206.

[0008] Citation document 3: Li Y, Huang Z, Chen S, et al. Blinkflow: A dataset to push the limits of event-based optical flow estimation[C] / / Proceedings of the IEEE / RSJInternational Conference on Intelligent Robots and Systems.2023.

[0009] Citation document 4: Luo X, Luo K, Luo A, et al. Learning Optical Flow from EventCamera with Rendered Dataset[C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision.2023:9847-9857.

[0010] Citation document 5: Liu H, Chen G, Qu S, et al.TMA: Temporal Motion Aggregation for Event-based Optical Flow[C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision.2023. Summary of the Invention

[0011] This invention provides a method, system, device, and medium for calculating optical flow based on an event camera. It fully utilizes the high temporal resolution and spatial sparsity of the event camera's data characteristics to efficiently and quickly estimate the motion of each pixel and output dense optical flow.

[0012] This invention is achieved through the following technical solution: a method for calculating optical flow based on an event camera, comprising the following steps:

[0013] Step S1: Construct a training dataset based on the data collected by the event camera and the optical flow labels. The training dataset includes the event sequence within the time interval and the displacement of all pixels in the spatial image plane of the event camera within the same time interval.

[0014] Step S2: Construct an optical flow computing network model, which includes a lightweight encoder, a hole feature correlation builder, and a refined decoder.

[0015] Step S3: Convert the event sequence into a discrete event cumulative image using an image and input it into the optical flow computing network model, then run the optical flow computing network model;

[0016] Step S4: Train the optical flow computing network model using the training dataset, use the L1 loss function to guide the optical flow computing model, pre-set the relevant hyperparameters of the optical flow computing network model, use the learning rate descent method to iteratively decay the loss value, optimize the weight parameters of the optical flow computing network model until the number of iterations equals the maximum number of iterations, stop training, and generate the trained optical flow computing network model.

[0017] Step S5: Use an event camera to collect two consecutive event stream data segments with the same time interval, then convert them into discrete-time cumulative images, input them into the trained optical flow calculation network model, calculate the displacement of all pixels in the spatial image plane within the time interval as optical flow, and store it.

[0018] To better implement the present invention, step S1 further includes:

[0019] The training dataset includes multiple data pairs, each data pair including a time interval t. k-1 →t k The sequence E of events recorded by the internal event camera k-1 The next time interval t k →t k+1 sequence of events E within k and time interval t k →t k+1 The displacement F of all pixels within the intra-event camera spatial image plane Ω k ;

[0020] Where k is the sequence number of the time series arranged in chronological order, the size of the event sequence E is N×4, and N is the length of the event sequence; the size of the pixel displacement F is H×W×2, where H and W are the width and height of the event camera spatial image plane Ω, respectively.

[0021] To better realize the present invention, the structure of the optical flow computing network model in step S2 further includes:

[0022] The lightweight encoder, the hole feature correlation builder, and the fine-grained decoder are connected sequentially from front to back;

[0023] The lightweight encoder includes a 6-layer pooling pyramid structure. The first three layers of the pooling pyramid structure contain two pooling convolutional layers, while the last three layers of the pooling pyramid structure contain only pooling layers. The feature resolution of each layer decreases layer by layer with a scale factor of 2.

[0024] The refined decoder includes two convolutional layers a, a detail supplementation module a, and two convolutional layers f connected sequentially from front to back. The two convolutional layers a and the two convolutional layers f have the same structure, and the detail supplementation module a and the detail supplementation module f have the same structure.

[0025] The pooling layer is connected to the hole feature correlation constructor, and the hole feature correlation constructor is connected to two convolutional layers a.

[0026] To better realize the present invention, the method of converting the event sequence into a discrete event cumulative image and inputting it into the optical flow calculation network model in step S3 further includes:

[0027] Record time interval t k-1 →t k sequence of events E within k The location (x) of each event in the image space e ,y e The triggering time t e and the polarity of brightness change p eThe four attributes ∈{-1,+1} will be used to determine the event sequence E. k Represented as:

[0028] E k ={(x e ,y e ,t e ,p e ) N},(x e ,y e )∈Ω,t e ∈[t k-1 ,t k ], p e ∈{-1,+1};

[0029] Using the discrete-time cumulative image representation method, the event sequence E in a data pair of the training dataset is represented. k-1 and E k Convert to V k-1 and V k , which serves as the input to the optical flow computing network.

[0030] To better realize the present invention, the method for running the optical flow computing network model in step S3 further includes:

[0031] The lightweight encoder accepts two network inputs V. k-1 and V k A pooling pyramid structure with shared weights was used to extract features at six scales from 1 / 2 full resolution to 1 / 64 full resolution. and Where i represents the layer number and N represents the total number of layers;

[0032] The feature and The input is fed into the hole feature correlation builder, which constructs the feature using inner product operations and methods to calculate hole feature correlation. and The correlation tensor between them {(C i ) N}={C 1 C 2 ,...,C 6};

[0033] The correlation tensor {(C i ) N}, the features and The data are then input into the refined decoder.

[0034] The correlation tensor {(C i ) N} Contains 6 sets of tensors {C 1 C 2 ,...,C 6 The features are sequentially input into two convolutional layers a and f according to their group. and The optical flow is divided into 6 pairs according to the group and sequentially input into the same detail supplementation module a and detail supplementation module f. The detail supplementation module a and detail supplementation module f upsample the optical flow.

[0035] The decoding process of the refined decoder is represented as follows:

[0036] Where C(·,·), W(·,·) and ↑ represent the correlation of hole features, optical flow distortion operation, and optical flow upsampling, respectively, and F i-1 F is the output optical flow of the (i-1)th layer decoder. i and Conv i It represents the output optical flow and convolution operation of the i-th layer decoder.

[0037] To better implement this invention, the optical flow distortion operation is further expressed as:

[0038]

[0039] k b (a) = max(0, 1-a);

[0040] Where p and d represent the i-th layer features before and after the optical flow warping operation, respectively. The coordinates of the pixel space image plane Ω, F i-1 It is the output optical flow of the (i-1)th layer decoder, ↑ indicates upsampling of the optical flow, k b (a) represents the bilinear interpolation operation, where a is the input for the interpolation operation.

[0041] To better realize the present invention, the method of converting the event sequence into a discrete event cumulative image and inputting it into the optical flow calculation network model in step S4 further includes:

[0042] The event sequence E k The sequence is divided into B small segments along the time dimension. The time dimension is scaled to the range [0, B-1] using bilinear interpolation. Then, the trigger times of events at the same spatial image location in each segment are accumulated and transformed into a spatial image V of size B×H×W. The calculation process is as follows:

[0043]

[0044] k b (a) = max(0, 1-a);

[0045]

[0046] Where x e ,y e ,t e and p e Represents the event sequence E k The four properties of any event e in the context of the event, It is t e Normalized to the range [0, B-1], N is the total number of events in the sequence, x and y represent the coordinates of the image plane of the discrete-time cumulative image V, b ∈ [0, B-1] represents the index of the discrete-time cumulative image V on the event axis, and k b (a) represents the bilinear interpolation operation, where a is the input for the interpolation operation;

[0047] t respectively k-1 →t k and t k →t k+1 Event sequence E within the time interval k and E k+1 Represented as a discrete-time cumulative image V k-1 and V k , which serves as the input to the optical flow computing network.

[0048] This invention also provides a system for calculating optical flow based on an event camera, comprising a data acquisition module, an optical flow calculation module, and a result storage module, wherein:

[0049] The data acquisition module is used to construct a training dataset based on the data acquired by the event camera and optical flow labels. The training dataset includes event sequences within time intervals and the displacements of all pixels in the spatial image plane of the event camera within the same time interval. It is used to acquire two consecutive event stream data segments with the same time interval using the event camera, then convert them into discrete-time cumulative images, and input them into the trained optical flow calculation network model.

[0050] An optical flow computation module is used to construct an optical flow computation network model, which includes a lightweight encoder, a hole feature correlation builder, and a refined decoder. It is used to: convert the event sequence into discrete event cumulative images and input them into the optical flow computation network model; run the optical flow computation network model; train the optical flow computation network model using the training dataset; guide the optical flow computation model using the L1 loss function; pre-set the relevant hyperparameters of the optical flow computation network model; iteratively decay the loss value using the learning rate descent method; optimize the weight parameters of the optical flow computation network model until the number of iterations equals the maximum number of iterations, then stop training and generate the trained optical flow computation network model; and calculate the displacement of all pixels in the spatial image plane within the same time interval as optical flow.

[0051] The result storage module is used to store the displacement of all pixels in the spatial image plane.

[0052] The present invention also provides an electronic device comprising a processor and a memory; the processor includes the system for calculating optical flow based on an event camera as described in the second aspect above.

[0053] The present invention also provides a computer-readable storage medium comprising instructions that, when executed on an electronic device described in the third aspect, cause the electronic device to perform the method described in the first aspect.

[0054] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0055] (1) This invention provides a method, system, device and medium for calculating optical flow based on event camera. Unlike networks that rely on recursive refinement structures, the network proposed in this invention is based on encoder-decoder structure and multi-scale global optimization scheme, which can generate full-resolution optical flow with minimal computational overhead.

[0056] (2) This invention proposes to combine a large-scale feature pyramid with other smaller-scale pooling pyramids to construct a lightweight encoder. This encoder can enhance the network's ability to extract large-scale features, obtain sufficient motion information for refined optical flow, and at the same time reduce the number of channels for low-scale features, eliminating unnecessary overhead for estimating coarse optical flow;

[0057] (3) This invention fully considers the spatial sparsity of event data and proposes a hole feature correlation builder to establish the correlation tensor between features. This builder performs dense sampling around the center and sparse sampling at greater distances, thereby reducing the computational load while maintaining a large radius cost volume;

[0058] (4) This invention proposes a confidence-guided detail enhancement module to address the mixed motion problem during optical flow upsampling. This module corrects the initial optical flow through a self-calibrator and an autocorrelation branch, and uses a confidence map to highlight error areas, thereby enhancing edge details and resolving the problem of blurred object edges. Attached Figure Description

[0059] The present invention will be further described in conjunction with the following drawings and embodiments. All inventive concepts of the present invention should be considered as disclosed content and within the scope of protection of the present invention.

[0060] Figure 1 A network structure diagram of an optical flow calculation network model in a method, system, device, and medium based on an event camera, provided in this application embodiment;

[0061] Figure 2 This application provides a supplementary module for confidence-guided calculation of optical flow based on an event camera, which is a method, system, device, and medium for this application embodiment.

[0062] Figure 3 A pooled pyramid structure for a lightweight encoder in a method, system, device, and medium for calculating optical flow based on an event camera, provided in the embodiments of this application;

[0063] Figure 4 A subjective comparison of experimental results in a method, system, device, and medium for calculating optical flow based on an event camera, provided as an embodiment of this application;

[0064] Figure 5 Experimental results of embodiments of the method, system, device, and medium for calculating optical flow based on an event camera provided in this application. Detailed Implementation

[0065] To more clearly illustrate the technical solutions of the embodiments of the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments, and therefore should not be regarded as a limitation on the scope of protection. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "set up," "connected," and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0067] Example 1:

[0068] This embodiment provides a method, system, device, and medium for calculating optical flow based on an event camera, such as... Figure 1 As shown, the training dataset is constructed by collecting a large amount of data from event cameras and optical flow labels to build the training dataset D. This training dataset should consist of multiple data pairs, each including a time interval t. k-1 →t k The sequence E of events recorded by the internal event camera k-1 The next time interval t k →t k+1 sequence of events E within k and time interval t k →t k+1 The displacement F of all pixels within the intra-event camera spatial image plane Ω k The size of any event sequence E is N×4, where N is the length of the event sequence; the size of any pixel displacement F is H×W×2, where H and W are the width and height of the event camera spatial image plane Ω, respectively.

[0069] This embodiment uses the DSEC dataset used in cited document 2 as the training dataset D. This training dataset consists of approximately 8,000 data pairs.

[0070] Example 2:

[0071] This embodiment is a further optimization based on embodiment 1, with a time interval t. k-1 →t k The event sequence E within the image consists of multiple events, each with a location (x, y) in the image space. e ,y e The triggering time t e and the polarity of brightness change p e The event sequence records only four attributes: x-position attribute, x-position attribute, trigger time attribute, and polarity attribute.

[0072] Event Sequence E k Represented as E k={(x e ,y e ,t e ,p e ) N},(x e ,y e )∈Ω,t e ∈[t k-1 ,t k ], p e ∈{-1,+1}; (1)

[0073] To facilitate the processing of event sequences by convolutional neural networks, they need to be transformed into spatial images. Preferably, this invention employs a discrete-time cumulative image representation method to transform the event sequence as input to the network. This invention first divides the event sequence into B segments along the time dimension, then scales the time dimension to the range [0, B-1] using bilinear interpolation. Next, the trigger times of events at the same spatial image location in each segment are accumulated and transformed into a spatial image V of size B×H×W. The calculation process is as follows:

[0074]

[0075] k b (a) = max(0, 1 - |a|) (3);

[0076]

[0077] Based on the above representation method, the present invention respectively represents t k-1 →t k and t k →t k+1 Event sequence E within the time interval k and E k+1 Represented as a discrete-time cumulative image V k-1 and V k This serves as the input for subsequent optical flow computing networks.

[0078] The other parts of this embodiment are the same as those in Embodiment 1, so they will not be described again.

[0079] Example 3:

[0080] This embodiment further optimizes upon Embodiment 1 or 2 described above, constructing an optical flow computation network. Unlike recent networks that rely on recursive refinement structures, the network proposed in this invention is based on an encoder-decoder structure and a multi-scale global optimization scheme, enabling the generation of full-resolution optical flow with minimal computational overhead. Figure 1 As shown, the network proposed in this invention uses a lightweight encoder to obtain data from V... k-1 and V kTwo sets of multi-scale features are extracted, and then the correlation tensor between the two sets of features is established using the hole feature correlation builder. Finally, the estimation results are gradually improved from coarse to fine residual optical flow using a fine-grained decoder, and the confidence-guided detail supplementation module supplements the boundary details during optical flow upsampling.

[0081] Specifically, the network proposed in this invention uses a lightweight encoder to obtain V k-1 and V k Two sets of multi-scale features are extracted, and then the correlation tensor between the two sets of features is established using the hole feature correlation builder. Finally, the estimation results are gradually improved from coarse to fine residual optical flow using a fine-grained decoder, and the confidence-guided detail supplementation module supplements the boundary details during optical flow upsampling.

[0082] The lightweight encoder used in this invention combines a large-scale feature pyramid with other smaller-scale pooling pyramids to enhance the network's ability to extract large-scale features, thereby obtaining sufficient motion information for refined optical flow and reducing the number of channels for low-scale features to eliminate unnecessary overhead in estimating coarse optical flow. Preferably, the lightweight encoder proposed in this invention employs a 6-layer pooling pyramid structure, with the first three layers containing two pooling convolutional layers and the last three layers containing only pooling layers. The feature resolution of each layer decreases progressively with a scale factor of 2, ultimately starting from V... k-1 and V k Six sets of scale features were extracted from 1 / 2 full resolution to 1 / 64 full resolution. and

[0083] Hollow Feature Correlation Constructor. The hollow feature correlation constructor constructs features using inner product operations. and The correlation tensor between them {(C i ) N} is used to sense motion information to estimate rough optical flow. The calculation process is shown in the following formula (5):

[0084]

[0085] Where u represents the feature Spatial coordinates, N represents the feature The search region in N is used to limit the calculation range, d represents the offset coordinate of an element in N, and M represents the number of elements in N. Typically, the search region N is defined as a rectangle of size (2r+1)×(2r+1), and it has been observed that increasing the radius r can improve the accuracy of optical flow estimation but increases the computational cost. This invention proposes a method for calculating the correlation of hole features, which involves dense sampling around the center and sparse sampling at greater distances, thereby reducing the computational cost while maintaining a large radius. The definition of the search region N is shown in formula (6):

[0086]

[0087] Where d=(d x ,d y ) represents the offset coordinate of an element in N, when N(d x ,d y When N(d) = 1, the similarity is calculated at u+d. x ,d y When ) = 0, it means no calculation is performed.

[0088] Refined Decoder. In the decoding process, the present invention adopts a coarse-to-fine residual method to gradually improve the optical flow, highlighting local motion and focusing on the details of object edges. The decoding process is shown in formula (7).

[0089]

[0090] Where C(·,·), W(·,·), and ↑ represent the hole feature correlation builder, optical flow warping operation, and optical flow upsampling, respectively. This invention employs a pyramid decoder for optical flow estimation, F... i-1 F is the output optical flow of the (i-1)th layer decoder. i and Conv i This refers to the output optical flow and convolution operation of the i-th layer decoder. The most noteworthy aspect during the decoding process is the application of F... i-1 Upsampling A common method is bilinear interpolation, but this can lead to the mixing of various motion errors at the edges of objects, resulting in blurring. Therefore, this invention proposes a confidence-guided detail enhancement module for optical flow upsampling to enhance edge details.

[0091] like Figure 1 As shown, the fine-grained decoder has 6 layers, each consisting of two convolutional layers and a detail supplementation module. Only the first and last layers of the decoder are shown; the structures of the intermediate layers are repetitive and are omitted.

[0092] The other parts of this embodiment are the same as any one of the embodiments 1-2 above, so they will not be described again.

[0093] Example 4:

[0094] This embodiment is a further optimization based on any one of embodiments 1-3 above, such as... Figure 1 and Figure 2 As shown, Figure 2 yes Figure 1A schematic diagram of the detail enhancement module in the decoder. The confidence-guided detail enhancement module proposed in this invention can effectively solve the problem of object edge blurring caused by the mixing of multiple motions at the intersection of different motions during optical flow upsampling. The optical flow F of the (i-1)th layer decoder... i-1 After input, the module first generates the initial optical flow of the (i-1)th layer through bilinear interpolation. Then, a self-calibrator and an autocorrelation branch are set to correct it. The self-calibrator is based on a dense convolutional network with a 5-layer structure, and it is modified by splicing... and Dense convolution is performed to capture motion information within the edge neighborhood, and the corrected optical flow ΔF is output. i And confidence plot W i The autocorrelation branch is based on a self-attention mechanism and uses a larger receptive field in the features. Find with The motion in the error region is the same fine region, and the output is from the attention weight A. i , and the initial flow field Multiplication. Using the corrected optical flow ΔF i and self-attention weight A i The fine optical flow can be calculated using formula (8).

[0095]

[0096] Where W(·,·) represents the optical flow distortion operation. This represents a multiplication operation, where α∈[0,1] is the weighting coefficient. Finally, based on formula (9), and according to the confidence plot W... i Highlight the edge regions of objects that are prone to errors and use them as weights to fuse the initial optical flow. and fine optical flow The final corrected optical flow was obtained.

[0097] Where ⊙ represents element-wise multiplication.

[0098] The other parts of this embodiment are the same as any one of the embodiments 1-3 above, so they will not be described again.

[0099] Example 5:

[0100] This embodiment further optimizes any one of embodiments 1-4 above by training, batch packaging data, loading it into the network for multiple rounds of supervised training to obtain the final trained model. Preferably, the present invention uses the L1 distance between the network predicted value and the true optical flow value as the loss function L to supervise the training of the network, as shown in formula (10):

[0101] Where l represents the number of layers in the network, F i This represents the optical flow prediction value of the i-th layer of the network. This indicates that the full-resolution optical flow is downsampled to the same scale as the prediction value of the i-th layer of the network. λ is used as a weighting coefficient to balance the distance loss of different layers. In this embodiment, λ = 0.8 is set as the weighting coefficient to balance the distance loss of different layers. The training is performed on two NVIDIA 2080Ti graphics cards using the PyTorch framework, employing the AdamW optimizer and the OneCycle strategy, with a learning rate of 1×10⁻⁶. -4 The weight decays to 5×10 -5 Other default parameters are set as follows: β1 = 0.9, β2 = 0.99, ε = 1 × 10⁻⁶. -4 The training was iterated 100,000 times.

[0102] The event camera device captured by this invention includes all models of event cameras that were on the market before the publication of this claim, and the shooting method includes, but is not limited to, handheld camera movement, fixed camera movement in a car or unmanned aerial vehicle, etc.

[0103] The other parts of this embodiment are the same as any one of the embodiments 1-4 above, so they will not be described again.

[0104] Example 6:

[0105] This embodiment further optimizes any one of embodiments 1-5 above, testing and inputting test data into the trained network to obtain prediction results. This embodiment uses the evaluation metrics adopted in citations 2, 3, 4, and 5 to assess the accuracy of the prediction results. The evaluation metrics use the average endpoint error (EPE) to measure the error between the predicted result and the true value. Additionally, NPE is used to measure the percentage of optical flow errors with an amplitude greater than N pixels (e.g., 3PE, 2PE, 1PE), and angular error (AE) is used to evaluate directional accuracy. Smaller EPE, 3PE, 2PE, 1PE, and AE indicate higher prediction accuracy. This embodiment also uses frames per second (FPS) to measure the network inference speed. Higher FPS indicates faster speed.

[0106] like Figure 4 As shown, the network trained in this embodiment exhibits superior performance and operational efficiency compared to other networks. Figure 3 As shown, a subjective comparison chart of the prediction results of the network trained in this embodiment is presented.

[0107] Furthermore, this invention also provides a system for calculating optical flow based on an event camera, which is compatible with the method. The data acquisition module uses an event camera device and shooting method to acquire event flow data. The optical flow calculation module uses an optical flow calculation network to process the acquired data in real time and calculate the optical flow. The result storage module saves the optical flow result as a floating-point array of size H×W×2, stores it as a readable and writable file in .flo format, and transmits it to the running terminal of downstream tasks (such as video stabilization, multi-exposure enhancement, etc.) through wired or wireless communication.

[0108] The present invention also provides an electronic device comprising a processor and a memory; the processor includes the aforementioned system for calculating optical flow based on an event camera.

[0109] Example 7:

[0110] The present invention also provides a computer-readable storage medium comprising instructions; when the instructions are executed on the electronic device described in the above embodiments, the electronic device causes the electronic device to perform the methods described in the above embodiments. Optionally, the computer-readable storage medium may be a memory.

[0111] The processor involved in the embodiments of this application can be a chip. For example, it can be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0112] The memory involved in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DRRAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0113] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0114] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0115] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0116] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or modules may be electrical, mechanical, or other forms.

[0117] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located on one device or distributed across multiple devices. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0118] In addition, the functional modules in the various embodiments of this application can be integrated into one device, or each module can exist physically separately, or two or more modules can be integrated into one device.

[0119] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0120] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications or equivalent changes made to the above embodiments based on the technical essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. A method for calculating optical flow based on an event camera, characterized in that, Includes the following steps: Step S1: Construct a training dataset based on the data collected by the event camera and the optical flow labels. The training dataset includes the event sequence within the time interval and the displacement of all pixels in the spatial image plane of the event camera within the same time interval. Step S2: Construct an optical flow computing network model, which includes a lightweight encoder, a hole feature correlation builder, and a refined decoder. Step S3: Convert the event sequence into a discrete event cumulative image using an image and input it into the optical flow computing network model, then run the optical flow computing network model; The method for running the optical flow computing network model includes: the lightweight encoder for two network inputs and Extracting from a pooled pyramid structure using shared weights Full resolution to Features at 6 scales in full resolution , and , ,in, Indicates the floor number. Indicates the total number of floors; The feature and The input is fed into the hole feature correlation builder, which constructs the feature using inner product operations and methods to calculate hole feature correlation. and Correlation tensor between ; The correlation tensor The features and The data are then input into the refined decoder. The correlation tensor Contains 6 sets of tensors The features are sequentially input into two convolutional layers a and f according to their group. and The optical flow is divided into 6 pairs according to the group and sequentially input into the same detail supplementation module a and detail supplementation module f. The detail supplementation module a and detail supplementation module f upsample the optical flow. The decoding process of the refined decoder is represented as follows: ; in , and These represent the correlation of hole features, optical flow distortion calculation, and optical flow upsampling, respectively. It is the first The output optical flow of the layer decoder and It is the first The output optical flow and convolution operations of the layer decoder; Step S4: Train the optical flow computing network model using the training dataset, use the L1 loss function to guide the optical flow computing model, pre-set the relevant hyperparameters of the optical flow computing network model, use the learning rate descent method to iteratively decay the loss value, optimize the weight parameters of the optical flow computing network model until the number of iterations equals the maximum number of iterations, stop training, and generate the trained optical flow computing network model. Step S5: Use an event camera to collect two consecutive event stream data segments with the same time interval, then convert them into discrete-time cumulative images, input them into the trained optical flow calculation network model, calculate the displacement of all pixels in the spatial image plane within the time interval as optical flow, and store it.

2. The method for calculating optical flow based on an event camera according to claim 1, characterized in that, Step S1 includes: the training dataset includes multiple data pairs, each data pair including a time interval. A sequence of events recorded by an internal event camera Next time interval sequence of events within and time interval Internal event camera spatial image plane Displacement of all pixels ; Where k is the sequence number of the time series arranged in ascending order, and the size of the event series E is... , The length of the event sequence; the magnitude of the pixel displacement F is , and These are the event camera spatial image planes. Width and height.

3. The method for calculating optical flow based on an event camera according to claim 1, characterized in that, The structure of the optical flow computing network model in step S2 includes: The lightweight encoder, the hole feature correlation builder, and the fine-grained decoder are connected sequentially from front to back; The lightweight encoder includes a 6-layer pooling pyramid structure. The first three layers of the pooling pyramid structure contain two pooling convolutional layers, while the last three layers of the pooling pyramid structure contain only pooling layers. The feature resolution of each layer decreases layer by layer with a scale factor of 2. The refined decoder includes two convolutional layers a, a detail supplementation module a, and two convolutional layers f connected sequentially from front to back. The two convolutional layers a and the two convolutional layers f have the same structure, and the detail supplementation module a and the detail supplementation module f have the same structure. The pooling layer is connected to the hole feature correlation constructor, and the hole feature correlation constructor is connected to two convolutional layers a.

4. The method for calculating optical flow based on an event camera according to claim 1, characterized in that, The method for converting the event sequence into a discrete event accumulation image and inputting it into the optical flow calculation network model in step S3 includes: recording time intervals. sequence of events within The location of each event in the image space The moment of triggering and the polarity of brightness changes Four attributes, for the event sequence Represented as: ； Using a discrete-time cumulative image representation method, the event sequence in a data pair of the training dataset is represented. and Transform into and , which serves as the input to the optical flow computing network.

5. The method for calculating optical flow based on an event camera according to claim 4, characterized in that, The optical flow distortion operation is expressed as: ； ； in and Representing the first and second halves of the optical flow distortion operation, respectively. Layer features pixel space image plane coordinates It is the first The output optical flow of the layer decoder Optical flow upsampling This represents bilinear interpolation. This is the input for the interpolation operation.

6. The method for calculating optical flow based on an event camera according to claim 1, characterized in that, The method in step S4 of converting the event sequence into a discrete event cumulative image and inputting it into the optical flow computing network model includes: converting the event sequence into a discrete event cumulative image and inputting it into the optical flow computing network model. Divide the sequence into B small segments along the time dimension, and scale the time dimension using bilinear interpolation. Within a certain range, the trigger times of events at the same spatial image location in each sequence are accumulated and transformed into a spatial image V of size V. The calculation process is as follows: ； ； ； in , , and Represents an event sequence any event The four attributes, It is Normalization to Within the range, It is the total number of events in the sequence. , These represent discrete-time cumulative images. The coordinates of the image plane, Represents discrete-time cumulative image The sequence number on the event axis This represents bilinear interpolation. This serves as the input for the interpolation operation; Each and Event sequence within time interval and Represented as discrete-time cumulative image and , which serves as the input to the optical flow computing network.

7. A system for calculating optical flow based on an event camera, characterized in that, It includes a data acquisition module, an optical flow calculation module, and a result storage module, wherein: The data acquisition module is used to construct a training dataset based on the data acquired by the event camera and optical flow labels. The training dataset includes event sequences within time intervals and the displacements of all pixels in the spatial image plane of the event camera within the same time interval. It is used to acquire two consecutive event stream data segments with the same time interval using the event camera, then convert them into discrete-time cumulative images, and input them into the trained optical flow calculation network model. The optical flow computing module is used to construct an optical flow computing network model, which includes a lightweight encoder, a hole feature correlation builder, and a refined decoder; it is used to convert the event sequence into discrete event cumulative images and input them into the optical flow computing network model, and run the optical flow computing network model. The method for running the optical flow computing network model includes: the lightweight encoder for two network inputs and Extracting from a pooled pyramid structure using shared weights Full resolution to Features at 6 scales in full resolution , and , ,in, Indicates the floor number. Indicates the total number of layers; the features and The input is fed into the hole feature correlation builder, which constructs the feature using inner product operations and methods to calculate hole feature correlation. and Correlation tensor between ; the correlation tensor The features and The correlation tensor is input into the refined decoder respectively. Contains 6 sets of tensors The features are sequentially input into two convolutional layers a and f according to their group. and The optical flow is divided into 6 pairs according to group and sequentially input into the same detail supplementation module a and detail supplementation module f. The detail supplementation module a and detail supplementation module f upsample the optical flow. The decoding process of the fine-tuning decoder is represented as follows: ;in , and These represent the correlation of hole features, optical flow distortion calculation, and optical flow upsampling, respectively. It is the first The output optical flow of the layer decoder and It is the first The output optical flow and convolution operations of the layer decoder; The method is used to train the optical flow computing network model using the training dataset, guide the optical flow computing model using the L1 loss function, pre-set the relevant hyperparameters of the optical flow computing network model, continuously iterate and decay the loss value using the learning rate descent method, optimize the weight parameters of the optical flow computing network model until the number of iterations equals the maximum number of iterations, stop the training, and generate the trained optical flow computing network model; calculate the displacement of all pixels in the spatial image plane within the same time interval as optical flow. The result storage module is used to store the displacement of all pixels in the spatial image plane.

8. An electronic device, characterized in that, It includes a processor and a memory; the processor is used to run the system for calculating optical flow based on an event camera as described in claim 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on the electronic device of claim 8, cause the electronic device to perform the method of any one of claims 1-6.

Images