A method and apparatus for optical flow estimation

Abstract

The application provides an optical flow estimation method and device, which can improve the accuracy of optical flow estimation from two adjacent image frames to any time between the two adjacent image frames. The method can include: obtaining a first image frame and a second image frame, the first image frame and the second image frame being any two adjacent image frames in an image sequence, the image sequence being obtained by shooting a target scene; obtaining a first event frame, the first event frame being used to describe the brightness change of the target scene in a time period between the first image frame and the second image frame; determining a target optical flow based on the first image frame, the second image frame and the first event frame, the target optical flow being an optical flow from the first image frame to a target time, the target time being any time between the first image frame and the second image frame.

Description

Technical Field

[0001] This application relates to the field of computer vision, and in particular to an optical flow estimation method and apparatus. Background Technology

[0002] Optical flow is a method that uses the temporal changes of pixels in an image sequence and the correlation between adjacent frames to find the correspondence between the previous frame and the current frame, thereby calculating the motion information of objects between adjacent frames.

[0003] Traditional optical flow estimation methods can only estimate the optical flow between the first image frame and the second image frame (including the optical flow from the first image frame to the second image frame and from the second image frame to the first image frame), where the first image frame and the second image frame are two adjacent image frames. For the optical flow at any time between the two image frames (including the optical flow from the first image frame to that arbitrary time and from the second image frame to that arbitrary time), it can only be averaged by using the linear motion assumption and weighting the time length.

[0004] Since linear motion is an assumed scenario that differs significantly from reality, traditional optical flow calculation methods for estimating optical flow at any time between two adjacent image frames have low accuracy. Summary of the Invention

[0005] This application provides an optical flow estimation method and apparatus that can improve the accuracy of optical flow estimation from two adjacent image frames to any time between those two adjacent image frames.

[0006] In a first aspect, this application provides an optical flow estimation method. The method may include: obtaining a first image frame and a second image frame, wherein the first image frame and the second image frame are any two adjacent image frames in an image sequence, the image sequence being captured of a target scene; obtaining a first event frame, the first event frame being used to describe the brightness change of the target scene during the time period between the first image frame and the second image frame; and determining a target optical flow based on the first image frame, the second image frame, and the first event frame, wherein the target optical flow is the optical flow from the first image frame to a target time, the target time being any time between the first image frame and the second image frame.

[0007] In one possible implementation, the method can be used in an optical flow estimation system, which may include a pixel sensor, an event sensor, and an optical flow estimation device, wherein the pixel sensor and the event sensor are respectively connected to the optical flow estimation device. The method can be performed, for example, by the aforementioned optical flow estimation device.

[0008] It should also be noted that, since the image sequence was captured by a pixel camera of the target scene, the first image frame and the second image frame contain pixel information of the target object in the target scene. Since the event stream data was captured by an event camera of the target scene, the first event stream data can capture the actual high-speed motion information (including linear and non-linear motion) of the target object in the target scene during the time interval between the first image frame and the second image frame.

[0009] Using the optical flow estimation method provided in this application, the optical flow estimation device first estimates the first optical flow between the first image frame and the second image frame based on the first image frame, the second image frame, and the first event frame; then, based on the second event frame, it determines the weight of the optical flow between the first image frame and the target moment compared to the optical flow between the first image frame and the second image frame (i.e., the first optical flow allocation mask). Since there is no motion assumption of the target object, the obtained first optical flow allocation mask has the characteristic of accurately allocating the real motion optical flow. Therefore, the target optical flow obtained by weighting the first optical flow through the first optical flow allocation mask has higher accuracy.

[0010] Optionally, the optical flow estimation device may obtain the first image frame and the second image frame in a variety of ways, and this application does not limit this.

[0011] In one possible implementation, the optical flow estimation device can receive the first image frame and the second image frame sent by the pixel camera.

[0012] In another possible implementation, the optical flow estimation device can obtain the first image frame and the second image frame from other devices or for input via an input interface.

[0013] Optionally, the target scene may include at least one target object, and some or all of the objects in the at least one target object are in motion.

[0014] Optionally, the optical flow estimation device may obtain the first event frame in a variety of ways, and this application does not limit this.

[0015] In one possible implementation, the optical flow estimation device can receive event stream data sent by an event camera. This event stream data includes event data for each of at least one event, each corresponding to at least one brightness change occurring in the target scene between a first image frame and a second image frame. The data for each event includes a timestamp, pixel coordinates, and polarity. The optical flow estimation device can then derive the first event frame based on this event stream data. In other words, the event camera can acquire the event stream data and send it to the optical flow estimation device.

[0016] In another possible implementation, the optical flow estimation device can receive the first event frame sent by the event camera. That is, the event camera can acquire the event stream data, generate the first event frame based on the event stream data, and send it to the optical flow estimation device.

[0017] It should be noted that the first image frame, the second image frame, and the first event frame have the same resolution.

[0018] In one possible implementation, taking the first image frame as comprising H×W pixels, where H and W are both integers greater than 1, the first event frame may include multiple channels, including a first channel, a second channel, a third channel, and a fourth channel. The first channel includes H×W first values, each corresponding to one of the positions of the H×W pixels, and these first values ​​represent the number of times the brightness of the corresponding pixel in the first image frame increases during the time interval between the first and second image frames. The second channel includes H×W second values, each corresponding to one of the positions of the H×W pixels, and these second values ​​represent the number of times the brightness of the corresponding pixel in the first image frame decreases during the time interval between the first and second image frames. The third channel includes H×W third values, each corresponding to one of the positions of H×W pixels. These third values ​​represent the time of the last increase in brightness of the corresponding pixel in the first image frame during the time interval between the first and second image frames. The fourth channel includes H×W fourth values, each corresponding to one of the positions of H×W pixels. These fourth values ​​represent the time of the last decrease in brightness of the corresponding pixel in the first image frame during the time interval between the first and second image frames.

[0019] In one possible implementation, before the optical flow estimation device determines the target optical flow based on the first image frame, the second image frame, and the first event frame, the optical flow estimation device may obtain a second event frame, which is used to describe the brightness change of the target scene during the time period between the first image frame and the target time. Accordingly, the optical flow estimation device determines the target optical flow based on the first image frame, the second image frame, and the first event frame, including: the optical flow estimation device determines the target optical flow based on the first image frame, the second image frame, the first event frame, and the second event frame.

[0020] Specifically, the optical flow estimation device can determine a first optical flow based on the first image frame, the second image frame, and the first event frame, wherein the first optical flow is the optical flow from the first image frame to the second image frame; determine a first optical flow allocation mask based on the second event frame, wherein the first optical flow allocation mask is used to indicate the weight of the target optical flow relative to the first optical flow; and determine the target optical flow based on the first optical flow and the first optical flow allocation mask.

[0021] Optionally, the first optical flow can be either a sparse optical flow or a dense optical flow; this application does not limit this.

[0022] In one possible implementation, taking dense optical flow as an example, the first optical flow represents different directions of motion through different colors of pixels and different rates of motion through different brightness of pixels.

[0023] Optionally, the optical flow estimation device can determine the first optical flow based on the first image frame, the second image frame, and the first event frame in various ways, and this application does not limit this.

[0024] In one possible implementation, the optical flow estimation device can input the first image frame, the second image frame, and the first event frame into a preset optical flow estimation model to obtain the first optical flow.

[0025] To meet the requirements of edge deployment and real-time performance, the network structure of the aforementioned optical flow estimation model cannot be too complex. At the same time, since optical flow estimation is a complex task, a simple network structure is difficult to achieve high-precision optical flow estimation. Therefore, this application uses a lightweight first convolutional neural network for training. This first convolutional neural network may include several processing layers such as dimensionality reduction, convolution, residual, deconvolution, and dimensionality increase. By iteratively training this first convolutional neural network, the accuracy of optical flow estimation can be improved, and it is easy to deploy on the edge.

[0026] Optionally, the optical flow estimation device can input the first image frame, the second image frame, and the first event frame into the optical flow estimation model and perform iterative iterations to obtain the first optical flow. That is, the optical flow estimation device can input the first image frame, the second image frame, and the first event frame into the optical flow estimation model to obtain the second optical flow; input the first image frame, the second image frame, the first event frame, and the second optical flow into the optical flow estimation model to obtain the third optical flow; and so on, iterating iteratively until the optical flow estimation model outputs the first optical flow, which satisfies the preset loss function of the optical flow estimation model, at which point the first optical flow is output.

[0027] It should be noted that the first optical flow allocation mask mentioned above has the same resolution as the first image frame, the second image frame, and the first event frame.

[0028] Optionally, the optical flow estimation device can determine the first optical flow allocation mask based on the second event frame in a variety of ways, and this application does not limit this.

[0029] In one possible implementation, the optical flow estimation device can input the second event frame into a preset optical flow allocation model to obtain the first optical flow allocation mask.

[0030] To meet the requirements of edge deployment and real-time performance, the network structure of the aforementioned optical flow allocation model cannot be too complex. Therefore, this application uses a lightweight second CNN for training. This second CNN may include processing layers such as fusion and convolution. By iterating over this second CNN, the accuracy of optical flow estimation can be improved, and it is easy to deploy on the edge.

[0031] Optionally, the optical flow estimation device can input the second event frame into the optical flow allocation model and perform iterative iterations to obtain the first optical flow allocation mask.

[0032] Secondly, this application also provides an optical flow estimation device, characterized in that it includes: an acquisition module and an optical flow estimation module; the acquisition module is used to acquire a first image frame and a second image frame, the first image frame and the second image frame being any two adjacent image frames in an image sequence, the image sequence being captured of a target scene; acquire a first event frame, the first event frame being used to describe the brightness change of the target scene during the time period between the first image frame and the second image frame; the optical flow estimation module is used to determine a target optical flow based on the first image frame, the second image frame and the first event frame, the target optical flow being the optical flow from the first image frame to a target time, the target time being any time between the first image frame and the second image frame.

[0033] In one possible implementation, the obtaining module is further configured to obtain a second event frame before determining the target optical flow based on the first image frame, the second image frame, and the first event frame. The second event frame is used to describe the brightness change of the target scene during the time period between the first image frame and the target time. The optical flow estimation module is specifically configured to determine the target optical flow based on the first image frame, the second image frame, the first event frame, and the second event frame.

[0034] In one possible implementation, the optical flow estimation module includes an inter-frame optical flow estimation submodule, an optical flow allocation submodule, and an inter-frame arbitrary-time optical flow estimation submodule. The inter-frame optical flow estimation submodule is used to determine a first optical flow based on the first image frame, the second image frame, and the first event frame, wherein the first optical flow is the optical flow from the first image frame to the second image frame. The optical flow allocation submodule is used to determine a first optical flow allocation mask based on the second event frame, wherein the first optical flow allocation mask is used to indicate the weight of the target optical flow relative to the first optical flow. The inter-frame arbitrary-time optical flow estimation submodule is used to determine the target optical flow based on the first optical flow and the first optical flow allocation mask.

[0035] In one possible implementation, the inter-frame optical flow estimation submodule is specifically used to input the first image frame, the second image frame, and the first event frame into a preset optical flow estimation model to obtain the first optical flow.

[0036] In one possible implementation, the inter-frame optical flow estimation submodule is specifically used to input the first image frame, the second image frame, and the first event frame into the optical flow estimation model and perform iterative iterations to obtain the first optical flow.

[0037] In one possible implementation, the optical flow allocation submodule is specifically used to input the second event frame into a preset optical flow allocation model to obtain the first optical flow allocation mask.

[0038] In one possible implementation, the optical flow allocation submodule is specifically used to input the second event frame into the optical flow allocation model and perform iterative loops to obtain the first optical flow allocation mask.

[0039] In one possible implementation, the first image frame includes H×W pixels, where H and W are both integers greater than 1. The first event frame includes multiple channels, including a first channel, a second channel, a third channel, and a fourth channel. The first channel includes H×W first values, each corresponding to one of the positions of the H×W pixels. Each first value represents the number of times the brightness of a pixel at a corresponding position in the first image frame increases during the time interval between the first and second image frames. The second channel includes H×W second values, each corresponding to one of the positions of the H×W pixels. Each second value represents the number of times the brightness of a pixel at a corresponding position in the first image frame increases during the time interval between the first and second image frames. The third channel includes H×W third values, each corresponding to one of the positions of the H×W pixels. Each third value represents the timestamp of the last increase in brightness of the pixel at the corresponding position in the first image frame during the time interval between the first and second image frames. The fourth channel includes H×W fourth values, each corresponding to one of the positions of the H×W pixels. Each fourth value represents the timestamp of the last decrease in brightness of the pixel at the corresponding position in the first image frame during the time interval between the first and second image frames.

[0040] In one possible implementation, the obtaining module is specifically used to: obtain event stream data, which includes event data for each of at least one event, the at least one event corresponding one-to-one with at least one brightness change that occurs in the target scene between the first image frame and the second image frame, the data for each event including a timestamp, pixel coordinates and polarity; and obtain the first event frame based on the event stream data.

[0041] Thirdly, this application also provides an optical flow estimation apparatus, which may include at least one processor and at least one communication interface, wherein the at least one processor and the at least one communication interface are coupled, the at least one communication interface is used to provide information and / or data to the at least one processor, and the at least one processor is used to run computer program instructions to perform the optical flow estimation method described in the first aspect and any possible implementation thereof.

[0042] Alternatively, the device can be a chip or an integrated circuit.

[0043] Fourthly, this application also provides a terminal that may include an optical flow estimation device as described in the second aspect and any possible implementation thereof, or an optical flow estimation device as described in the third aspect.

[0044] Fifthly, this application also provides a computer-readable storage medium, characterized in that it is used to store a computer program, which, when run by a processor, implements the optical flow estimation method described in the first aspect and any possible implementation thereof.

[0045] Sixthly, this application also provides a computer program product, characterized in that, when the computer program product is run on a processor, it implements the optical flow estimation method described in the first aspect and any possible implementation thereof.

[0046] The optical flow estimation device, computer storage medium, computer program product, chip and terminal provided in this application are all used to execute the optical flow estimation method provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects of the optical flow estimation method provided above, and will not be repeated here. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the event stream data provided in the embodiments of this application;

[0048] Figure 2 This is a schematic diagram of sparse optical flow and dense optical flow provided in the embodiments of this application;

[0049] Figure 3 This is a schematic block diagram of the optical flow estimation system 100 provided in an embodiment of this application;

[0050] Figure 4 This is a schematic flowchart of the optical flow estimation method 200 provided in the embodiments of this application;

[0051] Figure 5 This is another schematic diagram of the event stream data provided in the embodiments of this application;

[0052] Figure 6 This is a schematic diagram of the first event frame provided in an embodiment of this application;

[0053] Figure 7 This is a schematic block diagram of the optical flow estimation device 300 provided in the embodiments of this application;

[0054] Figure 8 This is a schematic flowchart of the optical flow estimation method provided in the embodiments of this application;

[0055] Figure 9 This is a schematic block diagram of the optical flow estimation device 400 provided in the embodiments of this application. Detailed Implementation

[0056] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0057] 1. Pixel camera

[0058] A pixel camera, also known as a traditional camera, captures the brightness values ​​of a scene at a fixed rate (i.e., frame rate) and outputs them as image data at a fixed rate.

[0059] 2. Event-based camera

[0060] An event camera is a new type of sensor that uses an event-driven approach to capture dynamic changes in the brightness of pixels in a scene.

[0061] Traditional cameras, to some extent, capture a static / still space, while the purpose of event cameras is to sensitively capture moving objects.

[0062] Unlike traditional cameras, event cameras only observe "motion" in a scene, or more precisely, "changes in brightness." Event cameras only output the brightness change (1 or 0) of the corresponding pixel when there is a change in brightness, offering advantages such as fast response and wide dynamic range.

[0063] For a single pixel, an event camera only outputs data when the light intensity changes. For example, if the brightness increases and exceeds a certain threshold, the corresponding pixel will output a brightness increase event. Event cameras do not have the concept of frames; they generate a series of pixel-level outputs as the scene changes. Their theoretical temporal resolution is as high as 1µs, resulting in very low latency, lower than the motion rates in most common scenes, thus eliminating motion blur. Furthermore, each pixel of an event camera operates independently and asynchronously, resulting in a large dynamic range. Event cameras also have the advantage of low power consumption.

[0064] In summary, traditional cameras capture a scene at a fixed frame rate, with all pixels working synchronously. Event cameras, on the other hand, allow each pixel to work independently and asynchronously, with a sampling rate as high as one megahertz (Hz). They only output changes in brightness (i.e., events), and each event is described by a quadruple of event data. The event data output by all pixels is aggregated to form an event list, which serves as the event stream data output by the camera.

[0065] For example, the event data of an event can be represented as (x,y,t,p), where (x,y) are the pixel coordinates where the event occurs, t is the time when the event occurs, and p is the polarity of the event (e.g., p=0 indicates that the brightness of the pixel has decreased compared to the previous sample, and p=1 indicates that the brightness of the pixel has increased compared to the previous sample).

[0066] Commonly used event cameras may include dynamic vision sensors (DVS) or dynamic and active-pixel vision sensors (DAVIS).

[0067] Example, Figure 1 A schematic diagram of event stream data is shown. Figure 1 Figure (a) in the figure represents the event stream data through a list of event data, which includes multiple event data, each describing an event. Each event data is represented by a quadruple, namely timestamp, x-coordinate, y-coordinate, and polarity. Figure 1 Figure (b) in the figure represents the event stream data through a visualization. The three-dimensional coordinate axes are frame width, frame height and time. The color of the coordinate point in the three-dimensional coordinate represents the number of times the brightness of the corresponding pixel point changes. The brighter the color, the more times the brightness changes.

[0068] 3. Optical flow

[0069] Optical flow, or the flow of light, is a method that uses the changes of pixels in an image sequence over time and the correlation between adjacent frames to find the correspondence between the previous frame and the current frame, thereby calculating the motion information of objects between adjacent frames.

[0070] Optical flow can be classified into sparse optical flow and dense optical flow depending on whether sparse points in the image are selected for optical flow estimation.

[0071] Example, Figure 2 A schematic diagram of sparse and dense optical flow is shown. Among them, Figure 2 Image (a) is a schematic diagram of sparse optical flow, which describes the optical flow of pixels with some salient features (such as corners) moving to the next frame. Figure 2 (b) is a schematic diagram of dense optical flow, which describes the optical flow of all pixels in the image moving to the next frame. Different colors in the dense optical flow diagram represent different directions of motion, and different brightness represents different speeds of motion.

[0072] In the prior art, FlowNet is a model for estimating optical flow, which is trained by a convolutional neural network. Optical flow estimation refers to estimating the pixel-level optical flow between any two adjacent image frames in an image sequence.

[0073] For example, taking two adjacent image frames as the first image frame and the second image frame, and the first image frame being the frame preceding the second image frame, an existing optical flow network can be used to estimate the bidirectional optical flow between the first image frame and the second image frame, that is, the optical flow from the first image frame to the second image frame and the optical flow from the second image frame to the first image frame.

[0074] Because traditional cameras capture images at a constant frequency (i.e., frame rate), even if the frame rate reaches 1kHz, there is still a 1ms delay. Within this 1ms delay, the target object may be moving at high speed.

[0075] For the optical flow at any time between the two image frames mentioned above, that is, the optical flow from the first image frame to the arbitrary time and the optical flow from the second image frame to the arbitrary time, it can only be evenly distributed according to the time length as a weight by the linear motion assumption.

[0076] However, in the above scenario, the target object may move in a nonlinear motion. Therefore, the accuracy of estimating the optical flow at any time between two adjacent image frames and the two image frames mentioned above is low when using existing linear motion assumption methods.

[0077] First, let me introduce the optical flow estimation system used in the optical flow estimation method and apparatus provided in this application.

[0078] Figure 3 A schematic block diagram of an optical flow estimation system 100 provided in an embodiment of this application is shown. Figure 3 As shown, the optical flow estimation system 100 may include a pixel sensor 110, an event sensor 120, and an optical flow estimation device 130, with the pixel sensor 110 and the event sensor 120 respectively connected to the optical flow estimation device 130.

[0079] The pixel sensor 110 is used to capture the target scene and obtain an image sequence; the first image frame and the second video frame are sent to the optical flow estimation device 130, where the first image frame and the second image frame are any two adjacent image frames in the image sequence.

[0080] For example, the pixel sensor can be a pixel camera. This application does not limit the model or type of the pixel camera.

[0081] Event sensor 120 is used to capture images of the target scene and obtain event stream data. The event stream data includes event data for each of at least one event, and each of the at least one event corresponds one-to-one with at least one brightness change of the target scene between the first image frame and the second image frame. The data for each event includes a timestamp, pixel coordinates, and polarity. Based on the event stream data, a first event frame is obtained. The first event frame is used to describe the brightness change of the target scene during the time period between the first image frame and the second image frame. The first image frame, the second image frame, and the first event frame have the same resolution. The first image frame, the second image frame, and the first event frame are sent to optical flow estimation device 130.

[0082] For example, the event sensor can be an event camera. This application does not limit the model or type of the event camera.

[0083] The optical flow estimation device 130 is used to determine the target optical flow based on the first image frame, the second image frame and the first event frame (the specific method can be referred to the optical flow estimation method provided in this application described below). The target optical flow is the optical flow from the first image frame to the target time, and the target time is any time between the first image frame and the second image frame.

[0084] It should be noted that the above description only uses the method of estimating the optical flow (i.e., the target optical flow) from the first image frame to the target time as an example. However, this application is not limited to this. The method of estimating the optical flow from the second image frame to the target time is similar to the method of estimating the target optical flow. You can refer to the method of estimating the target optical flow provided in this application. It will not be described again here.

[0085] Optionally, the event sensor 120 can directly send the event stream data to the optical flow estimation device 130; correspondingly, the optical flow estimation device 130 obtains the first event frame based on the event stream data.

[0086] Optionally, the above-mentioned devices can communicate with each other via wired or wireless means, and this application does not limit this.

[0087] For example, the above wired method can be to achieve communication through a data cable connection or through an internal bus connection.

[0088] For example, the aforementioned wireless method can be communication through a communication network, which can be a local area network (LAN), a wide area network (WAN) relayed through a relay device, or a combination of LAN and WAN. When the communication network is a LAN, it can be a wireless fidelity (Wi-Fi) hotspot network, a Wi-Fi peer-to-peer (P2P) network, a Bluetooth network, a Zigbee network, a near field communication (NFC) network, or a future general short-range communication network, etc. When the communication network is a WAN, for example, it can be a 3rd generation wireless telephone technology (3G) network, a 4th generation mobile communication technology (4G) network, a 5th generation mobile communication technology (5G) network, a public land mobile network (PLMN), or the Internet, etc., and this application does not limit it in this regard.

[0089] By employing the optical flow estimation system provided in this application, since the first event frame can capture the motion information of the target in the target scene with low latency between the first image frame and the second image frame, and the first image frame and the second image frame can capture the pixel information of the target scene, the accuracy of the target optical flow can be improved by combining the first event frame, the first image frame and the second image frame to determine the target optical flow.

[0090] The optical flow estimation system provided in this application has been introduced above. The optical flow estimation method applied to the above-mentioned optical flow estimation system will be further introduced below.

[0091] Please refer to Figure 4 , Figure 4 An optical flow estimation method 200 provided in an embodiment of this application is illustrated. This optical flow estimation method 200 can be used in the optical flow estimation system 100 described above. Figure 2 As shown, the optical flow estimation method 200 may include the following steps. It should be noted that the steps listed below may be performed in various orders and / or occur simultaneously, and are not limited to these steps. Figure 4 The execution order is shown.

[0092] S201. Obtain a first image frame and a second image frame, wherein the first image frame and the second image frame are any two adjacent image frames in an image sequence, and the image sequence is obtained by capturing the target scene.

[0093] Alternatively, the method 200 can be performed by an optical flow estimation device.

[0094] For example, the optical flow estimation device here can be the optical flow estimation device 130 in the optical flow estimation system 100 described above.

[0095] Optionally, the optical flow estimation device may obtain the first image frame and the second image frame in a variety of ways, and this application does not limit this.

[0096] In one possible implementation, the optical flow estimation device can receive the first image frame and the second image frame sent by the pixel camera.

[0097] For example, the pixel camera here can be the pixel camera 110 in the optical flow estimation system 100 described above.

[0098] In another possible implementation, the optical flow estimation device can obtain the first image frame and the second image frame from other devices or for input via an input interface.

[0099] Optionally, the target scene may include at least one target object, and some or all of the objects in the at least one target object are in motion.

[0100] S202. Obtain a first event frame, which is used to describe the brightness change of the target scene during the time period between the first image frame and the second image frame.

[0101] Optionally, the optical flow estimation device may obtain the first event frame in a variety of ways, and this application does not limit this.

[0102] In one possible implementation, the optical flow estimation device can receive event stream data sent by an event camera. This event stream data includes event data for each of at least one event, each corresponding to at least one brightness change occurring in the target scene between a first image frame and a second image frame. The data for each event includes a timestamp, pixel coordinates, and polarity. The optical flow estimation device can then derive the first event frame based on this event stream data. In other words, the event camera can acquire the event stream data and send it to the optical flow estimation device.

[0103] In another possible implementation, the optical flow estimation device can receive the first event frame sent by the event camera. That is, the event camera can acquire the event stream data, generate the first event frame based on the event stream data, and send it to the optical flow estimation device.

[0104] For example, the event camera here can be the event camera 120 in the optical flow estimation system 100 described above.

[0105] It should be noted that the first image frame, the second image frame, and the first event frame have the same resolution.

[0106] For example, taking the first image frame, the second image frame, and the first event frame as examples where the resolution is all 4×4. Figure 5 A schematic diagram of event stream data provided in an embodiment of this application is shown. For example... Figure 5 As shown, the event stream data includes 20 event data points. These 20 event data points describe 20 brightness changes (i.e., 20 events occurring in the target scene) that occur in pixels within the time period from the first image frame (i.e., time t1) to the second image frame (i.e., time t10), where t1 to t10 increase sequentially. Figure 5 As shown, taking event 1 as an example, the event data for event 1 includes the following four-tuple: timestamp t1+Δt1, x-coordinate 1, y-coordinate 1, and polarity 1 (i.e., increased brightness). Event 1 describes the increased brightness of the pixel at coordinate (1,1) at time t1+Δt1. The other 19 events have a similar interpretation and will not be elaborated here. The movement speed and direction of different pixels at different times can be estimated using the above event stream data.

[0107] In one possible implementation, taking the first image frame as comprising H×W pixels, where H and W are both integers greater than 1, the first event frame may include multiple channels, including a first channel, a second channel, a third channel, and a fourth channel. The first channel includes H×W first values, each corresponding to one of the positions of the H×W pixels, and these first values ​​represent the number of times the brightness of the corresponding pixel in the first image frame increases during the time interval between the first and second image frames. The second channel includes H×W second values, each corresponding to one of the positions of the H×W pixels, and these second values ​​represent the number of times the brightness of the corresponding pixel in the first image frame decreases during the time interval between the first and second image frames. The third channel includes H×W third values, each corresponding to one of the positions of H×W pixels. These third values ​​represent the time of the last increase in brightness of the corresponding pixel in the first image frame during the time interval between the first and second image frames. The fourth channel includes H×W fourth values, each corresponding to one of the positions of H×W pixels. These fourth values ​​represent the time of the last decrease in brightness of the corresponding pixel in the first image frame during the time interval between the first and second image frames.

[0108] Example, Figure 6 This illustration shows a schematic diagram of a first event frame provided in an embodiment of this application. The first event frame may include a first channel, a second channel, a third channel, and a fourth channel. The first event frame is based on... Figure 5 The event stream data shown is obtained. Figure 6 As shown, taking the pixel at coordinates (1,1) as an example, in Figure 5 In the event stream data, the polarity of this pixel is 1 (i.e., brightness increases) at times t2 and t9, respectively. This means the pixel's brightness increases 2 times and decreases 0 times between t1 and t10. The last increase occurs at t9, and the last decrease does not occur. Therefore, the pixel at coordinates (1,1) in the first channel (e.g., ...) Figure 6 The first value at the black background pixel in the first channel is 2, and the pixel at coordinates (1,1) in the second channel (as shown in the image) is 2. Figure 6 The second value at the black background pixel in the second channel is 0, and the pixel at coordinates (1,1) in the third channel (as shown in the image) is 0. Figure 6 The third value at the black background pixel in the third channel is t9, and the pixel at coordinates (1,1) in the fourth channel (as shown in the image) is t9. Figure 6 The fourth value at the black background pixel in the fourth channel is recorded as 0.

[0109] Similarly, taking the pixel at coordinates (2,2) as an example, in Figure 5 In the event stream data, the polarity of this pixel is 0 (i.e., brightness decreases) at times t2, t3, t4, t5, and t6, respectively. This means the pixel's brightness increases 0 times and decreases 5 times between t1 and t10. The time of the last increase is missing, and the time of the last decrease is t6. Therefore, the pixel at coordinates (2,2) in the first channel (e.g., ...) Figure 6 The first value at the location indicated by the thickened border in the first channel is 0, and the first value at the location of the pixel at coordinates (2,2) in the second channel is 0. Figure 6 The second value at the location indicated by the thickened border in the second channel is 5, and the value at the location indicated by the pixel in the third channel (2,2) is 5. Figure 6 The third value at the location indicated by the thickened border of the third channel is denoted as 0. The pixel at coordinates (2,2) in the fourth channel (as shown in the image) is denoted as... Figure 6 The fourth value at the point indicated by the thickened border of the fourth channel is t6.

[0110] S203. Based on the first image frame, the second image frame, and the first event frame, determine the target optical flow, wherein the target optical flow is the optical flow from the first image frame to the target time, and the target time is any time between the first image frame and the second image frame.

[0111] It should also be noted that, since the image sequence was captured by a pixel camera of the target scene, the first image frame and the second image frame contain pixel information of the target object in the target scene. Since the event stream data was captured by an event camera of the target scene, the first event stream data can capture the actual high-speed motion information (including linear and non-linear motion) of the target object in the target scene during the time interval between the first image frame and the second image frame.

[0112] In summary, by combining the first image frame, the second image frame, and the first event frame to estimate the optical flow of the target at that moment, the accuracy of the target's optical flow can be improved.

[0113] In one possible implementation, prior to S203, the optical flow estimation device can obtain a second event frame, which describes the brightness change of the target scene during the time period between the first image frame and the target time; correspondingly, in S203, the optical flow estimation device can determine the target optical flow based on the first image frame, the second image frame, the first event frame, and the second event frame.

[0114] It should be noted that the method for obtaining the second event frame can refer to the method for obtaining the first event frame described above, and will not be repeated here.

[0115] Specifically, the optical flow estimation device can determine a first optical flow based on the first image frame, the second image frame, and the first event frame, wherein the first optical flow is the optical flow from the first image frame to the second image frame; determine a first optical flow allocation mask based on the second event frame, wherein the first optical flow allocation mask is used to indicate the weight of the target optical flow relative to the first optical flow; and determine the target optical flow based on the first optical flow and the first optical flow allocation mask.

[0116] Optionally, the first optical flow can be either a sparse optical flow or a dense optical flow; this application does not limit this.

[0117] In one possible implementation, taking dense optical flow as an example, the first optical flow represents different directions of motion through different colors of pixels and different rates of motion through different brightness of pixels.

[0118] Optionally, the optical flow estimation device can determine the first optical flow based on the first image frame, the second image frame, and the first event frame in various ways, and this application does not limit this.

[0119] In one possible implementation, the optical flow estimation device can input the first image frame, the second image frame, and the first event frame into a preset optical flow estimation model to obtain the first optical flow.

[0120] To meet the requirements of edge deployment and real-time performance, the network structure of the aforementioned optical flow estimation model cannot be too complex. At the same time, since optical flow estimation is a complex task, a simple network structure is difficult to achieve high-precision optical flow estimation. Therefore, this application uses a lightweight first convolutional neural network (CNN) for training. This first CNN may include several processing layers such as dimensionality reduction, convolution, residual, deconvolution, and dimensionality increase. By iteratively training this first CNN, the accuracy of optical flow estimation can be improved, and it is easy to deploy on the edge.

[0121] Optionally, the optical flow estimation device can input the first image frame, the second image frame, and the first event frame into the optical flow estimation model and perform iterative iterations to obtain the first optical flow. That is, the optical flow estimation device can input the first image frame, the second image frame, and the first event frame into the optical flow estimation model to obtain the second optical flow; input the first image frame, the second image frame, the first event frame, and the second optical flow into the optical flow estimation model to obtain the third optical flow; and so on, iterating iteratively until the optical flow estimation model outputs the first optical flow, which satisfies the preset loss function of the optical flow estimation model, at which point the first optical flow is output.

[0122] It should be noted that the first optical flow allocation mask mentioned above has the same resolution as the first image frame, the second image frame, and the first event frame.

[0123] Optionally, the optical flow estimation device can determine the first optical flow allocation mask based on the second event frame in a variety of ways, and this application does not limit this.

[0124] In one possible implementation, the optical flow estimation device can input the second event frame into a preset optical flow allocation model to obtain the first optical flow allocation mask.

[0125] To meet the requirements of edge deployment and real-time performance, the network structure of the aforementioned optical flow allocation model cannot be too complex. Therefore, this application uses a lightweight second CNN for training. This second CNN may include processing layers such as fusion and convolution. By iterating over this second CNN, the accuracy of optical flow estimation can be improved, and it is easy to deploy on the edge.

[0126] For example, the fusion processing layer is used to fuse the multi-channel second event frame into a single-channel image, and the convolution processing layer is used to perform convolution processing on the single-channel image using convolution kernels in the X direction and the Y direction respectively to output the single-channel first optical flow allocation mask, the resolution of which is the same as that of the second event frame.

[0127] For example, the convolution kernel in the X direction can be... The convolution kernel in the Y direction can be

[0128] Optionally, the optical flow estimation device can input the second event frame into the optical flow allocation model and perform iterative iterations to obtain the first optical flow allocation mask. That is, the optical flow estimation device can input the second event frame into the optical flow allocation model to obtain a second optical flow allocation mask; input the second event frame and the second optical flow allocation mask into the optical flow allocation model to obtain a third optical flow allocation mask; and so on, iterating until the optical flow allocation model outputs the first optical flow allocation mask when it satisfies a preset loss function of the optical flow allocation model, at which point the first optical flow allocation mask is output.

[0129] In one possible implementation, the optical flow estimation device can obtain the target optical flow by weighting the optical flow at corresponding positions in the first optical flow using the first optical flow allocation mask.

[0130] Using the optical flow estimation method provided in this application, the optical flow estimation device first estimates the first optical flow between the first image frame and the second image frame based on the first image frame, the second image frame, and the first event frame; then, based on the second event frame, it determines the weight of the optical flow between the first image frame and the target moment compared to the optical flow between the first image frame and the second image frame (i.e., the first optical flow allocation mask). Since there is no motion assumption of the target object, the obtained first optical flow allocation mask has the characteristic of accurately allocating the real motion optical flow. Therefore, the target optical flow obtained by weighting the first optical flow through the first optical flow allocation mask has higher accuracy.

[0131] The above combination Figures 4 to 6 The optical flow estimation method provided in the embodiments of this application has been introduced. The optical flow estimation device provided in the embodiments of this application will be further described below.

[0132] Please refer to Figure 7 , Figure 7 A schematic block diagram of an optical flow estimation device 300 provided in an embodiment of this application is shown. The optical flow estimation device 300 may include an acquisition module 301 and an optical flow estimation module 302.

[0133] The acquisition module 301 is used to acquire a first image frame and a second image frame, which are any two adjacent image frames in an image sequence, which is obtained by capturing a target scene; and to acquire a first event frame, which is used to describe the brightness change of the target scene between the first image frame and the second image frame.

[0134] The optical flow estimation module 302 is used to determine a target optical flow based on the first image frame, the second image frame, and the first event frame. The target optical flow includes at least one of the optical flow of corresponding pixels between the first image frame and the target time and the optical flow of corresponding pixels between the target time and the second image frame. The target time is any time between the first image frame and the second image frame.

[0135] In one possible implementation, the obtaining module 301 is further configured to obtain a second event frame before determining the target optical flow based on the first image frame, the second image frame, and the first event frame. The second event frame is used to describe the brightness change of the target scene between the first image frame and the target time. The optical flow estimation module 302 is specifically configured to determine the target optical flow based on the first image frame, the second image frame, the first event frame, and the second event frame.

[0136] Optionally, the optical flow estimation module 302 may include an inter-frame optical flow estimation submodule 3021, an optical flow allocation submodule 3022, and an inter-frame arbitrary time optical flow estimation submodule 3023.

[0137] In one possible implementation, the inter-frame optical flow estimation submodule 3021 is used to determine a first optical flow based on the first image frame, the second image frame, and the first event frame, wherein the first optical flow is the optical flow of corresponding pixels between the first image frame and the second image frame; the optical flow allocation submodule 3022 is used to determine a first optical flow allocation mask based on the second event frame, wherein the first optical flow allocation mask is used to indicate the weight of the target optical flow relative to the first optical flow; and the inter-frame arbitrary time optical flow estimation submodule 3023 is used to determine the target optical flow based on the first optical flow and the first optical flow allocation mask.

[0138] In one possible implementation, the inter-frame optical flow estimation submodule 3021 is specifically used to input the first image frame, the second image frame and the first event frame into a preset optical flow estimation model to obtain the first optical flow.

[0139] In one possible implementation, the inter-frame optical flow estimation submodule 3021 is specifically used to input the first image frame, the second image frame and the first event frame into the optical flow estimation model and perform iterative iterations to obtain the first optical flow.

[0140] In one possible implementation, the optical flow allocation submodule 3022 is specifically used to input the second event frame into a preset optical flow allocation model to obtain the first optical flow allocation mask.

[0141] In one possible implementation, the optical flow allocation submodule 3022 is specifically used to input the second event frame into the optical flow allocation model and perform iterative loops to obtain the first optical flow allocation mask.

[0142] In one possible implementation, the first image frame includes H×W pixels, where H and W are both integers greater than 1. The first event frame includes multiple channels, including a first channel, a second channel, a third channel, and a fourth channel. The first channel includes H×W first values, each corresponding to one of the positions of the H×W pixels, and each first value represents the number of times the brightness of the corresponding pixel in the first image frame increases between the first image frame and the second image frame. The second channel includes H×W second values, each corresponding to one of the positions of the H×W pixels, and each second value represents one of the positions of the first image frame. The first image frame contains H×W values, each corresponding to one of the positions of the H×W pixels. Each value represents the time between the first and second image frames when the brightness of the corresponding pixel in the first image frame decreases. The second image frame contains H×W values, each corresponding to one of the positions of the H×W pixels. Each value represents the time between the first and second image frames when the brightness of the corresponding pixel in the first image frame last increases. The third image frame contains H×W values, each corresponding to one of the positions of the H×W pixels. Each value represents the time between the first and second image frames when the brightness of the corresponding pixel in the first image frame last decreases.

[0143] In one possible implementation, the obtaining module is specifically used to obtain event stream data, which includes event data for each of at least one event, the at least one event corresponding one-to-one with at least one brightness change that occurs in the target scene between the first image frame and the second image frame, and the data for each event includes a timestamp, pixel coordinates and polarity; based on the event stream data, the first event frame is obtained.

[0144] It should be noted that the information interaction and execution process between the above modules are based on the same concept as the method embodiments of this application. Their specific functions and technical effects can be found in the method embodiments section, and will not be repeated here. In an optional example, the optical flow estimation device 300 can specifically be the optical flow estimation device in the above-described optical flow estimation method 200 embodiment. The optical flow estimation device 300 can be used to execute the various processes and / or steps corresponding to the optical flow estimation device in the above-described optical flow estimation method 200 embodiment. To avoid repetition, these will not be repeated here.

[0145] Figure 7One or more of the modules in the illustrated embodiments can be implemented by software, hardware, firmware, or a combination thereof. The software or firmware includes, but is not limited to, computer program instructions or code, and can be executed by a hardware processor. The hardware includes, but is not limited to, various integrated circuits, such as a central processing unit (CPU), a digital signal processor (DSP), a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC).

[0146] Example, Figure 8 A schematic flowchart of the optical flow estimation method provided in an embodiment of this application is shown. Optionally, the steps in this flowchart can be... Figure 7 The optical flow estimation device 300 described herein performs the operation. It should be noted that the steps listed below can be performed in various orders and / or occur simultaneously, and are not limited to... Figure 8 The execution order is shown. The process includes the following steps:

[0147] (1) The module 301 obtains the first image frame, the second image frame, the first event frame, and the second event frame. For details, please refer to the relevant descriptions in steps 201 and 202 of the above method.

[0148] (2) The acquisition module 301 sends the first image frame, the second image frame and the first event frame to the inter-frame optical flow estimation submodule 3021.

[0149] (3) The inter-frame optical flow estimation submodule 3021 inputs the first image frame, the second image frame, and the first event frame into the optical flow estimation model and performs iterative iterations to obtain the first optical flow. For details, please refer to the relevant description in step 203 of the above method.

[0150] (4) The inter-frame optical flow estimation submodule 3021 sends the first optical flow to the inter-frame optical flow estimation submodule 3023 at any time.

[0151] (5) The acquisition module 301 sends the second event frame to the optical flow distribution submodule 3022.

[0152] (6) The optical flow allocation submodule 3022 inputs the second event frame into the optical flow allocation model and performs iterative iterations to obtain the first optical flow allocation mask. For details, please refer to the relevant description in step 203 of the above method.

[0153] (7) The optical flow allocation submodule 3022 sends the first optical flow allocation mask to the optical flow estimation submodule 3023 at any time between frames.

[0154] (8) The optical flow allocation submodule 3022 weights the first optical flow through the first optical flow allocation mask to obtain the target optical flow.

[0155] Please see Figure 9 , Figure 9 A schematic block diagram of an optical flow estimation device 400 provided in an embodiment of this application is shown. The optical flow estimation device 400 may include a processor 401 and a communication interface 402, wherein the processor 401 and the communication interface 402 are coupled.

[0156] The communication interface 402 is used to input image data to the processor 401 and / or output image data from the processor 401; the processor 401 runs a computer program or instructions to enable the optical flow estimation device 400 to implement the optical flow estimation method described in the above-described method 200 embodiments.

[0157] The processor 401 in this embodiment includes, but is not limited to, a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor, a microcontroller, or any conventional processor.

[0158] For example, the processor 401 is used to obtain a first image frame and a second image frame through the communication interface 402, wherein the first image frame and the second image frame are any two adjacent image frames in an image sequence, which is captured by shooting a target scene; obtain a first event frame through the communication interface 402, wherein the first event frame is used to describe the brightness change of the target scene between the first image frame and the second image frame; and determine a target optical flow based on the first image frame, the second image frame and the first event frame, wherein the target optical flow includes at least one of the optical flow of corresponding pixels between the first image frame and the target time and the optical flow of corresponding pixels between the target time and the second image frame, wherein the target time is any time between the first image frame and the second image frame.

[0159] In an alternative example, those skilled in the art will understand that the optical flow estimation device 400 may specifically be the optical flow estimation device in the above-described optical flow estimation method 200 embodiment. The optical flow estimation device 400 may be used to execute the various processes and / or steps corresponding to the optical flow estimation device in the above-described optical flow estimation method 200 embodiment. To avoid repetition, these will not be described again here.

[0160] Optionally, the optical flow estimation device 400 may also include a memory 403.

[0161] Memory 403 can be volatile memory or non-volatile memory, or may include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM).

[0162] Specifically, the memory 403 is used to store the program code and instructions of the optical flow estimation device. Optionally, the memory 403 is also used to store data obtained by the processor 401 during the execution of the above-described optical flow estimation method 200 embodiment, such as the first optical flow, the first optical flow allocation mask, the target optical flow, etc.

[0163] Alternatively, the memory 403 may be a separate device or integrated into the processor 401.

[0164] It should be noted that, Figure 9Only a simplified design of the optical flow estimation device 400 is shown. In practical applications, the optical flow estimation device 400 may also include other necessary components, including but not limited to any number of communication interfaces, processors, controllers, memories, etc., and all optical flow estimation devices 400 that can implement this application are within the protection scope of this application.

[0165] In one possible design, the optical flow estimation device 400 can be a chip. Optionally, the chip may also include one or more memories for storing computer-executable instructions. When the chip device is running, the processor can execute the computer-executable instructions stored in the memory to cause the chip to perform the optical flow estimation method described above.

[0166] Optionally, the chip device can be a field-programmable gate array, a dedicated integrated circuit, a system-on-a-chip, a central processing unit, a network processor, a digital signal processing circuit, a microcontroller, or a programmable controller or other integrated chip to implement the relevant functions.

[0167] This application also provides a computer-readable storage medium storing computer instructions that, when executed on a computer, implement the optical flow estimation method described in the above method embodiments.

[0168] This application also provides a computer program product that, when run on a processor, implements the optical flow estimation method described in the above method embodiments.

[0169] This application embodiment also provides a terminal, which includes the above-described optical flow estimation system. Optionally, the terminal may further include a display screen for displaying the target optical flow output by the optical flow estimation system.

[0170] The optical flow estimation device, computer-readable storage medium, computer program product, chip, or terminal provided in the embodiments of this application are all used to execute the corresponding optical flow estimation method provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding optical flow estimation method provided above, and will not be repeated here.

[0171] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0172] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0173] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0174] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0175] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0176] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0177] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0178] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An optical flow estimation method, characterized in that, include: A first image frame and a second image frame are obtained, wherein the first image frame and the second image frame are any two adjacent image frames in an image sequence, and the image sequence is obtained by capturing the target scene. A first event frame is obtained, which is used to describe the brightness change of the target scene during the time period between the first image frame and the second image frame. Based on the first image frame, the second image frame, and the first event frame, a target optical flow is determined, wherein the target optical flow is the optical flow from the first image frame to the target time, and the target time is any time between the first image frame and the second image frame; The method further includes, prior to determining the target optical flow based on the first image frame, the second image frame, and the first event frame: A second event frame is obtained, which is used to describe the brightness change of the target scene during the time period between the first image frame and the target time. The step of determining the target optical flow based on the first image frame, the second image frame, and the first event frame includes: The target optical flow is determined based on the first image frame, the second image frame, the first event frame, and the second event frame.

2. The method according to claim 1, characterized in that, Determining the target optical flow based on the first image frame, the second image frame, the first event frame, and the second event frame includes: Based on the first image frame, the second image frame, and the first event frame, a first optical flow is determined, wherein the first optical flow is the optical flow from the first image frame to the second image frame; Based on the second event frame, a first optical flow allocation mask is determined, which is used to indicate the weight of the target optical flow relative to the first optical flow; The target optical flow is determined based on the first optical flow and the first optical flow allocation mask.

3. The method according to claim 2, characterized in that, Determining the first optical flow based on the first image frame, the second image frame, and the first event frame includes: The first image frame, the second image frame, and the first event frame are input into a preset optical flow estimation model to obtain the first optical flow.

4. The method according to claim 3, characterized in that, The step of inputting the first image frame, the second image frame, and the first event frame into a preset optical flow estimation model to obtain the first optical flow includes: The first image frame, the second image frame, and the first event frame are input into the optical flow estimation model and iterated cyclically to obtain the first optical flow.

5. The method according to any one of claims 2-4, characterized in that, Determining the first optical flow allocation mask based on the second event frame includes: The second event frame is input into a preset optical flow allocation model to obtain the first optical flow allocation mask.

6. The method according to claim 5, characterized in that, The step of inputting the second event frame into a preset optical flow allocation model to obtain the first optical flow allocation mask includes: The second event frame is input into the optical flow allocation model and iterated in a loop to obtain the first optical flow allocation mask.

7. The method according to any one of claims 1-4, characterized in that, The first image frame includes H×W pixels, where H and W are both integers greater than 1. The first event frame includes multiple channels, including a first channel, a second channel, a third channel, and a fourth channel. The first channel includes H×W first values, and the H×W first values ​​correspond one-to-one with the positions of the H×W pixels. The first values ​​are used to indicate the number of times the brightness of the pixel at the corresponding position in the first image frame increases during the time period between the first image frame and the second image frame. The second channel includes H×W second values, each of which corresponds one-to-one with the position of the H×W pixels. The second values ​​are used to indicate the number of times the brightness of the corresponding pixel in the first image frame decreases during the time period between the first image frame and the second image frame. The third channel includes H×W third values, each of which corresponds one-to-one with the position of the H×W pixels. The third values ​​are used to represent the timestamp of the last increase in brightness of the corresponding pixel in the first image frame during the time period between the first image frame and the second image frame. The fourth channel includes H×W fourth values, each of which corresponds one-to-one with the position of one of the H×W pixels. The fourth values ​​are used to represent the timestamp of the last decrease in brightness of the corresponding pixel in the first image frame during the time period between the first image frame and the second image frame.

8. The method according to any one of claims 1-4, characterized in that, Obtaining the first event frame includes: Obtain event stream data, which includes event data for each of at least one event, wherein the at least one event corresponds one-to-one with at least one brightness change that occurs in the target scene between the first image frame and the second image frame, and the data for each event includes a timestamp, pixel coordinates, and polarity; The first event frame is obtained based on the event stream data.

9. An optical flow estimation device, characterized in that, include: Acquisition module and optical flow estimation module; The obtaining module is used to obtain a first image frame and a second image frame, wherein the first image frame and the second image frame are any two adjacent image frames in an image sequence, and the image sequence is obtained by capturing the target scene. A first event frame is obtained, which is used to describe the brightness change of the target scene during the time period between the first image frame and the second image frame. The optical flow estimation module is used to determine the target optical flow based on the first image frame, the second image frame, and the first event frame. The target optical flow is the optical flow from the first image frame to the target time, and the target time is any time between the first image frame and the second image frame. The obtaining module is further configured to obtain a second event frame before determining the target optical flow based on the first image frame, the second image frame, and the first event frame. The second event frame is used to describe the brightness change of the target scene during the time period between the first image frame and the target time. The optical flow estimation module is specifically used to determine the target optical flow based on the first image frame, the second image frame, the first event frame, and the second event frame.

10. The apparatus according to claim 9, characterized in that, The optical flow estimation module includes an inter-frame optical flow estimation submodule, an optical flow allocation submodule, and an inter-frame arbitrary time optical flow estimation submodule; The inter-frame optical flow estimation submodule is used to determine a first optical flow based on the first image frame, the second image frame, and the first event frame, wherein the first optical flow is the optical flow from the first image frame to the second image frame; The optical flow allocation submodule is used to determine a first optical flow allocation mask based on the second event frame, wherein the first optical flow allocation mask is used to indicate the weight of the target optical flow relative to the first optical flow; The inter-frame arbitrary time optical flow estimation submodule is used to determine the target optical flow based on the first optical flow and the first optical flow allocation mask.

11. The apparatus according to claim 10, characterized in that, The inter-frame optical flow estimation submodule is specifically used to input the first image frame, the second image frame, and the first event frame into a preset optical flow estimation model to obtain the first optical flow.

12. The apparatus according to claim 11, characterized in that, The inter-frame optical flow estimation submodule is specifically used to input the first image frame, the second image frame, and the first event frame into the optical flow estimation model and perform iterative iterations to obtain the first optical flow.

13. The apparatus according to any one of claims 10-12, characterized in that, The optical flow allocation submodule is specifically used to input the second event frame into a preset optical flow allocation model to obtain the first optical flow allocation mask.

14. The apparatus according to claim 13, characterized in that, The optical flow allocation submodule is specifically used to input the second event frame into the optical flow allocation model and perform iterative loops to obtain the first optical flow allocation mask.

15. The apparatus according to any one of claims 9-12, characterized in that, The first image frame includes H×W pixels, where H and W are both integers greater than 1. The first event frame includes multiple channels, including a first channel, a second channel, a third channel, and a fourth channel. The first channel includes H×W first values, and the H×W first values ​​correspond one-to-one with the positions of the H×W pixels. The first values ​​are used to indicate the number of times the brightness of the pixel at the corresponding position in the first image frame increases during the time period between the first image frame and the second image frame. The second channel includes H×W second values, each of which corresponds one-to-one with the position of the H×W pixels. The second values ​​are used to indicate the number of times the brightness of the corresponding pixel in the first image frame decreases during the time period between the first image frame and the second image frame. The third channel includes H×W third values, each of which corresponds one-to-one with the position of the H×W pixels. The third values ​​are used to represent the timestamp of the last increase in brightness of the corresponding pixel in the first image frame during the time period between the first image frame and the second image frame. The fourth channel includes H×W fourth values, each of which corresponds one-to-one with the position of one of the H×W pixels. The fourth values ​​are used to represent the timestamp of the last decrease in brightness of the corresponding pixel in the first image frame during the time period between the first image frame and the second image frame.

16. The apparatus according to any one of claims 9-12, characterized in that, The obtaining module is specifically used for: Obtain event stream data, which includes event data for each of at least one event, wherein the at least one event corresponds one-to-one with at least one brightness change that occurs in the target scene between the first image frame and the second image frame, and the data for each event includes a timestamp, pixel coordinates, and polarity; The first event frame is obtained based on the event stream data.

17. An optical flow estimation device, characterized in that, include: A processor and a communication interface, wherein the processor and the communication interface are coupled, the communication interface being used to provide data to the processor, and the processor being used to execute computer program instructions to perform the method of any one of claims 1-8.

18. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the method as described in any one of claims 1-8.

19. A computer program product, characterized in that, When the computer program product is run on a processor, it implements the method as described in any one of claims 1-8.

Images