Ultrahigh-speed imaging method, device and system of event camera

Abstract

The invention discloses an ultrahigh-speed imaging method, device and system of an event camera, and belongs to the technical field of image processing. The method comprises the following steps: acquiring a superpixel segmentation mask of a motion area according to an event trigger position and an image appearance-position feature, selecting event points in the superpixel segmentation area for continuous time point tracking, and calculating an optical flow field with continuous time and dense space by combining a point tracking time sequence track and the spatial superpixel mask. An imaging result of any to-be-imaged moment t between the two frames is calculated by using the optical flow field, and high-precision imaging of an ultra-high-speed motion scene can be realized.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and more specifically, relates to an ultra-high-speed imaging method, apparatus and system for an event camera. Background Technology

[0002] High frame rate imaging has immeasurable value in scientific research, industrial inspection, and security monitoring. However, ultra-high-speed cameras generate a large amount of redundant data (e.g., a typical 1920×1080 resolution, 10,000fps RGB high-speed camera can have a data rate exceeding 100Gb / s), requiring high-performance computing equipment for processing and storage. This increases operational complexity and maintainability, and makes long-term operation difficult. Event cameras, a novel neuromorphic sensor, feature asynchronous triggering of pixels, low latency (microsecond level), low data redundancy, and high dynamic range, making them highly valuable for ultra-high-speed imaging.

[0003] Although event cameras have the potential for ultra-high-speed imaging, they are difficult to capture static scene information due to their triggering nature based on external brightness (usually moving edges in the scene). Therefore, ultra-high-speed reconstruction imaging is often achieved by using event-low frame rate image sequence fusion. One common method is to first use the event information acquired by the event camera to estimate the inter-frame optical flow field, and then interpolate the low frame rate image sequence based on the estimated optical flow field to assist the image imaging process. However, event information is spatially sparse, and it is difficult to accurately estimate the spatially dense and temporally continuous optical flow field using only event information. Incorrect optical flow will directly affect the accuracy of the reconstruction results, thus limiting the quality of ultra-high-speed imaging. Summary of the Invention

[0004] In view of the above-mentioned defects or improvement needs of the prior art, the present invention provides an ultra-high-speed imaging method, device and system for an event camera, the purpose of which is to improve the accuracy of ultra-high-speed imaging by accurately estimating the spatially dense and temporally continuous optical flow field.

[0005] To achieve the above objectives, in a first aspect, the present invention provides an ultra-high-speed imaging method for an event camera, comprising: The system acquires raw image sequences captured by a traditional camera and event sequences captured by an event camera within the same time period and scene, respectively; the output frame rate of the event sensor is higher than that of the traditional camera. Select a first image frame imaged before the imaging time t from the original image sequence, and a second image frame imaged after t. Use t, the first image frame, the second image frame, and the corresponding event sequence as input to the optical flow field acquisition algorithm. Employ the optical flow field acquisition algorithm to obtain the optical flow field F from the imaging time corresponding to the first image frame to time t. t ; Using F t Perform optical flow transformation on the first image frame to obtain the imaging result at time t; The aforementioned optical flow field acquisition algorithm includes: Extract the event stream F between two input image frames I0 and I1 from the input event sequence. e And generate the corresponding event frame; I0 is imaged earlier than I1; For F e The corresponding event frames are sequentially binarized and morphologically closed to obtain the mask of the moving target in the scene; Perform superpixel segmentation on I0, and take the superpixels that overlap with the mask as the motion region; Event stream F e The data is divided into multiple consecutive segments, and event frames corresponding to each consecutive segment are generated; the binarized event frames are then used to form an event frame sequence. Arbitrarily select a location point in each motion region as the target to be tracked, and use the target tracking model to extract the motion trajectory of the target in the event frame sequence as the motion trajectory of each location point in the motion region. Then, obtain the optical flow values ​​of all points in the motion region from the imaging time corresponding to I0 to time t. Thus, construct the optical flow field from the imaging time corresponding to I0 to time t.

[0006] More preferably, the target tracking model is used to obtain a local region centered on the target to be tracked in each event frame of the event frame sequence, and extract the features of the local region; compare the similarity between the features of the local regions in two adjacent event frames in the event frame sequence, connect the targets to be tracked in two event frames with similarity less than a preset value, and then obtain the motion trajectory of the target to be tracked in the event frame sequence.

[0007] More preferably, the above-mentioned ultra-high-speed imaging method further includes: after obtaining the imaging result at time t, further optimizing the obtained imaging result using an image optimization network to obtain the final imaging result.

[0008] More preferably, the target tracking model used in the optical flow field acquisition algorithm is trained in the following way: Obtain the training set; each training sample in the training set includes: a pair of event sequences and a low frame rate image sequence, with the corresponding label being the real high frame rate image sequence; the event sequence and the real high frame rate image sequence corresponding to the same training sample are obtained by capturing scenes containing moving targets in the same time period using an event camera and a high-speed camera that are time-synchronized and spatially registered, respectively; the low frame rate image sequence is obtained by extracting frames from the corresponding real high frame rate image sequence, where the time in the real high frame rate image sequence that is not extracted is recorded as the verification time; For each training sample, one or more verification times are selected from the corresponding real high frame rate image sequence, and the imaging result of the training sample at each selected verification time is obtained; wherein, any verification time t of any training sample S c The imaging results are obtained by selecting a low frame rate image sequence from S at time t. c The previously imaged image frame I0', and a frame in t c The image frame I1' that was subsequently imaged; with t c The event sequences in I0', I1', and S serve as inputs to the optical flow field acquisition algorithm. This algorithm acquires the data from the imaging time corresponding to I0' to t. c The optical flow field F at time t tc ; Using F tc Perform optical flow transformation on I0' to obtain the training sample S at t c Imaging results at that moment; The target tracking model used in the optical flow field acquisition algorithm is trained by minimizing the difference loss between the imaging result of each training sample at each selected verification time and the real image frame at the selected verification time in the corresponding real high frame rate image sequence.

[0009] More preferably, the target's motion pattern includes nonlinear motion.

[0010] In a second aspect, the present invention provides an ultra-high-speed imaging device for an event camera, comprising: The data acquisition module is used to acquire the raw image sequence captured by the traditional camera and the event sequence captured by the event camera in the same time period and scene; the output frame rate of the event sensor is higher than that of the traditional camera. The optical flow field acquisition module selects a first image frame imaged before the imaging time t and a second image frame imaged after t from the original image sequence. Using t, the first image frame, the second image frame, and the corresponding event sequence as input to the optical flow field acquisition algorithm, the algorithm acquires the optical flow field F from the imaging time corresponding to the first image frame to time t. t ; Imaging module, used for employing F t Perform optical flow transformation on the first image frame to obtain the imaging result at time t; The optical flow field acquisition algorithm used in the optical flow field acquisition module includes: Extract the event stream F between two input image frames I0 and I0 from the input event sequence. e And generate the corresponding event frame; I0 is imaged earlier than I0; For F eThe corresponding event frames are sequentially binarized and morphologically closed to obtain the mask of the moving target in the scene; Perform superpixel segmentation on I0, and take the superpixels that overlap with the mask as the motion region; Event stream F e The data is divided into multiple consecutive segments, and event frames corresponding to each consecutive segment are generated; the binarized event frames are then used to form an event frame sequence. Arbitrarily select a location point in each motion region as the target to be tracked, and use the target tracking model to extract the motion trajectory of the target in the event frame sequence as the motion trajectory of each location point in the motion region. Then, obtain the optical flow values ​​of all points in the motion region from the imaging time corresponding to I0 to time t. Thus, construct the optical flow field from the imaging time corresponding to I0 to time t.

[0011] More preferably, the above-mentioned ultra-high-speed imaging device further includes: an image optimization module, used to optimize the obtained imaging results using an image optimization network to obtain the final imaging result.

[0012] Thirdly, the present invention provides an ultra-high-speed imaging system for an event camera, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the method provided in the first aspect of the present invention.

[0013] Fourthly, the present invention also provides a computer-readable storage medium comprising a stored computer program, wherein the computer program, when executed by a processor, controls the device in which the storage medium is located to perform the method provided in the first aspect of the present invention.

[0014] Fifthly, the invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the method provided in the first aspect of the invention.

[0015] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects: 1. This invention provides an ultra-high-speed imaging method for an event camera. Based on the event triggering position and the image appearance-position features, a superpixel segmentation mask for the motion region is obtained. Event points are selected in the superpixel segmentation region for continuous time point tracking. By combining the point tracking time sequence trajectory and the spatial superpixel mask, a continuous time and spatially dense optical flow field is calculated. The optical flow field is used to calculate the imaging result at any time t between two frames, which can achieve high-precision imaging of ultra-high-speed motion scenes.

[0016] 2. Furthermore, the ultra-high-speed imaging method provided by the present invention further includes: after obtaining the imaging result at time t, further optimizing the obtained imaging result using an image optimization network, which can obtain a more refined reconstructed image and further improve the accuracy of ultra-high-speed imaging.

[0017] 3. Furthermore, the ultra-high-speed imaging method provided by this invention uses a target tracking model in the optical flow field acquisition algorithm that is trained based on a corresponding training set constructed according to the task type itself, which can further improve the ultra-high-speed imaging effect of the event camera.

[0018] 4. Furthermore, the ultra-high-speed imaging method provided by this invention uses a training set for the target tracking model in the training optical flow field acquisition algorithm, which includes training samples collected from targets under complex nonlinear motion modes. This enables the trained target tracking model to learn the features of such complex motion modes, improves the accuracy of target tracking in such special scenarios, and further enhances the accuracy of ultra-high-speed imaging in such special scenarios. Attached Figure Description

[0019] Figure 1 This is a flowchart of the ultra-high-speed imaging method for an event camera provided in Embodiment 1 of the present invention.

[0020] Figure 2 This is a schematic diagram illustrating the process of obtaining the mask of a moving target in a scenario provided in Embodiment 1 of the present invention.

[0021] Figure 3 This is a schematic diagram of image superpixel segmentation provided in Embodiment 1 of the present invention.

[0022] Figure 4 This is a schematic diagram of an event-high-speed camera coaxial imaging system provided in Embodiment 1 of the present invention.

[0023] Figure 5 This is a schematic diagram of the projection process provided in Embodiment 1 of the present invention.

[0024] Figure 6 This is a schematic diagram of the continuous time-point tracking, optical flow densification, and optical flow optimization results provided in Embodiment 1 of the present invention.

[0025] Figure 7 This is a flowchart of constructing a low frame rate image sequence into an ultra-high frame rate image sequence, as provided in Embodiment 1 of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0027] Example 1 This embodiment provides an ultra-high-speed imaging method for an event camera, such as... Figure 1 As shown, it includes: The system acquires raw image sequences captured by a traditional camera and event sequences captured by an event camera within the same time period and scene, respectively; the output frame rate of the event sensor is higher than that of the traditional camera. Select a first image frame imaged before the imaging time t from the original image sequence, and a second image frame imaged after t. Use t, the first image frame, the second image frame, and the corresponding event sequence as input to the optical flow field acquisition algorithm. Employ the optical flow field acquisition algorithm to obtain the optical flow field F from the imaging time corresponding to the first image frame to time t. t ; Using F t Perform optical flow transformation on the first image frame to obtain the imaging result at time t; The aforementioned optical flow field acquisition algorithm includes: Extract the event stream F between two input image frames I0 and I1 from the input event sequence. e And generate the corresponding event frame; I0 is imaged earlier than I1; For F e The corresponding event frames are sequentially binarized and morphologically closed to obtain the mask of the moving target in the scene; I0 is segmented into superpixels, and superpixels that overlap with the mask are taken as moving regions, while superpixels that do not overlap with the mask are taken as stationary regions. Event stream F e The data is divided into multiple consecutive segments, and event frames corresponding to each consecutive segment are generated; the binarized event frames are then used to form an event frame sequence. Arbitrarily select a location point in each motion region as the target to be tracked, and use a target tracking model to extract the motion trajectory of the target in the event frame sequence as the motion trajectory of each location point in the motion region. Then, obtain the optical flow value of all points in the motion region from the imaging time corresponding to I0 to time t. Thus, construct the optical flow field from the imaging time corresponding to I0 to time t. Among them, the optical flow value of all points in the stationary region from the imaging time corresponding to I0 to time t is 0.

[0028] Images are spatially dense and temporally sparse, while events are spatially sparse and temporally continuous. Images also contain rich appearance information, and similar appearances mean similar motion. Superpixel partitioning can make superpixel blocks similar within each block but significantly different between them. By combining this with event point tracking, the advantages of each modality can be utilized to estimate the temporally continuous and spatially dense optical flow field.

[0029] This invention utilizes the high temporal resolution information provided by event streams and the dense appearance information provided by images to estimate spatially dense and temporally continuous optical flow between frames, thereby improving the accuracy of ultra-high-speed imaging. Based on the event triggering location and image appearance-position features, a superpixel segmentation mask for the moving region is obtained. Event points are selected within the superpixel segmentation region for continuous time-point tracking. Combining the point tracking temporal trajectory and the spatial superpixel mask, a continuous temporal and spatially dense optical flow field is calculated. The optical flow field is then used to calculate the imaging result at the time t between two frames, achieving high-precision imaging of ultra-high-speed motion scenes.

[0030] By using event streams and low-speed image sequences as input to reconstruct ultra-high-speed image sequences, the amount of data is reduced by more than 100 times compared to traditional ultra-high-speed cameras, which greatly reduces the data transmission and storage costs required for high-speed imaging, and also has the ability to work continuously for a long time.

[0031] This invention fully utilizes the high temporal resolution of event streams and the dense appearance information provided by images. It accurately estimates the inter-frame optical flow field through superpixel segmentation and event point tracking sets, and further optimizes the optical flow field through photometric loss. While possessing physical interpretability, it significantly improves the generalization and stability of the algorithm in different application scenarios. It avoids the problems of existing learning-based ultra-high-speed imaging methods that directly treat machine learning or deep learning networks as black boxes, lack clear physical meaning, and are heavily dependent on training data, resulting in limited ultra-high-speed imaging quality.

[0032] In this embodiment, the event stream F e = E i (x i ,y i ,t i ,p i An event frame (the size of the event frame is the same as the image size) is generated according to the event trigger location. The value of the event frame is 1 for locations where an event exists, and 0 otherwise. Then, a morphological closing operation is performed using a 5×5 kernel to delete isolated pixels and connect isolated pixels in the region without event triggers, forming a motion region mask M (i.e., the mask of moving objects in the scene). The entire process is as follows: Figure 2As shown, the number of superpixel segmentation regions is calculated based on the image's length H and width W, where H = (H / 30) × (W / 30). The image is divided into a 30×30 grid, and the center point of each grid is used as the initial cluster center. Superpixel clustering is performed using pixel value and pixel position as inputs to achieve superpixel segmentation, resulting in a region segmentation map S (i.e., superpixels), as shown. Figure 3 As shown.

[0033] Finally, the region where the segmentation map S intersects with the motion region mask M is taken as the motion region, which is the target template T to be tracked.

[0034] The event stream F will be based on the movement speed of the targets in the scene. e The event is divided into multiple continuous segments (e.g., a continuous event stream with a 1-second interval is divided into 1000 1-ms continuous event segments). An event is selected as the tracking point in the motion region. Continuous point tracking is performed based on the apparent similarity between adjacent event segments to obtain the continuous motion trajectory of the selected point in the tracking points between the time periods corresponding to image frames I0 and I1. Then, the optical flow values ​​of all points in the motion region from the imaging time corresponding to I0 to the imaging time t are obtained. Thus, the optical flow field from the imaging time corresponding to I0 to time t is constructed.

[0035] The point tracking results are used to assign values ​​to the motion region to which the tracking point belongs. The continuous motion trajectory of a point in time is regarded as the motion trajectory of its motion region. For the entire region where no event occurs (the above-mentioned static region), the value is directly assigned to 0. The above steps can obtain the spatial dense optical flow field, which is the process of optical flow field densification.

[0036] It should be noted that there are various target tracking models, such as the SIFT model, ORB model, CoTracker model, and TAPIR model. In one optional implementation, the target tracking model is used to acquire a local region centered on the target to be tracked in each event frame of the event frame sequence (the shape of the local region is not limited and can be circular, rectangular, square, etc. When the shape is circular, the local region is a circular region centered on the target to be tracked with a radius of a preset length; when the shape is rectangular, the local region is a rectangular region centered on the target to be tracked, and the center distance to the horizontal boundary and the vertical boundary of the region are respectively a first preset distance and a second preset distance; when the shape is square, the local region is a square region centered on the target to be tracked, and the center distance to the boundary is a preset distance). Features of this local region are extracted; the similarity between the features of the local regions in two event frames in the event frame sequence is compared, and the targets to be tracked in two event frames with a similarity less than a preset value (0.1 in one optional implementation) are connected to obtain the motion trajectory of the target to be tracked in the event frame sequence.

[0037] It should be noted that the target tracking model used in the optical flow field acquisition algorithm can be an existing pre-trained target tracking model; however, to further improve the ultra-high-speed imaging effect of the event camera, it is preferable to train it according to the task type itself. Preferably, in an optional implementation, the target tracking model used in the optical flow field acquisition algorithm is trained in the following way: Obtain the training set; each training sample in the training set includes: a pair of event sequences and a low frame rate image sequence, with the corresponding label being a real high frame rate image sequence; the event sequence and the real high frame rate image sequence corresponding to the same training sample are obtained by capturing a scene containing moving targets in the same time period using a time-synchronized and spatially registered event camera and a high-speed camera, respectively; the low frame rate image sequence is obtained by extracting frames from the corresponding real high frame rate image sequence, wherein the time in the real high frame rate image sequence that is not extracted is recorded as the verification time; preferably, the motion mode of the target includes nonlinear motion; For each training sample, one or more verification times are selected from the corresponding real high frame rate image sequence, and the imaging result of the training sample at each selected verification time is obtained; wherein, any verification time t of any training sample S c The imaging results are obtained by selecting a low frame rate image sequence from S at time t. c The previously imaged image frame I0', and a frame in t c The image frame I1' that was subsequently imaged; with t c The event sequences in I0', I1', and S serve as inputs to the optical flow field acquisition algorithm. This algorithm acquires the data from the imaging time corresponding to I0' to t. c The optical flow field F at time t tc ; Using F tc Perform optical flow transformation on I0' to obtain the training sample S at t c Imaging results at that moment; The target tracking model used in the optical flow field acquisition algorithm is trained by minimizing the difference loss between the imaging result of each training sample at each selected verification time and the real image frame at the selected verification time in the corresponding real high frame rate image sequence.

[0038] It should be noted that the training set can be an existing training set that has been pre-collected and used directly in this embodiment, or it can be collected by building an imaging system. This embodiment builds a coaxial imaging system for collecting paired data of events and high frame rate images, and collects event-image paired datasets that are synchronized and spatially registered.

[0039] In this embodiment, the following was constructed: Figure 4The event-high-speed camera coaxial imaging system shown includes an event camera, a high frame rate camera, a beam splitter cube, a non-polarizing beam splitter (supporting multiple bands such as visible light, near infrared, short-wave infrared, mid-wave infrared, and long-wave infrared), an imaging lens, a synchronization circuit, a synchronization cable, and a data connection line. The beam splitter evenly splits the incident light into two beams in a 50:50 ratio, which then pass through two imaging lenses to reach the event camera and the high-speed camera respectively. The incident light is split into two equal parts (50:50) by the flat beam-splitting mirror in the beam-splitting cube. These parts then pass through identical optical lenses to the event camera and high-speed camera, respectively. A synchronization circuit generates a high-frequency PWM signal to achieve precise time synchronization between the two cameras. By acquiring data from a checkerboard calibration board, the event is reconstructed into a grayscale image. The checkerboard corner points in both the reconstructed event image and the high-speed camera image are detected to calibrate the intrinsic and extrinsic parameters of the two cameras. Based on these parameters, the image planes of the two cameras are then projected onto the same plane in three-dimensional space. The projection process is as follows: Figure 5 As shown, due to the small baseline of the co-optical system, the parallax between the two image planes is almost zero, thus achieving pixel-level registration between the two cameras. Based on this, a time-synchronized and spatially registered event-high frame rate image pair dataset can be obtained. The high frame rate image sequence acquired by the high-speed camera serves as the ground truth, while the image input to the imaging algorithm is a low frame rate image sequence extracted from the original sequence.

[0040] The aforementioned event-image dual-mode coaxial imaging system can provide time-synchronized, spatially pixel-level aligned event-high frame-rate image pairs. The image data acquired by the ultra-high-speed camera is used as the ground truth to provide a supervisory signal for the network, which can effectively improve imaging quality and further enhance the performance of downstream application tasks.

[0041] Preferably, the image obtained after optical flow transformation is further optimized. In an optional implementation, the above-mentioned ultra-high-speed imaging method further includes: after obtaining the imaging result at time t, further optimizing the obtained imaging result using an image optimization network (such as the U-Net model) to obtain the final imaging result.

[0042] It should be noted that the image optimization network can be an existing pre-trained image optimization network, which can be used directly here.

[0043] Preferably, to further improve the ultra-high-speed imaging effect of the event camera, training is preferably performed according to the task type itself. The image optimization network used in this embodiment is a typical U-Net structure. After the image to be optimized is input into the network, the difference loss between the final imaging result of each training sample at each selected verification time and the corresponding real image frame at that selected verification time in the real high frame rate image sequence is minimized. Simultaneously, the target tracking model and the image optimization network are trained. The loss function is defined as follows: , where I t 'I' is the final imaging result output by the image optimization network after optimizing the imaging results at a certain time t. t This is the actual image frame at time t. Through the above steps, high-precision imaging of ultra-high-speed motion scenes can be achieved. A schematic diagram of the continuous time-point tracking, optical flow densification, and optical flow optimization results in this embodiment is shown below. Figure 6 As shown.

[0044] It should be noted that the ultra-high-speed imaging method for event cameras provided by this invention is not only applicable to ultra-high-speed imaging at any given time, but also applicable to constructing ultra-high frame rate image sequences from low frame rate image sequences. The specific process is as follows: Figure 7 As shown. Specifically, in one optional implementation, the image can be transformed to any time using a continuous optical flow field between frames, and then a finely reconstructed image is obtained using an image optimization network, thereby achieving high frame rate and high precision imaging of ultra-high speed scenes using low data rate event-low frame rate image sequences.

[0045] Example 2 This embodiment provides an ultra-high-speed imaging device for an event camera, including: The data acquisition module is used to acquire the raw image sequence captured by the traditional camera and the event sequence captured by the event camera in the same time period and scene; the output frame rate of the event sensor is higher than that of the traditional camera. The optical flow field acquisition module selects a first image frame imaged before the imaging time t and a second image frame imaged after t from the original image sequence. Using t, the first image frame, the second image frame, and the corresponding event sequence as input to the optical flow field acquisition algorithm, the algorithm acquires the optical flow field F from the imaging time corresponding to the first image frame to time t. t ; Imaging module, used for employing F t Perform optical flow transformation on the first image frame to obtain the imaging result at time t; The optical flow field acquisition algorithm used in the optical flow field acquisition module includes: Extract the event stream F between two input image frames I0 and I0 from the input event sequence. e And generate the corresponding event frame; I0 is imaged earlier than I0; For F e The corresponding event frames are sequentially binarized and morphologically closed to obtain the mask of the moving target in the scene; Perform superpixel segmentation on I0, and take the superpixels that overlap with the mask as the motion region; Event stream F e The data is divided into multiple consecutive segments, and event frames corresponding to each consecutive segment are generated; the binarized event frames are then used to form an event frame sequence. Arbitrarily select a location point in each motion region as the target to be tracked, and use the target tracking model to extract the motion trajectory of the target in the event frame sequence as the motion trajectory of each location point in the motion region. Then, obtain the optical flow values ​​of all points in the motion region from the imaging time corresponding to I0 to time t. Thus, construct the optical flow field from the imaging time corresponding to I0 to time t.

[0046] In one optional implementation, the above-mentioned ultra-high-speed imaging device further includes: an image optimization module, used to optimize the obtained imaging results using an image optimization network to obtain the final imaging result.

[0047] The related technical solutions are the same as the ultra-high-speed imaging method for event cameras provided in the first aspect of this invention, and will not be described in detail here.

[0048] Thirdly, the present invention provides an ultra-high-speed imaging system for an event camera, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the method provided in the first aspect of the present invention.

[0049] The related technical solutions are the same as the ultra-high-speed imaging method for event cameras provided in the first aspect of this invention, and will not be described in detail here.

[0050] Fourthly, the present invention also provides a computer-readable storage medium comprising a stored computer program, wherein the computer program, when executed by a processor, controls the device in which the storage medium is located to perform the method provided in the first aspect of the present invention.

[0051] The related technical solutions are the same as the ultra-high-speed imaging method for event cameras provided in the first aspect of this invention, and will not be described in detail here.

[0052] Fifthly, the invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the method provided in the first aspect of the invention.

[0053] The related technical solutions are the same as the ultra-high-speed imaging method for event cameras provided in the first aspect of this invention, and will not be described in detail here.

[0054] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for ultra-high-speed imaging with an event camera, characterized in that, include: Acquire the raw image sequence captured by a traditional camera and the event sequence captured by an event camera in the same scene and time period. The event sensor has a higher output frame rate than a traditional camera. Select a first image frame imaged before the imaging time t from the original image sequence, and a second image frame imaged after t. Use t, the first image frame, the second image frame, and the corresponding event sequence as input to the optical flow field acquisition algorithm. Employ the optical flow field acquisition algorithm to obtain the optical flow field F from the imaging time corresponding to the first image frame to time t. t ; Using F t Perform optical flow transformation on the first image frame to obtain the imaging result at time t; The optical flow field acquisition algorithm includes: Extract the event stream F between two input image frames I0 and I1 from the input event sequence. e And generate the corresponding event frame; I0 is imaged earlier than I1; For F e The corresponding event frames are sequentially binarized and morphologically closed to obtain the mask of the moving target in the scene; Perform superpixel segmentation on I0, and take the superpixels that overlap with the mask as the motion region; Event stream F e The data is divided into multiple consecutive segments, and event frames corresponding to each consecutive segment are generated; the binarized event frames are then used to form an event frame sequence. Arbitrarily select a location point in each motion region as the target to be tracked, and use the target tracking model to extract the motion trajectory of the target in the event frame sequence as the motion trajectory of each location point in the motion region. Then, obtain the optical flow values ​​of all points in the motion region from the imaging time corresponding to I0 to time t. Thus, construct the optical flow field from the imaging time corresponding to I0 to time t.

2. The ultra-high-speed imaging method according to claim 1, characterized in that, The target tracking model is used to obtain a local region centered on the target to be tracked in each event frame of the event frame sequence, and extract the features of the local region; compare the similarity between the features of the local regions in two adjacent event frames in the event frame sequence, connect the targets to be tracked in two event frames with similarity less than a preset value, and then obtain the motion trajectory of the target to be tracked in the event frame sequence.

3. The ultra-high-speed imaging method according to claim 1, characterized in that, The ultra-high-speed imaging method further includes: after obtaining the imaging result at time t, further optimizing the obtained imaging result using an image optimization network to obtain the final imaging result.

4. The ultra-high-speed imaging method according to any one of claims 1-3, characterized in that, The target tracking model used in the optical flow field acquisition algorithm is trained in the following way: Obtain the training set; each training sample in the training set includes: a pair of event sequences and a low frame rate image sequence, with the corresponding label being a real high frame rate image sequence; the event sequence and the real high frame rate image sequence corresponding to the same training sample are obtained by capturing a scene containing moving targets in the same time period using an event camera and a high-speed camera that are time-synchronized and spatially registered, respectively; the low frame rate image sequence is obtained by extracting frames from the corresponding real high frame rate image sequence, wherein the time in the real high frame rate image sequence that is not extracted is recorded as the verification time; For each training sample, one or more verification times are selected from the corresponding real high frame rate image sequence, and the imaging result of the training sample at each selected verification time is obtained; wherein, any verification time t of any training sample S c The imaging results are obtained by selecting a low frame rate image sequence from S at time t. c The previously imaged image frame I0', and a frame in t c The image frame I1' that was subsequently imaged; with t c The event sequences in I0', I1', and S serve as inputs to the optical flow field acquisition algorithm. This algorithm acquires the data from the imaging time corresponding to I0' to t. c The optical flow field F at time t tc ; Using F tc Perform optical flow transformation on I0' to obtain the training sample S at t c Imaging results at that moment; The target tracking model used in the optical flow field acquisition algorithm is trained by minimizing the difference loss between the imaging result of each training sample at each selected verification time and the real image frame at the selected verification time in the corresponding real high frame rate image sequence.

5. The ultra-high-speed imaging method according to claim 4, characterized in that, The target's motion patterns include nonlinear motion.

6. A high-speed imaging device for an event camera, characterized in that, include: The data acquisition module is used to acquire the raw image sequence captured by the traditional camera and the event sequence captured by the event camera in the same time period and scene. The event sensor has a higher output frame rate than a traditional camera. The optical flow field acquisition module selects a first image frame imaged before the imaging time t and a second image frame imaged after t from the original image sequence. Using t, the first image frame, the second image frame, and the corresponding event sequence as input to the optical flow field acquisition algorithm, the algorithm acquires the optical flow field F from the imaging time corresponding to the first image frame to time t. t ; Imaging module, used for employing F t Perform optical flow transformation on the first image frame to obtain the imaging result at time t; The optical flow field acquisition algorithm used by the optical flow field acquisition module includes: Extract the event stream F between two input image frames I0 and I0 from the input event sequence. e And generate the corresponding event frame; I0 is imaged earlier than I0; For F e The corresponding event frames are sequentially binarized and morphologically closed to obtain the mask of the moving target in the scene; Perform superpixel segmentation on I0, and take the superpixels that overlap with the mask as the motion region; Event stream F e The data is divided into multiple consecutive segments, and event frames corresponding to each consecutive segment are generated; the binarized event frames are then used to form an event frame sequence. Arbitrarily select a location point in each motion region as the target to be tracked, and use the target tracking model to extract the motion trajectory of the target in the event frame sequence as the motion trajectory of each location point in the motion region. Then, obtain the optical flow values ​​of all points in the motion region from the imaging time corresponding to I0 to time t. Thus, construct the optical flow field from the imaging time corresponding to I0 to time t.

7. The ultra-high-speed imaging device for an event camera according to claim 6, characterized in that, Also includes: The image optimization module is used to optimize the obtained imaging results using an image optimization network to obtain the final imaging result.

8. A high-speed imaging system for an event camera, characterized in that, include: A memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the method according to any one of claims 1-5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed by a processor, it controls the device on which the storage medium is located to perform the method according to any one of claims 1-5.

10. A computer program product, characterized in that, Includes a computer program / instruction that, when executed by a processor, implements the method described in any one of claims 1-5.

Images