A method and system for integrating event camera datasets with real motion

Abstract

The application relates to the field of image processing and computer vision, and discloses an event camera dataset integration method and system matching real motion, which comprises the following steps: S1, constructing an image dataset; S2, predicting forward and backward optical flow; S3, obtaining a depth map; S4, taking the depth map as a weight to synthesize a real image frame of real motion at an intermediate time; S5, continuously updating parameters to synthesize multiple real image frames at different intermediate times and form a video segment; and S6, taking all adjacent real image frames from the video segment, calculating descriptors of all triggered events, and generating an event camera dataset. The system comprises a dataset construction module, an optical flow prediction module, a depth map acquisition module, a real image frame acquisition module, an image frame synthesis module and an event camera dataset generation module. The application is used for solving the problems of insufficient training data of the current event camera and difficult acquisition of optical flow labels.

Description

Technical Field

[0001] This invention relates to the fields of image processing and computer vision, and in particular to a method and system for integrating event camera data to match real motion, which solves the data problems such as insufficient training data for current event cameras and difficulty in collecting optical flow labels. Background Technology

[0002] Event cameras are a new type of biomimetic vision sensor, whose working mechanism is more similar to that of the human eye, making them widely known. Unlike the working mechanism and output method of traditional frame-based cameras, the sensor pixels of event cameras can individually detect logarithmic changes in light intensity and output event information including position, time, and polarity when the change exceeds a certain threshold. They have the advantages of low latency, high dynamic range, and low power consumption. Their unique output method and working characteristics make them particularly suitable for occasions with high-speed movement, large changes in lighting conditions, or low energy consumption.

[0003] However, compared to traditional cameras, event cameras have shorter development histories, higher operating costs, and more complex data processing. Currently, the number of publicly available event camera datasets is still far smaller than that of image datasets. With the increasing commercial application of event cameras and the continuous development of deep learning algorithms for processing event camera data, the need to construct large-scale event camera datasets for training is becoming increasingly urgent. Therefore, a method for generating corresponding event camera data based on existing image datasets is of great significance for the research and application of event cameras.

[0004] In cited document 1, "Video to Events: Recycling video datasets for event cameras. [C]. Gehrig D., Gehrig M., Hidalgo-Carrio J., and Scaramuzza D., In 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3583–3592, 2020," Gehrig et al. proposed a method to generate synthetic event camera data from synthetic video or image datasets rendered by a renderer and apply it to tasks of event camera classification and semantic segmentation. The training method of pre-training with synthetic event camera data and then fine-tuning with real data improves the performance of the task model. In cited document 2, “v2e:From Video Frames to Realistic DVS Events.[C].Hu Y.,Liu S.-C.,and Delbruck T.,In Proceedings of the 2021IEEE / CVF Conference on Computer Vision and Pattern Recognition,pages 1312–1321,2021.”, Hu et al. considered the characteristics of motion blur and slower event generation speed in real event camera data due to differences in the sharpness of object edges under different lighting conditions when synthesizing event camera data, making the synthesized event camera data closer to reality.

[0005] The methods mentioned above all use frame interpolation to improve the temporal resolution of images and videos to synthesize event camera data. However, the motion process estimated by frame interpolation is difficult to match with the real motion of the original image dataset, resulting in differences between the synthesized event camera data and the real data.

[0006] This invention provides a method and system for integrating event camera data to match real motion, which solves the data problems such as insufficient training data for current event cameras and difficulty in collecting optical flow labels. Summary of the Invention

[0007] This invention provides a method and system for integrating event camera data to match real motion, which solves the data problems such as insufficient training data for current event cameras and difficulty in collecting optical flow labels.

[0008] This invention is achieved through the following technical solution: a method for integrating event camera data to match real motion, comprising the following steps:

[0009] Step S1: Construct a dataset of images showing real motion.

[0010] Step S2: Select an optical flow prediction network, select image pairs from the image dataset, and input them into the optical flow prediction network to predict forward and backward optical flow.

[0011] Step S3: Using a monocular depth estimation network, select image pairs from the image dataset and input them into the monocular depth estimation network to obtain depth maps;

[0012] Step S4: Set the parameters for the intermediate time point, use the depth map as weights, and input the forward optical flow and backward optical flow into the bidirectional image fusion module to synthesize the image frame of the real motion at the intermediate time point;

[0013] Step S5: Repeat step S4, continuously iterate and update the parameters of the intermediate time, synthesize multiple real image frames of different intermediate times, and form a set of high temporal resolution video clips.

[0014] Step S6: Extract all adjacent pairs of real image frames from the video segment in chronological order, calculate the descriptors of all triggering events between adjacent real image frames, and generate an event camera dataset.

[0015] To better implement the present invention, step S1 further includes:

[0016] Collect a large number of consecutive video frames of real motion to construct a real motion image dataset D. I The image dataset D I It is a sequence set consisting of multiple image pairs. Each image pair includes two frames (I0, I1) before and after the actual motion is generated, where I0 represents the image before the displacement and I1 represents the image after the displacement. The image size is W×H.

[0017] To better realize the present invention, step S2 further includes:

[0018] Construct an optical flow prediction network Θ from the image dataset D I Extract a pair of images The input is fed into the optical flow prediction network Θ to obtain the forward optical flow F. 0→1 =Θ(I0,I1) and backward optical flow F 1→0 =Θ(I1,I0).

[0019] To better realize the present invention, step S3 further includes:

[0020] Using a monocular depth estimation network Φ, from the image dataset D I Extract an image pair The data are sequentially input into the monocular depth estimation network Φ to obtain depth maps D0 = Φ(I0) and D1 = Φ(I1).

[0021] To better realize the present invention, step S4 further includes:

[0022] A piecewise linear fitting algorithm is employed, using a small displacement optical flow F 0→t The mapped vector is obtained by mapping the image I0 before displacement to the intermediate time t.

[0023] Using micro-displacement optical flow F 1→t The mapped vector is obtained by mapping the displacement image I1 to the intermediate time t.

[0024] The two mapping vectors before and after and V1 t Together with the depth maps D0 = Φ(I0) and D1 = Φ(I1), the data is input into the bidirectional image fusion module to generate the real image frame I at intermediate time t. t .

[0025] To better implement the present invention, the bidirectional image fusion module further generates an image frame I at intermediate time t. t The methods include:

[0026] Plot the image frames at the midpoint of time t in the time directions 0→t and 1→t. and

[0027] Use the snowball method to map vectors The floating-point coordinates in the image are interpolated to integer coordinates on the image frame, and the depth map D0 is used as the weights to adjust the mapping vector. By summing the weighted values ​​of multiple pixels pointing to the same position, an image frame representing the midpoint t in the time direction from 0 to t is plotted.

[0028] Use the snowball method to map vectors The floating-point coordinates in the image are interpolated to integer coordinates on the image frame, and the depth map D1 is used as weights to adjust the mapping vector V1. t By summing the weighted values ​​of multiple pixels pointing to the same position, an image frame representing the midpoint t along the time path from 1 to t is plotted.

[0029] Fuse the image frames and Synthesize the real image frame I at intermediate time t t .

[0030] To better realize the present invention, the image frames are further fused. and Synthesize the real image frame I at intermediate time t t The methods include:

[0031] The image frame was detected using a hole region detection method. The empty areas in the image are output as a binary image, where 0 represents an empty pixel and 1 represents a normal pixel.

[0032] Then the image frame The corresponding pixels are filled into the image frame. The hollow region is finally output as the true image frame I at the intermediate time t. t .

[0033] To better implement the present invention, step S5 further includes:

[0034] The parameters t at the intermediate time points are continuously updated iteratively to synthesize K. i Image frames at different intermediate times I t To form a set of high temporal resolution video clips

[0035] To better realize the present invention, further, from video clip V I Extract two adjacent real image frames I s and I s+1 Record the corresponding time parameter as t. s and t s+1 Set the event trigger reference time matrix T ref Initialize to t s The descriptor of the event is (p,t,σ), where p represents the image pixel position, t represents the time when the event occurs, and σ = ±1 represents the direction of the image pixel brightness change, +1 represents the brightening direction, and -1 represents the darkening direction.

[0036] For the real image frame I s and I s+1 Taking the logarithm gives L s and L s+1 A trigger threshold C is manually set, when the real image frame I... s and I s+1 An event is considered to have been triggered at a pixel location p if the intensity difference exceeds a threshold C. The descriptive factors p and σ are recorded, and the triggering time tk of the event is calculated based on the assumption of uniform brightness variation. An event triggering reference time matrix T is then set for the current location. ref (p) = tk; Traverse the image dataset D of the real motion. IFor the image pairs in the video segment V, repeat steps S2 to S6, in chronological order. I Extract all adjacent pairs of real image frames sequentially, calculate the descriptors of all trigger events between adjacent real image frames, arrange them in ascending order of trigger time, synthesize the event camera data E, organize them according to image pairs, and output the synthesized event camera dataset D. E .

[0037] This invention also provides an event camera data integration system for matching real motion, comprising a dataset construction module, an optical flow prediction module, a depth map acquisition module, a real image frame acquisition module, an image frame synthesis module, and an event camera dataset generation module, wherein:

[0038] The dataset construction module is used to construct image datasets of real motion.

[0039] An optical flow prediction module is used to select an optical flow prediction network and input image pairs from the image dataset into the optical flow prediction network to predict forward and backward optical flow.

[0040] The depth map acquisition module is used to construct a monocular depth estimation network and select image pairs from the image dataset to input into the monocular depth estimation network to obtain depth maps.

[0041] The intermediate moment image frame synthesis module is used to continuously iterate and update the intermediate moment parameters, and use the bidirectional image fusion module to synthesize multiple image frames of real motion at different intermediate moments to form a set of high temporal resolution video clips.

[0042] The event camera dataset generation module is used to extract all adjacent pairs of image frames from the video segment in chronological order, calculate the descriptors of all triggering events between adjacent image frames, and generate the event camera dataset.

[0043] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0044] (1) This invention provides a method for generating corresponding event camera data by simulating real pixel motion based on existing image datasets, which is applicable to all image datasets with continuous frames;

[0045] (2) This invention simulates the real pixel motion of the original image dataset. Based on the principle of piecewise linear fitting, it uses micro-displacement optical flow to map adjacent consecutive image frames of the original image dataset to multiple intermediate times, and then fuses them into images at different intermediate times to improve the temporal resolution of the image video. Finally, it synthesizes event camera data that matches the real motion.

[0046] (3) In view of the “many-to-one” problem caused by occlusion in optical flow mapping, the present invention uses the snowball method to distribute the intensity of each pixel in the original image to the pixels in the surrounding area of ​​the intermediate time image after optical flow mapping according to the proportion of spatial distance, and uses the depth map as weight to increase the proportion of foreground pixels in the intensity of each pixel in the intermediate time image.

[0047] (4) In view of the problem that some areas have holes when drawing intermediate time images due to the lack of corresponding pixel filling, the present invention uses a two-way image fusion method to fill these hole areas. Attached Figure Description

[0048] The present invention will be further described in conjunction with the following drawings and embodiments. All inventive concepts of the present invention should be considered as disclosed content and within the scope of protection of the present invention.

[0049] Figure 1 A flowchart of an event camera data integration method and system for matching real motion, provided in this application embodiment;

[0050] Figure 2 A schematic diagram of the overall framework of an event camera data integration method for matching real motion, provided in an embodiment of this application;

[0051] Figure 3 This application provides an example of an event camera data integration method for matching real motion and some examples of event camera datasets in the system. Detailed Implementation

[0052] To more clearly illustrate the technical solutions of the embodiments of the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments, and therefore should not be regarded as a limitation on the scope of protection. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "set up," "connected," and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0054] Example 1:

[0055] This embodiment presents a method and system for integrating event camera data to match real-world motion, such as... Figure 1 As shown, this invention provides a method for generating corresponding event camera data by simulating real pixel motion based on existing image datasets, which solves the data problems such as insufficient training data for current event cameras and difficulty in collecting optical flow labels.

[0056] Constructing an image dataset: Collect a large number of consecutive video frames to construct an image dataset D. I The dataset is a sequence set of multiple image pairs. Each image pair includes two frames (I0, I1) before and after the motion is generated. I0 represents the image before the displacement and I1 represents the image after the displacement. The image size is W×H.

[0057] This embodiment uses 22,232 image pairs from the publicly available image dataset FlyingChairs in the literature “Flownet: Learning optical flow with convolutional networks.” [C]. Dosovitskiy A., Fischer P., Ilg E., et al., In IEEE Int. Conference on Computer Vision (ICCV), pages 2758-2766, 2015, to construct the image dataset D. I Each image pair consists of two frames (I0, I1) before and after the motion is generated, with an image size of 384×512.

[0058] Example 2:

[0059] This embodiment further optimizes upon embodiment 1 by predicting forward and backward optical flow: An optical flow prediction network Θ is constructed, based on the image dataset D. I Extract a pair of images The input is fed into the optical flow prediction network Θ to obtain the forward optical flow F. 0→1 =Θ(I0,I1) and backward optical flow F 1→0 =Θ(I1,I0).

[0060] This embodiment uses the RAFT proposed in reference 3, "Raft: Recurrent all-pairs field transforms for optical flow." [C]. Zachary T. and Jia D. In European Conference on Computer Vision (ECCV), pages 402-419, 2020, as the optical flow prediction network Θ. After loading the pre-trained weights, the image dataset D is used...I Extract a pair of images The input is fed into the optical flow prediction network Θ to obtain the forward optical flow F. 0→1 =Θ(I0,I1) and backward optical flow F 1→0 =Θ(I1,I0).

[0061] The other parts of this embodiment are the same as those in Embodiment 1, so they will not be described again.

[0062] Example 3:

[0063] This embodiment further optimizes upon the above embodiment 1 or 2, obtaining a depth map: constructing a monocular depth estimation network Φ, and extracting depth maps from the image dataset D. I Extract an image pair The data are sequentially input into the monocular depth estimation network Φ to obtain depth maps D0=Φ(I0) and D1=Φ(I1).

[0064] This embodiment uses the DPT network proposed in cited document 4, “Vision transformers for dense prediction.”[C]. Ranftl, R., Bochkovskiy, A., Koltun, V. In IEEE Int. Conference on Computer Vision (ICCV), pages 12179-12188, 2021, to construct a monocular depth estimation network Φ from the image dataset D. I Extract an image pair The data are sequentially input into the monocular depth estimation network Φ to obtain depth maps D0=Φ(I0) and D1=Φ(I1).

[0065] The other parts of this embodiment are the same as any one of the embodiments 1-2 above, so they will not be described again.

[0066] Example 4:

[0067] This embodiment further optimizes any one of embodiments 1-3 above, and synthesizes image frame I at intermediate time t. t To simulate realistic pixel motion, this invention employs a piecewise linear fitting algorithm. This algorithm uses multiple tiny straight lines to fit the curve function of the real motion. Specifically, this invention uses a tiny displacement optical flow F... 0→t The mapped vector is obtained by mapping the image I0 before displacement to the intermediate time t. Similarly, optical flow F 1→t The shifted image I1 is mapped to the intermediate time t to obtain the mapped vector V1. t Then the two mapping vectors before and after and V1t Together with the depth maps D0 = Φ(I0) and D1 = Φ(I1), the data is input into the bidirectional image fusion module to generate the image frame I at the intermediate time t. t The calculation formulas are shown in formulas (1), (2), and (3), where... B represents optical flow mapping, and B represents the bidirectional image fusion module.

[0068] F 0→t =t·F 0→1 ,F 1→t = (1-t)·F 0→1 (1)

[0069]

[0070]

[0071] The bidirectional image fusion module is used to solve the occlusion problem (foreground and background overlap after displacement) and the hole problem (background is covered by foreground before displacement) caused by object movement. Preferably, the specific operation of the bidirectional image fusion module is as follows:

[0072] Plot the image frames at the midpoint of time t in the time directions 0→t and 1→t. and During optical flow mapping in step S4, due to the movement of the object, the moving part of the object will cover the background of the area that has not been moved after displacement, resulting in multiple pixels pointing to the same position in the mapping, producing a "many-to-one" problem.

[0073] For mapping vectors To address the existing "many-to-one" problem, this invention uses a snowballing method to map vectors. The floating-point coordinates in the image are interpolated to integer coordinates on the image frame, and the depth map D0 is used as the weight to map the vector. The "many-to-one" problem mentioned above is solved by weighted summation of multiple pixels pointing to the same position. Then, the image frame at the midpoint t along the time axis from 0 to t is plotted. The calculation formulas are shown in formulas 4, 5, and 6, where p = (x, y) represents the integer coordinates of pixel I0 in the image before displacement, and q = (x′, y′) represents the mapping vector. The floating-point coordinates of the elements, u = (u x ,u y The value represents the difference between the floating-point coordinates after optical flow mapping and the integer coordinates of the surrounding pixels.

[0074]

[0075] b(u) = max(0, 1 - |u|) x |)·max(0,1-|uy |); (5)

[0076]

[0077] Similarly, given a mapping vector V1 t The image frame at the midpoint t in the time direction from 1 to t is plotted using the depth map D1.

[0078] Fusion image frames and Synthesize the real image frame I at intermediate time t t . Plotting the image frame at intermediate time t and When the foreground and background are covered by the background during displacement, these covered areas lack corresponding image pixels for filling, resulting in the rendering of image frames. and Hollow areas with no pixel values ​​appear.

[0079] To address the aforementioned hole problem, this invention employs a hole region detection method as shown in formula (7) to detect image frames. The image contains hole regions, which are output as a binary image (0 represents a hole, 1 represents a normal pixel). Then, the image frames are... The corresponding pixels are filled into the image frame. The hollow region is finally output as the true image frame I at the intermediate time t. t As shown in formula (8).

[0080]

[0081]

[0082] Repeat step S4 of the method of the present invention, iteratively update the parameter t, and synthesize K. i Image frames at different intermediate times I t To form a set of high temporal resolution video clips Preferably, the number of synthesized image frames K i The choice can be calculated by formula (9) so that the maximum relative displacement of the frame at the intermediate time does not exceed 1 pixel.

[0083]

[0084] According to the definition of an event camera, an event consists of a (p,t,σ) descriptor, where p represents the image pixel position, t represents the time when the event occurs, and σ = ±1 represents the direction of the image pixel brightness change (+1 represents the direction of brightening, and -1 represents the direction of darkening).

[0085] From video clip V I Extract two adjacent real image frames I s and I s+1 Record the corresponding time parameter as t. s and t s+1 Set the event trigger reference time matrix T ref Initialize to t s For real image frame I s and I s+1 Taking the logarithm gives L s and L s+1 (See Formula 10), a trigger threshold C is manually set, when the real image frame I... s and I s+1 An event is considered to have been triggered at a pixel location p if the intensity difference exceeds a threshold C (see Formula 11). The descriptive factors p and σ are recorded, and the triggering time t of the event is calculated based on the assumption of uniform brightness variation. k (See Formula 12) Set the event trigger reference time matrix T for the current position. ref (p)=t k .

[0086] Starting from video clip V in chronological order I All adjacent real image frames are extracted sequentially, and the descriptors (p,t,σ) of all trigger events between adjacent real image frames are calculated. The descriptors are arranged in ascending order of trigger time to synthesize the event camera data dataset E.

[0087] L s =log(I s ),L s+1 =log(I s+1 ),;(10)

[0088] ΔL(p)=L s+1 (p)-L s (p)≥σC;(11)

[0089]

[0090] The other parts of this embodiment are the same as any one of the embodiments 1-3 above, so they will not be described again.

[0091] Example 5:

[0092] This embodiment further optimizes any one of embodiments 1-4 above by traversing the input image dataset D. I For each image pair in the input image dataset, repeat steps S2 to S6 of the method of the present invention to generate event camera data E corresponding to the input image pair. Arrange the images according to the order of the image pairs and output the synthesized event camera dataset D.E .like Figure 3 As shown, some examples of event camera datasets synthesized from the publicly available image dataset FlyingChairs are presented.

[0093] The other parts of this embodiment are the same as any one of the embodiments 1-4 above, so they will not be described again.

[0094] Example 6:

[0095] The present invention also provides an event camera data integration system that matches the method and matches real motion, including a dataset construction module, an optical flow prediction module, a depth map acquisition module, a real image frame acquisition module, an image frame synthesis module, and an event camera dataset generation module.

[0096] Example 7:

[0097] The present invention also provides an electronic device comprising a processor and a memory; the processor includes the event camera data integration system described above for matching real motion.

[0098] The present invention also provides a computer-readable storage medium comprising instructions; when the instructions are executed on the electronic device described in the above embodiments, the electronic device causes the electronic device to perform the methods described in the above embodiments. Optionally, the computer-readable storage medium may be a memory.

[0099] The processor involved in the embodiments of this application can be a chip. For example, it can be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0100] The memory involved in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0101] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0102] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0103] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0104] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or modules may be electrical, mechanical, or other forms.

[0105] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located on one device or distributed across multiple devices. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0106] In addition, the functional modules in the various embodiments of this application can be integrated into one device, or each module can exist physically separately, or two or more modules can be integrated into one device.

[0107] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0108] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications or equivalent changes made to the above embodiments based on the technical essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. A method for integrating event camera dataset matching real motion, characterized in that, Includes the following steps: Step S1: Construct a dataset of images showing real motion. Step S2: Select an optical flow prediction network, select image pairs from the image dataset, and input them into the optical flow prediction network to predict forward and backward optical flow. Step S3: Using a monocular depth estimation network, select image pairs from the image dataset and input them into the monocular depth estimation network to obtain depth maps; Step S4: Set the parameters for the intermediate time point, use the depth map as weights, and input the forward optical flow and backward optical flow into the bidirectional image fusion module to synthesize the image frame of the real motion at the intermediate time point; A piecewise linear fitting algorithm is employed, using micro-displacement optical flow. Image before displacement Mapping to intermediate time Obtain the mapping vector ; Using micro-displacement optical flow The displaced image Mapping to intermediate time Obtain the mapping vector ; The two mapping vectors before and after and Along with depth map and The input is fed into the bidirectional image fusion module to generate intermediate times. Real image frames ; The bidirectional image fusion module generates intermediate moments. Image frames The methods include: drawing and Midpoint in time Image frames and ; Use the snowball method to map vectors The floating-point coordinates in the image are interpolated to integer coordinates on the image frame, and the depth map is used. As weights, for the mapping vector The middle pointer is used to sum the weighted values ​​of multiple pixels at the same location, and then the result is drawn. Midpoint in time direction Image frames ; Use the snowball method to map vectors The floating-point coordinates in the image are interpolated to integer coordinates on the image frame, and the depth map is used. As weights, for the mapping vector The middle pointer is used to sum the weighted values ​​of multiple pixels at the same location, and then the result is drawn. Midpoint in time Image frames ; Fuse the image frames and Synthesis Intermediate Time Real image frames ; Fuse the image frames and Synthesis Intermediate Time Real image frames The method includes: detecting the image frame using a hole region detection method. The empty areas in the image are output as a binary image, where 0 represents an empty pixel and 1 represents a normal pixel. Then the image frame The corresponding pixels are filled into the image frame. The empty region is finally output as the intermediate time. Real image frames ; Step S5: Repeat step S4, continuously iterate and update the parameters of the intermediate time, synthesize multiple real image frames of different intermediate times, and form a set of high temporal resolution video clips. Step S6: Extract all adjacent pairs of real image frames from the video segment in chronological order, calculate the descriptors of all triggering events between adjacent real image frames, and generate an event camera dataset.

2. The method of claim 1, wherein, Step S1 includes: collecting a large number of continuous video frames of real motion to construct a real motion image dataset. The image dataset It is a sequence of multiple image pairs, each image pair consisting of two frames before and after generating the real motion. ,in, This represents the image before the displacement occurred. This represents the image after displacement, with an image size of [size missing]. .

3. The method of claim 1, wherein, Step S2 includes: constructing an optical flow prediction network. From the image dataset Extract a pair of images Input into the optical flow prediction network In the middle, the forward optical flow is obtained. and backward optical flow .

4. The method of claim 1, wherein, Step S3 includes: using a monocular depth estimation network From the image dataset Extract an image pair The data is then sequentially input into the monocular depth estimation network. Depth maps were obtained respectively. and depth map .

5. The method of claim 1, wherein, Step S5 includes: iteratively updating the parameters at the intermediate time points. ,synthesis Image frames at different intermediate times To form a set of high temporal resolution video clips .

6. The method of claim 1, wherein, Step S6 includes: from video clips Extract two adjacent real image frames and Record the corresponding time parameters as follows and Set the event trigger reference time matrix Initialize to The descriptor of the event is ,in, Indicates the position of image pixels. Indicates the time when the event occurred. Indicates the direction of change in image pixel brightness. Indicates the direction of brightening. Indicates the direction of darkening; For the real image frame and Taking the logarithm yields and Manually set trigger thresholds When the real image frame and a certain pixel position The intensity difference exceeds the threshold This position can be considered as An event was triggered, and the descriptor was recorded. and The trigger time of the event is calculated based on the assumption of uniform brightness variation. k Set the event trigger reference time matrix for the current location. k ; Traverse the image dataset of the real motion For the image pairs in the video clip, repeat steps S2 to S6, and extract the images in chronological order from the video clips. The process involves sequentially extracting all adjacent pairs of real image frames, calculating the descriptors of all trigger events between adjacent real image frames, arranging them in ascending order of trigger time, and then synthesizing the event camera data. Organize the images according to their order and output the synthesized event camera dataset. .

7. An event camera data integration system for matching real motion, characterized in that, It includes a dataset construction module, an optical flow prediction module, a depth map acquisition module, a real image frame acquisition module, an image frame synthesis module, and an event camera dataset generation module, among which: The dataset construction module is used to construct image datasets of real motion. An optical flow prediction module is used to select an optical flow prediction network and input image pairs from the image dataset into the optical flow prediction network to predict forward and backward optical flow. The depth map acquisition module is used to construct a monocular depth estimation network and select image pairs from the image dataset to input into the monocular depth estimation network to obtain depth maps. The intermediate moment image frame synthesis module is used to set the parameters of the intermediate moment, use the depth map as weight, and input the forward optical flow and backward optical flow to the bidirectional image fusion module to synthesize the image frame of real motion at the intermediate moment. A piecewise linear fitting algorithm is employed, using micro-displacement optical flow. Image before displacement Mapping to intermediate time Obtain the mapping vector ; Using micro-displacement optical flow The displaced image Mapping to intermediate time Obtain the mapping vector ; The two mapping vectors before and after and Along with depth map and The input is fed into the bidirectional image fusion module to generate intermediate times. Real image frames ; The bidirectional image fusion module generates intermediate moments. Image frames The methods include: drawing and Midpoint in time Image frames and ; Use the snowball method to map vectors The floating-point coordinates in the image are interpolated to integer coordinates on the image frame, and the depth map is used. As weights, for the mapping vector The middle pointer is used to sum the weighted values ​​of multiple pixels at the same location, and then the result is drawn. Midpoint in time Image frames ; Use the snowball method to map vectors The floating-point coordinates in the image are interpolated to integer coordinates on the image frame, and the depth map is used. As weights, for the mapping vector The middle pointer is used to sum the weighted values ​​of multiple pixels at the same location, and then the result is drawn. Midpoint in time direction Image frames ; Fuse the image frames and Synthesis Intermediate Time Real image frames ; Fuse the image frames and Synthesis Intermediate Time Real image frames The method includes: detecting the image frame using a hole region detection method. The empty areas in the image are output as a binary image, where 0 represents an empty pixel and 1 represents a normal pixel. Then the image frame The corresponding pixels are filled into the image frame. The empty region is finally output as the intermediate time. Real image frames ; The parameters at the intermediate moments are continuously iteratively updated to synthesize multiple real image frames at different intermediate moments, forming a set of high temporal resolution video clips; The event camera dataset generation module is used to extract all adjacent pairs of image frames from the video segment in chronological order, calculate the descriptors of all triggering events between adjacent image frames, and generate the event camera dataset.

Images