Method for generating light field event stream based on single event camera and multiple RGB cameras

Abstract

The invention provides a method for generating a light field event stream based on a single event camera and a plurality of RGB (Red, Green and Blue) cameras, and aims to reduce the manufacturing cost of a high-frame-rate RGB light field video (RGB-LFV) and enhance immersive video experience in VR (Virtual Reality). An existing RGB-LFV has the problems of low frame rate, motion blur and the like, in order to solve the problems, the invention provides a framework which is used for effectively extracting angle features, time features and long-distance dependence of the RGB-LFV through space-angle convolution, a convolution long-short time memory network and a Transform model by only using one event camera and cooperating with a plurality of RGB cameras, and generating a light field event stream. Therefore, the frame rate of the RGB-LFV is improved, the manufacturing cost of the high-frame-rate RGB-LFV is reduced, and the user experience in virtual reality is improved.

Description

Technical Field

[0001] The present invention relates to the technical fields of virtual reality (VR), light field image processing and event-based vision, and in particular to a method for generating a light field event stream based on a single event camera and multiple RGB cameras. The method aims to combine the high temporal resolution of the event camera with the multi-view light field capture capability of the RGB camera, reduce the production cost of high frame rate light field videos, solve the problem of low frame rate of current light field videos and easy motion blur in fast motion scenes, and improve the dynamic performance and immersive experience of virtual reality videos. Background Art

[0002] With the rapid development of virtual reality technology, users' demand for high-quality immersive video experience is growing. As a multi-view video technology that can simultaneously capture light intensity and direction, light field video (LFV) shows great potential in providing a 6-DOF immersive experience and has gradually become a research hotspot in the field of VR video processing. Traditional RGB light field video (RGB-LFV) uses multiple RGB cameras to synchronously acquire images from different perspectives, thereby providing rich spatial and angular information. However, due to the limited frame rate of RGB cameras, frame blur is prone to occur, especially in fast-motion scenes, affecting the user's viewing experience. In addition, the computing resources required to process multi-view RGB-LFV data are extremely large, further limiting its promotion in real-time applications.

[0003] Event cameras are an emerging type of bio-inspired imaging sensor that can capture brightness changes of each pixel in a scene with a temporal resolution of microseconds. Their high temporal accuracy makes them particularly good at capturing high-speed motion scenes, and can effectively avoid the motion blur problem encountered by traditional RGB cameras. However, the high price of event cameras and the need for multiple event cameras to capture multi-view event streams make their application cost on RGB-LFV platforms too high.

[0004] To solve the above problems, some studies in the prior art have attempted to use RGB cameras and event cameras to generate hybrid videos at the same time. However, this method faces the problem of how to introduce the high temporal resolution advantage of event cameras into multi-view light field video systems without significantly increasing hardware costs. In addition, although the existing event stream generation method has improved video clarity to a certain extent, there are still large errors when generating light field event streams, and it fails to fully explore the angular features and temporal dependencies in light field videos. Therefore, there is an urgent need for a new method and framework that can fully utilize the high temporal resolution advantage of event cameras while maintaining reasonable costs, and combine with multiple RGB cameras to generate high frame rate, low motion blur light field videos, thereby significantly improving the immersive experience in virtual reality. Summary of the invention

[0005] The present invention aims to solve the problems of low frame rate of existing RGB light field videos, easy motion blur in fast motion scenes, and high production cost of high frame rate RGB light field videos. A method for generating light field event streams based on a single event camera and multiple RGB cameras is provided, which can effectively improve the video quality and immersive experience in virtual reality.

[0006] The technical solution provided by the present invention is as follows:

[0007] A method for generating a light field event stream based on a single event camera and multiple RGB cameras is designed to reduce costs and improve immersive video experience in virtual reality (VR), specifically comprising the following steps:

[0008] 1) Data acquisition: An event camera and multiple RGB cameras are placed in the scene to capture multi-view RGB light field video (RGB-LFV) and event streams. The RGB camera records the RGB image sequence in the scene, while the event camera captures the brightness changes of each pixel in the scene to generate a high-temporal resolution event stream. After recording, the RGB-LFV and event stream are saved in corresponding formats.

[0009] 2) Spatial-angular feature extraction: Through the spatial-angular convolution operation, the angular features between multiple perspectives are extracted from the RGB-LFV to ensure that the information between different perspectives can be effectively combined to provide spatial information support for the subsequent event stream synthesis.

[0010] 3) Temporal feature extraction: A convolutional long short-term memory network (ConvLSTM) is used to extract temporal features from RGB-LFV to capture the temporal changes in the video sequence and ensure the continuity and consistency of the video in the temporal dimension.

[0011] 4) Long-distance dependency modeling: Through the Transformer model, long-distance dependencies in RGB-LFV and event stream data are modeled to capture the complex dynamic information between video frames and enhance the accuracy of video generation.

[0012] 5) Light field event stream synthesis: Combining the above-extracted angle features, time features, long-distance dependency information and event streams, a multi-perspective high-frame rate light field event stream is generated to ensure the clarity and high frame rate performance of the video in fast-motion scenes.

[0013] The innovation of the present invention lies in that, by combining a single event camera with multiple RGB cameras, the hardware cost is reduced, and a high-temporal-resolution light field event stream is generated through an efficient deep learning framework, which significantly reduces the blur phenomenon in fast-motion scenes. In addition, the present invention proposes a multi-view synthesis method for light field event streams for the first time, solving the problem of poor experience caused by insufficient frame rate in the prior art.

[0014] The technical effects of the present invention are:

[0015] The method provided by the present invention significantly improves the frame rate and clarity of videos in virtual reality by combining the high temporal resolution of event cameras with the multi-view characteristics of RGB light field videos, reduces the blurring problem caused by rapid motion, enhances the user's sense of immersion in virtual reality, and reduces the production cost of high-frame-rate RGB-LFV. The method is applicable to the fields of virtual reality, panoramic video processing, and can be widely used in various application scenarios that require high-frame-rate, high-quality videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 The device used in the present invention is the DAVIS 346 event camera.

[0017] Figure 2 It is the RGB camera array used in the present invention.

[0018] Figure 3 This is a design diagram of the deep learning model used in the present invention to combine event stream and light field data.

[0019] Figure 4 The overall flow chart for generating a light field event stream for the specific implementation of the present invention; DETAILED DESCRIPTION

[0020] The present invention will be further described below by way of embodiments in conjunction with the accompanying drawings, but the scope of the present invention is not limited in any way.

[0021] The present invention provides a method for generating light field event streams based on a single event camera and multiple RGB cameras, aiming to improve the immersive video experience in virtual reality (VR). Through the framework of the present invention, motion blur can be reduced in fast-motion scenes, and the clarity and frame rate of the video can be improved.

[0022] like Figure 1 and Figure 2As shown, the device used in the present invention includes a DAVIS 346 event camera and a camera array consisting of multiple RGB cameras. The event camera can capture brightness changes in the scene and generate event stream data with high temporal resolution, while the RGB camera array is used to record multi-view light field video (RGB-LFV). First, the equipment is arranged in the scene, the event camera captures the high-resolution event stream of the central view, and the RGB camera array captures static images and video frames of multiple viewpoints. After completing the data acquisition, the event stream and RGB-LFV data are saved separately.

[0023] like Figure 3 As shown, the present invention adopts an event light field deep learning model to combine event streams with light field video data. First, the spatial-angular convolution module is used to extract angle features in the RGB light field video. This operation utilizes the angle differences between multiple perspectives, combines image information from different perspectives, generates accurate angle features, and provides support for subsequent event stream synthesis. Next, the temporal features of the RGB light field video are extracted through a convolutional long short-term memory network (ConvLSTM). ConvLSTM can effectively process the temporal information in the video sequence, ensuring that the temporal continuity of the video is maintained in fast-motion scenes, avoiding the problem of temporal inconsistency. At the same time, it can efficiently capture the dynamic changes between frames in the video, thereby improving the temporal resolution of the video.

[0024] Afterwards, the long-distance dependencies between RGB-LFV frames are modeled through the Transformer module. The Transformer model can capture long-term dependencies in the video, ensuring that the video can accurately reflect these changes in scenes with complex dynamic changes, thereby improving the accuracy of video synthesis.

[0025] like Figure 4 As shown, the light field event stream generation process of the present invention is based on the angle features, time features, long-distance dependencies and event data extracted from RGB-LFV. The synthesis process makes full use of the high temporal resolution characteristics of the event camera, combines the captured central perspective event stream with other perspective data of the RGB light field video, and generates a high frame rate, low blur light field event stream.

[0026] It should be noted that the disclosure of the embodiments is to help better understand the present invention, but those skilled in the art may make various modifications and substitutions to the present invention without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the above embodiments, and the specific protection scope shall be subject to the claims.

Claims

1. A method for generating a light field event stream based on a single event camera and multiple RGB cameras, the method being used to enhance an immersive video experience in virtual reality (VR), characterized in that: The method comprises the following steps: 1) Use an event camera to capture high temporal resolution event streams and multiple RGB cameras to capture RGB light field video (RGB-LFV); 2) Process the event stream in step 1); 3) Extract the angle features of RGB-LFV in step 1) through a convolutional network (such as spatial-angle convolution, etc.); 4) Use a convolutional network (such as a convolutional long short-term memory network, etc.) to process the temporal features of the RGB-LFV in step 1); 5) Use attention models (such as Transformer models) to capture the long-distance dependencies of RGB-LFV in step 3); 6) Combining the event processing data in step 2), the angle features in step 3), the time features in step 4), and the long-distance dependency in step 5) to generate a light field event stream, thereby generating a low-cost and high-frame-rate light field video and reducing motion blur.

2. The method according to claim 1, characterized in that The event acquisition method described in step 1) is to capture video using a single event camera and multiple RGB cameras.

3. The method according to claim 1, characterized in that Step 3) The convolutional network is used to process the angular correlation of different views in RGB-LFV.

4. The method according to claim 1, characterized in that Step 4) The convolutional network is used to capture the temporal variation characteristics of RGB-LFV to improve the temporal resolution of the video.

5. The method according to claim 1, characterized in that Step 5) The attention model is used to capture the long-range dependencies of RGB-LFV to improve the overall coherence and frame rate of the video.

6. The method according to claim 1, characterized in that The generated light field event stream can be used to directly generate high frame rate light field video, which effectively reduces the cost, reduces blur in RGB-LFV, and improves the user experience in virtual reality.

Images