Light field time domain super-resolution method based on event camera

Abstract

The invention discloses a light field time domain super-resolution method based on an event camera, and belongs to the field of optical engineering. A light field camera and an event camera are used for collecting light field event data, decoding the light field data, denoising the event data and generating an event image, the light field data and an event grayscale image are subjected to joint calibration in a sub-aperture image mode, joint optimization is carried out through calibration results of multiple viewpoints, a checkerboard is shot, and a target object is obtained. And obtaining an affine transformation matrix and a perspective transformation matrix, and obtaining a super-resolution result of projecting the event image onto a light field image plane. Starting from the expression mode of the light field macro pixel, the method has applicability to light field data of any macro pixel size; compared with a traditional time domain super-resolution method, the time domain super-resolution of the light field camera at any frame rate can be realized by segmenting the event stream at different scales.

Description

Technical Field

[0001] This invention relates to the fields of computational photography, event cameras, and optical engineering, and specifically to a temporal super-resolution method and data acquisition device for light fields based on an event camera. Background Technology

[0002] Light field is a novel visual expression method that can describe the distribution of light in all directions in three-dimensional space and capture and reproduce the intensity and direction information of light, thereby enabling depth perception of the three-dimensional world and free change of perspective.

[0003] While light field imaging holds immense potential for providing rich visual information and depth perception, its applications still face several challenges. The primary challenge is the hardware limitations imposed on light field images, which are constrained by both spatial resolution (image resolution per viewpoint) and angular resolution (sampling density of viewpoints). Light field imaging involves the coupling of multiple relationships across the angular, spatial, and temporal domains, with each frame containing images acquired from multiple viewpoints. However, light field cameras use microlens arrays to capture light, and each pixel can only measure a limited number of light directions. This means that in some cases, light field cameras may fail to accurately capture very subtle angular changes, thus limiting angular resolution. In practical applications, since the image sensor size is fixed, achieving higher angular resolution typically requires choosing smaller microlens sizes, but this may come at the cost of some spatial resolution. Conversely, higher spatial resolution necessitates larger microlens spacing, which may also sacrifice some angular resolution. To overcome this limitation, choosing ultra-high resolution sensors is the primary solution. However, ultra-high resolution introduces bandwidth limitations, inevitably reducing the temporal frame rate of the light field camera. Existing methods for achieving temporal super-resolution of light fields using hybrid imaging systems employ a combination of a standard frame rate SLR camera and a low-resolution light field camera. However, the super-resolution achieved by this method is limited by the frame rate of the SLR camera, which is only 30 frames per second.

[0004] The Dynamic Vision Sensor (DVS) is a biomimetic camera that mimics the characteristics of the biological retina. It boasts a wider dynamic range and outputs data more sparsely and rapidly than traditional cameras. The key difference lies in its operation: instead of capturing images continuously at time intervals, it records data only when significant changes occur within the scene. This camera operates based on event perception and response, enabling it to capture rapid dynamic changes in a scene with extremely high temporal resolution. It is widely used for video frame interpolation tasks in hybrid imaging systems.

[0005] However, event cameras only respond to pixel-level changes in light intensity, making it difficult to recover the complete angular and structural information of light rays in space. Therefore, there is currently no mature method to combine event response mechanisms with light field imaging to simultaneously acquire high-frequency changing light intensity and angular direction information. To achieve temporal super-resolution of light fields, the purpose of this invention is to fully leverage the advantages of event cameras in the field of temporal super-resolution of light fields and to design a temporal super-resolution method for light fields using the characteristics of event cameras. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, this invention provides a temporal super-resolution method for light fields based on an event camera, belonging to the field of optical engineering. It utilizes a light field camera and an event camera to acquire light field event data, decodes the light field data, denoises the event data, and generates event images. The light field data is jointly calibrated with the event grayscale image in the form of a sub-aperture image, and jointly optimized using calibration results from multiple viewpoints. A checkerboard pattern is captured, and affine transformation and perspective transformation matrices are obtained to obtain a super-resolution result projecting the event image onto the light field image plane. This invention starts from the representation method of light field macropixels, making it applicable to light field data of arbitrary macropixel size. By segmenting the event stream at different scales, compared to traditional temporal super-resolution methods, it can achieve temporal super-resolution for light field cameras at any frame rate.

[0007] A temporal super-resolution method for light field based on an event camera, the method comprising the following steps:

[0008] Step 1, data acquisition using a light field camera and an event camera: the data acquisition is achieved by either using a beam splitter or by arranging the cameras in parallel.

[0009] An event sequence e is obtained through an event camera, and the event sequence e is integrated over time intervals to generate an event pseudo-image; a light field image LF(u,v,x,y) is obtained through a light field camera, and a light field sub-aperture image is generated from the light field image LF(u,v,x,y); where (u,v) is the viewpoint number of a small aperture in the camera array or microlens array; and (x,y) is a pixel position in the image captured from the small aperture viewpoint.

[0010] Step 2: Jointly calibrate the light field sub-aperture image and the event pseudo-image:

[0011] Step 3, Obtain the light field tensor:

[0012] For each (u,v) in the light field image LF(u,v,x,y), extract the sub-aperture image and organize the sub-aperture images into a C×H×W light field tensor, where H and W are the spatial dimensions of the light field sub-aperture image, representing the length and width respectively; C is the number of channels of the image, and the dataset has 3 channels of RGB values.

[0013] Step 4, obtain the event tensor E:

[0014] For a set of N input event sequences e = {(X_j,Y_j,t)|j∈[0,N-1]}, where N ranges from 2 to 1000, and the N input event sequences e occur between keyframes F_(t_i) and F_(t_(i+1)), where X and Y are pixel coordinates and t is the timestamp of the event; the event sequence e is divided into equally spaced parts by the number of accumulated event frames, and a tensor is constructed using a fixed dimension W×H×B, where B is the accumulated value of the event at each position; finally, based on the number of event frames N, an event tensor E is formed, with the dimension of event tensor E being N×W×H×B;

[0015] Step 5: Input the event tensor and the light field tensor into the Event-Light Field Network (ELF-Net) for temporal super-resolution processing to obtain the predicted image;

[0016] Step 6: Use the EPI optimization network to optimize the predicted image to obtain a high-quality temporal super-resolution video of the light field.

[0017] Furthermore, in step 2, the joint calibration process is as follows:

[0018] Step 2.1: Use the grayscale mode of the light field camera and the event camera to photograph the checkerboard pattern;

[0019] Step 2.2: Use OpenCV to calibrate the chessboard grid.

[0020] Furthermore, in step 2.2, the OpenCV calibration process is as follows:

[0021] Step 2.2.1: Use the built-in function cv2.findChessboardCorners() to detect the checkerboard corner points of the light field sub-aperture image and the event pseudo-image;

[0022] Step 2.2.2: Use cv2.getAffineTransform() to obtain the affine mapping matrix and perform the affine transformation.

[0023] Step 2.2.3: Use cv2.warpPerspective() to perform perspective transformation, thereby projecting the event image onto the plane of the light field image, establishing the mapping relationship between the light field sub-aperture image and the event pseudo-image, and providing alignment data for subsequent temporal super-resolution.

[0024] Furthermore, in step 2.2.3, the perspective transformation formula is as follows:

[0025] [x dst,y dst ,z dst ] = [x src ,y src ,z src ]*H

[0026] Among them, (x src ,y src ,z src (x) represents the coordinates of a point on the source plane of the light field sub-aperture image. dst ,y dst ,z dst ) are the coordinates of a point on the target plane of the event pseudo-image, and H is a 3×3 perspective transformation matrix.

[0027] Furthermore, in step 5, the event light field network includes three sub-networks: a feature extraction network, a light field synthesis network, and an EPI optimization network; the feature extraction network uses a ResNet-50 network; the light field synthesis network uses a UNet network to define a DoubleConv sub-network; and the EPI optimization sub-network uses a UNet network as its main framework.

[0028] Furthermore, the temporal super-resolution processing procedure of the event light field network is as follows:

[0029] Step 5.1, using a pre-trained model based on ResNet-50: using ResNet-50 to extract features from the event image and the light field image respectively, to obtain feature maps;

[0030] Step 5.2: The feature maps are stitched together and then input into the light field synthesis network to generate a predicted image from a single viewpoint;

[0031] Step 5.3: Use an optical field synthesis network to generate predicted images for all viewpoints; the predicted images are predicted sub-aperture images.

[0032] Furthermore, in step 5.2, the light field synthesis network uses a UNet network to define a DoubleConv module, which includes two consecutive convolutional layers. Each convolutional layer is followed by a Batch Normalization (BN) layer and a ReLU activation function. The UNet network is a standard U-shaped image segmentation network. The UNet network contains encoder and decoder parts. The encoder includes pooling layers and five DoubleConv modules, each of which performs convolution and feature extraction on the input feature map. The decoder gradually restores the resolution through upsampling and skip connections, and fuses the features from the encoder with the features from the decoder. Finally, a 1x1 convolutional layer is used to generate the predicted image.

[0033] Furthermore, in step 6, the predicted image optimization process is as follows:

[0034] Step 6.1: Use the UNet network as a framework to extract EPI images from the predicted sub-aperture images in Step 5.3:

[0035] A sub-aperture image is fixed with rows (or columns) and a view dimension. Pixels of the rows (or columns) are extracted from multiple different viewpoints. The pixels are then stacked in viewpoint order to form a two-dimensional image, namely an EPI image. The EPI image shows the parallax changes of objects at different depths in the scene under the change of viewpoint. The slope of the EPI image reflects the depth information of the object.

[0036] Step 6.2: Spatial angle alignment is performed on the extracted EPI image and the predicted sub-aperture image to achieve position and angle consistency, and finally a high-quality light field temporal super-resolution video is obtained.

[0037] An electronic device, characterized in that it comprises: one or more processors; a memory; one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs being configured to perform the above-described optical field temporal super-resolution method.

[0038] A computer-readable storage medium, characterized in that the computer-readable storage medium stores program code, which can be called by a processor to execute the above-described optical field temporal super-resolution method.

[0039] The beneficial effects of this invention are:

[0040] (1) Significantly reduces light field interpolation artifacts and defocusing issues. Traditional light field temporal super-resolution methods, such as those based on optical flow or depth estimation, suffer from artifacts caused by inconsistent viewpoints. Event data naturally possesses sparse response characteristics, enabling consistent temporal event alignment across multiple sub-aperture maps. It can better maintain edge sharpness and geometric consistency between multiple viewpoint frames during multi-view synthesis.

[0041] (2) Reduce training dependency and improve system generalization. Existing deep learning-based super-resolution models require a large amount of high frame rate light field training data, which is costly to collect. This method is mainly event-driven, requiring little or no learning, and has physical universality beyond end-to-end modeling. It also improves applicability in different shooting environments and different types of camera systems.

[0042] (3) Temporal super-resolution and spatial light field structure are naturally integrated. By utilizing the high temporal sampling of event data and the multi-view spatial decomposition of microlens arrays, a five-dimensional event light field flow can be constructed; it supports simultaneous temporal frame interpolation and spatial viewpoint decoding, enhancing the spatial consistency of the overall reconstruction; it is conducive to the subsequent realization of higher-level applications such as continuous temporal domain light field video modeling, three-dimensional reconstruction and dynamic volume rendering. Attached Figure Description

[0043] Figure 1 This is the event light field network structure used in the embodiments of the present invention. Detailed Implementation

[0044] The present invention will be further described below with reference to the accompanying drawings and embodiments. The present invention includes, but is not limited to, the following embodiments.

[0045] A temporal super-resolution method for light field based on an event camera, the method comprising the following steps:

[0046] Step 1, data acquisition using a light field camera and an event camera: the data acquisition is achieved by either using a beam splitter or by arranging the cameras in parallel.

[0047] An event sequence e is obtained through an event camera, and the event sequence e is integrated over time intervals to generate an event pseudo-image; a light field image LF(u,v,x,y) is obtained through a light field camera, and a light field sub-aperture image is generated from the light field image LF(u,v,x,y); where (u,v) is the viewpoint number of a small aperture in the camera array or microlens array; and (x,y) is a pixel position in the image captured from the small aperture viewpoint.

[0048] Step 2: Jointly calibrate the light field sub-aperture image and the event pseudo-image:

[0049] Furthermore, in step 2, the joint calibration process is as follows:

[0050] Step 2.1: Use the grayscale mode of the light field camera and the event camera to photograph the checkerboard pattern;

[0051] Step 2.2: Use OpenCV to calibrate the chessboard grid.

[0052] Furthermore, in step 2.2, the OpenCV calibration process is as follows:

[0053] Step 2.2.1: Use the built-in function cv2.findChessboardCorners() to detect the checkerboard corner points of the light field sub-aperture image and the event pseudo-image;

[0054] Step 2.2.2: Use cv2.getAffineTransform() to obtain the affine mapping matrix and perform the affine transformation.

[0055] Step 2.2.3: Use cv2.warpPerspective() to perform perspective transformation, thereby projecting the event image onto the plane of the light field image, establishing the mapping relationship between the light field sub-aperture image and the event pseudo-image, and providing alignment data for subsequent temporal super-resolution.

[0056] Furthermore, in step 2.2.3, the perspective transformation formula is as follows:

[0057] [x dst ,y dst ,z dst ] = [x src ,y src ,z src ]*H

[0058] Among them, (x src ,y src ,z src (x) represents the coordinates of a point on the source plane of the light field sub-aperture image. dst ,y dst ,z dst ) are the coordinates of a point on the target plane of the event pseudo-image, and H is a 3×3 perspective transformation matrix.

[0059] Step 3, Obtain the light field tensor:

[0060] For each (u,v) in the light field image LF(u,v,x,y), extract the sub-aperture image and organize the sub-aperture images into a C×H×W light field tensor, where H and W are the spatial dimensions of the light field sub-aperture image, representing the length and width respectively; C is the number of channels of the image, and the dataset has 3 channels of RGB values.

[0061] Step 4, obtain the event tensor E:

[0062] For a set of N input event sequences e = {(X_j,Y_j,t)|j∈[0,N-1]}, where N ranges from 2 to 1000, and the N input event sequences e occur between keyframes F_(t_i) and F_(t_(i+1)), where X and Y are pixel coordinates and t is the timestamp of the event; the event sequence e is divided into equally spaced parts by the number of accumulated event frames, and a tensor is constructed using a fixed dimension W×H×B, where B is the accumulated value of the event at each position; finally, based on the number of event frames N, an event tensor E is formed, with the dimension of event tensor E being N×W×H×B;

[0063] Step 5: Input the event tensor and light field tensor into the Event-Light Field Network (ELF-Net) for temporal super-resolution processing to obtain the predicted image; the Event-Light Field Network is as follows: Figure 1 As shown;

[0064] Furthermore, in step 5, the event light field network includes three sub-networks: a feature extraction network, a light field synthesis network, and an EPI optimization network; the feature extraction network uses a ResNet-50 network; the light field synthesis network uses a UNet network to define a DoubleConv sub-network; and the EPI optimization sub-network uses a UNet network as its main framework.

[0065] Furthermore, the temporal super-resolution processing procedure of the event light field network is as follows:

[0066] Step 5.1, using a pre-trained model based on ResNet-50: using ResNet-50 to extract features from the event image and the light field image respectively, to obtain feature maps;

[0067] Step 5.2: The feature maps are stitched together and then input into the light field synthesis network to generate a predicted image from a single viewpoint;

[0068] In step 5.2, the light field synthesis network uses a UNet network to define a DoubleConv module, which includes two consecutive convolutional layers. Each convolutional layer is followed by a Batch Normalization (BN) layer and a ReLU activation function. The UNet network is a standard U-shaped image segmentation network. The UNet network includes an encoder and a decoder. The encoder includes a pooling layer and five DoubleConv modules. Each DoubleConv module performs convolution and feature extraction on the input feature map. The decoder gradually restores the resolution through upsampling and skip connections, and fuses the features from the encoder with the features from the decoder. Finally, a 1x1 convolutional layer is used to generate the predicted image.

[0069] Step 5.3: Use an optical field synthesis network to generate predicted images for all viewpoints; the predicted images are predicted sub-aperture images.

[0070] Step 6: Use the EPI optimization network to optimize the predicted image to obtain a high-quality temporal super-resolution video of the light field.

[0071] Step 6.1: Use the UNet network as a framework to extract EPI images from the predicted sub-aperture images in Step 5.3:

[0072] A sub-aperture image is fixed with rows (or columns) and a view dimension. Pixels of the rows (or columns) are extracted from multiple different viewpoints. The pixels are then stacked in viewpoint order to form a two-dimensional image, namely an EPI image. The EPI image shows the parallax changes of objects at different depths in the scene under the change of viewpoint. The slope of the EPI image reflects the depth information of the object.

[0073] Step 6.2: Spatial angle alignment is performed on the extracted EPI image and the predicted sub-aperture image to achieve position and angle consistency, and finally a high-quality light field temporal super-resolution video is obtained.

Claims

1. A temporal super-resolution method for light fields based on an event camera, characterized in that, The optical field temporal super-resolution method includes the following steps: Step 1, data acquisition using a light field camera and an event camera: the data acquisition is achieved by either using a beam splitter or by arranging the cameras in parallel. An event sequence e is obtained through an event camera, and the event sequence e is integrated over time intervals to generate an event pseudo-image; a light field image LF(u,v,x,y) is obtained through a light field camera, and a light field sub-aperture image is generated from the light field image LF(u,v,x,y); where (u,v) is the viewpoint number of a small aperture in the camera array or microlens array; and (x,y) is a pixel position in the image captured from the small aperture viewpoint. Step 2: Jointly calibrate the optical field sub-aperture image and the event pseudo-image; Step 3, Obtain the light field tensor: For each (u,v) in the light field image LF(u,v,x,y), extract the sub-aperture image and organize the sub-aperture images into a C×H×W light field tensor, where H and W are the spatial dimensions of the light field sub-aperture image, representing the length and width respectively; C is the number of channels of the image, and the dataset has 3 channels of RGB values. Step 4, obtain the event tensor E: For a set of N input event sequences e = {(X_j,Y_j,t)|j∈[0,N-1]}, where N ranges from 2 to 1000, and the N input event sequences e occur between keyframes F_(t_i) and F_(t_(i+1)), where X and Y are pixel coordinates and t is the timestamp of the event; the event sequence e is divided into equally spaced parts by the number of accumulated event frames, and a tensor is constructed using a fixed dimension W×H×B, where B is the accumulated value of the event at each position; finally, based on the number of event frames N, an event tensor E is formed, with the dimension of event tensor E being N×W×H×B; Step 5: Input the event tensor and the light field tensor into the event light field network respectively for temporal super-resolution processing to obtain the predicted image; Step 6: Use the EPI optimization network to optimize the predicted image to obtain a high-quality temporal super-resolution video of the light field.

2. The temporal super-resolution method for optical fields according to claim 1, characterized in that, In step 2, the joint calibration process is as follows: Step 2.1: Use the grayscale mode of the light field camera and the event camera to photograph the checkerboard pattern; Step 2.2: Use OpenCV to calibrate the chessboard grid.

3. The temporal super-resolution method for optical fields according to claim 2, characterized in that, In step 2.2, the OpenCV calibration process is as follows: Step 2.2.1: Use the built-in function cv2.findChessboardCorners() to detect the checkerboard corner points of the light field sub-aperture image and the event pseudo-image; Step 2.2.2: Use cv2.getAffineTransform() to obtain the affine mapping matrix and perform the affine transformation. Step 2.2.3: Perform perspective transformation using cv2.warpPerspective().

4. The temporal super-resolution method for optical fields according to claim 3, characterized in that, In step 2.2.3, the perspective transformation formula is as follows: [x dst ,y dst ,z dst ]=[x src ,y src ,z src ]*H Among them, (x src y src , z src (x) represents the coordinates of a point on the source plane of the light field sub-aperture image. dst y dst , z dst ) are the coordinates of a point on the target plane of the event pseudo-image, and H is a 3×3 perspective transformation matrix.

5. The temporal super-resolution method for optical fields according to claim 1, characterized in that, In step 5, the event light field network includes three sub-networks: a feature extraction network, a light field synthesis network, and an EPI optimization network. The feature extraction network uses a ResNet-50 network. The light field synthesis network uses a UNet network to define a DoubleConv sub-network. The EPI optimization sub-network uses a UNet network as its framework.

6. The temporal super-resolution method for optical fields according to claim 1, characterized in that, In step 5, the event light field network performs temporal super-resolution processing as follows: Step 5.1, using a pre-trained model based on ResNet-50: using ResNet-50 to extract features from the event image and the light field image respectively, to obtain feature maps; Step 5.2: The feature maps are stitched together and then input into the light field synthesis network to generate a predicted image from a single viewpoint; Step 5.3: Use an optical field synthesis network to generate predicted images for all viewpoints; the predicted images are predicted sub-aperture images.

7. The temporal super-resolution method for optical fields according to claim 6, characterized in that, In step 5.2, the light field synthesis network uses a UNet network to define a DoubleConv module, which includes two consecutive convolutional layers. Each convolutional layer is followed by a BN layer and a ReLU activation function. The UNet network is a standard U-shaped image segmentation network. The UNet network includes an encoder and a decoder. The encoder includes pooling layers and five DoubleConv modules. Each DoubleConv module performs convolution and feature extraction on the input feature map. The decoder gradually restores the resolution through upsampling and skip connections, and fuses the features from the encoder with the features from the decoder. Finally, a predicted image is generated through a 1x1 convolutional layer.

8. The temporal super-resolution method for optical fields according to claim 1, characterized in that, In step 6, the predicted image optimization process is as follows: Step 6.1: Use the UNet network as a framework to extract EPI images from the predicted sub-aperture images: fix the row and view dimension of a sub-aperture image, extract the pixels of the row from multiple different viewpoints, and then stack the pixels in viewpoint order to form a two-dimensional image, namely the EPI image. Step 6.2: Spatial angle alignment is performed on the extracted EPI image and the predicted sub-aperture image to achieve position and angle consistency, and finally a high-quality light field temporal super-resolution video is obtained.

9. An electronic device, characterized in that, include: One or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs being configured to perform the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code that can be invoked by a processor to execute the method as described in any one of claims 1-8.

Images