A simulation system and method for a high-dimensional light field event camera

Abstract

The present invention discloses a simulation system and method for a high-dimensional light field event camera. The simulation method comprises the following steps: (1) constructing a world coordinate system, building a three-dimensional object model to be simulated, and setting the object's motion state to meet the dynamic scene requirements of a dynamic vision sensor; (2) constructing an optical camera model in front of the three-dimensional object model according to a focused Kepler light field structure, so that the light field camera can capture the angle and position of light and obtain high-dimensional light field information including the light position and incident angle; (3) mapping the obtained high-dimensional light field information to a two-dimensional plane to form an image frame, simulating it as an event according to the brightness difference between each frame, and finally outputting a light field event stream. The present invention can accurately simulate the light field event stream, avoid errors caused by processes such as microlens array calibration in actual hardware construction, and provide a simple and feasible method for the production of large-scale light field event data sets.

Description

Technical Field

[0001] The present invention relates to the field of computational photography, and in particular to a simulation system and method for a high-dimensional light field event camera. Background Art

[0002] Both traditional and digital cameras operate on a frame-based image acquisition principle. Video is composed of multiple continuously captured frames. For example, the current standard for movie projection is 24 frames per second. With the development of society, 24 frames per second no longer meets people's needs, so cameras with higher frame rates have been developed and put into use. However, frame-based image acquisition often suffers from data redundancy, and the higher the frame rate, the more severe the data redundancy. This data redundancy often leads to a series of problems, such as high memory consumption, high latency, and high computing power consumption. Therefore, research on new cameras has begun to address this problem.

[0003] As early as the early 20th century, researchers sought to develop novel sensors inspired by biological vision. Over time, event cameras, based on dynamic vision sensors, gradually emerged. Unlike traditional frame-based vision sensors, each pixel in this dynamic vision sensor independently detects changes in the logarithm of light intensity. When the change reaches a certain threshold, an event is generated. This event can be represented by a four-dimensional signal (x, y, p, t), where (x, y) represents the two-dimensional coordinates of the event on the dynamic vision sensor, p represents the polarity of the event, and t represents the time of the event. This format allows event cameras to only record portions of a scene where brightness changes, omitting data for portions where brightness remains constant, significantly reducing data redundancy. With their advantages of high temporal resolution, high dynamic range, low power consumption, and wide readout bandwidth, event cameras hold great promise for applications in high-speed motion and high dynamic range scenarios, and are currently the subject of extensive research.

[0004] If event cameras revolutionized the frame-based imaging of traditional cameras, the emergence of light field cameras has transcended the traditional two-dimensional imaging framework. Traditional two-dimensional imaging frameworks only record the locations along which light rays pass, losing information about the scene's depth and the geometry of objects, thus presenting significant limitations. Light field imaging, on the other hand, can capture the distribution of light radiation throughout space. Through phase transformation and projection integration, images at different spatial locations can be obtained. Direct error correction of light radiation can be performed, removing geometric phase differences and calculating the three-dimensional structure of an object. The concept of the light field was first proposed by Gershun in 1936. In 1991, Professor Adelson of MIT first proposed the plenoptic camera model, representing the light field as a seven-dimensional plenoptic function (x, y, z, θ, φ, λ, t), which describes light at any spatial position (x, y, z), angle (θ, φ), wavelength λ, and time t. To address diverse application conditions, the more concise four-dimensional plenoptic function (u, v, s, t) was proposed to describe the distribution of light, where (u, v) and (s, t) are the intersection points of a ray passing through two planes. Early light field acquisition systems used multi-camera arrays, which suffered from drawbacks such as large size and difficulty in synchronized control. However, handheld light field cameras based on the four-dimensional plenoptic function reduce the cost of light field acquisition systems by adding a microlens array in front of the sensor, enriching their application scenarios.

[0005] A light field event camera is a new type of camera that combines a light field camera and an event camera. It combines the high-dimensional light field of a light field camera with the high temporal resolution and high dynamic characteristics of an event camera, enabling three-dimensional reconstruction of high-speed moving objects. However, the current hardware construction of light field event cameras faces many problems, such as errors in microlens array calibration and low availability of collected data, which are not conducive to the development of high-dimensional light field event extraction algorithms. Therefore, the study of a simulation system for high-dimensional light field event cameras has become a very important topic. The simulation data of the simulation system can avoid the errors caused by the hardware construction and microlens array calibration processes, which is of great significance to the development of high-dimensional light field event extraction algorithms. Summary of the Invention

[0006] In response to the difficulties and shortcomings of the above-mentioned light field event camera hardware system, the present invention provides a simulation system and method for a high-dimensional light field event camera.

[0007] The technical solution adopted by the simulation system of the present invention is as follows:

[0008] A simulation system for a high-dimensional light field event camera includes an object model, a focused Keplerian light field structure, and a dynamic vision sensor. The focused Keplerian light field structure is composed of a primary lens and a microlens array. The model used in the simulation system is described as follows:

[0009] ∑E i =G e [∫∫r(u,v,s,t)dsdt]+σ noise

[0010] Among them, ∑E i is the event flow of the simulation, G e Represents the conversion function from high-dimensional light field information to events, r(u,v,s,t) is the light [u,v,s,t] T The radiance, σ noise is the noise event in the dynamic vision sensor.

[0011] The present invention also provides a simulation method using the above simulation system, the simulation method comprising the following steps:

[0012] Step 1: Construct a world coordinate system, build a 3D object model to be simulated, and set the object's motion state to meet the dynamic scene required by the dynamic vision sensor;

[0013] Step 2: Build an optical camera model in front of the three-dimensional object model according to the focused Kepler light field structure, so that the light field camera can capture the angle and position of the light and obtain high-dimensional light field information [u, v, s, t] containing the light position and incident angle. t ;

[0014] Step 3: Map the obtained high-dimensional light field information to a two-dimensional plane to form an image frame, simulate it as an event according to the brightness difference between each frame, and finally output a light field event stream.

[0015] Furthermore, in step 2, a virtual image of the three-dimensional object model is generated after passing through the main lens, and the virtual image is then secondary focused by the microlens array and finally transmitted to the dynamic vision sensor; the distance relationship between the virtual image, the microlens array, and the dynamic vision sensor is as follows:

[0016]

[0017] b MLA >f MLA

[0018] Among them, f MLA is the focal length of each sub-lens in the microlens array, a MLA The distance from the virtual image plane of the main lens to the microlens array, b MLA is the distance from the microlens array to the dynamic vision sensor.

[0019] Furthermore, in step 3, the event simulation calculation formula is as follows:

[0020]

[0021]

[0022] Among them, f i and f i+1 are two adjacent image frames, Y represents the brightness value of the RGB image converted into a brightness image, ΔL is the logarithmic difference between the brightness images of two adjacent frames, and L m is the preset brightness logarithmic threshold, E (i,i+1) It is an event that occurs between two adjacent frames.

[0023] Furthermore, in step 3, the total light field event flow E sum for:

[0024] E sum =∑ i (E (i,i+1) +E hot +E leak )

[0025] Among them, E (i,i+1) is a common event between frames, E hot is the event generated by the hot pixel of the dynamic vision sensor, E leak It consists of leakage events spontaneously generated by the dynamic vision sensor.

[0026] This paper proposes a simulation system and method for a high-dimensional light field event camera, enabling the complete transformation process from a three-dimensional model to high-dimensional light field information and then to an event stream. This method accurately simulates light field event streams, avoiding errors in hardware construction and microlens array calibration. It also provides a simple and feasible method for validating light field event camera algorithms and generating large datasets for deep learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Schematic diagram of the structure of the simulation system of the present invention;

[0028] Figure 2 is an input model diagram of the simulation system of the present invention;

[0029] Figure 3 It is a simulation effect diagram of the simulation system of the present invention. DETAILED DESCRIPTION

[0030] Reference Figure 1 The present invention provides a simulation system for a high-dimensional light field event camera, which can simulate the entire process of a high-dimensional light field event camera. The overall model is described as follows:

[0031] ∑E i =G e[∫∫r(u,v,s,t)dsdt]+σ noise

[0032] Among them, ∑E i is the event flow of the simulation, G e Represents the conversion function from high-dimensional light field information to events, which is used to characterize the conversion process from image to event; r(u,v,s,t) is the light [u,v,s,t] t The radiance, σ noise is the noise event in the dynamic vision sensor.

[0033] The specific steps of the simulation method are as follows:

[0034] Step 1: The primary goal of a light field event camera is to reconstruct moving 3D objects. To achieve this, a unified world coordinate system is first constructed in the simulation software Blender, and 3D object models are built. These models can range from simple objects like spheres and dolls to complex indoor or natural scenes. Models of varying complexity provide valuable validation of the light field event camera's reconstruction algorithm. Furthermore, by inserting keyframes and other operations, object motions such as translation and rotation are implemented to meet the dynamic input requirements of dynamic vision sensors.

[0035] Step 2: The main purpose of this step is to capture the high-dimensional light field information of moving objects. A light field camera model is built in front of the object model according to the focused Kepler light field structure. This model includes a main lens, a microlens array, and a dynamic vision sensor. A microlens array is added behind the main lens according to a certain geometric relationship to realize the collection of high-dimensional light field information. After the object model passes through the main lens, a virtual image is generated. The virtual image is then refocused by the microlens array and finally transmitted to the dynamic vision sensor. The distance relationship between the virtual image, microlens array, and dynamic vision sensor is as follows:

[0036]

[0037] b MLA >f MLA

[0038] Among them, f MLA is the focal length of each sub-lens in the microlens array, a MLA The distance from the virtual image plane of the main lens to the microlens array, b MLA is the distance from the microlens array to the dynamic vision sensor.

[0039] Step 3: Map the captured high-dimensional light field information to a two-dimensional plane to form a light field image. The formula is as follows:

[0040] I(x,y)=∫∫r(u,v,s,t)dsdt

[0041] Where I(x,y) represents the image brightness at the pixel coordinate (x,y), and r(u,v,s,t) represents the light [u,v,s,t] T radiance.

[0042] The movement of an object will cause brightness differences between consecutive light field images, and the dynamic vision sensor generates events based on the brightness changes between consecutive images. Based on this principle, the obtained consecutive light field image frames are simulated as event streams. This process is represented by G in the model. e Description, including the following processes: RGB→Y→E (i,i+1) →E sum , the specific function of each process is expressed as follows:

[0043] First, the light field camera obtains an RGB image. To compare the brightness difference between two consecutive frames, the RGB image needs to be converted into a brightness image. The formula is as follows:

[0044] Y=0.299R+0.587G+0.114B

[0045] Among them, Y is the brightness value, and R, G, and B are the three channels of the color image.

[0046] Take the logarithm of the image brightness value obtained, compare the logarithmic difference between two consecutive frames of brightness images, and compare the obtained difference with the preset threshold. If the brightness increases by more than the threshold, a positive polarity event will be generated. If the brightness decreases by more than the threshold, a negative polarity event will be generated. If the difference does not exceed the threshold, no event will be generated. The formula is as follows:

[0047]

[0048]

[0049] Among them, f i and f i+1 are two adjacent image frames, ΔL is the logarithmic difference between the brightness images of two adjacent frames, L m is the preset brightness logarithmic threshold, E (i,i+1) It is an event that occurs between two adjacent frames.

[0050] However, real dynamic vision sensors generate not only events caused by brightness changes but also noise events from the sensor itself, including hot pixel events and leakage events. Hot pixel events are generated by hot pixels in the dynamic vision sensor. These hot pixels continuously trigger events at a high rate due to their abnormally low threshold or very high dark current. In the simulation, a small number of pixels are set to continuously generate events to simulate hot pixels. Leakage events are positive polarity events that the dynamic vision sensor spontaneously generates. In the simulation, positive polarity events are generated at a rate of 0.1Hz to simulate leakage events. Therefore, the total event flow can be expressed as follows:

[0051] E sum =∑ i (E (i,i+1) +E hot +E leak )

[0052] Among them, E sum is the total light field event flow, consisting of the common events ∑ i E (i,i+1) , the hot pixel event E of the dynamic vision sensor hot , leakage events E generated spontaneously by dynamic vision sensors leak composition.

[0053] The actual effect of this simulation system is as follows Figure 2 and Figure 3 As shown, Figure 2 A green flying chess piece is used as the input of this system. At the same time, the translation motion of the aircraft is set in Blender to meet the dynamic input requirements of the dynamic vision sensor. In the focused Kepler light field camera, the ratio of the distance from the virtual image plane to the microlens array to the distance from the microlens array to the dynamic vision sensor is set to 3, that is,

[0054]

[0055] Among them, a MLA The distance from the virtual image plane of the main lens to the microlens array, b MLA is the distance from the microlens array to the dynamic vision sensor.

[0056] Therefore, the number of viewing angles of the Kepler-type light field camera is 3*3, that is, each point of the model is photographed 9 times in the microlens array, showing a multi-view state. Finally, the dynamic vision sensor converts the input high-dimensional light field information into an event stream containing multi-view information, namely, the light field event stream, such as Figure 3 shown.

Claims

1. A simulation system for a high-dimensional light field event camera, characterized in that: The simulation system includes an object model, a focused Kepler light field structure, and a dynamic vision sensor. The focused Kepler light field structure is composed of a main lens and a microlens array. The model used in the simulation system is described as follows: ∑E i =G e [∫∫r(u,v,s,t)dsdt]+σ noise Among them, ∑E i is the event flow of the simulation, G e Represents the conversion function from high-dimensional light field information to events, r(u,v,s,t) is the light [u,v,s,t] T The radiance, σ noise is a noise event in the dynamic vision sensor; The simulation method of the simulation system comprises the following steps: Step 1: Construct a world coordinate system, build a 3D object model to be simulated, and set the object's motion state to meet the dynamic scene required by the dynamic vision sensor; Step 2: Build an optical camera model in front of the three-dimensional object model according to the focused Kepler light field structure, so that the light field camera can capture the angle and position of the light and obtain high-dimensional light field information [u, v, s, t] containing the light position and incident angle. T ; Step 3: Map the obtained high-dimensional light field information to a two-dimensional plane to form an image frame, simulate it as an event according to the brightness difference between each frame, and finally output a light field event stream; In step 3, the event simulation calculation formula is as follows: Among them, f i and f i+1 are two adjacent image frames, Y represents the brightness value of the RGB image converted into a brightness image, ΔL is the logarithmic difference between the brightness images of two adjacent frames, and L m is the preset brightness logarithmic threshold, E (i,i+1) It is the event generated between two adjacent frames; In step 3, the total light field event flow E sum for: AND sum =∑ i (AND (i,i+1) +E hot +E leak ) Among them, E (i,i+1) is a common event between frames, E hot is the event generated by the hot pixel of the dynamic vision sensor, E leak It consists of leakage events spontaneously generated by the dynamic vision sensor.

2. The high-dimensional light field event camera simulation system according to claim 1, characterized in that: In step 2, a virtual image of the three-dimensional object model is generated after passing through the main lens, and the virtual image is then secondary focused by the microlens array and finally transmitted to the dynamic vision sensor; The distance relationship between the virtual image, microlens array, and dynamic vision sensor is as follows: b MLA >f MLA Among them, f MLA is the focal length of each sub-lens in the microlens array, a MLA The distance from the virtual image plane of the main lens to the microlens array, b MLA is the distance from the microlens array to the dynamic vision sensor.

Images