Event camera physical property simulation method and device

Abstract

The present application relates to the technical field of camera simulation, and particularly relates to an event camera physical characteristic simulation method and device, wherein the method comprises the following steps: a description file of a lens group of a target event camera is established to select a plurality of sampling interval points; an exit pupil condition of each sampling interval point is calculated, and the exit pupil condition on the entire straight line is generated according to the exit pupil condition of each sampling interval point; whether a light beam of a target scene is in the exit pupil condition on the entire straight line is judged based on a three-dimensional scene model obtained by modeling the target scene and the target event camera, if the light beam is in the exit pupil condition on the entire straight line, the light beam is converted into a photoelectric current, otherwise the light beam is ignored; and the photoelectric current is simulated by using a sensor of the target event camera to obtain a three-dimensional simulation result. Thus, a physical-based full-link event simulator is obtained, which can provide data for various event camera vision tasks, and solves the problem that a traditional camera is difficult to detect with high precision in high-speed motion, excessive illumination and low-illumination scenes.

Description

Technical Field

[0001] This invention relates to the field of camera simulation technology, and in particular to a method and apparatus for simulating the physical characteristics of an event camera. Background Technology

[0002] Event cameras, or bio-inspired vision sensors, operate fundamentally differently from traditional cameras. Instead of capturing intensity images at a fixed rate, event cameras respond to a series of asynchronous events in response to changes in brightness. With their advantages of high temporal resolution, high dynamic range, low power consumption, and minimal redundancy, event cameras are widely used in various computer vision tasks.

[0003] However, despite significant progress in event-based vision, training deep learning methods still requires substantial amounts of simulated event data. Notably, the high cost and deployment difficulties of event cameras in high-speed or low-light scenes limit the availability and scale of real-world datasets. Therefore, some event camera simulators attempt to generate large amounts of affordable and reliable event data.

[0004] Existing event camera simulators mainly fall into two categories. One category consists of optimization-based event camera simulators, which aim to design handcrafted modules (i.e., methods based on mathematical mechanisms rather than neural networks) to generate event data. For example, the V2E (From video frames to realistic DVS events, InCVPRW) method converts intensity frames into event data through multiple handcrafted modules. These event camera simulators directly treat three-channel video as input, ignoring the entire optical path in a real camera system. This can lead to the loss of optical information, severely impacting the simulator's accuracy. While some solutions (e.g., Esim: an open event camerasimulator, Event camerasimulator improvements via characterized parameters) offer offline rendering support in 3D scenes, they neglect spectral data considerations in their rendering modules and lack concepts such as lens simulation and quantum efficiency, resulting in less than ideal accuracy. The other category is learning-based event camera simulators, which aim to generate event representations that directly improve the generalization ability of deep learning models in the target domain, without relying on the original events. For example, Eventgan (Leveraging large-scale image datasets for event cameras) is an end-to-end neural network that directly converts images into event representations for downstream computer vision tasks. However, these simulators require retraining for different purposes and have limited generalization capabilities across various scenarios. Furthermore, the event representations generated by such simulators may limit their broader applications when they are not in the original signal domain. Summary of the Invention

[0005] This invention provides a method and apparatus for simulating the physical characteristics of an event camera, in order to solve the problems of traditional cameras' difficulty in high-precision detection in high-speed motion, over-illuminated and low-illuminated scenes.

[0006] A first aspect of the present invention provides a method for simulating the physical characteristics of an event camera, comprising the following steps: establishing a description file for the lens group of a target event camera to select multiple sampling interval points; calculating the exit pupil condition at each sampling interval point, and generating the exit pupil condition along the entire straight line based on the exit pupil condition at each sampling interval point; determining whether the light beam of the target scene is within the exit pupil condition along the entire straight line based on the target scene and the three-dimensional scene model obtained by modeling the target event camera; if it is within the exit pupil condition along the entire straight line, converting the light beam into a photocurrent; otherwise, ignoring the light beam; and simulating the photocurrent using the sensor of the target event camera to obtain the three-dimensional simulation result of the light beam.

[0007] Optionally, the description file for establishing the lens group of the target event camera, to select multiple sampling interval points, includes:

[0008] A description file for the lens group of the target event camera is established, wherein the description file includes the relative position, concavity / convexity, aperture size, and radius of curvature of each chip;

[0009] Based on the description file, the plurality of sampling interval points are uniformly selected on the direction plane of the beam.

[0010] Optionally, the three-dimensional scene model obtained based on the target scene and the target event camera modeling determines whether the light beam of the target scene is within the exit pupil of the entire straight line. If it is within the exit pupil of the entire straight line, the light beam is converted into photocurrent; otherwise, the light beam is ignored. This includes:

[0011] Based on the three-dimensional scene model, after the target lens group receives the light beam from the target scene, it is determined whether the light beam is within the exit pupil of the entire straight line;

[0012] If the exit pupil is within the entire straight line, the contribution of the beam to the photocurrent is considered, the beam is ray-traced using a multispectral renderer to generate a multispectral image, and the photocurrent is calculated based on the multispectral image and a preset quantum efficiency; otherwise, the beam is ignored.

[0013] Optionally, calculating the photocurrent based on the multispectral image and a preset quantum efficiency includes:

[0014] When the multispectral image has a fixed number of channels, the multispectral image and the preset quantum efficiency are numerically integrated to obtain the photocurrent;

[0015] When the multispectral image is a discrete channel, the Monte Carlo algorithm is used to perform numerical integration calculation on the multispectral image and the preset quantum efficiency to obtain the photocurrent.

[0016] Optionally, it also includes:

[0017] The similarity between the 3D simulation results and the 3D ground truth results of the target event camera is evaluated using a spatiotemporal asynchronous event metric method, wherein the spatiotemporal asynchronous event metric method includes block distance metric and Gaussian kernel distance metric.

[0018] Optionally, evaluating the similarity between the 3D simulation results and the 3D ground truth results of the target event camera using a spatiotemporal asynchronous event metric method includes:

[0019] Normalize each component of the three-dimensional simulation result and the three-dimensional truth result to obtain normalized three-dimensional simulation result and normalized three-dimensional truth result;

[0020] The two coordinate components, polarity components, and time components of the normalized three-dimensional simulation result and the normalized three-dimensional truth result are multiplied by a preset amplification value to obtain the processed simulation four-dimensional points and the processed truth four-dimensional points.

[0021] Based on the aforementioned spatiotemporal asynchronous event measurement method, a KD tree is established according to the processed simulated four-dimensional points and the processed true four-dimensional points;

[0022] Traverse the KD tree to calculate the average distance between each point in the processed simulated four-dimensional points and the processed ground truth four-dimensional points;

[0023] The similarity between the 3D simulation results and the 3D ground truth results is evaluated based on the average distance between each point.

[0024] A second aspect of the present invention provides an event camera physical characteristic simulation device, comprising: a creation module for creating a description file of the lens group of a target event camera to select multiple sampling interval points; a generation module for calculating the exit pupil condition at each sampling interval point and generating the exit pupil condition along the entire straight line based on the exit pupil condition at each sampling interval point; a conversion module for determining whether a beam of light from the target scene is within the exit pupil condition along the entire straight line based on a three-dimensional scene model obtained by modeling the target scene and the target event camera; if it is within the exit pupil condition along the entire straight line, converting the beam of light into photocurrent; otherwise, ignoring the beam of light; and a simulation module for simulating the photocurrent using the sensor of the target event camera to obtain a three-dimensional simulation result of the beam of light.

[0025] A third aspect of the present invention provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the event camera physical characteristics simulation method as described in the above embodiments.

[0026] A fourth aspect of the present invention provides a computer program product, which, when executed by a processor, implements the above-described method for simulating the physical characteristics of an event camera.

[0027] A fifth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for simulating the physical characteristics of an event camera.

[0028] The event camera physical characteristic simulation method and apparatus proposed in this invention constructs a physics-based full-link event simulator that can generate highly realistic event streams through direct interface interaction with 3D scenes, providing data for various event camera visual tasks. The physics-based renderer designs a realistic lens simulation module and improves rendering quality and speed by optimizing pupil sampling, resulting in a more realistic spectral distribution on the chip. High-precision photocurrent is generated through quantum efficiency and Monte Carlo integration, leading to more accurate event data after subsequent electronic circuit simulation. Two asynchronous spatiotemporal metrics are proposed to measure the distance between simulated and original events, thus demonstrating the high signal fidelity of the simulation method in this invention.

[0029] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0030] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0031] Figure 1 A flowchart of an event camera physical characteristics simulation method provided in an embodiment of the present invention;

[0032] Figure 2 This is a schematic diagram illustrating the specific execution of the event camera physical characteristic simulation method provided in an embodiment of the present invention;

[0033] Figure 3 This is a schematic diagram of exit pupil sampling provided in an embodiment of the present invention, wherein red represents light that cannot reach the chip smoothly, and blue represents light that can reach the chip;

[0034] Figure 4This is a schematic diagram of a rotating scene test provided in an embodiment of the present invention;

[0035] Figure 5 This is a schematic diagram of a translation scene test provided in an embodiment of the present invention;

[0036] Figure 6 This is a block diagram of the event camera physical characteristic simulation device provided in an embodiment of the present invention;

[0037] Figure 7 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0038] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0039] The method and apparatus for simulating the physical characteristics of an event camera according to embodiments of the present invention are described below with reference to the accompanying drawings.

[0040] Figure 1 This is a flowchart illustrating a method for simulating the physical characteristics of an event camera, as provided in an embodiment of the present invention.

[0041] like Figure 1 As shown, the simulation method for the physical characteristics of the camera in this event includes the following steps:

[0042] In step S101, a description file for the lens group of the target event camera is created to select multiple sampling interval points.

[0043] In step S102, the exit pupil situation at each sampling interval point is calculated, and the exit pupil situation on the entire straight line is generated based on the exit pupil situation at each sampling interval point.

[0044] In practice, in computer graphics, real camera lens simulation mimics the behavior of actual camera lenses. Ray tracing is typically used to accurately simulate the path of light through multiple lenses. This approach takes into account the complexity of the lens system, including refraction at the interface, and enhances image quality by estimating the incident radiance along any light ray.

[0045] Therefore, as Figure 2As shown, this embodiment of the invention models a lens system with different elements and tracks light rays using multispectral information, considering the interaction between the light rays and the optical system until the light rays either leave the optical system or are absorbed by the aperture or lens housing. To reduce computational waste, this embodiment of the invention performs pupil sampling before performing actual simulations to pre-calculate the set of light rays that can be emitted. The definition of the pupil differs slightly from that in existing optics; the pupil defined in this embodiment of the invention represents all directions (not just the center point on the membrane) of light rays emitted from the lens of the target event camera at a specific point on the sensor membrane.

[0046] The sensor plane coordinates of the target event camera are defined as (u,v), and the pupil is the set of feasible directions represented by a point (x,y) on the direction plane. Each point (x,y) in the pupil corresponds to a unique direction. Therefore, the mathematical definition of the pupil can be expressed as:

[0047] Pupil(u,v)={(x,y)|Ray({u,v},{x,y})}

[0048] Pupil sampling is an algorithm for determining the correct pupil at each point on the sensor membrane. While typical lens systems present challenges due to the complex pupil shape, the lens assembly of a target event camera typically has a centrally symmetrical structure with a circular aperture. This allows the pupil shape to be simplified to an ellipse, significantly simplifying pupil sampling. Therefore, the pupil approximation in this embodiment is:

[0049] Pupil(u,v)={(x,y)|{x,y}∈Elli(p,a,b)}

[0050] Where Elli is an ellipse in the plane, p is the center of the ellipse, a is the length of the semi-major axis of the ellipse, and b is the length of the semi-minor axis of the ellipse.

[0051] like Figure 3 As shown, by uniformly sampling points on the beam direction plane, the exit pupil of each sampling point on the sensor film is efficiently calculated, and light is generated within the pupil, significantly improving the speed of lens system simulation. For other points, the exit pupil is determined by linear interpolation between the center of the ellipse and its major and minor axes. In summary, this invention pioneers the combination of realistic lens simulation and target event camera simulation, improving the realism of event camera simulation and making the visual scene more lifelike.

[0052] In some embodiments, a description file for the lens group of the target event camera is created to select multiple sampling interval points, including:

[0053] Create a description file for the lens group of the target event camera, wherein the description file includes the relative position, concavity / convexity, aperture size and radius of curvature of each chip;

[0054] Based on the description file, multiple sampling interval points are uniformly selected on the direction plane of the beam.

[0055] Specifically, Blender software can be used to model the geometry and lighting information of the target scene, as well as the pose information of the target event camera at various times, to obtain a three-dimensional scene model, which is used to determine whether the beam of the target scene is within the exit pupil of the entire straight line. A lens description file is established based on the relative position, concavity and convexity, aperture size and radius of curvature of each chip in the lens group of the target event camera. According to the description file, appropriate sampling interval points are selected on the sensor plane of the target event camera, and the exit pupil of each point is calculated. Then, the exit pupil of the entire straight line is obtained by interpolation. In this embodiment of the invention, a total of 1024 samples were taken on the straight line.

[0056] In step S103, based on the three-dimensional scene model obtained by the target scene and the target event camera modeling, it is determined whether the beam of the target scene is within the exit pupil of the entire straight line. If it is within the exit pupil of the entire straight line, the beam is converted into photocurrent; otherwise, the beam is ignored.

[0057] In some embodiments, based on the 3D scene model obtained by modeling the target scene and the target event camera, it is determined whether the light beam of the target scene is within the exit pupil of the entire straight line. If it is within the exit pupil of the entire straight line, the light beam is converted into photocurrent; otherwise, the light beam is ignored. This includes:

[0058] Based on the 3D scene model, after the target lens group receives the beam of light from the target scene, it is determined whether the beam is within the exit pupil of the entire straight line.

[0059] If the exit pupil is along the entire straight line, the contribution of the beam to the photocurrent is considered. The beam is then ray-traced using a multispectral renderer to generate a multispectral image. The photocurrent is then calculated based on the multispectral image and the preset quantum efficiency. Otherwise, the beam is ignored.

[0060] Specifically, based on the 3D scene model, after the target lens group receives the light beam from the target scene, if the light beam is outside the exit pupil of the entire straight line, it is ignored; if the light beam appears within the exit pupil of the entire straight line, its contribution to the photocurrent is considered. Based on the refraction of light by the lens group, the photocurrent of the light beam at the sensor chip is calculated.

[0061] In some embodiments, calculating the photocurrent based on a multispectral image and a preset quantum efficiency includes:

[0062] With a fixed number of channels in the multispectral image, the photocurrent is obtained by numerical integration of the multispectral image and the preset quantum efficiency.

[0063] When the multispectral image is a discrete channel, the Monte Carlo algorithm is used to perform numerical integration calculations on the multispectral image and the preset quantum efficiency to obtain the photocurrent.

[0064] It should be noted that the multispectral renderer converts the light signal received by the sensor of the target event camera into photocurrent, which is usually adjusted for display convenience. However, in this embodiment of the invention, a multispectral renderer is used to detect photons of different frequencies in the sensor of the target event camera and then generate photocurrent through the photoelectric effect.

[0065] Existing simulation systems typically use optical forcing to estimate photocurrent, but real photon receivers do not convert light of different frequencies into the same number of electrons, a fact revealed by quantum efficiency curves. The absolute quantum efficiency is defined as:

[0066]

[0067]

[0068] Therefore, this invention fully utilizes known spectral information and quantum efficiency to accurately estimate the photocurrent of multispectral images. It directly calculates the photocurrent by transforming the integral equation into a numerical summation formula. This method overcomes the limitations of existing simulation systems, ensuring a more accurate representation of the photocurrent. The numerical integral solution formula is as follows:

[0069] i p =C·∫λ·QE(λ)·L(λ)·dλ

[0070] Among them, i p λ is the photocurrent, C is a constant related to the unit and circuit constant, λ is the wavelength, QE is the relative quantum efficiency, and L is the spectral curve.

[0071] Since numerical integration formulas typically lack analytical solutions when multispectral images are discrete channels, Monte Carlo integration is usually used to calculate the final photocurrent. The specific formula is as follows:

[0072]

[0073] Among them, I N The photocurrent is calculated from the samples, where N is the number of samples and λ is the number of samples. k Let p be the corresponding sampling point, and p be the sampling probability function. The sampling probability function is often pre-calculated using equally spaced sampling with a low number of samples.

[0074] This invention reduces error sources in this way, ensuring that the main error originates only from the input spectrum. Therefore, accuracy is significantly improved compared to existing methods that directly use intensity. It is worth noting that with advancements in rendering technology and the development of hyperspectral cameras, the error of this method is expected to decrease further. In contrast, due to the large error in equivalent quantum efficiency, substantial improvements in intensity-based methods are unlikely. In summary, this invention innovatively estimates photocurrent by fusing spectral data and considering quantum efficiency, enhancing simulation and more accurately representing optical signal conversion.

[0075] For example, the multispectral image on the chip obtained from multispectral rendering is first stored using `exr`, and then opened and processed in Python using `openexr`. Since the multispectral image has discrete channels, the Monte Carlo algorithm may not be used; in other words, only the pre-calculation process of Monte Carlo sampling can be used. A fixed number of channels is selected for simple numerical integration to obtain the photocurrent. In this embodiment, a 31-channel spectral representation between 400 and 700 channels is implemented. For continuous spectral representation, weights can be obtained through pre-calculation before Monte Carlo sampling. This embodiment uses built-in functions of NumPy to implement this process. For subsequent simulation processing of the generated photocurrent, any mature electronic circuit simulation device can be used, such as ICNS.

[0076] In step S104, the photocurrent is simulated using the sensor of the target event camera to obtain the three-dimensional simulation result of the beam.

[0077] In some embodiments, the method further includes: evaluating the similarity between the 3D simulation results and the 3D ground truth results of the target event camera using a spatiotemporal asynchronous event metric method, wherein the spatiotemporal asynchronous event metric method includes a block distance metric and a Gaussian kernel distance metric.

[0078] In practical implementation, to measure the similarity between simulated camera events and original camera events, this embodiment of the invention proposes two asynchronous spatiotemporal event metrics (i.e., block distance and Gaussian kernel distance), which is similar to the metric part in the iterative nearest point task. The new set of metrics is as follows:

[0079]

[0080] d1(r,q)=||rq||2

[0081]

[0082] Where R and Q are two event data to be compared, r is an event in R, q is an event in Q, d1(r,q) is a measure of block distance, d2(r,q) is a measure of Gaussian kernel distance, e is the natural logarithm, and σ is an adjustable hyperparameter.

[0083] It should be noted that, compared to street distance, Gaussian distance exhibits relative stability against outliers, making it more robust to noise. However, its measurement results have a narrower range of variation, resulting in poorer discriminative power when simultaneously evaluating multiple simulators, especially those with lower performance. Therefore, this embodiment of the invention uses both street distance and Gaussian distance to comprehensively and objectively evaluate the similarity between 3D simulation results and 3D ground truth results.

[0084] Specifically, since the units of the various components of the input event data are different, it is necessary to first normalize each component of the 3D simulation results and the 3D truth results to eliminate the influence of different units and obtain normalized 3D simulation results and normalized 3D truth results.

[0085] Furthermore, after the normalization process is completed, the two coordinate components, polar component, and time component of the normalized three-dimensional simulation result and the normalized three-dimensional true value result are multiplied by a preset amplification value to obtain the processed simulation four-dimensional point and the processed true value four-dimensional point. For example, the two coordinate components and the polar component are multiplied by 100, and the time component is multiplied by 1000.

[0086] The processed simulated 4D points and processed ground truth 4D points are used as inputs to a spatiotemporal asynchronous event metric method. A KD-tree is constructed, and the KD-tree is traversed to calculate the average distance between each point in both the processed simulated and ground truth 4D points. The similarity between the 3D simulation results and the 3D ground truth results is evaluated based on the average distance between each point. The use of the KD-tree method can significantly reduce computational complexity.

[0087] To quantitatively evaluate the effectiveness of our method, the following is presented in Table 1. Figure 4 and Figure 5 In this paper, the embodiments of the present invention are compared with three open-source event camera simulators, and the simulation results of the three open-source event camera simulators, the simulation results obtained by the embodiments of the present invention, and the results obtained by the actual EVK4 camera are compared.

[0088] Table 1

[0089]

[0090]

[0091] like Figure 4As shown, this set of rotating scene test experiments involves a turntable with different rotation speeds (60 / 360 rpm) and lighting conditions (H: high, L: low), and the corresponding simulation results are displayed. From left to right, the results are: scene illustration, three benchmark methods, simulation results obtained from the embodiment of this invention, and ground truth values ​​obtained by the EVK4 camera.

[0092] like Figure 5 As shown, this set of translation scene test experiments involves translational checkerboard patterns under different movement speeds (0.6 / 1m / s) and lighting conditions (H: high, L: low), and the corresponding simulation results are displayed. From left to right, the results are: scene illustration, three benchmark methods, simulation results obtained from the embodiment of this invention, and ground truth values ​​obtained by the EVK4 camera.

[0093] Through Table 1, Figure 4 and Figure 5 As can be seen, the event camera physical characteristic simulation method proposed in this invention outperforms the three state-of-the-art simulators in both quantitative and visual evaluation. More specifically, in terms of Gaussian metrics, compared to methods (TODO) ESIM, V2E, and ICNS, the embodiments of this invention achieve averages of 0.434, 0.236, and 0.208, respectively. Meanwhile, in terms of street metrics, compared to ESIM, V2E, and ICNS, the embodiments of this invention reduce averages by 48.749, 8.212, and 3.435, respectively. This is because the sensor models of V2E and ESIM are too simple, resulting in concentrated and discontinuous data distribution in three-dimensional space. ESIM's event data is entirely concentrated on the timestamps of the input frames, lacking continuity, which is the main reason for its poor performance. Although ICNS has an advanced sensor model and supports scene input, the lack of this part of the algorithm leads to insufficient accuracy in event data generation.

[0094] In summary, the event camera physical characteristic simulation method proposed according to the embodiments of the present invention has the following beneficial effects:

[0095] (1) Construct a physics-based full-link event simulator that can generate highly realistic event streams by directly interacting with the 3D scene, and can provide data for various event camera vision tasks;

[0096] (2) The physical renderer is designed with a realistic lens simulation module and optimizes pupil sampling to improve the quality and speed of rendering, and obtains a more realistic spectral distribution on the chip.

[0097] (3) High-precision photocurrent is generated through quantum efficiency and Monte Carlo integration, so that more accurate event data can be obtained through subsequent electronic circuit simulation;

[0098] (4) Two asynchronous spatiotemporal metrics are proposed to measure the distance between simulated events and original events, thereby proving that the simulation method of the present invention has high signal fidelity.

[0099] Next, the event camera physical characteristic simulation device according to an embodiment of the present invention is described with reference to the accompanying drawings.

[0100] Figure 6 This is a block diagram of an event camera physical characteristic simulation device according to an embodiment of the present invention.

[0101] like Figure 6 As shown, the event camera physical characteristics simulation device 60 includes: a setup module 601, a generation module 602, a conversion module 603, and a simulation module 604.

[0102] The system comprises the following modules: Module 601 establishes a description file for the lens group of the target event camera to select multiple sampling interval points. Module 602 calculates the exit pupil condition at each sampling interval point and generates the exit pupil condition along the entire straight line based on the exit pupil condition at each sampling interval point. Module 603, based on the 3D scene model obtained from the target scene and the target event camera model, determines whether the beam of the target scene is within the exit pupil condition along the entire straight line. If it is, the beam is converted into photocurrent; otherwise, the beam is ignored. Module 604 simulates the photocurrent using the sensor of the target event camera to obtain the 3D simulation result of the beam.

[0103] It should be noted that the foregoing explanation of the embodiment of the event camera physical characteristic simulation method also applies to the event camera physical characteristic simulation device of this embodiment, and will not be repeated here.

[0104] The event camera physical characteristic simulation device proposed according to an embodiment of the present invention has the following beneficial effects:

[0105] (1) Construct a physics-based full-link event simulator that can generate highly realistic event streams by directly interacting with the 3D scene, and can provide data for various event camera vision tasks;

[0106] (2) The physical renderer is designed with a realistic lens simulation module and optimizes pupil sampling to improve the quality and speed of rendering, and obtains a more realistic spectral distribution on the chip.

[0107] (3) High-precision photocurrent is generated through quantum efficiency and Monte Carlo integration, so that more accurate event data can be obtained through subsequent electronic circuit simulation;

[0108] (4) Two asynchronous spatiotemporal metrics are proposed to measure the distance between simulated events and original events, thereby proving that the simulation method of the present invention has high signal fidelity.

[0109] Figure 7 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. The electronic device may include:

[0110] The memory 701, the processor 702, and the computer program stored on the memory 701 and executable on the processor 702.

[0111] When the processor 702 executes the program, it implements the event camera physical characteristic simulation method provided in the above embodiments.

[0112] Furthermore, electronic devices also include:

[0113] Communication interface 703 is used for communication between memory 701 and processor 702.

[0114] The memory 701 is used to store computer programs that can run on the processor 702.

[0115] The memory 701 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0116] If the memory 701, processor 702, and communication interface 703 are implemented independently, then the communication interface 703, memory 701, and processor 702 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0117] Optionally, in a specific implementation, if the memory 701, processor 702, and communication interface 703 are integrated on a single chip, then the memory 701, processor 702, and communication interface 703 can communicate with each other through an internal interface.

[0118] The processor 702 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0119] This invention also provides a computer program product, which, when executed by a processor, implements the above-described method for simulating the physical characteristics of an event camera.

[0120] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for simulating the physical characteristics of an event camera.

[0121] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0122] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0123] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0124] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0125] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0126] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it includes one or a combination of the steps of the method embodiments.

[0127] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0128] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for event camera physical property simulation, the method comprising: The method comprises the following steps: establishing a description file of a lens group of a target event camera to select a plurality of sampling interval points; calculating an exit pupil condition of each sampling interval point and generating an exit pupil condition on a whole straight line according to the exit pupil condition of each sampling interval point; judging whether a light beam of a target scene is within the exit pupil condition on the whole straight line based on a three-dimensional scene model obtained by modeling the target scene and the target event camera, and converting the light beam into a photocurrent if the light beam is within the exit pupil condition on the whole straight line, or ignoring the light beam; simulating the photocurrent by using a sensor of the target event camera to obtain a three-dimensional simulation result of the light beam.

2. The event camera physical property simulation method of claim 1, wherein, The establishment of the description file of the lens group of the target event camera to select the plurality of sampling interval points comprises: establishing a description file of a lens group of a target event camera, wherein the description file comprises the relative position, concave-convex property, aperture size and curvature radius of each chip; selecting the plurality of sampling interval points uniformly on a directional plane of the light beam based on the description file.

3. The event camera physical property simulation method of claim 1, wherein, The judgment of whether the light beam of the target scene is within the exit pupil condition on the whole straight line based on the three-dimensional scene model obtained by modeling the target scene and the target event camera, and the conversion of the light beam into the photocurrent if the light beam is within the exit pupil condition on the whole straight line, or the ignoring of the light beam comprises: judging whether a light beam of a target scene is within the exit pupil condition on the whole straight line after the target lens group receives the light beam of the target scene based on the three-dimensional scene model; if the light beam is within the exit pupil condition on the whole straight line, considering the contribution of the light beam to the photocurrent, performing ray tracing on the light beam by using a multi-spectral renderer to generate a multi-spectral image, and calculating the photocurrent according to the multi-spectral image and a preset quantum efficiency, or ignoring the light beam.

4. The event camera physical property simulation method of claim 3, wherein, The calculation of the photocurrent according to the multi-spectral image and the preset quantum efficiency comprises: in a case where the multi-spectral image has a fixed channel number, performing numerical integral calculation on the multi-spectral image and the preset quantum efficiency to obtain the photocurrent; in a case where the multi-spectral image has discrete channels, performing numerical integral calculation on the multi-spectral image and the preset quantum efficiency by using a Monte Carlo algorithm to obtain the photocurrent.

5. The event camera physical property simulation method of claim 1, wherein, Further comprising: evaluating the similarity between the three-dimensional simulation result and a three-dimensional ground truth result of the target event camera by using a spatiotemporal asynchronous event measurement method, wherein the spatiotemporal asynchronous event measurement method comprises a block distance measurement and a Gaussian kernel distance measurement.

6. The event camera physical property simulation method of claim 5, wherein, The evaluation of the similarity between the three-dimensional simulation result and the three-dimensional ground truth result of the target event camera by using the spatiotemporal asynchronous event measurement method comprises: performing normalization processing on each component of the three-dimensional simulation result and the three-dimensional ground truth result to obtain a normalized three-dimensional simulation result and a normalized three-dimensional ground truth result; multiplying two coordinate components, a polarity component and a time component of the normalized three-dimensional simulation result and the normalized three-dimensional ground truth result by a preset amplification value to obtain a processed simulation four-dimensional point and a processed ground truth four-dimensional point; Based on the spatio-temporal asynchronous event metric method, a KD tree is established according to the processed simulation four-dimensional points and the processed true value four-dimensional points; The KD tree is traversed to calculate the distance average value between each point in the processed simulation four-dimensional points and the processed true value four-dimensional points; The similarity of the three-dimensional simulation result and the three-dimensional true value result is evaluated according to the distance average value between each point.

7. An event camera physical properties emulation device, comprising: Comprise: The establishing module is used for establishing a description file of a lens group of a target event camera to select a plurality of sampling interval points; The generating module is used for calculating an exit pupil condition of each sampling interval point and generating an exit pupil condition on an entire straight line according to the exit pupil condition of each sampling interval point; The converting module is used for judging whether a light beam of a target scene is within the exit pupil condition on the entire straight line based on a three-dimensional scene model obtained by modeling the target scene and the target event camera, and if the light beam is within the exit pupil condition on the entire straight line, the light beam is converted into a photocurrent, otherwise the light beam is ignored; The simulation module is used for simulating the photocurrent by using a sensor of the target event camera to obtain a three-dimensional simulation result of the light beam.

8. An electronic device, comprising: Comprise: A memory, a processor and a computer program stored in the memory and executable on the processor, the processor executes the program to implement the event camera physical property simulation method according to any one of claims 1-6.

9. A computer program product, characterised in that, The computer program / instructions are executed by the processor to implement the event camera physical property simulation method according to any one of claims 1-6.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the event camera physical property simulation method according to any one of claims 1-6.

Images