Pulse-to-event method, device and equipment based on multi-stage space-time filtering

Abstract

The invention discloses a pulse-to-event conversion method, device and equipment based on multi-stage space-time filtering. The method comprises the following steps: acquiring an input pulse stream, and calculating a first light intensity estimation value based on the pulse stream; inputting the first light intensity estimation value into a Markov random field for time domain and space domain filtering to obtain a second light intensity estimation value; and calculating a light intensity difference based on the second light intensity estimation value, generating an event when the light intensity difference is greater than or equal to a preset threshold value, and obtaining an event stream. The invention provides a motion event characterization and conversion method and device based on a pulse camera. A brand new defined event type motion characterization (hereinafter referred to as a pulse event) is generated from a pulse stream output by the pulse camera through the method. The representation can generate an event stream which is low in noise and high in fidelity and completely retains moving target information, is compatible with an existing event vision algorithm, and can be directly used for high-speed reconstruction, dynamic obstacle avoidance, unmanned aerial vehicle navigation, augmented reality and other applications.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of neuromorphic vision sensor, in particular to a pulse-to-event method, device and equipment based on multi-stage spatiotemporal filtering. BACKGROUND

[0002] At present, neuromorphic vision sensor has become a hot spot in computer vision research. The neuromorphic vision sensor simulates the biological retina based on the pulse coding mechanism: when the input light intensity changes, each pixel triggers a pulse asynchronously. This coding mechanism has ultra-low delay and ultra-high dynamic range while achieving microsecond-level time resolution. In contrast, the traditional camera outputs low frame rate images based on exposure. In the scene of fast motion, it is easy to produce blur and inter-frame information loss, which seriously restricts the performance of tasks such as optical flow estimation or autonomous driving perception. An increasing number of studies have shown that neuromorphic vision sensors can effectively alleviate these problems.

[0003] The current mainstream neuromorphic camera is divided into two categories: event cameras output ON / OFF events when the change of light intensity at a pixel point exceeds a set threshold; and pulse cameras generate a binary pulse stream by accumulating the input light intensity to trigger a pulse. Although both of them break the traditional imaging paradigm, there is a fundamental difference in the essence of their output information. The inter-pulse interval (ISI) of the pulse camera is inversely proportional to the light intensity of the corresponding pixel, which can achieve complete texture reconstruction. The event camera discards static information in design and only represents the light intensity change caused by motion. The mapping from light intensity to event is unidirectional. Therefore, how to realize pulse-to-event is a technical problem to be solved by those skilled in the art. SUMMARY

[0004] Embodiments of the present application provide a pulse-to-event method, device and equipment based on multi-stage spatiotemporal filtering, to at least solve the technical problem that it is difficult to realize pulse-to-event in related technologies.

[0005] According to an aspect of an embodiment of the present application, a pulse-to-event method based on multi-stage spatiotemporal filtering is provided, comprising: obtaining an input pulse stream, and calculating a first light intensity estimate based on the pulse stream; inputting the first light intensity estimate into a Markov random field for time domain and spatial domain filtering to obtain a second light intensity estimate; calculating a light intensity difference based on the second light intensity estimate, and generating an event when the light intensity difference is greater than or equal to a preset threshold to obtain an event stream.

[0006] In one implementation, calculating a first light intensity estimate based on the pulse stream comprises: The pulse stream is input into a short-time plasticity model and filtered in the time domain to obtain the first light intensity estimate.

[0007] In one implementation, the pulse stream is input into a short-time plasticity model for time-domain filtering to obtain the first light intensity estimate, including: The pulse flow is used as input to the short-time plasticity model, and intermediate variables are calculated based on the difference equation. The first light intensity estimate is calculated based on the intermediate variable.

[0008] In one implementation, the first light intensity estimate is input into a Markov random field for time-domain and spatial-domain filtering to obtain the second light intensity estimate, including: Input the first light intensity estimate into the Markov random field to obtain the intermediate variables of the Markov random field; The second light intensity estimate is calculated based on the intermediate variables of the Markov random field.

[0009] In one implementation, a light intensity difference is calculated based on the second light intensity estimate, and an event is generated when the light intensity difference is greater than or equal to a preset threshold, resulting in an event stream, including: Calculate the difference in light intensity between the second light intensity estimate at the current moment and at the moment of the last event emission; Determine whether the light intensity difference is greater than or equal to a preset threshold; An event is generated when the light intensity difference is greater than or equal to a preset threshold, and an event stream is obtained.

[0010] In one implementation, a light intensity difference is calculated based on the second light intensity estimate, and an event is generated when the light intensity difference is greater than or equal to a preset threshold, resulting in an event stream, including: Calculate the light intensity difference between the second light intensity estimate of the current frame and the second light intensity estimate of several previous frames; Determine whether the light intensity difference is greater than or equal to a preset threshold; An event is generated when the light intensity difference is greater than or equal to a preset threshold, and an event stream is obtained.

[0011] In one implementation, calculating a first light intensity estimate based on the pulse stream includes: Calculate the first light intensity estimate based on the pulse interval; or, The first light intensity estimate is calculated using a sliding window reconstruction method; or, Based on the image segmentation reconstruction method, the first light intensity estimate is calculated; or, The first light intensity estimate is calculated using a motion-compensated reconstruction method.

[0012] According to another aspect of the embodiments of this application, a pulse-to-event device based on multi-stage spatiotemporal filtering is provided, comprising: The first estimation module is used to acquire the input pulse stream and calculate the first light intensity estimate based on the pulse stream; The second estimation module is used to input the first light intensity estimate into a Markov random field for time-domain and spatial-domain filtering to obtain the second light intensity estimate. The event generation module is used to calculate the light intensity difference based on the second light intensity estimate, and generate an event when the light intensity difference is greater than or equal to a preset threshold, thereby obtaining an event stream.

[0013] In one implementation, the event generation module is configured to calculate the light intensity difference between the second light intensity estimate at the current time and the time of the last event issuance; determine whether the light intensity difference is greater than or equal to a preset threshold; and generate an event if the light intensity difference is greater than or equal to the preset threshold, thereby obtaining an event stream.

[0014] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-described pulse-to-event method based on multi-stage spatiotemporal filtering through the computer program.

[0015] The technical solutions provided in this application embodiment may include the following beneficial effects: The proposed method first calculates a first light intensity estimate based on the input pulse stream; then, it inputs the first light intensity estimate into a Markov random field for temporal and spatial filtering to obtain a second light intensity estimate. Next, it calculates the light intensity difference based on the second light intensity estimate, generating an event stream when the light intensity difference is greater than or equal to a preset threshold. Intensity information is reconstructed from the pulse data and then resynthesized into an event stream emphasizing motion contours. This application employs a multi-stage spatiotemporal filtering light intensity estimation method, which can generate a high-quality event stream.

[0016] Furthermore, this application proposes an asynchronous differential method based on dual filtering of short-time plasticity and Markov random fields, and pulse-driven analysis of the final light intensity estimate. This method fully leverages the high sampling rate of pulse cameras, conforms to the working principle of event cameras, reduces the impact of quantization errors and noise, and can generate low-noise, high-quality event streams in various motion scenarios while completely preserving the information of moving objects. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1This is a flowchart of an optional pulse-to-event method based on multi-stage spatiotemporal filtering according to an embodiment of this application; Figure 2 This is a flowchart of an optional pulse-to-event method based on multi-stage spatiotemporal filtering according to an embodiment of this application; Figure 3 This is a schematic diagram of a pulse-to-event device based on multi-stage spatiotemporal filtering according to an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an optional electronic device according to an embodiment of this application. Detailed Implementation

[0018] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0019] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0020] Event cameras generate events by detecting changes in the light intensity of pixels. When the light intensity of a pixel changes, the event camera records an event, including the time, location, and direction of the light intensity change (increase or decrease). This representation method makes event cameras highly sensitive to motion, enabling them to quickly capture information such as the trajectory and speed of moving objects.

[0021] The mapping from light intensity to events by event cameras is unidirectional; that is, events can be generated from changes in light intensity, but complete light intensity information cannot be directly reconstructed from events. This is because event cameras only record partial information about the changes in light intensity, losing the light intensity information of the static parts. This unidirectional mapping presents certain challenges to the processing and application of event data, but it also provides inspiration for new technological concepts.

[0022] The concept of S2E (Simulation-to-Event) technology reconstructs intensity information from pulse data. Pulse data contains temporal and spatial information about light intensity changes. Using specific algorithms and models, the light intensity distribution of the scene can be inferred from this pulse data. This step essentially supplements the static light intensity information lost by the event camera, enabling subsequent processing to obtain more complete scene information.

[0023] After reconstructing the intensity information, S2E technology resynthesizes this intensity information into an event stream that emphasizes motion contours. This process is not simply about restoring the output of the event camera, but rather about optimizing and processing the synthesized event stream to better highlight motion features, making it more beneficial for subsequent computer vision tasks. For example, edge detection and motion estimation can be performed on the reconstructed intensity information to generate a more accurate and clearer event stream with motion contours.

[0024] In some implementations, light intensity is first estimated using pulse intervals, and then an event is generated by comparing the difference between the light intensity estimates of the current frame and the previous frame. This method has several drawbacks: (1) The light intensity estimate obtained directly through the pulse interval is not stable enough and is easily affected by quantization error and the noise of the pulse camera itself.

[0025] (2) This method is inconsistent with the working principle of event cameras.

[0026] (3) Events are generated by comparing the difference between the light intensity estimates of the current frame and the previous frame. The resulting event stream is too sparse. Even in areas with high-speed motion, there are only very sparse events, while in areas with low-speed motion, there are almost no events. This also causes this method to lose a lot of information about moving objects.

[0027] This application proposes a pulse-to-event method based on dual filtering using a short-time plasticity model and a Markov random field, and driven by pulses. The method first applies a short-time plasticity model to the input pulse stream for time-domain filtering to obtain a preliminary light intensity estimate. Then, it applies a Markov random field to the preliminary light intensity estimate for time-spatial-domain filtering to obtain a more accurate light intensity estimate. Finally, the final light intensity estimate is subjected to pulse-driven asynchronous differential processing to obtain the output event stream.

[0028] This method aligns with the working principle of event cameras, mitigating the effects of quantization errors and noise. It can generate low-noise, high-quality event streams in various motion scenarios while fully preserving the information of moving objects.

[0029] The pulse-to-event method based on multi-stage spatiotemporal filtering according to embodiments of this application will be described in detail below with reference to the accompanying drawings. Figure 1As shown, the method mainly includes the following steps: S101 acquires the input pulse stream and calculates the first light intensity estimate based on the pulse stream.

[0030] In a preferred embodiment, the pulse stream is input into a short-time plasticity model and filtered in the time domain to obtain a first light intensity estimate.

[0031] Specifically, the pulse flow is used as the input to the short-time plasticity model, and the intermediate variables R and u are calculated based on the difference equation. The intermediate variables R and u of short-time plasticity can be calculated based on the following formulas (1) and (2).

[0032]

[0033] in, , Represents pixels In the The and the first Between pulses and The value, Represents pixels The first pulse intervals, These are the parameters of the STP model.

[0034] Further, the first light intensity estimate is calculated based on the intermediate variables. Using formulas (3), (4), and (5), according to... and The first light intensity estimate is obtained. .

[0035]

[0036] in, express The weight. , Indicates an intermediate variable.

[0037] Short-Term Plasticity (STP) models are used to describe the dynamic changes of neuronal synapses over short periods (typically milliseconds to seconds). They primarily simulate how synaptic transmission efficiency is affected by previous activity over a short time, thus influencing the transmission and processing of neural signals. They can simulate the adaptation and adjustment of neurons to signals over a short period. In applications such as light intensity estimation, this model calculates intermediate variables R and u to obtain a preliminary estimate of light intensity, and then optimizes the estimation result by adjusting the weights, thereby more accurately reflecting the actual light intensity.

[0038] S102 inputs the first light intensity estimate into a Markov random field for time and spatial filtering to obtain the second light intensity estimate.

[0039] A Markov Random Field (MRF) is a probabilistic graphical model used to model the interdependencies between random variables.

[0040] In a preferred embodiment, the first light intensity estimate is input into a Markov random field for time-domain and spatial-domain filtering to obtain a second light intensity estimate.

[0041] Specifically, the initial light intensity estimate is input into a Markov random field to obtain intermediate variables of the Markov random field. The initial light intensity estimate is then used... As input to the Markov random field, the intermediate variables of the Markov random field are obtained using formula (6). :

[0042] in, It is a constant used to control the input intensity. This represents the hidden state of the Markov random field (initially set to 0). Represents pixels The set of all adjacent pixels, Represents pixels and The connection weights between nodes determine the spatial connection method of the Markov Random Field. In this embodiment, the Markov Random Field adopts uniform spatial connection, and the total connection weight of each node is denoted as... .

[0043] Furthermore, based on the intermediate variables of the Markov random field, the second light intensity estimate is calculated. Using formula (7), according to... To obtain a more accurate light intensity estimate :

[0044] Inputting the first light intensity estimate into a Markov Random Field (MRF) for temporal and spatial filtering significantly improves the accuracy and robustness of light intensity estimation. Through its undirected graph structure, the MRF fully utilizes the spatial correlation and temporal dependence between pixels to effectively suppress noise, smooth light intensity changes, and preserve important edge and detail information. In temporal filtering, the MRF captures the continuity of light intensity over time, reducing errors caused by time jitter; spatial filtering utilizes the local correlation between pixels to further optimize light intensity estimation, making the image closer to the real scene. This process not only improves the accuracy of light intensity estimation but also enhances its adaptability to complex scenes.

[0045] S103 calculates the light intensity difference based on the second light intensity estimate, and generates an event when the light intensity difference is greater than or equal to a preset threshold, thus obtaining an event stream.

[0046] In one implementation, the light intensity difference is calculated based on the second light intensity estimate, and an event is generated when the light intensity difference is greater than or equal to a preset threshold to obtain an event stream, including: performing pulse-driven asynchronous differential on the final light intensity estimate and outputting the event stream.

[0047] Specifically, the difference between the estimated second light intensity at the current time and the time of the last event is calculated; it is determined whether the difference in light intensity is greater than or equal to a preset threshold; if the difference in light intensity is greater than or equal to the preset threshold, an event is generated, and an event stream is obtained.

[0048] First, the difference between the current estimated light intensity and the estimated light intensity at the time of the last event is calculated. Next, it is determined whether this light intensity difference is greater than or equal to a preset threshold. If the difference is greater than or equal to the threshold, an event is generated at the current time and added to the event stream. In this way, an event stream can be dynamically generated based on changes in light intensity, thereby achieving efficient capture and representation of dynamic changes in the scene and providing a foundation for subsequent event-driven processing.

[0049] Calculate using formula (8) The difference between the current moment and the last event dispatch time is used to generate events based on whether the difference exceeds a threshold, ultimately resulting in a generated event stream. .

[0050]

[0051] in, Represents pixels At any moment The most recent time this event was held, It is a very small constant (used to avoid) (and division by zero error) Indicates the threshold of the event camera. The event generation function is defined as shown in equation (9):

[0052] Specifically, for each pixel x at the current time t, the calculated light intensity difference ΔI(x,t) is compared with a preset threshold θ. If ΔI(x,t) is greater than or equal to the threshold θ, a positive event is generated at pixel x with a value of 1, indicating an increase in light intensity; if ΔI(x,t) is less than or equal to the negative threshold -θ, a negative event is generated at pixel x with a value of -1, indicating a decrease in light intensity; if ΔI(x,t) is between -θ and θ, no event is generated with a value of 0, indicating that the change in light intensity is insufficient to trigger an event. In this way, an event stream E can be dynamically generated based on changes in light intensity, thereby achieving efficient capture and representation of dynamic changes in the scene.

[0053] In one alternative implementation, the light intensity difference is calculated based on the second light intensity estimate, and an event is generated when the light intensity difference is greater than or equal to a preset threshold to obtain an event stream, including using a sliding window-based event generation method.

[0054] Set the sliding window to, for example, 3, calculate the light intensity difference between the second light intensity estimate of the current frame and the second light intensity estimate of several previous frames, such as the first three frames; determine whether the light intensity difference is greater than or equal to a preset threshold; generate an event if the light intensity difference is greater than or equal to the preset threshold, and obtain the event stream.

[0055] This application employs a pulse-driven asynchronous differential method to calculate the difference between the estimated second light intensity at the current moment and the light intensity at the moment of the previous event emission. When the difference is greater than or equal to a preset threshold, an event is generated, forming an event stream. This method effectively improves the density and information content of the event stream.

[0056] In one optional implementation, calculating the first light intensity estimate based on the pulse flow includes: calculating the first light intensity estimate based on the pulse interval; or, calculating the first light intensity estimate based on a sliding window reconstruction method; or, calculating the first light intensity estimate based on an image segmentation reconstruction method; or, calculating the first light intensity estimate based on a motion compensation reconstruction method. This application does not impose specific limitations.

[0057] This application proposes a pulse-to-event method that combines short-time plasticity and Markov random field dual filtering. This method not only utilizes the short-time plasticity model to simulate the dynamic characteristics of biological neurons, but also incorporates the spatiotemporal filtering capability of Markov random fields to generate an event stream through a pulse-driven mechanism.

[0058] This method not only effectively mitigates the impact of quantization errors and noise, but also generates low-noise, high-quality event streams in various motion scenarios, completely preserving the information of moving objects. By using a short-time plasticity model to simulate the dynamic characteristics of biological neurons, it enhances sensitivity to dynamic changes; while the spatiotemporal filtering capability of Markov random fields further optimizes the generation of event streams, improving their density and accuracy. This dual filtering mechanism fully leverages the high sampling rate advantage of pulse cameras, aligns with the working principle of event cameras, and significantly improves the performance of event streams in downstream tasks such as motion detection and optical flow estimation, demonstrating significant application value.

[0059] To facilitate understanding of the pulse-to-event method provided in the embodiments of this application, the following is in conjunction with the appendix. Figure 2 Further description.

[0060] like Figure 2 As shown in the image, this illustrates a flowchart of a pulse-to-event method. The process begins with pulse stream data, first obtaining an initial light intensity estimate through temporal filtering based on short-time plasticity. Then, temporal-spatial filtering based on Markov random fields further optimizes the light intensity estimate, resulting in a more accurate estimate. Next, asynchronous differential calculation driven by pulses calculates the difference between the current light intensity estimate and the light intensity estimate at the time of the previous event. When the light intensity difference exceeds a preset threshold, event data is generated. The entire process aims to generate a low-noise, high-quality event stream while fully preserving information about moving objects.

[0061] According to another aspect of the embodiments of this application, a pulse-to-event device based on multi-stage spatiotemporal filtering for implementing the above-described pulse-to-event method based on multi-stage spatiotemporal filtering is also provided. For example... Figure 3 As shown, the device includes: The first estimation module 301 is used to acquire the input pulse stream and calculate the first light intensity estimate based on the pulse stream; The second estimation module 302 is used to input the first light intensity estimate into a Markov random field for time-domain and spatial-domain filtering to obtain the second light intensity estimate. The event generation module 303 is used to calculate the light intensity difference based on the second light intensity estimate, generate an event when the light intensity difference is greater than or equal to a preset threshold, and obtain an event stream.

[0062] The event generation module is used to calculate the difference in light intensity between the current time and the time of the last event; determine whether the difference in light intensity is greater than or equal to a preset threshold; generate an event if the difference in light intensity is greater than or equal to the preset threshold, and obtain an event stream.

[0063] It should be noted that the pulse-to-event device based on multi-stage spatiotemporal filtering provided in the above embodiments is only illustrated by the division of the functional modules described above when executing the pulse-to-event method based on multi-stage spatiotemporal filtering. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the pulse-to-event device based on multi-stage spatiotemporal filtering and the pulse-to-event method embodiments based on multi-stage spatiotemporal filtering provided in the above embodiments belong to the same concept, and the implementation process is detailed in the method embodiments, which will not be repeated here.

[0064] According to another aspect of the embodiments of this application, an electronic device corresponding to the pulse-to-event method based on multi-stage spatiotemporal filtering provided in the foregoing embodiments is also provided, so as to execute the pulse-to-event method based on multi-stage spatiotemporal filtering described above.

[0065] Please refer to Figure 4 This illustrates a schematic diagram of an electronic device provided by some embodiments of this application. For example... Figure 4 As shown, the electronic device includes: a processor 400, a memory 401, a bus 402, and a communication interface 403. The processor 400, the communication interface 403, and the memory 401 are connected via the bus 402. The memory 401 stores a computer program that can run on the processor 400. When the processor 400 runs the computer program, it executes the pulse-to-event method based on multi-stage spatiotemporal filtering provided in any of the foregoing embodiments of this application.

[0066] The memory 401 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 403 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.

[0067] Bus 402 can be an ISA bus, PCI bus, or EISA bus, etc. Buses can be divided into address buses, data buses, control buses, etc. Memory 401 is used to store programs. After receiving execution instructions, processor 400 executes the program. The pulse-to-event method based on multi-stage spatiotemporal filtering disclosed in any of the foregoing embodiments of this application can be applied to processor 400, or implemented by processor 400.

[0068] The processor 400 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 400 or by instructions in software form. The processor 400 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 401. The processor 400 reads the information in memory 401 and, in conjunction with its hardware, completes the steps of the above method.

[0069] The electronic device provided in this application embodiment and the pulse-to-event method based on multi-stage spatiotemporal filtering provided in this application embodiment are based on the same inventive concept and have the same beneficial effects as the methods they adopt, operate or implement.

[0070] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0071] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this patent should be determined by the appended claims.

Claims

1. A pulse-to-event method based on multi-stage spatiotemporal filtering, characterized in that, include: Acquire the input pulse stream, and calculate the first light intensity estimate based on the pulse stream; The first light intensity estimate is input into a Markov random field for time and spatial filtering to obtain the second light intensity estimate. The light intensity difference is calculated based on the second light intensity estimate. An event is generated when the light intensity difference is greater than or equal to a preset threshold, and an event stream is obtained.

2. The method according to claim 1, characterized in that, The calculation of the first light intensity estimate based on the pulse stream includes: The pulse stream is input into a short-time plasticity model and filtered in the time domain to obtain the first light intensity estimate.

3. The method according to claim 2, characterized in that, The pulse stream is input into a short-time plasticity model and filtered in the time domain to obtain the first light intensity estimate, including: The pulse flow is used as input to the short-time plasticity model, and intermediate variables are calculated based on the difference equation. The first light intensity estimate is calculated based on the intermediate variable.

4. The method according to claim 1, characterized in that, The first light intensity estimate is input into a Markov random field for time and spatial filtering to obtain the second light intensity estimate, including: Input the first light intensity estimate into the Markov random field to obtain the intermediate variables of the Markov random field; The second light intensity estimate is calculated based on the intermediate variables of the Markov random field.

5. The method according to claim 1, characterized in that, The light intensity difference is calculated based on the second light intensity estimate. When the light intensity difference is greater than or equal to a preset threshold, an event is generated, resulting in an event stream, including: Calculate the difference in light intensity between the second light intensity estimate at the current moment and at the moment of the last event emission; Determine whether the light intensity difference is greater than or equal to a preset threshold; An event is generated when the light intensity difference is greater than or equal to a preset threshold, and an event stream is obtained.

6. The method according to claim 1, characterized in that, The light intensity difference is calculated based on the second light intensity estimate. When the light intensity difference is greater than or equal to a preset threshold, an event is generated, resulting in an event stream, including: Calculate the light intensity difference between the second light intensity estimate of the current frame and the second light intensity estimate of several previous frames; Determine whether the light intensity difference is greater than or equal to a preset threshold; An event is generated when the light intensity difference is greater than or equal to a preset threshold, and an event stream is obtained.

7. The method according to claim 1, characterized in that, The calculation of the first light intensity estimate based on the pulse stream includes: Calculate the first light intensity estimate based on the pulse interval; or, The first light intensity estimate is calculated using a sliding window reconstruction method; or, Based on the image segmentation reconstruction method, the first light intensity estimate is calculated; or, The first light intensity estimate is calculated using a motion-compensated reconstruction method.

8. A pulse-to-event device based on multi-stage spatiotemporal filtering, characterized in that, include: The first estimation module is used to acquire the input pulse stream and calculate the first light intensity estimate based on the pulse stream; The second estimation module is used to input the first light intensity estimate into a Markov random field for time-domain and spatial-domain filtering to obtain the second light intensity estimate. The event generation module is used to calculate the light intensity difference based on the second light intensity estimate, and generate an event when the light intensity difference is greater than or equal to a preset threshold, thereby obtaining an event stream.

9. The apparatus according to claim 8, characterized in that, The event generation module is used to calculate the difference in light intensity between the second light intensity estimate at the current time and the time of the last event; determine whether the light intensity difference is greater than or equal to a preset threshold; and generate an event if the light intensity difference is greater than or equal to the preset threshold, thereby obtaining an event stream.

10. An electronic device, characterized in that, It includes a processor and a memory storing program instructions, the processor being configured to, when executing the program instructions, perform the pulse-to-event method based on multi-stage spatiotemporal filtering as described in any one of claims 1 to 7.

Images