A scene gray image recovery system and method based on a dynamic vision sensor
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2023-11-21
- Publication Date
- 2026-06-09
Smart Images
Figure CN117579946B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision computational imaging technology, specifically to a scene grayscale image restoration system and method based on a dynamic vision sensor. Background Technology
[0002] In recent years, dynamic vision sensors (or event cameras) have played a crucial role in numerous fields of computer vision due to their high temporal resolution, such as optical flow estimation, motion vector estimation, and motion blur removal. Unlike traditional image sensors, dynamic vision sensors do not rely on a fixed exposure time to acquire light intensity information. Instead, they sense changes in brightness within a scene with a microsecond-level response speed, outputting positive events representing brightening and negative events representing darkening. Therefore, dynamic vision sensors do not possess the traditional concept of image grayscale; the raw information they acquire are discrete points in space and time.
[0003] However, acquiring grayscale information at different locations within a scene is crucial. Most modern computer vision applications are built upon the analysis of grayscale images. Image grayscale is a linear mapping of brightness changes in a scene, representing the most direct visual information humans acquire. Therefore, scenarios using dynamic vision sensors also heavily rely on grayscale information provided by traditional image sensors. Currently, methods for simultaneously acquiring grayscale images in dynamic vision sensor applications can be broadly categorized into three types: 1. Directly deploying a traditional image sensor on top of the dynamic vision sensor, such as Inivation's Davis series cameras. This approach naturally achieves spatial registration and temporal synchronization between grayscale images and events, allowing the device itself to simultaneously output the spatial and temporal characteristics of scene changes. However, this implementation places a significant burden on transmission bandwidth, as grayscale image transmission occupies a large portion of the original dynamic vision sensor information transmission space. 2. Coupling the dynamic vision sensor and traditional image sensor together using a beam splitting system to share an optical path for spatial registration, and achieving temporal synchronization through external triggering. In this approach, the dynamic vision sensor and the traditional image sensor do not share a transmission link, thus both can achieve high data transmission rates. However, the overall system complexity is significantly increased, and system stability is difficult to guarantee. 3. Using algorithms, discrete event information acquired by the dynamic vision sensor is restored into grayscale images. A representative algorithm is E2VID. Its basic principle is to use the brightness change information acquired by the dynamic vision sensor during motion, and through a certain mapping relationship, use a neural network to recover the grayscale information of the scene. This method has two major drawbacks: first, it depends on motion and is completely unsuitable when the scene and camera are stationary; second, the restoration effect is poor, and the restoration effect varies greatly between different devices. Summary of the Invention
[0004] To overcome the drawbacks of traditional grayscale image restoration methods based on dynamic vision sensors, which rely on motion and have poor image restoration results, this invention proposes a scene grayscale image restoration system and method based on dynamic vision sensors.
[0005] The technical solution of the present invention is as follows:
[0006] On one hand, the present invention provides a method for scene grayscale image restoration based on a dynamic vision sensor, which includes the following steps:
[0007] 1) An image acquisition system is arranged, the image acquisition system including an optical imaging system, an incident light intensity control component and a dynamic vision sensor; wherein, the incident light intensity control component is arranged in the imaging optical path, and the incident light intensity control component is controlled by the dynamic vision sensor to change the transmittance of the optical imaging system, thereby changing the light intensity illuminating the sensor plane.
[0008] 2) Before the dynamic vision sensor starts collecting scene information, the incident light intensity control component completely blocks the light from the external scene from shining onto the sensor plane.
[0009] 3) When the dynamic vision sensor starts to collect scene information, the incident light intensity control component is gradually turned on until it is fully turned on. The dynamic vision sensor acquires the brightness change events on the sensor plane during this process.
[0010] 4) Based on the relationship between brightness and time, the timestamps of the brightness change events collected in step 3) are converted into gray levels on the corresponding pixels to obtain a time-mapped grayscale image.
[0011] 5) Based on the relationship between the number of events and brightness, using the brightness change events collected in step 3), count the number of positive events generated by each pixel of the dynamic vision sensor, and use the logarithmic mapping relationship to obtain the event number-mapped grayscale image.
[0012] 6) Perform gamma correction and denoising on the time-mapped grayscale image obtained in step 4) and the event number-mapped grayscale image obtained in step 5), and then fuse the two to obtain the final scene grayscale image.
[0013] As a preferred embodiment of the present invention, the incident light intensity control component can be implemented by any device or combination of devices that can change the transmittance of the optical imaging system, such as an electrically adjustable aperture, an electrically rotating polarizing reducer, or a liquid crystal shutter.
[0014] According to an optional embodiment of the present invention, the incident light intensity control component is an electrically adjustable aperture with continuously variable aperture size. The electrically adjustable aperture is installed at the aperture stop position of the optical imaging system. A dynamic visual sensor sends a trigger signal to the electrically adjustable aperture to continuously change the aperture size, thereby controlling the transmittance of the optical imaging system. The dynamic visual sensor collects brightness change information during the aperture size change.
[0015] According to another alternative embodiment of the present invention, the incident light intensity control component is a liquid crystal shutter, and the dynamic vision sensor sends a signal to adjust the transmittance of the liquid crystal shutter and collects the brightness change information when the transmittance changes.
[0016] On the other hand, the present invention provides a scene grayscale image restoration system for implementing the method, comprising:
[0017] An image acquisition system includes an optical imaging system, an incident light intensity control component, and a dynamic vision sensor; wherein the incident light intensity control component is arranged in the imaging optical path and is controlled by the dynamic vision sensor to change the transmittance of the optical imaging system, thereby changing the light intensity illuminating the sensor plane.
[0018] The time mapping module converts the timestamps of the collected brightness change events into gray levels on the corresponding pixels based on the relationship between brightness and time, thus obtaining a time-mapped grayscale image.
[0019] The event number mapping module establishes a mapping relationship between grayscale and event number based on the relationship between event number and brightness; it uses the collected brightness change events to count the number of positive events generated by each pixel of the dynamic vision sensor, and uses the logarithmic mapping relationship to obtain the event number mapped grayscale image.
[0020] The gamma correction module performs gamma correction on time-mapped grayscale images and event-mapped grayscale images;
[0021] The filtering and noise reduction module denoises the time-mapped grayscale image and the event-mapped grayscale image after gamma correction.
[0022] The fusion module fuses the denoised time-mapped grayscale image and the event-mapped grayscale image to obtain the final scene grayscale image.
[0023] This invention enables dynamic vision sensors, which previously could only output discrete information about brightness changes, to also output grayscale images. Compared to existing grayscale image restoration methods for dynamic vision sensors, this invention is compatible with static scenes and offers superior restoration results. Compared to existing conventional image sensors, this invention enables high dynamic range and high-speed imaging in dark and moving scenes, showing broad prospects in fields such as intelligent driving and security. Attached Figure Description
[0024] Figure 1 This is a schematic diagram of the electric variable aperture image acquisition system proposed in this invention;
[0025] Figure 2 This is a schematic diagram of the liquid crystal shutter image acquisition system proposed in this invention;
[0026] Figure 3 This is the high dynamic range time-mapped grayscale image obtained in step 4 of this embodiment of the invention;
[0027] Figure 4 It is the grayscale image obtained in steps 4, 5, and 6 of the embodiment of the present invention.
[0028] Figure 5 This is the low-light resolution plate time-mapped grayscale image obtained in step 4 of the embodiment of the present invention. Detailed Implementation
[0029] The present invention will be further described and illustrated below with reference to specific embodiments. The embodiments described are merely examples of the content of this disclosure and do not limit the scope of the invention. The technical features of each embodiment in the present invention can be combined accordingly, provided that there is no mutual conflict.
[0030] The method of this invention is implemented based on an image acquisition system, which includes an optical imaging system, an incident light intensity control component, and a dynamic vision sensor. The incident light intensity control component is arranged in the imaging optical path and is controlled by the dynamic vision sensor to change the transmittance of the optical imaging system, thereby changing the light intensity illuminating the sensor plane. The incident light intensity control component can be implemented using a motorized variable aperture, a motorized rotary polarizing neutral density filter, a liquid crystal shutter, etc. Figure 1 and 2 The diagram shown is a schematic representation of two representative image acquisition systems provided by this invention.
[0031] Figure 1 This invention presents an electrically operated variable aperture image acquisition system, with an incident light intensity control component being an electrically operated variable aperture 13. 1 is an optical imaging system, including a front lens group 11 and a rear lens group 12. The electrically operated variable aperture 13 is positioned at the aperture stop of the optical imaging system. 2 is a dynamic vision sensor, which sends a trigger signal to the electrically operated variable aperture 13, causing the aperture size to continuously change. The dynamic vision sensor acquires brightness change information during the aperture size change.
[0032] Figure 2This invention relates to a liquid crystal shutter image acquisition system, with the incident light intensity control component being a liquid crystal shutter 2. 1 is an optical imaging system. 2 is the liquid crystal shutter, composed of two layers of liquid crystal molecules. The orientation of the first layer of liquid crystal molecules is fixed, converting the incident light into linearly polarized light. The orientation of the second layer of liquid crystal molecules changes according to an external electrical signal, thereby altering the intensity of the linearly polarized light emitted from the first layer. When the orientations of the two layers of liquid crystal molecules are the same, the light transmittance of the liquid crystal shutter is maximum; when the orientations are 90 degrees, the light transmittance is 0. 3 is a dynamic vision sensor that sends signals to the second layer of the liquid crystal shutter 2 to adjust the transmittance of the liquid crystal shutter and collects brightness change information under different transmittance levels during the process.
[0033] See Figure 1 Components 1 and 2 together form a traditional dynamic vision sensor information acquisition system. However, an electrically operated variable aperture 13 controlled by component 2 is added to component 1. Its function is to adjust the brightness of the light illuminating the plane of the dynamic vision sensor in a time-division manner.
[0034] The selection of the motorized variable aperture 13 has the following requirements: the motorized variable aperture must not leak light when the aperture is fully closed. The compliance of the motorized variable aperture can be evaluated as follows: with the aperture fully closed, place the acquisition system in front of a flickering screen. If the dynamic vision sensor does not detect any brightness change events, it indicates that the motorized variable aperture meets the requirements.
[0035] The motorized variable aperture 13 has the following installation requirements: the motorized variable aperture must coincide with the aperture stop plane designed in the optical imaging system. The purpose is to ensure that the brightness change is the same at all points in the field of view during aperture size changes.
[0036] See Figure 2 This paper provides a system installation scheme in which a liquid crystal shutter 2 is installed between an optical imaging system 1 and a dynamic vision sensor 3. The liquid crystal shutter 2 controls the transmittance of light emitted from the optical imaging system 1 and illuminating the plane of the dynamic vision sensor, and its function is to perform time-division control of the brightness of the light illuminating the plane of the dynamic vision sensor.
[0037] The selection of the LCD shutter 2 has the following requirements: the light leakage coefficient of the LCD shutter should not be too high when it is fully closed. The evaluation of whether the LCD shutter meets the standard can be done as follows: with the LCD shutter fully closed, place the acquisition system in front of a flickering screen. If the dynamic vision sensor does not detect any brightness change events, it indicates that the LCD shutter meets the standard.
[0038] In the LCD shutter image acquisition system, the installation position of the LCD shutter 2 is relatively flexible. It can be installed in front of the optical imaging system, inside the optical imaging system, or between the optical imaging system and the dynamic vision sensor. Its function is to adjust the light intensity illuminating the plane of the dynamic vision sensor in a time-division manner.
[0039] The most crucial component in a dynamic vision sensor grayscale recovery and acquisition system is the incident light intensity control unit, as shown in the attached figure. Figure 1 The motorized variable aperture 13 and its attachment Figure 2 The purpose of the LCD shutter 2 is to ensure that the brightness of all points on the dynamic vision sensor plane can gradually increase from 0 with the same degree of change.
[0040] Combination Figure 1 and Figure 2 The scene grayscale image restoration method based on a dynamic vision sensor provided by this invention can be implemented according to the following steps:
[0041] Step 1: Construction of the dynamic visual sensor grayscale recovery and acquisition system. (Based on the example in this patent...) Figure 1 Or example Figure 2 A dynamic visual grayscale recovery and acquisition system was built. For example... Figure 1 In the system shown, the electrically adjustable aperture 13 must be positioned at the aperture stop of the optical imaging system 1. For example... Figure 2 In the system shown, the light leakage coefficient of the LCD shutter cannot be too high when it is closed.
[0042] Step 2: Completely prevent light from the external scene from entering the acquisition system. Before the dynamic vision sensor begins acquiring information, [the system will be configured to prevent light from entering the system]. Figure 1 The motorized variable aperture 13 in the middle is completely closed, or for example... Figure 2 The LCD shutter in the image is completely closed.
[0043] Step 3: Gradually open the light source to the acquisition system. The dynamic vision sensor sends a start signal, which will... Figure 1 The electrically operated variable aperture 13 gradually opens to its maximum aperture, or... Figure 2 The LCD shutter gradually opens until it is fully open. Simultaneously, the dynamic vision sensor captures the brightness changes on the sensor plane during this process.
[0044] Step 4: Calculate the time-mapped grayscale image. Based on the relationship between brightness and time, the timestamps of the brightness change events collected in Step 3 are converted into grayscale levels on the corresponding pixels to obtain the time-mapped grayscale image.
[0045] Step 5: Calculate the event count mapping to the grayscale image. Based on the relationship between the event count and brightness, establish a mapping relationship between grayscale and the event count. Using the brightness change events acquired in Step 3, count the number of positive events generated by each pixel of the dynamic vision sensor, and use the logarithmic mapping relationship to obtain the event count mapping to the grayscale image.
[0046] Step 6: Image post-processing. Gamma correction and denoising are performed on the time-mapped grayscale image obtained in Step 4 and the event-mapped grayscale image obtained in Step 5, and the two are then fused to obtain the final scene grayscale image.
[0047] In one specific embodiment of the present invention, step three is as follows:
[0048] 3.1) The dynamic vision sensor sends a signal to the electrically operated variable aperture, causing it to open at a constant speed. The dynamic vision sensor records this moment as t0.
[0049] 3.2) During the aperture opening process, the dynamic vision sensor records the brightness changes of each pixel on the sensor plane, i.e., events. Since the brightness received on the sensor plane gradually increases during the aperture opening process, the generated events can only be positive events representing brightening. If negative events occur, they are removed as noise.
[0050] 3.3) Open the aperture to its maximum, and the dynamic vision sensor records this moment as t. k The time taken for the opening process is T = t k -t0.
[0051] The specific content of step four is as follows:
[0052] 4.1) Based on the characteristics of the dynamic vision sensor, a brightness change event is triggered when the brightness change of a pixel exceeds a fixed threshold. Since the brightness of each pixel in the dynamic vision sensor increases from 0 during step three, the timestamp of the first positive event generated by each pixel on the sensor plane is related to the brightness of that pixel in the scene, as follows:
[0053] 4.2) Let the brightness of pixel (x, y) on the sensor plane at time t be L. img (x, y, t), and when the incident light intensity control component is fully open, the brightness of the sensor plane pixel (x, y) is L. obj (x, y), this brightness is proportional to the brightness of the corresponding point in the scene. Since the motorized variable aperture is set on the aperture stop, the aperture change contributes equally to the brightness change of each pixel. The relationship between brightness and aperture change is abstracted as a transmittance function P(t). When the aperture is fully closed, there is no light leakage, and the transmittance function value is 0. When the aperture is fully open, the transmittance function value is defined as 1. Therefore, the relationship can be obtained: Limg (x, y, t) = L ob j(x, y)·P(t).
[0054] Depending on the brightness control method, P(t) can be manually controlled. In this embodiment, the aperture radius increases at a constant rate, so the transmittance function P(t) ∝ t. 2 .
[0055] 4.3) When the aperture opens from fully closed to a very small position, L img (x, y, t) is sufficient to reach the threshold C for triggering the event. According to formula 4.2), the time to trigger the first positive event at different pixel positions is... P -1 (*) means the inverse function of P(*). In this embodiment,
[0056] 4.4) Derived from 4.3) The occurrence time of the first positive event of each pixel during step 3) and the scene brightness L were established. obj (x, y The mapping relationship between scene brightness L and 0. obj (x, y According to the logarithmic mapping relationship Convert to I obj (x, y), I obj (x, y) represents the light intensity of each pixel on the sensor plane when the incident light intensity control component is fully open, and it is proportional to the light intensity of the corresponding point in the scene. All pixels together constitute the time-mapped grayscale image I. time The image linearly reflects the light intensity of corresponding points in the scene. According to the derivation in 4.3), the location with the lowest brightness in the scene generates the first positive event last. Once positive events have occurred at all locations, a high dynamic range grayscale image covering the entire scene is obtained, avoiding the overexposure or underexposure issues common in traditional image sensors. (Appendix) Figure 3 This refers to the time-mapped grayscale image obtained in step 4) of this embodiment, which depicts a typical high dynamic range scene: a person holding a flash in a dark environment with an average illuminance of 1100 lux. (From the attached image...) Figure 3 As can be seen, not only can the extremely bright diffraction starbursts from the flash be captured clearly without overexposure, but details in extremely dark background shadows can also be captured clearly.
[0057] When shooting in bright lighting conditions, the time-mapped grayscale image obtained in step 4) is acquired using a very small aperture. This small aperture introduces significant diffraction, especially in brighter areas of the scene, leading to a decrease in resolution. To supplement high-frequency details in bright areas and mitigate the effects of diffraction, step 5) is used to obtain an event-mapped grayscale image.
[0058] The specific details of step five are as follows:
[0059] 5.1) Consider the entire duration of aperture opening, the total time T = t k -t0 represents all positive events generated at each pixel location. The number of positive events generated by each pixel is counted to obtain an event count matrix E. Event count matrix E is a matrix with dimensions consistent with the sensor resolution, and the value at each position in the matrix is the number of positive events generated by that pixel.
[0060] 5.2) Throughout the entire aperture opening range, the initial brightness of each pixel is 0, and the final brightness is L. obj (x, y). The threshold of the dynamic vision sensor is C, therefore the number of events generated by each pixel in this process is...
[0061] 5.3) Map the event graph E according to the logarithmic mapping relationship L obj (x, y) = log(I) obj (x, y)) can be used to obtain the grayscale image I mapped to the number of events. event_count =e CE .
[0062] 5.4) See Appendix Figure 4 The left image is the time-mapped grayscale image obtained in step 4. Due to the bright shooting scene, the tree branches in the upper right corner of the image have low resolution due to diffraction. The middle image is the event-mapped grayscale image obtained in step 5, which effectively supplements the high-frequency details in the tree branches.
[0063] Event number mapping grayscale image I event_count Dynamic range and time-mapped grayscale image I time While consistent, its quantization accuracy is limited by the threshold C of the dynamic vision sensor, resulting in a relatively small value. Furthermore, the number of events is more susceptible to leakage current noise from the dynamic vision sensor. event_count The noise compared to I time The difference is more pronounced. Therefore, in static shooting scenarios, the results from both methods can be fused.
[0064] In a specific embodiment of the present invention, step six is as follows:
[0065] 6.1) Gamma correction. To better match the visual experience of the human eye, it is necessary to perform gamma correction on the time-mapped grayscale image I. time Event number mapping grayscale image I event_count Gamma correction is performed. In this embodiment, the time-mapped grayscale image I... time The selected gamma value is 2.2, and the number of events is mapped to the grayscale image I. event_count The selected gamma value is 1.
[0066] 6.2) Filtering and noise reduction. Time-mapped grayscale image I time The noise is mainly salt-and-pepper noise; therefore, in this embodiment, a median filter with a kernel size of 3 is used for noise reduction. The event count is mapped to the grayscale image I. event_count The noise primarily originates from leakage current in the dynamic vision sensor, resulting in a lower signal-to-noise ratio in dark areas of the image. Therefore, this embodiment employs a grayscale adaptive Gaussian filter G... adp (σ) is used to select a 5×5 window. The gray level is calculated for each position in the image. The smaller the mean gray level in the window, the larger the variance. A Gaussian filter parameter with a larger σ is used for noise reduction.
[0067] 6.3) Image Fusion. Temporally Mapped Grayscale Image I time The characteristic is that bright details are blurred due to diffraction, while the event number maps to the grayscale image I. event_count A key characteristic is that dark areas are significantly affected by noise. Therefore, in image fusion, the time-mapped grayscale image I... time The bright areas are given a smaller blending weight. t (x, y), mapping the number of events to image I event_count The dark areas are given a smaller blending weight. e (x, y).
[0068]
[0069] See example Figure 4 The image on the right is the final grayscale image I after image processing and fusion in step 6). fusion In the final grayscale image I after fusion fusion In the middle, the grayscale image I mapped by the number of events is preserved. event_count It provides high-frequency information in the bright areas while preserving the time-mapped grayscale image I. time Dark area information with medium to high signal-to-noise ratio.
[0070] See example Figure 5The image is a time-mapped grayscale image obtained by photographing a standard resolution board in a dark environment with an average illuminance of 0.7 lux and undergoing step 4). Under low-light conditions, even with an electrically adjustable aperture, the effect of diffraction is relatively weak. It can be seen that the image resolution at all field-of-view positions can reach the level of 7 line pairs.
[0071] The two image acquisition systems proposed in this invention each have their own advantages and disadvantages, with the core difference lying in the incident light intensity control component. The motorized variable aperture system has the advantage of ensuring minimal light leakage when the aperture is fully closed, but its disadvantage is that under high light intensity conditions, since events mainly occur when the aperture is relatively open, the imaging effect is easily affected by diffraction caused by the small aperture, leading to a decrease in resolution. The liquid crystal shutter has the advantage of achieving fully solid-state light intensity control without mechanical jitter, but its disadvantage is that when the liquid crystal shutter is fully closed, the light leakage rate is relatively high, making it difficult to meet the requirements of the time-mapped grayscale image generation principle.
[0072] The present invention further provides a scene grayscale image restoration system, which includes an image acquisition system, a time mapping module, an event number mapping module, and a post-processing module. The image acquisition system includes an optical imaging system, an incident light intensity control component, and a dynamic vision sensor. The incident light intensity control component is arranged in the imaging optical path and is controlled by the dynamic vision sensor to change the transmittance of the optical imaging system, thereby changing the light intensity illuminating the sensor plane.
[0073] The time mapping module converts the timestamps of the acquired brightness change events into gray levels on the corresponding pixels based on the relationship between brightness and time, thus obtaining a time-mapped grayscale image. The event number mapping module establishes a mapping relationship between gray levels and event numbers based on the relationship between event numbers and brightness. Using the acquired brightness change events, the module counts the number of positive events generated by each pixel of the dynamic vision sensor and obtains an event number-mapped grayscale image using a logarithmic mapping relationship.
[0074] The post-processing module further includes a gamma correction module, a filtering and denoising module, and a fusion module. The gamma correction module performs gamma correction on the time-mapped grayscale image and the event-mapped grayscale image; the filtering and denoising module denoises the gamma-corrected time-mapped grayscale image and the event-mapped grayscale image; and the fusion module fuses the denoised time-mapped grayscale image and the event-mapped grayscale image to obtain the final scene grayscale image.
[0075] The system of this invention enables dynamic visual sensors, which could originally only output discrete information about brightness changes, to also output grayscale images.
[0076] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. Those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.
Claims
1. A method for scene grayscale image restoration based on a dynamic visual sensor, characterized in that... Includes the following steps: 1) An image acquisition system is arranged, the image acquisition system including an optical imaging system, an incident light intensity control component and a dynamic vision sensor; wherein, the incident light intensity control component is arranged in the imaging optical path, and the incident light intensity control component is controlled by the dynamic vision sensor to change the transmittance of the optical imaging system, thereby changing the light intensity illuminating the sensor plane. 2) Before the dynamic vision sensor starts collecting scene information, the incident light intensity control component completely blocks the light from the external scene from shining onto the sensor plane. 3) When the dynamic vision sensor starts to collect scene information, the incident light intensity control component is gradually turned on until it is fully turned on. The dynamic vision sensor acquires the brightness change events on the sensor plane during this process. 4) Based on the relationship between brightness and time, the timestamps of the brightness change events collected in step 3) are converted into gray levels on the corresponding pixels to obtain a time-mapped grayscale image. 5) Based on the relationship between the number of events and brightness, using the brightness change events collected in step 3), count the number of positive events generated by each pixel of the dynamic vision sensor, and use the logarithmic mapping relationship to obtain the event number-mapped grayscale image. 6) Perform gamma correction and denoising on the time-mapped grayscale image obtained in step 4) and the event number-mapped grayscale image obtained in step 5), and then fuse the two to obtain the final scene grayscale image.
2. The scene grayscale image restoration method based on a dynamic visual sensor according to claim 1, characterized in that, The incident light intensity control component is an electrically adjustable aperture with continuously variable aperture size. The electrically adjustable aperture is installed at the aperture stop position of the optical imaging system. The dynamic vision sensor sends a trigger signal to the electrically adjustable aperture to make the aperture size continuously change, thereby controlling the transmittance of the optical imaging system. The dynamic vision sensor collects the brightness change information during the aperture size change.
3. The scene grayscale image restoration method based on a dynamic visual sensor according to claim 1, characterized in that, The incident light intensity control component is a liquid crystal shutter. The dynamic vision sensor sends a signal to adjust the transmittance of the liquid crystal shutter and collects the brightness change information when the transmittance changes.
4. The scene grayscale image restoration method based on a dynamic visual sensor according to claim 1, characterized in that, In step 3), the incident light intensity control component is turned on. Initially, the dynamic visual sensor records this moment as t0. When it reaches its maximum opening, the dynamic visual sensor records this moment as t... k The time taken for the opening process is T = t k -t0; During the opening process, the brightness received on the sensor plane gradually increases, so the generated events can only be positive events representing brightening. If a negative event occurs, it is removed as noise.
5. The scene grayscale image restoration method based on a dynamic vision sensor according to claim 1, characterized in that, Step 4) is specifically as follows: According to the characteristics of dynamic vision sensors, a brightness change event is triggered when the brightness change of a pixel exceeds a fixed threshold C. The timestamp of the first positive event generated by each pixel on the sensor plane is related to the brightness of the corresponding point of that pixel in the scene, and the relationship is as follows: L img (x,y,t)=L obj (x,y)·P(t) Among them, L img (x, y, t) represents the brightness of pixel (x, y) on the sensor plane at time t; P(t) is the transmittance function, representing the relationship between the brightness change of each pixel and the opening time t of the incident light intensity control component. The transmittance is defined as 1 when the incident light intensity control component is fully open and 0 when the incident light intensity control component is fully closed; L obj (x, y) is the brightness of the pixel (x, y) on the sensor plane when the incident light intensity control component is fully turned on. This brightness is proportional to the brightness of the corresponding point in the scene. The time when the first positive event is triggered at different pixel positions P-1(*) means the inverse function of P(*); since the relationship between P(t) and t is known, we can establish the relationship between the occurrence time t(x, y) of the first positive event of each pixel and L. obj The mapping relationship between (x, y); L obj (x, y) are related by the logarithmic mapping. Convert to I obj (x, y), I obj (x, y) represents the light intensity at pixel (x, y) on the sensor plane when the incident light intensity control component is fully open, and it is proportional to the light intensity at the corresponding point in the scene; the entire set of all pixels constitutes the time-mapped grayscale image I. time The image linearly reflects the light intensity at each corresponding point in the scene.
6. The scene grayscale image restoration method based on a dynamic vision sensor according to claim 1, characterized in that, Step 5) includes: 5.1) Consider the entire duration of the incident light intensity control component being turned on, with a total time length T = t k -t0 represents all positive events generated at each pixel position on the sensor plane; the number of positive events generated by each pixel is counted to obtain the event count matrix E. The event count matrix E is a matrix with the same length and width as the sensor resolution, and the value at each position of the matrix is the number of positive events generated by each pixel. 5.2) Throughout the entire aperture opening range, the initial brightness of each pixel is 0, and the final brightness is L. obj (x, y); The threshold of the dynamic vision sensor is C, therefore the number of events generated by each pixel in this process is... 5.3) Apply the logarithmic mapping relationship L to the event number matrix E. obj (x, y) = log(I) obj (x, y)) can be used to obtain the grayscale image I mapped to the number of events. event_count =e CE .
7. The scene grayscale image restoration method based on a dynamic vision sensor according to claim 1, characterized in that, The noise reduction in step 6) specifically involves: Time-mapped grayscale image I time The noise is mainly salt-and-pepper noise, and a median filter with a kernel size of 3 is used for noise reduction. Event number mapping grayscale image I event_count The noise mainly comes from the leakage current phenomenon of the dynamic vision sensor, and a gray-scale adaptive Gaussian filter G is used. adp (σ) is used to select a 5×5 window. The gray level is calculated for each position in the image. The smaller the mean gray level in the window, the larger the variance. A Gaussian filter parameter with a larger σ is used for noise reduction.
8. The scene grayscale image restoration method based on a dynamic vision sensor according to claim 1, characterized in that, The image fusion in step 6) specifically refers to: Time-mapped grayscale image I time The bright areas are given a smaller blending weight. t (x, y), mapping the number of events to image I event_count The dark areas are given a smaller blending weight. e (x, y); the final fused scene grayscale image I fusion (x, y) is obtained through the following relationship:
9. A scene grayscale image restoration system implementing the method of claim 1, characterized in that... include: An image acquisition system includes an optical imaging system, an incident light intensity control component, and a dynamic vision sensor; wherein the incident light intensity control component is arranged in the imaging optical path and is controlled by the dynamic vision sensor to change the transmittance of the optical imaging system, thereby changing the light intensity illuminating the sensor plane. The time mapping module converts the timestamps of the collected brightness change events into gray levels on the corresponding pixels based on the relationship between brightness and time, thus obtaining a time-mapped grayscale image. The event number mapping module establishes a mapping relationship between grayscale and event number based on the relationship between event number and brightness; it uses the collected brightness change events to count the number of positive events generated by each pixel of the dynamic vision sensor, and uses the logarithmic mapping relationship to obtain the event number mapped grayscale image. The gamma correction module performs gamma correction on time-mapped grayscale images and event-mapped grayscale images; The filtering and noise reduction module denoises the time-mapped grayscale image and the event-mapped grayscale image after gamma correction. The fusion module fuses the denoised time-mapped grayscale image and the event-mapped grayscale image to obtain the final scene grayscale image.