Electronic device for compensating for time delay of a dynamic vision sensor

Abstract

Disclosed is an electronic device including a dynamic vision sensor including a first pixel that senses a light intensity change and generates an event signal based on the sensed light intensity change, an illuminance estimator that estimates an illuminance of light, and a time delay compensator that calculates a time delay between a first time at which the light intensity change occurs and a second time at which the first pixel senses the light intensity change based on the illuminance of the light and compensates for the time delay.

Description

[0001] Cross-reference of related applications

[0002] This application claims priority to Korean Patent Application No. 10-2021-0099717, filed with the Korean Intellectual Property Office on July 29, 2021, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The embodiments of this disclosure described herein relate to an electronic device for compensating for time delays in a dynamic vision sensor, and more specifically, to an electronic device for compensating for event occurrence time delays in a dynamic vision sensor. Background Technology

[0004] With the development of semiconductor technology, sensors are becoming increasingly diverse. Sensors include charge-coupled device (CCD) image sensors, dynamic vision sensors (DVS), ambient light sensors (ALS), proximity sensors (PS), and / or the like.

[0005] Electronic devices can respond to external inputs using sensors. External inputs can include changes in light intensity, user touch, and / or similar objects. When receiving input from the outside, the sensor can output an electrical signal. The electronic device can then identify movement of external objects or changes in the surrounding environment based on these electrical signals.

[0006] For example, when a dynamic vision sensor senses a change in light intensity, the timing of this sensing change can vary depending on the surrounding environment. Therefore, the time data included in the event signal of the dynamic vision sensor may differ from the actual time of the event. This can lead to performance degradation in electronic devices using dynamic vision sensors. Furthermore, in electronic devices where a different sensor is implemented alongside the dynamic vision sensor, synchronization between the different sensors may not be achieved. Summary of the Invention

[0007] Embodiments of this disclosure provide an electronic device for compensating for the reaction time delay of a dynamic vision sensor due to illuminance or changes in illuminance.

[0008] According to an example embodiment, an electronic device includes: a dynamic vision sensor, the dynamic vision sensor including a first pixel that senses changes in light intensity and generates an event signal based on the sensed changes in light intensity; an illuminance estimator that estimates the illuminance of light; and a time delay compensator that calculates a time delay between a first time when the changes in light intensity occur and a second time when the first pixel senses the changes in light intensity based on the illuminance of the light, and compensates for the time delay.

[0009] According to an example embodiment, an electronic device includes: a dynamic vision sensor including a first pixel that senses changes in light intensity and generates a first event signal based on the sensed changes in light intensity; an image processing apparatus including a complementary metal-oxide-semiconductor (CMOS) image sensor that generates first image data based on light intensity; an illuminance calculator that calculates at least one of light illuminance and a change in illuminance based on the first image data; and a time delay compensator that calculates a time delay between a first time when the light intensity change occurs and a second time when the first pixel senses the light intensity change based on at least one of the light illuminance and the change in illuminance, and generates a second event signal for which the time delay is compensated. The CMOS image sensor includes a second pixel.

[0010] According to an example embodiment, an electronic device includes: a camera module group comprising a plurality of camera modules, each camera module generating an electrical signal based on light reflected from an object; and an application processor processing the electrical signals. A first camera module among the plurality of camera modules utilizes a dynamic vision sensor, and a second camera module among the plurality of camera modules utilizes an image sensor. The first camera module generates an event signal based on a change in light intensity, and the second camera module generates image data in frames. The application processor estimates at least one of illuminance and a change in illuminance based on the image data, and compensates for the time delay between a first time when the change in light intensity occurs and a second time when the first camera module senses the change in light intensity based on the illuminance and the change in illuminance.

[0011] According to an example embodiment, a method of operating an electronic device includes: sensing a change in light intensity to generate an event signal; estimating illuminance or a change in illuminance; and compensating for a time delay of the event signal based on the estimated illuminance or the estimated change in illuminance. Attached Figure Description

[0012] The above and other objects and features of this disclosure will become apparent from the detailed description of exemplary embodiments of this disclosure with reference to the accompanying drawings.

[0013] Figure 1 This is a configuration diagram of an electronic device according to an exemplary embodiment of the present disclosure.

[0014] Figure 2A This is a graph illustrating the time delay of a dynamic vision sensor based on pixel illumination.

[0015] Figure 2BThis is a graph illustrating the time delay of a dynamic vision sensor based on changes in pixel illumination.

[0016] Figure 3 This is a conceptual diagram illustrating a method for compensating for time delays occurring in a dynamic vision sensor according to an example embodiment of the present disclosure.

[0017] Figure 4 More detailed illustrations Figure 1 Configuration diagram of the dynamic vision sensor 100.

[0018] Figure 5 It is a diagram. Figure 4 The circuit diagram for the configuration of the pixel PX.

[0019] Figure 6 More detailed illustrations Figure 4 The circuit diagram for the configuration of the pixel PX.

[0020] Figure 7 More detailed illustrations Figure 1 Configuration diagram of the illuminance estimator 200.

[0021] Figure 8 This is a configuration diagram used to describe an illuminance calculator method for calculating illuminance according to an example embodiment of the present disclosure.

[0022] Figure 9 This is a configuration diagram illustrating an electronic device for performing motion deblur according to an exemplary embodiment of the present disclosure.

[0023] Figure 10 This is a configuration diagram illustrating an electronic device for performing motion blur removal according to other exemplary embodiments of the present disclosure.

[0024] Figure 11 The illustration shows raw image data according to an example embodiment of this disclosure.

[0025] Figure 12A The illustration shows image data for which motion blur removal is performed using the original event signal, according to an example embodiment of this disclosure.

[0026] Figure 12B The illustration shows image data for which motion blur removal is performed using a compensation event signal, according to an example embodiment of the present disclosure.

[0027] Figure 13 This is a configuration diagram illustrating an electronic device performing Simultaneous Localization and Mapping (SLAM) according to an exemplary embodiment of the present disclosure.

[0028] Figure 14The illustration shows the configuration of the electronic device for implementing the dynamic vision sensor of this disclosure, including a camera module.

[0029] Figure 15 This is a flowchart illustrating an operation method of an electronic device according to an exemplary embodiment of the present disclosure.

[0030] Figure 16 This is a flowchart illustrating an operation method of an electronic device for performing motion blur removal according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0031] Below, exemplary embodiments of the present disclosure will be described in detail and clearly so that those skilled in the art can readily implement the present disclosure.

[0032] In the detailed description, the components described by reference to terms such as "unit," "module," "block," "or device," etc., and the functional blocks illustrated in the accompanying drawings, will be implemented using software, hardware, or a combination thereof. For example, software can be machine code, firmware, embedded code, and application software. For example, hardware can include circuits, electronic circuits, processors, computers, integrated circuits, integrated circuit cores, pressure sensors, inertial sensors, microelectromechanical systems (MEMS), passive components, or a combination thereof.

[0033] Figure 1 This is a configuration diagram illustrating an electronic device 10 according to an exemplary embodiment of the present disclosure. The electronic device 10 may include a dynamic vision sensor 100, an illuminance estimator 200, and / or a time delay compensator 300. The electronic device 10 may be implemented as part of a variety of electronic devices such as desktop computers, laptop computers, tablets, smartphones, wearable devices, smart speakers, home security Internet of Things (IoT) devices, video game consoles, workstations, servers, and autonomous vehicles. The electronic device 10 can compensate for the time delay of event occurrence time data of the dynamic vision sensor 100 caused by illuminance or changes in illuminance.

[0034] The dynamic vision sensor 100 can sense changes in light intensity. For example, the dynamic vision sensor 100 can detect events of increased light intensity (hereinafter referred to as "on-event") and / or events of decreased light intensity (hereinafter referred to as "off-event"). When a change in light intensity exceeding an event threshold is sensed, the dynamic vision sensor 100 can generate a signal. The dynamic vision sensor 100 can generate an event signal ES as a result of processing the generated signal. According to an example embodiment of this disclosure, the dynamic vision sensor 100 may include a plurality of pixels. For example, the event signal ES may include: a value indicating the location of the pixel where the event occurred, a polarity value including information about the on-event or off-event, and a timestamp value (e.g., time data) and / or the like indicating the time when the dynamic vision sensor 100 sensed the change in light intensity. (See reference...) Figure 4 The configuration and operation of the dynamic vision sensor 100 are described in detail.

[0035] Illuminance estimator 200 can estimate the illuminance of incident light. For example, illuminance estimator 200 can sense light incident from an external light source on a pixel-by-pixel basis and estimate the illuminance of the sensed light. Additionally, illuminance estimator 200 can calculate changes in illuminance based on the illuminance at multiple times. Illuminance estimator 200 can generate an illuminance information ID based on the estimated illuminance value. For example, the illuminance information ID may include a value indicating the illuminance or changes in illuminance, corresponding to an event signal ES generated by dynamic vision sensor 100. For example, illuminance estimator 200 can be implemented using a sensor that measures illuminance (e.g., a photoresistor). However, this disclosure is not limited thereto. For example, illuminance estimator 200 can be implemented using any means that directly measures illuminance or indirectly estimates illuminance.

[0036] According to an example embodiment of this disclosure, the illumination estimator 200 may be implemented using a complementary metal-oxide-semiconductor (CMOS) image sensor (hereinafter referred to as "CIS"). The CIS may include a plurality of pixels. Pixels of the CIS may correspond to pixels of the dynamic vision sensor 100. For example, when the aspect ratio of the pixels of the CIS is the same as that of the pixels of the dynamic vision sensor 100 and the resolution of the CIS is the same as that of the dynamic vision sensor 100, the pixels of the CIS may correspond to pixels of the dynamic vision sensor 100 individually (e.g., a ratio of 1:1). Conversely, when the aspect ratio of the pixels of the CIS is different from that of the pixels of the dynamic vision sensor 100 and the resolution of the CIS is different from that of the dynamic vision sensor 100, the number of pixels of the CIS may be different from the number of pixels of the dynamic vision sensor 100 (e.g., a ratio of "1:n" or "n:1" (where n is an integer of 2 or greater)). For example, the dynamic vision sensor 100 may include a first pixel and a second pixel, and the CIS may include a third pixel and a fourth pixel. In some example embodiments, the first pixel may correspond to the third pixel, and the second pixel may correspond to the fourth pixel. In some example embodiments, the illuminance estimator 200 may generate an illuminance information ID corresponding to each pixel of the dynamic vision sensor 100 based on the illuminance values ​​of the pixels of the CIS corresponding to the pixels of the dynamic vision sensor 100. (Refer to...) Figure 7 The configuration and operation of an example embodiment of the illuminance estimator 200 implemented using CIS are described in detail.

[0037] The time delay compensator 300 can calculate and compensate for the time delay of the time data (e.g., timestamp value) included in the event signal ES based on the illuminance information ID. For example, the event signal ES may include pixel position values, polarity values, and timestamp values. Here, the time corresponding to the timestamp value may be the time when the dynamic vision sensor 100 senses a change in light intensity. The time when the dynamic vision sensor 100 senses a change in light intensity may differ from the time when the change in light intensity actually occurs (e.g., the actual event occurrence time). Therefore, a time delay exists between the time when the dynamic vision sensor 100 senses a change in light intensity and the time when the change in light intensity actually occurs.

[0038] As the illuminance of the light incident on the dynamic vision sensor 100 decreases, the delay time (e.g., the degree of time delay) can increase. When high-illuminance light is incident on the dynamic vision sensor 100, the dynamic vision sensor 100 can respond immediately or adequately to changes in light intensity. In some example embodiments, the time delay can be slight (negligible or negligible). In contrast, when low-illuminance light (e.g., less than 5 lux) is incident on the dynamic vision sensor 100, the dynamic vision sensor 100 may not respond immediately or adequately to changes in light intensity. In some example embodiments, the performance of the dynamic vision sensor 100 may be degraded due to the delay time (e.g., 10 ms or more). Furthermore, as the illuminance change of the light incident on the dynamic vision sensor 100 becomes smaller, the delay time can increase. (Refer to...) Figure 2A and Figure 2B Describe in detail the delay time based on the illuminance or illuminance change.

[0039] The time delay compensator 300 can calculate the time delay of the time data included in the event signal ES based on the illuminance information ID, and can adjust the timestamp value included in the received event signal ES based on the calculated time delay. The time delay compensator 300 can generate a compensated event signal ES' with the adjusted timestamp value applied. (Refer to...) Figure 3 This section provides a detailed description of how the time delay compensator 300 adjusts timestamp values.

[0040] According to example embodiments of this disclosure, when the calculated time delay is less than any, given, predetermined, or desired reference level, the time delay compensator 300 may not adjust the timestamp value. In example embodiments, when the delay time is less than 10 ms, the performance degradation of the dynamic vision sensor 100 may be slight (negligible or negligible), therefore, the time delay compensator 300 may not adjust the timestamp value. In some example embodiments, the compensation event signal ES' generated by the time delay compensator 300 may be the same as the event signal ES.

[0041] According to exemplary embodiments of this disclosure, and Figure 1 Unlike the examples illustrated, at least some of the functions of the illuminance estimator 200 and the time delay compensator 300 can be implemented in the dynamic vision sensor 100. For example, the functions of the illuminance estimator 200 and the time delay compensator 300 can be implemented using hardware, software, and / or combinations thereof to be included in the dynamic vision sensor 100. In some example embodiments, the dynamic vision sensor 100 can generate an event signal ES by synchronizing the time data (e.g., timestamp values) included in the event signal with the actual time of the event occurrence.

[0042] Figure 2A This is a graph illustrating the delay time of a dynamic vision sensor based on pixel illumination. (Reference) Figure 1 and Figure 2A The difference between the time corresponding to the timestamp value and the actual time of the event (i.e., the time delay) can occur based on the illuminance of the light incident on the pixels of the dynamic vision sensor 100 (hereinafter referred to as "pixel illuminance"). The time delay based on pixel illuminance can vary depending on the characteristics of the dynamic vision sensor 100, changes in light illuminance, and / or the event threshold. Figure 2A The graph shown in the figure illustrates a specific time delay of the dynamic vision sensor 100 as an example, but this disclosure is not limited thereto.

[0043] refer to Figure 2A The x-axis can correspond to pixel illuminance, and the y-axis can correspond to delay time. It is assumed that the pixel illuminance variation is fixed at twice the event threshold to reduce or prevent the delay time from varying with pixel illuminance. Here, the pixel illuminance variation can be the logarithm of the pixel illuminance variation sensed during a unit event sensing cycle of the dynamic vision sensor 100, following the relationship in Equation 1 below. Events can be classified as on-events or off-events. Because the formation of the on-event curve is similar to that of the off-event curve, for ease of description, a description will be given with reference to on-events. When the pixel illuminance exceeds a certain level (e.g., approximately 500 lux), the time delay can be uniform because the pixel illuminance exceeds the input limit of the dynamic vision sensor 100. When the pixel illuminance is less than or equal to the input limit of the dynamic vision sensor 100, the time delay can decrease as the pixel illuminance increases.

[0044] Figure 2B This is a graph illustrating the time delay of a dynamic vision sensor based on changes in pixel illumination. (References omitted) Figure 2A The description provided is to avoid redundancy. As an example, it is assumed that the pixel illuminance is 3 lux and the event threshold is 15%. Reference Figure 1 and Figure 2B The x-axis can correspond to "pixel illumination change". The y-axis, "100", can correspond to the delay time. As the pixel illumination change decreases, the delay time can increase; as the pixel illumination change increases, the delay time can decrease. The delay time based on the pixel illumination change can vary depending on the characteristics of the dynamic vision sensor 100, the pixel illumination, and / or the event threshold. The pixel illumination change can follow the relationship of Equation 1 below.

[0045] [Equation 1]

[0046] Referring to Equation 1, “t” is defined as the time when the dynamic vision sensor 100 senses an event. Therefore, when t1 is the current time, t2 can be defined as the next time, and (t2 – t1) can be any, given, predetermined, or desired unit event sensing period of the dynamic vision sensor 100. In Equation 1, “I” is defined as the pixel illuminance sensed during the sensing time. Therefore, I(t1) is defined as the pixel illuminance sensed at the current time, and I(t2) is defined as the pixel illuminance sensed at the next time. Thus, the pixel illuminance change is the logarithm of the pixel illuminance sensed during a unit event sensing period.

[0047] Figure 3 This is a conceptual diagram illustrating a method for compensating for time delays occurring in a dynamic vision sensor according to an example embodiment of this disclosure. (Reference) Figure 1 and Figure 3 The x-axis and y-axis can correspond to the position values ​​of the pixels in the dynamic vision sensor 100 where the event occurs. The t-axis can correspond to the time when the event occurs. The "p" value can correspond to the polarity value of the dynamic vision sensor 100. For example, when the position value of the pixel where the event occurs is (x1, y1), the actual event occurrence time can be t0. However, due to pixel illumination and / or pixel illumination variations, the time corresponding to the timestamp value included in the event signal ES (e.g., t1) may be delayed relative to t0, which is the actual event occurrence time. A time delay as much as (t1-t0) can occur in the dynamic vision sensor 100. Therefore, the time delay compensator 300 can adjust the timestamp value to the value corresponding to t0 by compensating for the time delay (t1-t0) at t1.

[0048] Figure 4 More detailed illustrations Figure 1 A configuration diagram of the dynamic vision sensor 100. The dynamic vision sensor 100 may include a DVS pixel array 110, an address event representation (AER) controller 120, and / or input / output (I / O) circuitry 130.

[0049] DVS pixel array 110 may include multiple pixels arranged in a matrix along multiple rows and multiple columns. Pixel PX from the multiple pixels of DVS pixel array 110 that senses an event can generate a column request ReqX indicating an event of increased or decreased light intensity. (See reference...) Figure 5 and Figure 6 The configuration and operation of the pixel PX are described in detail.

[0050] AER controller 120 may include column AER circuit 121, row AER circuit 122, and / or timestamp 123. AER controller 120 can control pixels PX that sense events. AER controller 120 can generate a timestamp TS, polarity information PI, and pixel address ADDR based on multiple signals ReqX, ReqON, and ReqOFF received from pixel PX. AER controller 120 can generate the timestamp TS, polarity information PI (including polarity values ​​corresponding to on event requests ReqON and / or off event requests ReqOFF), and pixel address ADDR (including column and row addresses corresponding to pixel position values) in the form of digital signals.

[0051] The column AER circuit 121 can receive a column request ReqX from a pixel PX. In response to the received column request ReqX, the column AER circuit 121 can send a column response signal AckX to the pixel PX. The pixel PX that receives the column response signal AckX can send an on event request ReqON and / or an off event request ReqOFF to the row AER circuit 122. The column AER circuit 121 can generate a column address for the pixel PX that sensed the event based on the column request ReqX received from the pixel PX that sensed the event, and this column address can form part of the address ADDR.

[0052] The row AER circuit 122 can receive an on event request ReqON and / or an off event request ReqOFF from a pixel PX. In response to the on event request ReqON and / or the off event request ReqOFF, the row AER circuit 122 can send a row response signal AckY. The pixel PX can generate a reset signal in response to the column response signal AckX and the row response signal AckY. The reset signal can reset the pixel PX where the event occurred. The row AER circuit 122 can generate a row address for the pixel PX that sensed the event based on the on event request ReqON and / or the off event request ReqOFF received from the pixel PX, and this row address can form part of the address ADDR.

[0053] The operation of column AER circuit 121 and row AER circuit 122 is not limited to the description given above. Figure 4 The examples illustrated in the diagram differ, and the operations of column AER circuit 121 and row AER circuit 122 can be interchanged. Therefore, in some example embodiments, column AER circuit 121 can receive an on event request ReqON and / or an off event request ReqOFF from pixel PX.

[0054] Timestamp 123 can generate a timestamp TS that includes information about the time when the event occurred. For example, timestamp 123 can be implemented using clock ticks generated in units of a few microseconds to tens of microseconds. However, the above units for generating clock ticks are examples, and clock ticks can vary depending on the manufacturer's settings or the user's request. According to an example embodiment of this disclosure, pixel PX may not be able to react immediately or adequately to the occurrence of an event, resulting in a time delay. Therefore, a time delay can occur between the actual time of the event and the time of the event corresponding to the timestamp TS.

[0055] The input / output circuit 130 can generate an event signal ES based on a timestamp TS, a pixel address ADDR, and polarity information PI. The input / output circuit 130 can add a header indicating the start of a notification signal transmission to the beginning of the event signal ES and a tail indicating the end of a notification signal transmission to the end of the event signal ES.

[0056] Figure 5 It is a diagram. Figure 4 A circuit diagram of the configuration of pixel PX. For ease of description, a pixel PX will be described, but this disclosure is not limited thereto. For example, it includes a DVS pixel array 110 (see reference 110). Figure 4 Each of the remaining pixels in ) can have the same as Figure 5 The structure of the pixel PX is similar. The pixel PX may include a photoreceptor 111, a differentiator 112, a comparator 113, and / or handshake logic 114. However, for the sake of convenience, reference will be made to... Figure 4 describe Figure 5 .

[0057] The photoreceptor 111 may include: a photodiode PD that converts light energy into electrical energy, a logarithmic amplifier LA that amplifies the voltage corresponding to the photocurrent IPD to output a logarithmic voltage VLOG on a logarithmic scale, and / or a feedback transistor FB that isolates the photoreceptor 111 from the differentiator 112.

[0058] Differentiator 112 can amplify voltage VLOG to generate voltage Vdiff. For example, differentiator 112 may include capacitors C1 and C2, a differential amplifier DA, and / or a switch SW, and the switch SW may operate in response to a reset signal RST. For example, capacitors C1 and C2 may store electrical energy generated by a photodiode PD. For example, the capacitances of capacitors C1 and C2 may be appropriately selected considering the shortest time between two consecutive events occurring at a pixel (e.g., the refractory period). When the switch SW is turned on by the reset signal RST, the pixel can be initialized.

[0059] Comparator 113 compares the output voltage Vdiff of the differential amplifier DA with the reference voltage Vref, and determines whether the event sensed by the pixel is an on or off event. When an event of increased light intensity is sensed, comparator 113 outputs a signal VON indicating that the sensed event is an on event; when an event of decreased light intensity is sensed, comparator 113 outputs a signal VOFF indicating that the sensed event is an off event.

[0060] Handshake logic 114 can send a column request ReqX to column AER circuit 121 in response to a signal VON indicating an on event or a signal VOFF indicating a off event. When column AER circuit 121 sends a column response signal AckX, handshake logic 114 can send an on event request ReqON corresponding to the signal VON indicating an on event or a off event request ReqOFF corresponding to the signal VOFF indicating a off event to row AER circuit 122. When row AER circuit 122 sends a row response signal AckY, handshake logic 114 can generate a reset signal RST based on the column response signal AckX and the row response signal AckY. To reduce workload or prevent workload from increasing due to a large number of events, the period for generating the reset signal RST can be controlled so that no events occur during a specific period.

[0061] Figure 6 More detailed illustrations Figure 5 The circuit diagram for the configuration of the pixel PX. Figure 6 The pixel PX shows Figure 5 The present disclosure includes transistor-level circuitry for the pixel PX, but is not limited thereto. (See references.) Figure 4 , Figure 5 and Figure 6 Pixel PX may include a photosensor 111, a differentiator 112, a comparator 113, and / or handshake logic 114. (See reference) Figure 5 The operation or function of each of the photoreceptor 111, differentiator 112, comparator 113 and / or handshake logic 114 is described, therefore additional descriptions will be omitted to avoid redundancy.

[0062] The photoreceptor 111 may include a photodiode PD, first NMOS transistors MN1 through MN5, a logarithmic current bias ILOG, and / or a source follower current bias ISF. The first NMOS transistors MN1 through MN4 may form a logarithmic amplifier. The first NMOS transistor MN1 may be connected between the power supply voltage VDD and the third node N3, and may operate in response to the voltage at the first node N1 (e.g., the logarithmic amplification voltage VLOG). The second NMOS transistor MN2 may be connected between the third node N3 and the fourth node N4, and may operate in response to the voltage at the second node N2. The third NMOS transistor MN3 may be connected between the first node N1 and the second node N2, and may operate in response to the voltage at the third node N3. The fourth NMOS transistor MN4 may be connected between the second node N2 and ground, and may operate in response to the voltage at the fourth node N4. The fifth NMOS transistor MN5 may operate as a source follower. The fifth NMOS transistor MN5 can be connected between the power supply voltage VDD and the fifth node N5, and can operate in response to the logarithmic amplification voltage VLOG. The logarithmic current bias ILOG can be connected between the power supply voltage VDD and the first node N1. The source follower current bias ISF can be connected between the fifth node N5 and ground.

[0063] Differentiator 112 can be connected between a fifth node N5 having a source follower voltage VSF and an eighth node N8 having an output voltage VOUT. Differentiator 112 may include a first capacitor C1, a second capacitor C2, a first PMOS transistor MP1, a reset transistor MRS, and / or an amplifier current bias IA. The first PMOS transistor MP1 can operate as an amplifier. The first PMOS transistor MP1 can be connected between the power supply voltage VDD and the eighth node N8 and can operate in response to the input voltage VIN. The reset transistor MRS can act as a switch to determine whether to reset a pixel. The reset transistor MRS can be connected between a sixth node N6 and a seventh node N7 and can operate in response to a reset signal RST. The amplifier current bias IA can be connected between the seventh node N7 and ground.

[0064] Comparator 113 may include a second PMOS transistor MP2 and a third PMOS transistor MP3, an on-event current bias ION, and / or an off-event current bias IOFF. The second PMOS transistor MP2 may be connected between the supply voltage VDD and the on-event node NON, and may operate in response to the output voltage VOUT. The third PMOS transistor MP3 may be connected between the supply voltage VDD and the off-event node NOFF, and may operate in response to the output voltage VOUT. The on-event current bias ION may be connected between the on-event node NON and ground. The off-event current bias IOFF may be connected between the off-event node NOFF and ground.

[0065] The connection event signal VON corresponding to the connection event can be provided to the handshake logic 114 through the connection event node NON, and the closing event signal VOFF corresponding to the closing event can be provided to the handshake logic 114 through the closing event node NOFF.

[0066] Figure 7 More detailed illustrations Figure 1 A configuration diagram of the illuminance estimator 200. According to an example embodiment of this disclosure, the illuminance estimator 200 may include an image processing device 210 and / or an illuminance calculator 220.

[0067] Image processing device 210 can be implemented as part of various electronic devices such as smartphones, digital cameras, laptops, and desktop computers. Image processing device 210 may include lens 211, image sensor 212, ISP (image signal processing) front-end block 213, and / or image signal processor 214.

[0068] Lens 211 can receive light reflected from objects, landscapes, etc., that are the subjects of the photograph. Lens 211 can be moved, for example, by an actuator (not shown). When the position of lens 211 changes according to the movement of lens 211, the focal length of lens 211 can change. Therefore, the focus on the object can be adjusted. Lens 211 can be configured to be aligned with the dynamic vision sensor 100 (see reference 100) in both direction and focus. Figure 1 The lenses included in ) are the same.

[0069] Image sensor 212 may include ordinary pixels for obtaining color information about an object. The ordinary pixels of image sensor 212 may generate electrical signals based on light received through lens 211. Additionally, image sensor 212 may include phase detection pixels for obtaining phase information about the object. Image sensor 212 may include phase detection pixels for adjusting focus. The phase detection pixels of image sensor 212 may generate electrical signals for performing phase detection autofocus (PDAF) based on light received through lens 211. Image sensor 212 may output image data IDAT1 including color and phase information about the object.

[0070] Figure 7 A lens 211 and an image sensor 212 are shown. However, in another example, the image processing device 210 may include multiple lenses, multiple ISP front-end blocks, and multiple image sensors. The multiple image sensors may be configured to have different functions, different performance, and / or different characteristics. In some example embodiments, the multiple image sensors may include lenses, each with a different field of view (FOV).

[0071] ISP front-end block 213 can process image data IDAT output from image sensor 212. l Various preprocessing operations are performed. For example, the ISP front-end block 213 can perform the following processing on the signal output from the image sensor 212: crosstalk compensation, and automatic black level compensation (ADLC) for removing fixed pattern noise (FPN). The ISP front-end block 213 can generate preprocessed image data IDAT2.

[0072] The image signal processor 214 can perform various processing operations on the image data IDAT2 processed by the ISP front-end block 213. For example, the image signal processor 214 can perform various processing operations such as color interpolation, automatic white balance, gamma correction, color saturation correction, formatting, bad pixel correction, and / or chroma correction. The image signal processor 214 can generate the final image data IDAT after undergoing various processing operations.

[0073] The illuminance calculator 220 can generate illuminance information ID based on image data IDAT2. For example, the illuminance calculator 220 can calculate the illuminance of light incident on each pixel (e.g., pixel illuminance) based on the pixel values ​​included in the image data IDAT2. (See reference...) Figure 8 The method for calculating illuminance using an illuminance calculator 220 according to an example embodiment of the present disclosure is described in detail. However, the present disclosure is not limited thereto; for example, the illuminance calculator 220 can also calculate changes in illuminance. Although in Figure 7Not illustrated, but in other example embodiments, the illuminance calculator 220 may generate an illuminance information ID based on image data IDAT1 or the final image data IDAT. In example embodiments, at least a portion of the functionality of the illuminance calculator 220 may be implemented in the ISP front-end block 213. In other example embodiments, at least a portion of the functionality of the illuminance calculator 220 may be implemented using a main processor (e.g., an application processor) that receives and processes the image data IDAT. The functionality of the illuminance calculator 220 may be implemented using hardware and / or software.

[0074] Figure 8 This is a configuration diagram for describing an illuminance calculation method of an illuminance calculator 220 according to an example embodiment of the present disclosure. (Reference) Figure 7 and Figure 8 The reflected light from the object can be input to the image processing device 210. The reflected light input to the image processing device 210 is converted into image data IDAT2 by the lens 211, image sensor 212, and ISP front-end block 213. The illuminance calculator 220 can calculate the pixel illuminance based on the image data IDAT2. The pixel illuminance can be calculated using the following equation 2.

[0075] [Equation 2]

[0076] Referring to Equation 2, "p" is defined as the pixel value. Each pixel of the image sensor 212 can output an electrical signal based on the light incident upon it, and the pixel value can correspond to the level of the electrical signal output by each pixel. v "Defined as the gain of the image sensor 212 and the ISP front-end block 213. Here, the gain can be based on analog gain and digital gain." exposure "This is defined as the time that a pixel of the image sensor 212 is exposed to light (hereinafter referred to as 'exposure time')." v " and "t exposure "This can vary depending on the manufacturer's settings or the user's request. α and β are defined as constant values ​​for the image sensor. Because α and β vary depending on the type, characteristics, and function of the image processing device of the image sensor, they can be adjusted in the electronic device 10 (reference)." Figure 1 α and β are determined in advance before the product is shipped.

[0077] In an example embodiment, the image processing device 210 and / or electronic device 10 (see reference) can be transported via image processing device 210 and / or electronic device 10 (see reference). Figure 1 The image processing device 210 was previously tested to calculate α and β. For example, α and β could be calculated using a separate testing device (not shown). Return to Figure 8Since light reflected from the object is incident on the pixels, chart 12 can be used as a test object for the image processing apparatus 210 in order to set reference pixel values ​​(e.g., pixel values ​​corresponding to bright areas and pixel values ​​corresponding to dark areas). For example, chart 12 can be a test board. Direct light DL from light source 11 can be reflected at a specific portion 13 of chart 12, and the reflected light RL reflected by the specific portion 13 can be incident on the image processing apparatus 210 including image sensor 212.

[0078] For example, assume that the reflectance of the black portion of Chart 12 is 3% and the reflectance of its white portion is 97%. The reflectance in the chart is not limited to this and can vary depending on the characteristics of the chart. Furthermore, assume an environment where the illuminance of the direct light DL is 100 lux. Therefore, the illuminance of the reflected light RL incident on the pixel after reflection at the black portion (i.e., the pixel illuminance) (hereinafter referred to as "I1") is 100 lux. 0.03, the pixel illuminance corresponding to the white area (hereinafter referred to as "I2") is 100. 0.97. According to an example embodiment of this disclosure, the image processing apparatus 210 can output image data IDAT2, and the image data IDAT2 can include pixel values. The pixel value corresponding to the black portion (hereinafter referred to as "p1") can be "0", and the pixel value corresponding to the white portion (hereinafter referred to as "p2") can be 1023. Furthermore, assuming the image sensor 212's "A" v "It is 30 and its "t exposure "It is 40 ms. Equations 3 and 4, which are used to obtain α and β, can be derived based on Equation 2 and the above assumptions."

[0079] [Equation 3]

[0080] [Equation 4]

[0081] By solving the simultaneous equations (i.e., equations 3 and 4), α is 110.264 and β is 32.649. The values ​​corresponding to α and β obtained in the above testing process can be stored in a memory (not shown) located inside or outside the illuminance estimator 200. The illuminance calculator 220 can load the values ​​corresponding to α and β from the memory (not shown). The illuminance calculator 220 can calculate pixel illuminance based on α and β according to pixel values. However, the obtained α and β are merely examples. For example, α and β can vary depending on the illuminance of the direct light DL and the characteristics of the image sensor in the image processing device. Therefore, the new simultaneous equations can be used to determine the illuminance based on the illuminance environment, the image sensor settings, or the characteristics of the transport electronics 10 (see reference 10). Figure 1 The model of the previous image sensor is changed to calculate new α and β, and the values ​​corresponding to the new α and β can be stored in memory (not shown).

[0082] Figure 9 This is a configuration diagram illustrating an electronic device 20 performing motion blur removal according to an exemplary embodiment of the present disclosure. References will be omitted. Figure 1 The description provided is for the purpose of avoiding redundancy. Electronic device 20 may include a dynamic vision sensor 100, an illuminance estimator 200, a time delay compensator 300, an image processing block 400, and / or a motion blur removal block 500. The time delay compensator 300 can compensate for the time delay of the dynamic vision sensor 100's time data caused by illuminance or changes in illuminance. The time delay compensator 300 can provide a compensation event signal ES' to the motion blur removal block 500.

[0083] Image processing block 400 can generate image data IDAT. Image processing block 400 can perform various processing operations for generating image data IDAT. Image processing block 400 may include an image sensor. For example, the image sensor may be implemented using a CIS or CCD image sensor and / or the like. The image sensor may include multiple pixels. The pixels of the image sensor may face the same direction as the pixels of the dynamic vision sensor 100. Therefore, the pixels of the image sensor and the pixels of the dynamic vision sensor 100 may receive light reflected from the same object. The image sensor may generate frame-unit image data IDAT based on the charge acquired during a given or predetermined or desired exposure time. In some example embodiments where there is motion of the object during the exposure time, the image data may be a blurred image with motion blur. The relationship between continuous exposure time and the blurred image in frame units is expressed by the following Equation 5.

[0084] [Equation 5]

[0085] Referring to Equation 5, “B” is defined as the blurred image, and “T” is defined as the exposure time of the image sensor included in image processing block 400. L(t) is defined as the pixel output data at time point “t” belonging to the exposure time “T”, and “f” is defined as the reference time point. That is, a blurred image in frame units can be generated as a value obtained by integrating the pixel output data over the exposure time over a continuous period and dividing the result of the integration by the exposure time.

[0086] Motion blur removal block 500 can generate deblurred image data IDAT' based on the compensation event signal ES' and image data IDAT. For example, in an example embodiment where motion blur occurs in image data IDAT, motion blur removal block 500 can remove motion blur from image data IDAT based on the compensation event signal ES'. Although in Figure 9 Although not illustrated, the motion blur removal block 500 can generate deblurred image data IDAT' based on the event signal ES. However, because the time delay of the dynamic vision sensor 100 based on illumination or illumination changes is not compensated, the event signal ES may not be synchronized with the image data IDAT. Therefore, the motion blur removal effect of the deblurred image data IDAT' based on the event signal ES may be lower than that based on the compensated event signal ES. The motion blur removal block 500 is illustrated as a separate block, but at least a portion of the functionality of the motion blur removal block 500 can be implemented within the image processing block 400.

[0087] The compensation event signal ES' can be a set of discontinuous data sampled in time units shorter than the frame period of the generated image data IDAT. The method for obtaining light intensity changes up to a specific time point based on the discontinuous event signal can follow the following equation 6.

[0088] [Equation 6]

[0089] Referring together to Equations 5 and 6, “s” is defined as the event occurrence time. The event occurrence time “s” can correspond to the timestamp value included in the compensation event signal ES’. e(s) can be the polarity information at the event occurrence time “s”. Here, the polarity information can be “1” when the light intensity increases and “-1” when the light intensity decreases. Therefore, E(t) corresponding to the cumulative change in light intensity at a specific time point “t” can be the sum of the polarity information from the reference time point “f” to the specific time point “t”.

[0090] The motion blur removal block 500 can extract a still image at a specific time point corresponding to the exposure time of a frame in the image data IDAT, based on the compensation event signal ES'. The method for obtaining the still image can follow the following equation 7.

[0091] [Equation 7]

[0092] Referring together to Equations 5, 6, and 7, the pixel output data L(t) at a specific time point "t" (e.g., still image data) can be obtained by multiplying the pixel output data L(f) at reference time point "f" by an exponential form E(t) corresponding to the change in light intensity at "t". The letter "c" is the constant multiplied by E(t). When rearranging the equations for the pixel output data L(f) at reference time point "f" using Equations 5 through 7, the relationship shown in Equation 8 below can be followed.

[0093] [Equation 8]

[0094] Referring to equations 5 to 8, by substituting the pixel output data L(f) at reference time point "f" into equation 7, pixel output data L(t) at a specific time point "t" can be generated, that is, the deblurred image data IDAT' corresponding to the still image. For the electronic device 20 according to an example embodiment of this disclosure, reference will be made to... Figures 10 to 12B A detailed description of the visualization data for image data IDAT and deblurred image data IDAT'.

[0095] According to an example embodiment of this disclosure, at least some of the functions of the other components of the electronic device 20 besides the dynamic vision sensor 100 can be implemented in an application processor, namely, at least some of the functions of the illuminance estimator 200, the time delay compensator 300, the image processing block 400, and the motion blur removal block 500.

[0096] Figure 10 This is a configuration diagram illustrating an electronic device 20 performing motion blur removal according to other exemplary embodiments of the present disclosure. References will be omitted. Figure 1 and Figure 9 The description provided is for the purpose of avoiding redundancy. Electronic device 20 may include a dynamic vision sensor 100, an illuminance estimator 200, a time delay compensator 300, and / or a motion blur removal block 500. The illuminance estimator 200 may include an image processing device 210 and / or an illuminance calculator 220. The image processing device 210 may generate image data IDAT and may provide the image data IDAT to the motion blur removal block 500. (See reference...) Figure 9In the electronic device 20 described according to an example embodiment of the present disclosure, image data IDAT (reference) Figure 9 ) can be processed from a separate image processing block 400 (reference) Figure 9 This is provided to motion blur removal block 500. In contrast, in Figure 10 In the electronic device 20 illustrated in the example embodiment of the present disclosure, image data IDAT (reference) Figure 10 The image processing device 210 included in the illumination estimator 200 can be provided to the motion blur removal block 500.

[0097] Figure 11 The illustration shows image data with motion blur according to an example embodiment of the present disclosure. (See also:) Figure 9 and Figure 11 In an example embodiment, an object can move at a speed of 0.5 m / s at a distance of 1 m from the image processing block 400. In an example embodiment, the image processing block 400 can capture the object in a low-light environment of 5 lux for an exposure time of 100 ms. Because the object moves continuously during the exposure time, the image processing block 400 can generate motion-blurred image data IDAT, i.e., like... Figure 11 The same blurry image.

[0098] Figure 12A The illustration shows a still image to which motion blur removal is performed using the original event signal, according to an example embodiment of this disclosure. (Reference) Figure 9 and Figure 11 Assuming the event signal ES generated by the dynamic vision sensor 100 is directly provided to the motion blur removal block 500, as referenced... Figure 9 As described. In some example embodiments, the motion blur removal block 500 can extract a still image at a specific time point from the image data IDAT based on the event signal ES, such as... Figure 12A As shown. With Figure 11 In comparison, according to Figure 12A It was confirmed that motion blur was partially removed.

[0099] Figure 12B The illustration shows a still image to which motion blur removal is performed using a compensation event signal, according to an example embodiment of this disclosure. Reference Figure 9 and Figure 11 The compensation event signal ES' generated by the time delay compensator 300 can be provided to the motion blur removal block 500. Because the compensation event signal ES' provides polarity information along with a timestamp value without time delay, the motion blur removal block 500 can extract a still image at a specific time point with minimal error. Therefore, with... Figure 12A In comparison, according to Figure 12BMotion blur was confirmed to have been further removed.

[0100] Figure 13 This is a configuration diagram illustrating an electronic device 30 performing Simultaneous Localization and Mapping (SLAM) according to an exemplary embodiment of the present disclosure. References will be omitted. Figure 1 The description given is to avoid redundancy. Electronic device 30 may include dynamic vision sensor 100, illuminance estimator 200, time delay compensator 300, inertial measurement unit (IMU) sensor 600 and / or SLAM block 700.

[0101] The IMU sensor 600 can generate inertial data IMD by measuring the tilt angle of the electronic device 30. For example, the IMU sensor 600 may include at least one of a gyroscope, an accelerometer, and a geomagnetic sensor. Therefore, the inertial data IMD may include at least one of angular rate information, acceleration information, and geomagnetic information.

[0102] The SLAM block 700 can create a map by sensing the surrounding environment of the electronic device 30 and can estimate the current position of the electronic device 30. The SLAM block 700 can generate SLAM data D that includes map and location information. SLAM For example, the SLAM block 700 can reduce sensing errors by sensing the surrounding environment based on the compensation event signal ES' and utilizing the inertial data IMD. Here, because the compensation event signal ES' is in a state where the time delay is compensated, the compensation event signal ES' can be synchronized with the inertial data IMD.

[0103] Figure 14 The illustration shows the configuration of an electronic device for a camera module that utilizes the dynamic vision sensor disclosed herein.

[0104] refer to Figure 14 The electronic device 1000 may include a camera module group 1100, an application processor 1200, a PMIC 1300, and / or an external memory 1400.

[0105] Camera module group 1100 may include multiple camera modules 1100a, 1100b, and 1100c. Figure 14 The diagram illustrates an electronic device comprising three camera modules 1100a, 1100b, and 1100c, but this disclosure is not limited thereto. Multiple camera modules 1100a, 1100b, and 1100c can generate electrical signals based on light reflected from an object, landscape, and / or the like. In some example embodiments, the camera module group 1100 can be modified to include only two camera modules. Additionally, in some example embodiments, the camera module group 1100 can be modified to include "n" camera modules (n is a natural number of 4 or greater). Although in Figure 14Not shown in the figure, but each of the multiple camera modules 1100a, 1100b and 1100c may include a prism, an optical path folding element, an actuator, an image sensing device and a storage device.

[0106] In some example embodiments, at least one of camera modules 1100a, 1100b, and 1100c may be implemented using a dynamic vision sensor. For example, camera module 1100a may be implemented using a dynamic vision sensor, and camera module 1100b may be implemented using an image sensor (e.g., a CIS). In some example embodiments, camera module 1100a may generate an event signal based on changes in light intensity, and camera module 1100b may generate image data. According to example embodiments of this disclosure, multiple camera modules 1100a, 1100b, and 1100c may have the same field of view.

[0107] In some example embodiments, at least two camera modules (e.g., 1100a and 1100b) of the plurality of camera modules 1100a, 1100b and 1100c may have different fields of view. In some example embodiments, at least two camera modules (e.g., 1100a and 1100b) of the plurality of camera modules 1100a, 1100b and 1100c may include different optical lenses, but are not limited thereto.

[0108] Additionally, in some example embodiments, the fields of view of the multiple camera modules 1100a, 1100b, and 1100c may be different. In some example embodiments, the multiple camera modules 1100a, 1100b, and 1100c may include different optical lenses, but are not limited thereto.

[0109] In some example embodiments, multiple camera modules 1100a, 1100b, and 1100c can be configured to be physically separate from each other. That is, multiple camera modules 1100a, 1100b, and 1100c may not use the sensing area of ​​a single image sensor, but rather each of the multiple camera modules 1100a, 1100b, and 1100c may include an independent image sensor.

[0110] Application processor 1200 may include image processing device 1210, memory controller 1220, and / or internal memory 1230. Application processor 1200 may be implemented separately from multiple camera modules 1100a, 1100b, and 1100c. For example, application processor 1200 and multiple camera modules 1100a, 1100b, and 1100c may be implemented using separate semiconductor chips.

[0111] Image processing apparatus 1210 may include a plurality of sub-image processors 1212a, 1212b and 1212c, an image generator 1214 and / or a camera module controller 1216. Image processing apparatus 1210 may include a plurality of sub-image processors 1212a, 1212b and 1212c in a number corresponding to the number of the plurality of camera modules 1100a, 1100b and 1100c.

[0112] Image data generated from camera modules 1100a, 1100b, and 1100c can be provided to corresponding sub-image processors 1212a, 1212b, and 1212c via separate image signal lines ISLa, ISLb, and ISLc, respectively. For example, image data generated from camera module 1100a can be provided to sub-image processor 1212a via image signal line ISLa, image data generated from camera module 1100b can be provided to sub-image processor 1212b via image signal line ISLb, and image data generated from camera module 1100c can be provided to sub-image processor 1212c via image signal line ISLc. For example, this image data transmission can be performed using a camera serial interface (CSI) based on MIPI (Mobile Industrial Processor Interface), but this disclosure is not limited thereto.

[0113] In some example embodiments, camera module 1100a may be implemented using a dynamic vision sensor. In some example embodiments, event signals generated from camera module 1100a may be provided to sub-image processor 1212a via image signal line ISLa. Alternatively, camera module 1100b may be implemented using an image sensor (e.g., CIS). Image data generated from camera module 1100b may be provided to sub-image processor 1212b via image signal line ISLb. Sub-image processor 1212b may estimate pixel illuminance based on the image data generated from camera module 1100b. Although not illustrated, sub-image processor 1212b may provide information about the estimated pixel illuminance to sub-image processor 1212a. Sub-image processor 1212a may compensate for the event occurrence time delay of the dynamic vision sensor based on the information about the estimated pixel illuminance.

[0114] In some example embodiments, a sub-image processor can be configured to correspond to multiple camera modules. For example, sub-image processors 1212a and 1212c can be implemented integrally, rather than separately as shown in FIG12; in some example embodiments, one of multiple image data provided from camera module 1100a and camera module 1100c respectively can be selected by a selection element (e.g., a multiplexer), and the selected image data can be provided to the integrated sub-image processor.

[0115] Event signals or image data provided to sub-image processors 1212a, 1212b, and 1212c respectively can be provided to image generator 1214. Image generator 1214 can generate an output image based on image generation information or mode signals by using image data provided from sub-image processors 1212a, 1212b, and 1212c respectively.

[0116] For example, image generator 1214 can generate an output image by merging at least a portion of image data generated from camera modules 1100a, 1100b, and 1100c, which have different fields of view, based on image generation information or a pattern signal. Alternatively, image generator 1214 can generate an output image by selecting one of the image data generated from camera modules 1100a, 1100b, and 1100c, which have different fields of view, based on image generation information or a pattern signal.

[0117] In some example embodiments, the image generator 1214 may remove motion blur present in image data output from the camera module 1100b (e.g., CIS) based on event signals output from the camera module 1100a (e.g., a dynamic vision sensor).

[0118] The camera module controller 1216 can provide control signals to camera modules 1100a, 1100b, and 1100c respectively. The control signals generated by the camera module controller 1216 can be provided to the corresponding camera modules 1100a, 1100b, and 1100c respectively via separate control signal lines CSLa, CSLb, and CSLc.

[0119] Application processor 1200 can store the received image signal (i.e., the encoded image signal) in internal memory 1230 or in external memory 1400 located outside application processor 1200. Subsequently, application processor 1200 can read and decode the encoded image signal from internal memory 1230 or external memory 1400, and can display image data generated based on the decoded image signal. For example, the corresponding sub-image processors among sub-image processors 1212a, 1212b, and 1212c of image processing apparatus 1210 can perform decoding and can also perform image processing on the decoded image signal.

[0120] The image signal provided to the application processor 1200 may be an unencoded signal. The application processor 1200 may perform image processing on the received image signal, or it may store the image signal in internal memory 1230 or external memory 1400.

[0121] PMIC 1300 can supply power, for example, power supply voltage, to multiple camera modules 1100a, 1100b, and 1100c respectively. For example, under the control of application processor 1200, PMIC 1300 can supply first power to camera module 1100a via power signal line PSLa, supply second power to camera module 1100b via power signal line PSLb, and supply third power to camera module 1100c via power signal line PSLc.

[0122] In response to a power control signal PCON from application processor 1200, PMIC 1300 can generate power corresponding to each of the plurality of camera modules 1100a, 1100b, and 1100c, and can adjust the power level. The power control signal PCON may include a power adjustment signal for each operating mode of the plurality of camera modules 1100a, 1100b, and 1100c. For example, the operating mode may include a low-power mode. In some example embodiments, the power control signal PCON may include information about the camera module operating in low-power mode and the set power level. The power levels provided to the plurality of camera modules 1100a, 1100b, and 1100c may be the same or different from each other. Additionally, the power level can be changed dynamically.

[0123] Figure 15 This is a flowchart illustrating an operation method of an electronic device according to an exemplary embodiment of the present disclosure. However, for the sake of convenience, reference will be made to... Figure 1 describe Figure 15 .

[0124] In operation S110, the dynamic vision sensor 100 can sense changes in light intensity (e.g., an event) to generate an event signal ES. The time corresponding to the timestamp value included in the event signal ES may differ from the actual time the event occurred.

[0125] In operation S120, the illuminance estimator 200 can estimate illuminance or illuminance changes. For example, the illuminance estimator 200 can estimate the illuminance (e.g., pixel illuminance) of light incident on each pixel of the dynamic vision sensor 100. According to exemplary embodiments of this disclosure, the illuminance estimator 200 can be implemented using a device including an image sensor (e.g., a CIS). In some exemplary embodiments, the illuminance estimator 200 can estimate pixel illuminance based on image data output from the image sensor, and can generate an illuminance information ID that includes information about illuminance or illuminance changes based on the pixel illuminance.

[0126] In operation S130, the time delay compensator 300 can compensate for the time delay of the event signal ES based on the estimated illuminance or the estimated change in illuminance. That is, the time delay compensator 300 can compensate for the time delay between the time when the light intensity change occurs and the time when the light intensity change is sensed. For example, the time delay compensator 300 can adjust the timestamp value to correspond to the time when the light intensity change occurs. As a result, the time delay compensator 300 can generate a compensated event signal ES'.

[0127] Figure 16 This is a flowchart illustrating an operation method of an electronic device performing motion blur cancellation according to an exemplary embodiment of the present disclosure. Operations S210 to S230 and... Figure 15 Operations S110 to S130 are the same; therefore, additional descriptions will be omitted to avoid redundancy. For ease of description, reference will be made to... Figure 9 describe Figure 16 .

[0128] In operation S240, image processing block 400 can generate image data IDAT. According to an example embodiment of this disclosure, image processing block 400 can be as follows: Figure 10 The illuminance estimator 200 shown includes an image processing device 210 (see image processing device 210). Figure 10 ).

[0129] In operation S250, motion blur removal block 500 can remove motion blur from image data IDAT based on compensation event signal ES'.

[0130] According to this disclosure, a dynamic vision sensor can provide consistent performance regardless of external illuminance or changes in illuminance.

[0131] According to exemplary embodiments of this disclosure, the motion blur removal performance utilizing dynamic vision sensors can be improved.

[0132] The exemplary embodiments are not necessarily mutually exclusive. For example, some exemplary embodiments may include one or more features described with reference to one or more of the accompanying drawings, and may also include one or more other features described with reference to one or more other of the accompanying drawings.

[0133] One or more of the elements disclosed above may include or be implemented with one or more processing circuits such as: hardware including logic circuits; hardware / software combination such as a processor executing software; or combinations thereof. For example, the processing circuits may more specifically include, but are not limited to, a central processing unit (CPU), an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field-programmable gate array (FPGA), a system-on-a-chip (SoC), a programmable logic unit, a microprocessor, an application-specific integrated circuit (ASIC), etc.

[0134] Although this disclosure has been described with reference to exemplary embodiments thereof, it will be apparent to those skilled in the art that various changes and modifications may be made thereto without departing from the spirit and scope of this disclosure as set forth in the appended claims.

Claims

1. An electronic device comprising: A dynamic vision sensor, the dynamic vision sensor including a first pixel configured to sense changes in light intensity, and the dynamic vision sensor configured to generate an event signal based on the sensed changes in light intensity; An illuminance estimator, configured to estimate the illuminance of light; as well as A time delay compensator is configured to: calculate the time delay between a first time when the light intensity change occurs and a second time when the first pixel senses the light intensity change based on the illuminance of the light, and compensate for the time delay. The illuminance estimator includes: An image processing apparatus, comprising an image sensor configured to generate an electrical signal based on incident light, and the image processing apparatus configured to generate image data based on the electrical signal; and An illuminance calculator, configured to calculate pixel illuminance based on the image data.

2. The electronic device according to claim 1, wherein, The event signal includes: the position value of the first pixel, a polarity value including information about the change in light intensity, and a timestamp value indicating the time when the first pixel sensed the change in light intensity.

3. The electronic device according to claim 2, wherein, The time delay compensator adjusts the timestamp value based on the time delay.

4. The electronic device according to claim 1, wherein, The image sensor is implemented using a complementary metal-oxide-semiconductor image sensor.

5. The electronic device according to claim 1, wherein, The image sensor includes at least one second pixel, and Wherein, the first pixel corresponds to the at least one second pixel.

6. The electronic device according to claim 5, wherein, The electrical signal is generated by at least one second pixel. The image data includes pixel values ​​corresponding to the electrical signal, and The illuminance calculator calculates the pixel illuminance based on the pixel value.

7. The electronic device according to claim 5, wherein, The at least one second pixel and the first pixel receive light reflected from the same object.

8. The electronic device according to claim 1, wherein, As the illuminance of the light decreases, the time delay increases, and the time delay compensator is further configured to compensate for the increased time delay.

9. The electronic device according to claim 1, wherein, The illuminance calculator is also configured to calculate illuminance changes based on the estimated illuminance of the light, and Wherein, as the change in illuminance decreases, the time delay increases, and the time delay compensator is further configured to compensate for the increased time delay.

10. An electronic device comprising: A dynamic vision sensor includes a first pixel configured to sense changes in light intensity, and the dynamic vision sensor is configured to generate a first event signal based on the sensed changes in light intensity. An image processing apparatus, the image processing apparatus including a CMOS image sensor configured to generate first image data based on light intensity, wherein the CMOS is a complementary metal-oxide-semiconductor. An illuminance calculator configured to calculate at least one of illuminance and illuminance variation based on the first image data; and A time delay compensator is configured to: calculate a time delay between a first time when the light intensity change occurs and a second time when the first pixel senses the light intensity change, based on at least one of the light illuminance and the light illuminance change, and generate a second event signal for which the time delay is compensated. The CMOS image sensor includes a second pixel.

11. The electronic device according to claim 10, wherein, The first pixel corresponds to the second pixel.

12. The electronic device according to claim 11, wherein, The dynamic vision sensor further includes a third pixel configured to sense changes in light intensity, and the dynamic vision sensor generates a third event signal based on the changes in light intensity sensed by the third pixel. The CMOS image sensor further includes a fourth pixel, and The third pixel corresponds to the fourth pixel.

13. The electronic device according to claim 12, wherein, The first event signal includes first-time data. The third event signal includes second time data. The first image data includes: a first pixel value corresponding to a first electrical signal generated by the second pixel, and a second pixel value corresponding to a second electrical signal generated by the fourth pixel. The illuminance calculator calculates at least one of the first pixel illuminance and the change in first pixel illuminance based on the first pixel value, and calculates at least one of the second pixel illuminance and the change in second pixel illuminance based on the second pixel value. The time delay compensator generates a second event signal obtained by compensating for a first time delay based on at least one of the first pixel illuminance and the change in the first pixel illuminance, and generates a fourth event signal obtained by compensating for a second time delay based on at least one of the second pixel illuminance and the change in the second pixel illuminance.

14. The electronic device of claim 10, further comprising: An image processing block configured to generate second image data; as well as Motion blur removal block, which is configured to remove motion blur present in the second image data.

15. The electronic device according to claim 14, wherein, The image processing block generates the second image data based on the charge obtained during the exposure time corresponding to the frame, according to the manufacturer's settings or the user's request. The motion blur removal block extracts a still image at a specific time point belonging to the exposure time based on the second event signal, and generates deblurred image data based on the still image at the specific time point.

16. The electronic device according to claim 14, wherein, The second image data is generated in units of frames, and The second event signal is a set of discontinuous data sampled at a time period shorter than the frame generation period.

17. The electronic device of claim 10, further comprising: Motion blur removal block, which is configured to remove motion blur present in the first image data.

18. The electronic device of claim 10, further comprising: An inertial measurement unit sensor, the inertial measurement unit sensor being configured to generate inertial data by measuring tilt angles; as well as Simultaneous localization and mapping (SLTAR) blocks are configured to estimate their current position by sensing the surrounding environment and creating a map based on the second event signal and the inertial data.

19. An electronic device, the electronic device comprising: A camera module group, comprising multiple camera modules, each camera module generating an electrical signal based on light reflected from an object; as well as An application processor configured to process the electrical signal. The first camera module among the plurality of camera modules utilizes a dynamic vision sensor, and the second camera module among the plurality of camera modules utilizes an image sensor. The first camera module generates event signals based on changes in light intensity. The second camera module generates image data in units of frames. The application processor estimates at least one of light intensity and light intensity change based on the image data, and compensates for the time delay between the first time when the light intensity change occurs and the second time when the first camera module senses the light intensity change based on the light intensity and light intensity change.

Images