Imaging device and information processing method

Abstract

The present invention provides an imaging device and an information processing method that can detect changes in brightness with lower latency. [Solution] The imaging device comprises a first image sensor and a second image sensor arranged on the same optical axis, which receive light during exposure and have an NDR function that allows non-destructive reading of pixel signals based on the received light during the exposure; and a control unit which alternately executes a first operation pattern in each frame, in which the first image sensor performs a first operation including resetting the pixel charge and exposure, and the second image sensor performs a second operation including exposure but without resetting the pixel charge, and a second operation pattern, in which the first image sensor performs a second operation and the second image sensor performs a first operation. When an event causing a change in brightness occurs, the image sensor performing the second operation outputs a brightness value based on the event based on the pixel signal read out by the NDR function.

Description

Technical Field

[0001] The present invention relates to an imaging device and an information processing method.

Background Art

[0002] Conventionally, an imaging device that performs imaging using a plurality of imaging elements has been known. For example, Patent Document 1 describes a camera system that alternately performs exposure, reset, and readout for each frame using a first imaging element and a second imaging element.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the camera system described in Patent Document 1, in each imaging element, information is read out in the frame next to the frame in which exposure is performed. Therefore, in this camera system, even when an event accompanied by a change in luminance occurs in the subject to be imaged, at least a delay corresponding to one frame occurs before the event is detected. As a result, in the camera system described in Patent Document 1, there are cases where the delay cannot be made sufficiently small.

[0005] According to some aspects of the present disclosure, it is possible to provide an imaging device and an information processing method capable of detecting a change in luminance with lower latency.

Means for Solving the Problems

[0006] <An imaging device according to one embodiment of the present invention comprises a first image sensor and a second image sensor arranged on the same optical axis, which receive light during exposure and have an NDR (Non Destructive Readout) function that allows non-destructive reading of pixel signals based on the received light during exposure; and a control unit that alternately executes, in each frame, a first operation pattern in which the first image sensor performs a first operation including resetting the pixel charge and exposure, and a second operation pattern in which the first image sensor performs the second operation and the second image sensor performs the first operation, wherein the image sensor performing the second operation when an event causing a change in brightness occurs outputs a brightness value based on the event based on the pixel signal read out by the NDR function.

[0007] An information processing method according to one embodiment of the present disclosure is an information processing method performed by an imaging device comprising: a first image sensor and a second image sensor arranged on the same optical axis, which receive light during exposure and have an NDR (Non Destructive Readout) function that allows non-destructive reading of pixel signals based on the received light during exposure; and a control unit that controls the operation of the first image sensor and the second image sensor, wherein in each frame, a first operation pattern is performed alternately, in which the first image sensor is made to perform a first operation including resetting the pixel charge and exposure, and the second image sensor is made to perform a second operation including exposure but not resetting the pixel charge; and a second operation pattern is performed in which the first image sensor is made to perform the second operation and the second image sensor is made to perform the first operation; and when an event causing a change in brightness occurs, the image sensor that is performing the second operation outputs a brightness value based on the event based on the pixel signal read out by the NDR function. [Brief explanation of the drawing]

[0008] [Figure 1]This is a schematic diagram showing an example of the configuration of an imaging device according to one embodiment. [Figure 2] This diagram shows the operation of the first and second image sensors shown in Figure 1. [Figure 3] Figure 1 is a flowchart showing an example of the processing performed by the control unit. [Modes for carrying out the invention]

[0009] The embodiments will be described below with reference to the drawings. In the drawings, the same or equivalent elements are denoted by the same reference numerals, and redundant descriptions are omitted. The embodiments described below are not intended to unduly limit the content described in the claims. Furthermore, not all of the configurations described in these embodiments are essential components of this disclosure.

[0010] <Example of imaging device configuration> Figure 1 is a schematic diagram showing an example of the configuration of an imaging device 1 according to one embodiment. As shown in Figure 1, the imaging device 1 comprises a plurality of imaging optical systems 11, an event sensor unit 12, a first image sensor 13a and a second image sensor 13b, a half mirror 14, a control unit 15, and a storage unit 16. The imaging device 1 is configured with these mechanisms housed inside a housing. In this embodiment, the event sensor unit 12 comprises two event sensors, a first event sensor 12a and a second event sensor 12b. In Figure 1, the dashed lines indicate light rays incident on the imaging device 1.

[0011] The imaging optical system 11 includes, for example, one or more optical components such as lenses and apertures. The imaging optical system 11 forms an image of the subject on the light-receiving surfaces of each event sensor included in the event sensor unit 12, as well as the first image sensor 13a and the second image sensor 13b. Specifically, in the imaging device 1, an imaging optical system 11 is arranged in a one-to-one correspondence with each event sensor included in the event sensor unit 12, and one imaging optical system 11 is arranged for common use with the first image sensor 13a and the second image sensor 13b. In this embodiment, since the event sensor unit 12 includes two event sensors, the imaging device 1 includes a total of three imaging optical systems 11.

[0012] More specifically, as shown in Figure 1, the imaging device 1 includes a first imaging optical system 11a that forms an image of a subject on the light-receiving surface of the image sensor of the first event sensor 12a, a second imaging optical system 11b that forms an image of a subject on the light-receiving surface of the image sensor of the second event sensor 12b, and a third imaging optical system 11c that forms an image of a subject on the light-receiving surfaces of the image sensors of the first image sensor 13a and the second image sensor 13b. If the number of event sensors in the event sensor unit 12 is other than two, the imaging device 1 may have an imaging optical system 11 equal to the number of event sensors in the event sensor unit 12 plus one imaging optical system 11c used by the first image sensor 13a and the second image sensor 13b.

[0013] The event sensor unit 12 detects brightness changes of each pixel based on an event and outputs an event signal indicating the coordinates of the pixel whose brightness has changed. In this embodiment, as described above, the event sensor unit 12 is equipped with two event sensors. The first event sensor 12a and the second event sensor 12b are image sensors equipped with, for example, a pixel array unit which comprises an event detection pixel group having multiple event pixels that detect changes in the amount of light received by a light receiving element that exceed a predetermined amount as an event.

[0014] The first event sensor 12a and the second event sensor 12b are positioned at different locations within the housing of the imaging device 1. Specifically, the first event sensor 12a and the second event sensor 12b are positioned within the housing of the imaging device 1 so that they can capture the same event from different locations within a common field of view. The first event sensor 12a and the second event sensor 12b are positioned so that the location where the event occurred can be identified using epipolar constraint. The first event sensor 12a and the second event sensor 12b output event signals based on events captured from different locations within a common field of view.

[0015] The first image sensor 13a and the second image sensor 13b are arranged on the same optical axis. Specifically, light incident after passing through the third imaging optical system 11c is split into a first and second light beam by the half mirror 14, and the first image sensor 13a and the second image sensor 13b are positioned to receive the first and second light beams, respectively. This allows the first image sensor 13a and the second image sensor 13b to be arranged on the same optical axis. Note that the half mirror 14 can be a half mirror with a known configuration.

[0016] The first image sensor 13a and the second image sensor 13b have common functions and configurations. The first image sensor 13a and the second image sensor 13b are composed of a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal-Oxide Semiconductor device) image sensor, etc. Each pixel arranged in the light-receiving area of ​​the first image sensor 13a and the second image sensor 13b generates a pixel signal according to the amount of light received. In this embodiment, the first image sensor 13a and the second image sensor 13b have an NDR (Non-Destructive Readout) function that allows light to be received during exposure and the pixel signal based on the received light to be read out non-destructively during the exposure. Known examples of image sensors with an NDR function include the application of a floating gate amplifier (FGA) to a CCD image sensor. However, the first image sensor 13a and the second image sensor 13b are not limited to this example and can be configured using any known image sensor with an NDR function.

[0017] The first image sensor 13a and the second image sensor 13b perform exposure, reset, and readout. Exposure causes charge to accumulate in each pixel of the first image sensor 13a and the second image sensor 13b according to the amount of light received. Reset resets the capacitance of each pixel of the first image sensor 13a and the second image sensor 13b. Readout reads out the charge accumulated in each pixel of the first image sensor 13a and the second image sensor 13b, and a pixel signal is generated. The first image sensor 13a and the second image sensor 13b can perform two types of exposure: full-screen exposure and NDR exposure. Full-screen exposure is normal exposure that accumulates charge in each pixel, while NDR exposure is exposure performed in a state where the above NDR function can be executed.

[0018] The control unit 15 controls and manages the entire imaging device 1, including each functional unit of the imaging device 1. The control unit 15 performs various controls, such as operating a control program stored in the storage unit 16. For example, the control unit 15 can be constituted by a control device such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). The control unit 15 executes the processes described in this embodiment using the first imaging element 13a and the second imaging element 13b. Specific processing details will be described later.

[0019] The storage unit 16 is a storage medium capable of storing programs and data. The storage unit 16 can be constituted by, for example, a semiconductor memory or a magnetic memory. Specifically, the storage unit 16 can be constituted by, for example, an EEPROM (Electrically Erasable Programmable Read-Only Memory). The storage unit 16 stores, for example, a program for operating the control unit 15.

[0020] <Basic Processing> Next, an example of the process executed by the imaging device 1 will be described. FIG. 2 is a diagram showing the operations of the first imaging element 13a and the second imaging element 13b shown in FIG. 1. The first imaging element 13a and the second imaging element 13b execute the operations shown in FIG. 2 based on the control by the control unit 15, for example.

[0021] FIG. 2 shows the operations of the first imaging element 13a and the second imaging element 13b over time. In FIG. 2, the horizontal axis represents time. In FIG. 2, in particular, the operations for three frames, namely frame N, frame N + 1, and frame N + 2, are shown.

[0022] In each frame, the control unit 15 causes the first imaging device 13a and the second imaging device 13b to alternately execute the first operation pattern and the second operation pattern. The first operation pattern is a pattern in which the first imaging device 13a executes the first operation and the second imaging device 13b executes the second operation. The second operation pattern is a pattern in which the first imaging device 13a executes the second operation and the second imaging device 13b executes the first operation. The first operation is an operation including reset and exposure. The exposure in the first operation is full-frame exposure. The second operation is an operation including exposure but not including reset. The exposure in the second operation is NDR exposure. That is, in the first operation pattern, the first imaging device 13a executes reset and exposure, and the second imaging device 13b executes only exposure without performing reset. In the second operation pattern, the first imaging device 13a executes only exposure without performing reset, and the second imaging device 13b executes reset and exposure.

[0023] In the example shown in FIG. 2, frame N starts at time T1. In frame N, the first operation pattern is executed. Therefore, in frame N, the first imaging device 13a executes the first operation and the second imaging device 13b executes the second operation.

[0024] Specifically, the first imaging device 13a performs reset at the start time T1 of frame N. The reset at this time is a reset of the charge accumulated in the pixels by the exposure (NDR exposure) of the frame N−1 immediately before frame N. Then, the first imaging device 13a starts the exposure (full-frame exposure) in frame N at time T2 of frame N. Furthermore, the first imaging device 13a starts reset at time T3 of frame N. The reset at this time is a reset of the charge accumulated in the pixels by the exposure (full-frame exposure) from time T2 to time T3 of frame N. By this reset, the first imaging device 13a can perform exposure (NDR exposure) from the start of frame N+1 starting at time T4. In this way, the first imaging device 13a executes the first operation in frame N.

[0025] Furthermore, as shown in Figure 2, the first image sensor 13a performs readout in frame N. At this time, the charge accumulated in the pixels by the exposure (NDR exposure) of the frame N-1 immediately preceding frame N is read out by the first image sensor 13a, and a pixel signal is generated. This generates the pixel signal captured in frame N-1. Therefore, in frame N, exposure (full-screen exposure) of frame N is performed, and the pixel signal of frame N-1 is read out.

[0026] Meanwhile, the second image sensor 13b starts exposure (NDR exposure) in frame N from the start time T1 of frame N. As shown in Figure 2, the second image sensor 13b performs only exposure during frame N from time T1 to time T4. In this way, the second image sensor 13b performs the second operation in frame N.

[0027] As described above, the first and second operation patterns are executed alternately in each frame. Therefore, in frame N+1, which immediately follows frame N, the second operation pattern is executed. Consequently, in frame N+1, the first image sensor 13a performs the second operation, and the second image sensor 13b performs the first operation.

[0028] Specifically, the first image sensor 13a starts exposure (NDR exposure) in frame N+1 from the start time T4 of frame N+1. Since the first image sensor 13a is reset between time T3 and time T4 of frame N, in frame N+1, the first image sensor 13a can perform exposure without resetting. As shown in Figure 2, the first image sensor 13a performs only exposure during frame N+1 from time T4 to time T9. In this way, the first image sensor 13a performs the second operation in frame N+1.

[0029] Meanwhile, the second image sensor 13b performs a reset at the start time T4 of frame N+1. This reset resets the charge accumulated in the pixels by the exposure (NDR exposure) of frame N. Then, at time T5 of frame N+1, the second image sensor 13b starts exposure (full-screen exposure) of frame N+1. Furthermore, the second image sensor 13b starts a reset at time T8 of frame N+1. This reset resets the charge accumulated in the pixels by the exposure (full-screen exposure) from time T5 to time T8 of frame N+1. This reset allows the second image sensor 13b to perform exposure (NDR exposure) from the start of frame N+2, which starts at time T9. In this way, the second image sensor 13b performs the first operation in frame N+1.

[0030] Furthermore, as shown in Figure 2, the second image sensor 13b performs readout in frame N+1. At this time, the second image sensor 13b reads out the charge accumulated in the pixels due to the exposure of frame N (NDR exposure), and a pixel signal is generated. This generates the pixel signal captured in frame N. Therefore, in frame N+1, exposure of frame N+1 is performed, and the pixel signal of frame N is read out.

[0031] In frame N+2, immediately following frame N+1, the first operation pattern is executed. The details of the operation in the first operation pattern are the same as those described in frame N above, so a detailed explanation is omitted here.

[0032] In this manner, the control unit 15 causes the first image sensor 13a and the second image sensor 13b to alternately execute the first operation pattern and the second operation pattern in each frame. The control unit 15 outputs the video based on the pixel signals read out by the execution of the first operation pattern and the second operation pattern. The control unit 15 may store the read out pixel signals in the storage unit 16.

[0033] <Processing when an event occurs> Next, we will explain an example of how to handle an event when it occurs, continuing to refer to Figure 2.

[0034] As shown in Figure 2, suppose an event occurs between time T6 and time T7 in frame N+1. The event is detected by the event sensor unit 12. Specifically, when the first event sensor 12a and the second event sensor 12b of the event sensor unit 12 detect a change in the amount of light received by the photodetector that exceeds a predetermined amount, they output an event signal indicating the temporal change in light brightness based on the event.

[0035] The control unit 15 outputs a brightness value based on the event, based on the pixel signal read out by the NDR function from the image sensor performing the second operation when an event causing a brightness change occurs. The control unit 15 can detect that an event causing a brightness change has occurred by the event signal. Therefore, based on the event signal, the control unit 15 outputs a brightness value based on the event, based on the pixel signal read out by the NDR function from the image sensor performing the second operation when the event sensor unit 12 outputs the event signal. In the example in Figure 2, the first image sensor 13a performs the second operation in frame N+1. Therefore, based on the event signal, the control unit 15 outputs a brightness value based on the event, based on the pixel signal read out by the first image sensor 13a using the NDR function.

[0036] Specifically, based on the event signal, the control unit 15 causes the first image sensor 13a to read out pixel signals based on the light received during exposure, using the NDR function, during the exposure. In other words, the control unit 15 causes the first image sensor 13a to read out the pixel signals acquired during the exposure. The first image sensor 13a reads out pixel signals corresponding to the integrated light received by each pixel through NDR exposure.

[0037] The control unit 15 then calculates the luminance value by comparing the pixel signal read out by the first image sensor 13a using the NDR function with the pixel signal read out in the frame before the execution of the second operation. In this example, the frame before the execution of the second operation is frame N. The pixel signal read out in frame N is a pixel signal obtained by reading out the charge accumulated in the pixel due to the exposure of frame N-1, and is therefore a pixel signal acquired based on the exposure of frame N-1. Accordingly, the control unit 15 calculates the luminance value by comparing the pixel signal read out by the first image sensor 13a using the NDR function with the pixel signal read out in frame N, which was acquired based on the exposure of frame N-1. Specifically, the control unit 15 calculates the luminance value caused by the event by taking the difference between the luminance value of the pixel signal read out by the NDR function and the pixel value (integrated light received value of each pixel) of the pixel signal read out in frame N. The control unit 15 then outputs the luminance value calculated in this way.

[0038] The control unit 15 can output video using the output luminance values. For example, the control unit 15 corrects the video using the luminance values, and displays the corrected video, that is, the video that takes into account the effects caused by the event, on a display screen (not shown). Note that video output is performed each time a luminance value is output, rather than by raster scanning.

[0039] The same processing is performed if an event occurs between frames N+2. For example, suppose an event occurs between time T11 and time T12 in frame N+2, as shown in Figure 2. In this case, when the event sensor unit 12 detects the event, the control unit 15 outputs a brightness value based on the event, based on the pixel signal read out by the NDR function from the image sensor (i.e., the second image sensor 13b) that is performing the second operation when the event occurred (i.e., when the event sensor unit 12 outputs the event signal). The specific processing is the same as the processing described above, so a detailed explanation is omitted here.

[0040] Thus, in this embodiment, when an event occurs, the control unit 15 reads out the pixel signal using the NDR function and outputs a brightness value based on the event based on the pixel signal. Therefore, the control unit 15 can output a brightness value based on the event in response to the occurrence of an event. Consequently, the imaging device 1 can detect changes in brightness with lower latency. For example, the control unit 15 can reflect changes in the brightness of light based on an event without waiting for the next frame.

[0041] Furthermore, in both the first and second operation patterns, the pixel signals are read out by the NDR function using the first image sensor 13a and the second image sensor 13b, respectively. Therefore, no matter when an event occurs, the control unit 15 can output a brightness value based on the event. For example, while the image sensor performing the first operation is resetting, no exposure occurs. Therefore, even if an event occurs during this time, the change in light brightness based on the event cannot be captured. However, even while the image sensor performing the first operation is resetting, the other image sensor performing the second operation can capture the change in light brightness based on the event. Therefore, no matter when an event occurs, the change in light brightness based on the event can be captured.

[0042] In the above embodiment, the case in which the control unit 15 determines that an event has occurred based on the output of an event signal was described. However, the control unit 15 does not necessarily have to determine that an event has occurred based on an event signal. For example, the control unit 15 may estimate the occurrence of an event from all frames and, as a result of the estimation, determine that an event has occurred, thereby executing the process described in the above embodiment. For example, the control unit 15 can estimate the occurrence of an event from the difference in exposure between the two previous frames and the previous frame. Specifically, the control unit 15 can estimate that an event has occurred if the event is continuing from the previous frame and is predicted to continue occurring. Alternatively, the control unit 15 can estimate that an event has occurred if the probability of the event occurring is statistically high. If the probability of the event occurring is statistically high, it can be estimated from past frames, for example, a road merging section, but is not limited to this. Alternatively, the control unit 15 can estimate that an event has occurred if the probability of the event occurring is high using object recognition technology based on image analysis. For example, if the control unit 15 recognizes a person's face using object recognition technology, it can estimate that there is a high probability of an event occurring for that person's face and thus estimate that an event has occurred.

[0043] <Identifying the location of the event> In the processing when the above event occurs, the control unit 15 may identify the position on the image plane where the event-based light is detected (in this embodiment, also called the "event position") on the image plane captured by the first image sensor 13a and the second image sensor 13b. In this case, the control unit 15 identifies the event position based on the event signals output from the two event sensors, namely the first event sensor 12a and the second event sensor 12b. Specifically, the control unit 15 identifies the position where the event occurred using epipolar constraints based on the coordinates indicated by the event signals output from the first event sensor 12a and the second event sensor 12b, and identifies which position (coordinates) on the image plane captured by the first image sensor 13a and the second image sensor 13b corresponds to the position where the event occurred. This identifies the event position on the image plane of the first image sensor 13a and the second image sensor 13b.

[0044] When the control unit 15 identifies the event location as described above, it outputs a brightness value based on the pixel signal read out by the NDR function from the pixel located at the event location in the image sensor that is performing the second operation when the event occurs. In this case, the control unit 15 outputs the brightness value based on the pixel signal read out from the pixel located at the event location, rather than from all pixels of the image sensor that is performing the second operation.

[0045] For example, in frame N+1 of Figure 2, the first image sensor 13a is performing the second operation. When an event occurs in frame N+1, the control unit 15 identifies the event location on the image plane of the first image sensor 13a based on the event signals output from the first event sensor 12a and the second event sensor 12b. The control unit 15 reads out pixel signals from the pixels located at the identified event location on the first image sensor 13a using the NDR function. The control unit 15 outputs a brightness value based on the readout pixel signals. In this case, since the pixel signals read out by the NDR function are limited to signals from pixels located at the event location, the readout range is narrowed compared to when all pixels are read out. As a result, the imaging device 1 can shorten the readout time and detect changes in brightness with lower latency.

[0046] <Example of processing by the control unit> Next, an example of a process performed by the control unit 15 will be described. Figure 3 is a flowchart showing an example of a process performed by the control unit 15 in Figure 1.

[0047] The control unit 15 causes the first image sensor 13a and the second image sensor 13b to alternately execute the first operation pattern and the second operation pattern in each frame (step S11). At this time, the control unit 15 determines whether or not an event has occurred (step S12). For example, the control unit 15 determines whether or not an event signal has been output from the event sensor unit 12.

[0048] If the control unit 15 determines that no event has occurred, that is, if it determines that no event signal has been output from the event sensor unit 12 (No. in step S12), it outputs (displays) an image based on the pixel signals read out by the execution of the first operation pattern and the second operation pattern (step S16).

[0049] On the other hand, if the control unit 15 determines that an event has occurred, that is, if it determines that an event signal has been output from the event sensor unit 12 (Yes in step S12), it identifies the event location based on the event signal (step S13). Then, the control unit 15 reads out the pixel signal from the pixel located at the event location using the NDR function (step S14).

[0050] The control unit 15 uses the read pixel signal to calculate and output the brightness value caused by the event (step S15). Then, the control unit 15 uses the output brightness value to correct the image and outputs the corrected image (step S16).

[0051] As described above in detail, this embodiment will be readily apparent to those skilled in the art, and many modifications are possible without substantially departing from the novelty and effects of this embodiment. Therefore, all such modifications are included within the scope of this disclosure. For example, any term that appears at least once in the specification or drawings alongside a broader or synonymous term may be replaced with that different term anywhere in the specification or drawings. Furthermore, all combinations of this embodiment and its modifications are also included within the scope of this disclosure. The configuration and operation of the decoding device are also not limited to those described in this embodiment, and various modifications are possible. [Explanation of Symbols]

[0052] 1…Imaging device, 11…Imaging optical system, 11a…First imaging optical system, 11b…Second imaging optical system, 11c…Third imaging optical system, 12…Event sensor unit, 12a…First event sensor, 12b…Second event sensor, 13a…First image sensor, 13b…Second image sensor, 14…Half mirror, 15…Control unit, 16…Storage unit

Claims

1. A first image sensor and a second image sensor are arranged on the same optical axis, receive light during exposure, and have an NDR (Non-Destructive Readout) function that allows for non-destructive reading of the pixel signal based on the received light during the exposure. A control unit alternately executes a first operation pattern in each frame, which causes the first image sensor to perform a first operation including resetting the pixel charge and exposure, and the second image sensor to perform a second operation including exposure but without resetting the pixel charge, and a second operation pattern in which the first image sensor performs the second operation and the second image sensor performs the first operation. Equipped with, Of the first and second image sensors, the image sensor performing the second operation when an event causing a change in brightness occurs outputs a brightness value based on the event based on the pixel signal read out by the NDR function. Imaging device.

2. The system further includes an event sensor unit that detects changes in the brightness of each pixel based on the aforementioned event and outputs an event signal indicating the coordinates of the pixels whose brightness has changed. When the event sensor unit outputs the event signal, the image sensor performing the second operation outputs a brightness value based on the event based on the pixel signal read by the NDR function. The imaging apparatus according to claim 1.

3. The event sensor unit comprises two event sensors, each outputting an event signal based on the event captured from different positions within a common field of view. Based on the event signals output from the two event sensors, the control unit identifies the event location on the image plane captured by the first and second image sensors where light based on the event is detected. When the event signal is output by the event sensor unit, the second operation is performed on the image sensor, and the brightness value is output based on the pixel signal read out by the NDR function from the pixel located at the event position. The imaging apparatus according to claim 2.

4. The imaging device according to claim 1, wherein the image sensor performing the second operation when the event causing the change in brightness occurs calculates the brightness value by comparing the pixel signal read out by the NDR function with the pixel signal read out in the frame before the second operation was performed.

5. A first image sensor and a second image sensor are arranged on the same optical axis, receive light during exposure, and have an NDR (Non-Destructive Readout) function that allows for non-destructive reading of the pixel signal based on the received light during the exposure. A control unit that controls the operation of the first image sensor and the second image sensor, An information processing method performed by an imaging device comprising: In each frame, a first operation pattern is performed, in which the first image sensor performs a first operation including resetting the pixel charge and exposure, and the second image sensor performs a second operation including exposure but without resetting the pixel charge; and a second operation pattern is performed, in which the first image sensor performs the second operation and the second image sensor performs the first operation. Of the first and second image sensors, the image sensor performing the second operation when an event causing a change in brightness occurs outputs a brightness value based on the event based on the pixel signal read out by the NDR function. Information processing methods.

Images