Imaging device, control method, and program

Abstract

To provide an imaging device, etc., that allows exposure adjustment to improve the ability to track changes in brightness and significantly reduce overexposure, underexposure, etc. [Solution] The imaging device 100 comprises an imaging unit 101 having an image sensor in which pixels for capturing an image and specific pixels that output an address event signal indicating the occurrence of a brightness change and the location of this occurrence are arranged in a matrix on the same plane, and the exposure can be controlled by analyzing the image captured by the imaging unit 101. It is characterized by comprising an address event acquisition unit 105 that acquires an address event signal, an address event integration unit 106 that generates integration information by integrating the acquired address event signals, a slope calculation unit 107 that generates address event slope information indicating the slope of the change in the address event signal based on the generated integration information, and an exposure control unit 104 that controls the imaging unit with the exposure determined based on the generated address event slope information.

Description

Technical Field

[0001] The present invention relates to an imaging device, a control method, and a program.

Background Art

[0002] Conventionally, an imaging element that captures an image in synchronization with a synchronization signal such as a vertical synchronization signal is used in an imaging device. A general imaging element outputs image data in the next frame after being exposed and accumulating charges. Therefore, there is a delay of one frame in the image data output by the imaging element with respect to the movement of the subject. Further, when performing exposure control by analyzing the image data, there is an additional delay of one frame in order to output the image data in which the exposure control is reflected. Eventually, when performing exposure control from the image output from the imaging element, it is two frames after imaging until an image in which the exposure setting is actually reflected is output.

[0003] In this regard, Patent Document 1 proposes a method of capturing frames with long / short exposure times, analyzing the image data of both long / short exposure times, and performing exposure control. Further, Patent Document 2 discloses an event-based sensor that outputs, in real time, changes in luminance for each pixel as address event signals.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, when performing exposure control by analyzing image data, it was not possible to track brightness changes of two frames or less due to the output delay of the image data. For example, when performing exposure control using the conventional technology disclosed in Patent Document 1, there was a limitation on the exposure time of the frame used as the recorded image. Because of this limitation on the exposure time of the frame used as the recorded image, the ability to track brightness changes was poor, and even after adjusting the exposure, many cases of overexposure and underexposure occurred. Furthermore, although the conventional technology disclosed in Patent Document 2 discloses an event-based sensor, it detects the subject and does not perform exposure control using the event-based sensor.

[0006] The present invention aims to provide an imaging device, control method, and program that can adjust exposure to improve responsiveness to changes in brightness and significantly reduce overexposure, underexposure, and other issues. [Means for solving the problem]

[0007] To achieve the above objective, one embodiment of the imaging device according to the present invention comprises an imaging unit having an image sensor in which pixels for capturing an image and specific pixels that output an address event signal indicating the occurrence of a change in brightness and the location of such change are arranged in a matrix on the same plane, and an imaging device capable of controlling exposure by analyzing an image captured by the imaging unit, comprising: an acquisition unit for acquiring the address event signal; an integration unit for generating integrated information by integrating the address event signal acquired by the acquisition unit; a slope calculation unit for generating slope information indicating the slope of the change in the address event signal based on the integrated information generated by the integration unit; and an exposure control unit for controlling the imaging unit with exposure determined based on the slope information generated by the slope calculation unit. [Effects of the Invention]

[0008] According to the present invention, exposure adjustment becomes possible in which the ability to track changes in brightness is improved and blown-out highlights, crushed blacks, etc., is significantly reduced. [Brief explanation of the drawing]

[0009] [Figure 1] This is a block diagram of an imaging device according to an embodiment of the present invention. [Figure 2] This flowchart shows the overall process according to an embodiment of the present invention. [Figure 3] This is a flowchart showing the address event signal integration process according to an embodiment of the present invention. [Figure 4] This flowchart shows the exposure control process according to an embodiment of the present invention. [Figure 5] This flowchart shows the operation of the exposure control post-processing according to an embodiment of the present invention. [Figure 6] This is a timing chart showing the results when exposure is controlled using only image analysis. [Figure 7] This is a timing chart showing the operation when exposure is controlled using address event tilt information according to an embodiment of the present invention. [Figure 8] This is a diagram showing the configuration of the imaging unit 101 according to an embodiment of the present invention. [Modes for carrying out the invention]

[0010] Embodiments of the present invention will be described in detail below with reference to the drawings. However, the configurations described in the following embodiments are merely illustrative, and the scope of the present invention is not limited to the configurations described in the embodiments.

[0011] (Summary of the invention) An I-pixel (see Figure 8) is used that outputs an address event signal (+1) indicating the occurrence of a brightness change due to brightness increase when the brightness of the subject increases, and an address event signal (-1) indicating the occurrence of a brightness change due to brightness decrease when the brightness of the subject decreases. The I-pixel is implemented using an EVS (event-based vision sensor) as an example. The image sensor has the I-pixel and the pixels that acquire the image arranged in a matrix on the same plane. The address event acquisition unit 105 acquires address event signals, and the address event integration unit 106 integrates them to generate "integration information (address event integration information)". Then, the tilt calculation unit 107 generates "tilt information (address event tilt information)" that indicates the tilt of the change in the address event signal based on the integration information.

[0012] Tilt information indicates a change in brightness, and the exposure control unit 104 performs exposure control based on the exposure determined by the tilt information. Specifically, when the tilt indicated by the tilt information is positive and constant, the exposure control unit 104 determines that the brightness of the subject has become high and shortens the exposure period or controls the aperture to close it. On the other hand, when the tilt indicated by the "tilt information" is negative and constant, the exposure control unit 104 determines that the brightness of the subject has become low and controls the exposure period to lengthen or opens the aperture.

[0013] More specifically, the "address event signal" is defined as a signal indicating the "direction," "position," and "time" in which the brightness change occurred at the pixel where the brightness change occurred. However, in this invention, it can be understood as a signal indicating at least the occurrence of the brightness change and the location of this occurrence. Furthermore, in this specification, "address" refers to the "position (coordinates) of the pixel where the brightness change occurred," that is, the "coordinates of the I pixel (see Figure 8) where the brightness change occurred." Also, for example, the system control unit 108, which will be described later, keeps track of the address of each I pixel.

[0014] (Figure 1: Configuration of imaging device 100) FIG. 1 is a block configuration diagram of imaging device 100. Imaging device 100 shown in FIG. 1 can be applied to various uses. For example, it can be applied to smartphones, head-mounted displays, industrial cameras, mobile robots, self-driving vehicles, etc., but the applicable range is not limited to these.

[0015] Referring to FIG. 1, the configuration of imaging device 100 will be described. Imaging device 100 includes imaging unit 101, image acquisition unit 102, exposure determination unit 103, exposure control unit 104, address event acquisition unit 105, address event integration unit 106, tilt calculation unit 107, system control unit 108, and image recording unit 109. Imaging device 100 further includes lens unit 111 and user interface unit 112. These components constituting imaging device 100 are communicably connected to each other via communication bus 110 so as to communicate necessary information.

[0016] (Imaging unit 101: Image acquisition unit 102) Imaging unit 101 is an imaging device provided on the same surface with a pixel array arranged in a matrix for imaging an image, and a pixel array that outputs an “address event signal” indicating the direction, position, and time at which the luminance change has occurred in a pixel where the luminance change has occurred. Details of the configuration of the imaging device will be described later with reference to FIG. 8. Image acquisition unit 102 acquires an image from imaging unit 101 and outputs the data of the acquired image to exposure determination unit 103 and image recording unit 109.

[0017] (Address event acquisition unit 105: Address event integration unit 106) Address event acquisition unit 105 (acquisition unit) acquires an address event signal from imaging unit 101 and outputs it to address event integration unit 106. Address event integration unit 106 (integration unit) generates address event integration information based on the address event signal acquired by address event acquisition unit 105 and outputs it to tilt calculation unit 107. For example, when the luminance change direction of the address event signal is the increasing direction, “1” is added to the address event integration information, while when the luminance change direction of the address event signal is the decreasing direction, “1” is subtracted.

[0018] Also, when the address event integration unit 106 is in the following situation, it weights the numerical value to be added or subtracted. That is, when it is determined that the address corresponding to the address event signal is included in the "range" instructed by the user based on the instruction received from the user via the user interface unit 112. Here, the "range" is a range of certain coordinates of the pixel array of the imaging unit 101, and refers to the conversion of the specific area on the screen instructed by the user into the coordinates of the pixel array in the imaging device. In the present embodiment, as an aspect of the "range", the "range" in which the user instructs to adjust the exposure will be taken as an example for explanation.

[0019] Also, when the address event integration unit 106 continuously outputs address event signals in which pixels of the same address have luminance changes in the same direction, it can also weight the numerical value to be added or subtracted. The weighting performed by the address event integration unit 106 is realized, for example, by increasing the numerical value to be added or subtracted more than usual.

[0020] Also, the address event integration unit 106 outputs the integrated address event integration information to the slope calculation unit 107 every unit time. This unit time is preferably shorter than the period in which the imaging unit 101 captures an image. Also, the address event integration unit 106 can cancel the weighting of the address event signal based on the instruction of the exposure determination unit 103. Also, the address event integration unit 106 can change the address event integration information to a predetermined value based on the instruction of the exposure determination unit 103. An example of changing the address event integration information to a predetermined value by the address event integration unit 106 is to change the address event integration information to the initial state (initial value).

[0021] (Slope calculation unit 107: Address event slope information) The tilt calculation unit 107 generates address event tilt information (tilt information) that indicates the tilt of the change in the address event signal, based on the address event integration information generated by the address event integration unit 106. More specifically, the tilt calculation unit 107 calculates address event tilt information that indicates the tilt of the direction of the brightness change in the address event signal and the number of events, based on the address event integration information output by the address event integration unit 106. Furthermore, as mentioned above, in the address event integration information, "1" is added when the brightness of the address event signal has increased, and "1" is subtracted when the brightness of the address event signal has decreased, so the following can be said. In other words, the address event tilt information can be obtained by calculating the amount of change in the address event integration information per "unit time". Furthermore, it is preferable that the "unit time" used for the address event tilt information is shorter than the period in which the imaging unit 101 captures an image, similar to the unit time used by the address event integration unit 106.

[0022] (Exposure determining section 103) The exposure determination unit 103 determines the exposure of the image captured by the imaging unit 101. The exposure determination method determined by the exposure determination unit 103 is determined by factors such as the charge accumulation period of the imaging unit 101 and the amount the aperture of the lens unit 111 is opened. The exposure determination unit 103 also analyzes the image output by the imaging unit 101 and determines the exposure so that the average brightness value and histogram reach target values. Furthermore, the exposure determination unit 103 determines the exposure according to the magnitude of the slope indicated by the address event slope information, which shows the change in brightness. For example, the exposure determination unit 103 determines the exposure so that the address event slope information output by the slope calculation unit 107 reaches the "target value". The exposure determination unit 103 can switch between determining the exposure based on the results of analyzing the image captured by the imaging unit 101 and determining the exposure based on the address event slope information.

[0023] The exposure determination unit 103 outputs the determined exposure to the exposure control unit 104. The exposure determination unit 103 also instructs the address event integration unit 106 to eliminate the weighting of the address event signals at a predetermined timing. One example of this predetermined timing is when the user changes the exposure.

[0024] (Exposure control unit 104: System control unit 108) The exposure control unit 104 controls the charge accumulation time of the imaging unit 101 and the aperture opening amount of the lens unit 111 based on the exposure determined by the exposure determination unit 103. The system control unit 108 controls the entire imaging device 100 via the communication bus 110. The processor (CPU, DSP, etc.) executes a program stored in a non-volatile memory (not shown), thereby executing each of the processes according to the embodiment of the present invention, which will be described later. In other words, the various functions of the imaging device 100 according to the embodiment of the present invention are realized.

[0025] (Image recording unit 109: Lens unit 111: Aperture; User interface unit 112) The image recording unit 109 is implemented using a recording medium such as a memory card and records images acquired by the image acquisition unit 102. The lens unit 111 consists of a focus / zoom lens group and the like, and constitutes an optical system that guides the optical image of the subject to the imaging unit 101. The lens unit 111 also has an aperture, and the amount of light incident on the imaging unit 101 can be controlled by controlling the aperture opening. The user interface unit 112 is implemented using switches, buttons, a touch panel, etc. that accept user operations, and receives the range of exposure to be adjusted instructed by the user and outputs this to the address event accumulator 106.

[0026] (Figure 2: Overall process) Next, an example of the overall processing of the imaging device 100 will be described with reference to Figure 2. The imaging device 100 starts the processing shown in Figure 2, for example, when the power is turned on. Alternatively, the imaging device 100 may be configured to start processing in response to instructions from the user.

[0027] (Step S201) First, in step S201, the system control unit 108 determines whether it has received an instruction from the user via the user interface unit 112 regarding the exposure range. If the system control unit 108 determines that it has received an instruction, it sets the exposure range for the address event accumulator 106 and proceeds to step S202.

[0028] (Step S202; Step S203) In step S202, the system control unit 108 controls the imaging unit 101 to capture an image and outputs it to the image acquisition unit 102. The image acquisition unit 102 outputs the acquired image to the exposure determination unit 103 and proceeds to step S203. In step S203, the system control unit 108 performs address event signal integration processing and proceeds to step S204. The "address event signal integration processing" in step S203 will be explained later with reference to Figure 3.

[0029] (Step S204) In step S204, the system control unit 108 instructs the tilt calculation unit 107 to generate address event tilt information. In response to this instruction, the tilt calculation unit 107 generates address event tilt information based on the address event integration information output by the address event integration unit 106, and then proceeds to step S205.

[0030] (Step S205: Step S206) In step S205, the system control unit 108 performs exposure control processing and proceeds to step S206. The "exposure control processing" in step S205 will be explained later with reference to Figure 4. In step S206, the system control unit 108 performs exposure control post-processing and proceeds to step S207. The "exposure control post-processing" in step S206 will be explained later with reference to Figure 5.

[0031] (Step S207) Then, in step S207, the system control unit 108 determines whether or not to continue imaging. The determination process of whether or not to continue imaging, performed by the system control unit 108, is executed, for example, in response to instructions from the user. Specifically, if the user determines to continue imaging (Yes), the system control unit 108 proceeds to step S202, while if it determines not to continue imaging (No), it terminates the process shown in the example in Figure 2.

[0032] (Figure 3: Address event signal integration processing: Step S301) Next, an example of the address event signal integration process (S203) will be explained with reference to Figure 3. First, in step S301, the system control unit 108 determines whether the address event acquisition unit 105 has acquired an address event signal and output it to the address event integration unit 106. If the system control unit 108 determines that an address event signal has been output (Yes), it proceeds to S302; otherwise, it proceeds to S301.

[0033] (Step S302: Step S303) In step S302, the system control unit 108 determines whether or not a range for adjusting exposure has been set by the user in step S201. If the system control unit 108 determines that a range has been set (in other words, if there is an address instruction from the user: Yes), it proceeds to step S303; otherwise, it proceeds to step S304. In step S303, the system control unit 108 weights the address event signals that were determined to be included in the exposure adjustment range in step S302. In other words, the system control unit 108 weights the address event signals for which the user has given an exposure adjustment instruction.

[0034] (Step S304) In step S304, the system control unit 108 checks the address of the address event signal and determines whether it is an address event signal for an address where the brightness has increased continuously over a predetermined period. If the system control unit 108 determines that it is an address event signal for an address where the brightness has increased continuously over a predetermined period (Yes), it proceeds to S305; otherwise, it proceeds to S306. The predetermined period in step S304 refers, for example, to the period from the last exposure control to the acquisition of the address event signal. Also, in step S304, "continuously increasing brightness" means that multiple address event signals for the same address always show an increase in brightness. Alternatively, it may also mean that the brightness increases on average.

[0035] (Step S305) In step S305, the system control unit 108 assigns weights to address event signals that were determined in step S304 to be addresses whose brightness has been continuously increasing over a predetermined period.

[0036] (Step S306) In step S306, the system control unit 108 checks the address of the address event signal and determines whether it is an address event signal for an address where the brightness has decreased continuously over a predetermined period. If the system control unit 108 determines that it is an address event signal for an address where the brightness has increased continuously over a predetermined period (Yes), it proceeds to S307; otherwise, it proceeds to S308. The predetermined period in step S306 refers, for example, to the period from the last exposure control to the acquisition of the address event signal. Also, in step S306, "continuous decrease in brightness" means that multiple address event signals for the same address always show a decrease in brightness. Alternatively, it may also mean that the brightness decreases on average.

[0037] (Step S307) In step S307, the system control unit 108 assigns weights to address event signals that were determined in step S306 to be addresses whose brightness has been continuously decreasing over a predetermined period.

[0038] (Step S308: Step S309) In step S308, the system control unit 108 accumulates the address event signals and proceeds to step S309. Specifically, the system control unit 108 instructs the address event accumulation unit 106 to accumulate the address event signals. In step S309, the system control unit 108 determines whether a unit of time has elapsed. If it determines that the unit of time has elapsed (Yes), it proceeds to step S310; otherwise, it proceeds to step S301. The unit of time in step S309 is the same as the unit of time described in the explanation of the address event accumulation unit 106.

[0039] (Step S310) In step S310, the system control unit 108 issues an instruction to the address event accumulation unit 106 to generate address event accumulation information and an instruction to the slope calculation unit 107 to output address event accumulation information, and then proceeds to step S311.

[0040] (Step S311) In step S311, the system control unit 108 determines whether the generated address event integration information is in a state where the slope can be calculated by the slope calculation unit 107. If the system control unit 108 determines that the slope can be calculated (Yes), it terminates the address event signal integration process. If it determines that the slope cannot be calculated (No), it proceeds to step S301. In step S311, one example of a state in which the slope calculation unit 107 can calculate the slope is when multiple address event integration information is output. If the system control unit 108 determines "Yes" in step S311, it calculates the address event slope information based on the address event integration information output by the address event integration unit 106.

[0041] Furthermore, if the address event integration unit 106 includes an address event signal from a specific pixel (see 805) in the address event integration information, it can also weight the address event integration information and generate the address event integration information as described below. That is, the address event integration unit 106 generates the address event integration information such that the slope indicated by the address event slope information becomes larger, or generates the address event integration information such that the slope becomes smaller.

[0042] (Figure 4: Exposure control process: Step S401) Next, the exposure control process (S205) will be explained with reference to Figure 4. First, in step S401, the system control unit 108 determines whether the slope indicated by the address event slope information is greater than a reference value. If it is determined to be greater than the reference value (Yes), the process proceeds to step S402; otherwise, it proceeds to step S405. One form of the reference value in step S401 is a predetermined value, but a value set by the user may also be used.

[0043] (Step S402) In step S402, the system control unit 108 determines the direction of the tilt indicated by the address event tilt information calculated and generated in step S204. If the system control unit 108 determines that the "tilt" indicates an increase in brightness, it proceeds to step S403; however, if it determines that the "tilt" indicates a decrease in brightness, it proceeds to step S404.

[0044] (Step S403: Increasing slope direction) In step S403, since the tilt indicates an increase in brightness, the system control unit 108 instructs the exposure determination unit 103 to reduce the exposure according to the magnitude of the tilt indicated by the address event tilt information. In response, the exposure determination unit 103 determines the exposure according to the magnitude of the tilt and instructs the exposure control unit 104 to reduce the exposure. The exposure control unit 104 instructs the imaging unit 101 to shorten the charge accumulation time in the image sensor, or instructs the lens unit 111 to stop down the aperture. After that, the exposure control process (S205) is terminated.

[0045] (Step S404: Direction of decreasing slope) In step S404, since the tilt indicates a decrease in brightness, the system control unit 108 instructs the exposure determination unit 103 to increase the exposure according to the magnitude of the tilt indicated by the "address event tilt information". In response, the exposure determination unit 103 determines the exposure according to the magnitude of the tilt and instructs the exposure control unit 104 to increase the exposure. The exposure control unit 104 instructs the imaging unit 101 to lengthen the charge accumulation time of the image sensor, or instructs the lens unit 111 to open the aperture. After that, the exposure control process (S205) is terminated.

[0046] (Step S405) In step S405, the system control unit 108 determines whether the slope indicated by the "address event slope information" is continuously smaller than a reference value for a predetermined period (a predetermined number of frames). If it is determined to be continuously smaller than the reference value (Yes), the system proceeds to step S406; otherwise, it proceeds to step S407.

[0047] (Step S406; Step S407) In step S406, the system control unit 108 instructs the exposure determination unit 103 to analyze the image output by the image acquisition unit 102 and determine the exposure. The exposure determination unit 103 analyzes the image and determines the exposure, and instructs the exposure control unit 104 to control the imaging unit 101 or the lens unit 111 based on the determined exposure. After that, the exposure control process S205 is terminated. In step S407, the system control unit 108 instructs the exposure determination unit 103 not to change the exposure, and terminates the exposure control process (S205).

[0048] (Figure 5: Exposure control post-processing: Step S501) Next, the exposure control post-processing (S206) will be explained with reference to Figure 5. First, in step S501, the system control unit 108 determines whether or not exposure control has been performed based on the tilt indicated by the address event tilt information. If the system control unit 108 determines that exposure control has been performed (Yes), it proceeds to S502, while if it determines that exposure control has not been performed (No), it terminates the exposure control post-processing (S206).

[0049] (Step S502: Step S503) In step S502, the system control unit 108 eliminates the weighting of the address event signals whose brightness has changed continuously and proceeds to step S503. That is, the system control unit 108 eliminates the weighting if it determines that the exposure has been controlled based on the address event tilt information. In step S503, the system control unit 108 changes the address event cumulative information to a predetermined value and terminates the exposure control post-processing (S206). An example of the predetermined value in step S503 is a pre-set initial value.

[0050] (Figure 6: Exposure control using image analysis only) Next, referring to Figure 6, we will explain a conventional timing chart for when the exposure determination unit 103 determines the exposure using only the results of image analysis. In Figure 6, the vertical direction shows, from top to bottom, "subject, frame period, charge accumulation period, imaging time, and image accumulation period used for exposure determination," and the horizontal direction shows the passage of time. As shown in "t601," the brightness of the subject changes rapidly. Figure 6 shows the case where the brightness of the subject increases, but even if the brightness of the subject decreases, the same operation is performed, only the direction of brightness change is different. "Frame period" is the imaging period of the imaging unit 101, for example, a period of "1 / 60 (seconds)."

[0051] "t602" indicates the relationship between "charge accumulation period" and "captured image". The diagonally downward arrow pointing from "charge accumulation period" to "captured image" indicates the relationship between "charge accumulation period" and "captured image". The "charge accumulation period" is the period during which the imaging unit 101 can accumulate charge when taking an image. For example, the imaging unit 101 outputs the charge accumulated during the charge accumulation time "1" as the captured image of the second frame. The image that the image acquisition unit 102 can acquire is an image based on the charge accumulated by the imaging unit 101 during the period of the previous frame.

[0052] "t603" indicates the relationship between the timing at which the exposure determination unit 103 analyzes the image, determines the exposure, and issues an exposure control instruction to the exposure control unit 104, and the "charge accumulation period" in which this is reflected. The diagonal upward arrow pointing from the "accumulation period of the image used for exposure determination" to the "charge accumulation period" shows the relationship between the "accumulation period of the image used for exposure determination" and the "charge accumulation period". The "charge accumulation period" of the image used by the exposure determination unit 103 for analysis corresponds to the previous frame, and exposure control is applied to the next frame.

[0053] For example, "t604" is an image output by the imaging unit 101 based on the charge accumulation period "4," and the exposure for "t604" is determined based on the image output by the imaging unit 101 based on the charge accumulation period "2." Also, because the subject in charge accumulation period "2" is low brightness, the exposure determination unit 103 determines the exposure so that the brightness of the image is high. However, as can be seen by referring to "t601," the subject in charge accumulation period "4" is high brightness, so the image of "t604" is overexposed.

[0054] "t605" is the image output by the imaging unit 101 based on the charge accumulation period "5". The exposure for "t605" is determined based on the image output by the imaging unit 101 based on the charge accumulation period "3". Since the subject in charge accumulation period "3" is high brightness, the exposure determination unit 103 determines the exposure so that the brightness of the image is low. However, as can be seen by referring to 601, the subject in charge accumulation period "5" is low brightness, so the image of "t605" is underexposed.

[0055] "t606" is the image output by the imaging unit 101 based on the charge accumulation period "6". "t605" is the image whose exposure is determined based on the image output by the imaging unit 101 based on the charge accumulation period "4". Since the subject in charge accumulation period "4" is high brightness, the exposure determination unit 103 determines the exposure so that the brightness of the image is low. However, as can be seen by referring to "t601", the subject in charge accumulation period "6" is low brightness, so the image in "t605" is underexposed, similar to "t605". Thus, it can be seen that with conventional methods, when exposure adjustment is performed based on the image output from the image sensor, there are problems such as overexposure and underexposure occurring over multiple frames when brightness changes occur.

[0056] (Figure 7: When exposure is controlled using address event tilt information) Next, referring to Figure 7, we will explain the timing chart when the exposure determination unit 103 determines the exposure using "address event tilt information". The explanation for "t601" is omitted as it is the same as in Figure 6. Note that, as in Figure 6, the vertical direction shows the subject, frame period, charge accumulation period, etc., but "address event integration information" is also newly shown. The horizontal direction in Figure 7 also represents the passage of time, but the change in "address event integration information" with respect to the time change in brightness is also shown.

[0057] "t701" indicates that the system control unit 108 has determined that the slope (upward to the right) indicated by the "address event slope information" for charge accumulation period "3" is greater than a predetermined value. In this case, the system control unit 108 instructs the exposure determination unit 103 to reduce the exposure according to the magnitude of the slope indicated by the "address event slope information" for charge accumulation period "3". In response, the exposure determination unit 103 determines the exposure according to the magnitude of the slope and instructs the exposure control unit 104 to reduce the exposure (t701). With the exposure adjusted in this way, the captured image of the fifth frame corresponding to charge accumulation period "4" (see diagonally downward arrow) is output.

[0058] The period during which the "address event accumulation information" is accumulated is shorter than the period during which the imaging unit 101 captures an image. As a result, in "t701", the exposure determination unit 103 can determine the exposure corresponding to the subject's condition during the charge accumulation period "3" before the imaging unit 101 outputs an image based on the charge accumulation period "3".

[0059] "t702" shows the image output by the imaging unit 101 based on the charge accumulation period "4". The exposure for "t702" is determined based on the "address event tilt information" from the charge accumulation period "3". This allows the system to track the rapid brightness changes in "t601" and output an image with appropriate brightness.

[0060] "t703" indicates that the system control unit 108 has determined that the slope (downward to the right) indicated by the "address event slope information" for charge accumulation period "5" is smaller than a predetermined value. In this case, the system control unit 108 instructs the exposure determination unit 103 to increase the exposure according to the magnitude of the slope indicated by the "address event slope information" for charge accumulation period "5". In response, the exposure determination unit 103 determines the exposure according to the magnitude of the slope and instructs the exposure control unit 104 to increase the exposure (t703). With the exposure adjusted in this way, the captured image of the 7th frame corresponding to charge accumulation period "6" (see diagonally downward arrow) is output.

[0061] The period during which address event accumulation information is accumulated is shorter than the period during which the imaging unit 101 captures an image. This allows the exposure determination unit 103 to determine the exposure corresponding to the subject's condition during the charge accumulation period "5" before the imaging unit 101 outputs an image based on the charge accumulation period "5".

[0062] "t704" is the image output by the imaging unit 101 based on the charge accumulation period "6". The exposure for "t704" is determined based on the "address event tilt information" from the charge accumulation period "5". This allows the output of an image with appropriate brightness in response to the rapid brightness changes in "t601".

[0063] "t705" is an image captured by the system control unit 108, which determines the exposure by analyzing the image because the slope indicating "address event slope information" is smaller than the reference value for a predetermined number of frames.

[0064] (Figure 8: Configuration diagram of the imaging unit 101) Next, an example of the configuration of the imaging unit 101 will be described with reference to Figure 8. The imaging unit 101 is an image sensor, such as a CMOS sensor, and comprises a peripheral circuit unit 802 and a pixel unit 803. The peripheral circuit unit 802 includes a drive circuit for the pixel unit 803, a circuit that outputs an image to the image acquisition unit 102, and a circuit that outputs an "address event signal" to the address event acquisition unit 105.

[0065] The pixel unit 803 has pixels for capturing images and pixels that output an "address event signal" indicating the direction, position, and time of the brightness change of a pixel where a brightness change has occurred, arranged in a matrix on the same plane. The RGB pixels 804 are used to capture images and are covered, for example, with RGB color filters to output the image to the peripheral circuit unit 802. For ease of understanding, the code "804" is not assigned to all "RGB pixels" in Figure 8.

[0066] Furthermore, the I-pixel 805 (specific pixel) is a pixel that outputs an "address event signal," and outputs an "address event signal" to the peripheral circuit section 802 that indicates the direction, position, and time of the brightness change of the pixel where a brightness change has occurred. Also, Figure 8 is merely an example of the arrangement of RGB pixels 804 and I-pixel 805. For example, if we assume that four pixels constitute one set, the configuration may include both RGB pixels 804 and I-pixel 805 in all sets, or, as shown in Figure 8, the configuration may not necessarily include I-pixel 805 in all sets. Thus, various arrangements of I-pixel 805 can be adopted.

[0067] As described above, the imaging device 100 according to the embodiment of the present invention can sufficiently track brightness changes of two frames or less by determining the exposure based on the slope of the brightness change indicated by the address event slope information.

[0068] Furthermore, the exposure determination unit 103 may be configured to switch between determining exposure by analyzing the acquired image in accordance with the meeting of predetermined criteria, or determining exposure according to the magnitude of the tilt indicated by the address event tilt information. The predetermined criteria are, for example, that at least one of the "number of frames of the image" captured by the imaging unit 101 and the "tilt" indicated by the address event tilt information is smaller than the corresponding reference value. Also, if multiple I pixels are arranged on the image sensor, the system may be configured to allow the user to adjust the utilization rate of the I pixels using an operator.

[0069] <Addendum> This embodiment includes the following configurations, methods, and programs. (Composition 1) An imaging device comprising an imaging unit having an image sensor in which pixels for capturing an image and specific pixels that output at least the occurrence of brightness changes and an address event signal indicating the location of such occurrences are arranged in a matrix on the same plane, and an imaging device capable of exposure control by analyzing the image captured by the imaging unit, An acquisition unit that acquires the aforementioned address event signal, An integration unit generates integrated information by accumulating the address event signals acquired by the acquisition unit, A slope calculation unit generates slope information indicating the slope of the change in the address event signal based on the integrated information generated by the integrated unit, An imaging device comprising: an exposure control unit that controls the imaging unit with an exposure determined based on the tilt information generated by the tilt calculation unit. (Configuration 2) The imaging device according to configuration 1, further comprising an exposure determination unit that determines the exposure according to the magnitude of the tilt indicated by the tilt information, wherein the tilt information is information indicating a change in brightness. (Composition 3) The aforementioned calculation unit is The imaging device according to configuration 1 or 2, characterized in that if the tilt information indicates an increase in brightness, 1 is added to the integrated information, and if the tilt information indicates a decrease in brightness, 1 is subtracted from the integrated information to generate the tilt information. (Composition 4) The imaging apparatus according to configuration 1 or 2, further comprising a control unit that weights the address event signals output by specific pixels within a range set by the user. (Composition 5) The imaging apparatus according to configuration 1 or 2, further comprising a control unit that weights address event signals output from specific pixels that have been determined to be continuously increasing or decreasing in brightness over a predetermined period of time. (Composition 6) The aforementioned slope calculation unit, The imaging device according to configuration 1 or 2, characterized in that it generates the tilt information by calculating the amount of change in the cumulative information per unit time. (Composition 7) The control unit further, The imaging device according to configuration 5, characterized in that, when it is determined that the exposure has been controlled based on the tilt information, the weighting is removed. (Composition 8) The aforementioned accumulating unit further, The imaging apparatus according to configuration 2, characterized in that the integrated information is changed to a predetermined value based on the instructions of the exposure determination unit. (Composition 9) The system further includes an image acquisition unit that acquires the image captured by the imaging unit, The exposure determination unit further, The imaging device according to configuration 2, characterized in that it can switch between determining the exposure by analyzing the acquired image in accordance with whether a predetermined standard is met, or determining the exposure according to the magnitude of the tilt. (Composition 10) The imaging apparatus according to configuration 9, characterized in that the predetermined standard is that at least one of the number of frames of the image captured by the imaging unit and the tilt indicated by the tilt information is smaller than the corresponding standard value. (Composition 11) The aforementioned accumulating unit further, The imaging apparatus according to configuration 1 or 2, characterized in that, when the integrated information includes the address event signal of the specific pixel, the integrated information is weighted to generate the integrated information such that the slope indicated by the slope information becomes larger, or to generate the integrated information such that the slope indicated by the slope information becomes smaller. (method) A control method for an imaging device comprising an imaging unit having an image sensor in which pixels for capturing an image and specific pixels that output at least the occurrence of brightness changes and an address event signal indicating the location of such occurrences are arranged in a matrix on the same plane, wherein exposure control is possible by analyzing the image captured by the imaging unit, The acquisition step of acquiring the aforementioned address event signal, An integration step generates integrated information by accumulating the address event signals acquired in the acquisition step, A slope calculation step generates slope information indicating the slope of the change in the address event signal based on the integrated information generated by the integration step, A control method for an imaging device, comprising: an exposure control step of controlling the imaging unit with an exposure determined based on the tilt information generated by the tilt calculation step. (program) A program that causes a computer to execute a control method for an imaging device, which includes an imaging unit having an image sensor in which pixels for capturing an image and specific pixels that output at least the occurrence of brightness changes and an address event signal indicating the location of such occurrence are arranged in a matrix on the same plane, and which can control exposure by analyzing an image captured by the imaging unit, The control method described above is The acquisition step of acquiring the aforementioned address event signal, An integration step generates integrated information by accumulating the address event signals acquired in the acquisition step, A slope calculation step generates slope information indicating the slope of the change in the address event signal based on the integrated information generated by the integration step, A program characterized by comprising: an exposure control step that controls the imaging unit with an exposure determined based on the tilt information generated by the tilt calculation step.

[0070] Although preferred embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above, and various modifications and changes are possible within the scope of its gist. For example, the present invention can also be realized by supplying a program that implements one or more of the functions of the above embodiments to a system or device via a network or recording medium, and the processor of the computer in that system or device reads and executes the program. Furthermore, the present invention can also be realized by a circuit (e.g., ASIC) that implements one or more functions. [Explanation of Symbols]

[0071] 100 imaging devices 101 Imaging Unit 102 Image acquisition unit 103 Exposure determining section 104 Exposure Control Unit 105 Address Event Acquisition Unit 106 Address Event Accumulation Unit 107 Slope Calculation Unit 108 System Control Unit 109 Image Recording Unit 110 Communications Bus 111 Lens section 112 User Interface Section

Claims

1. An imaging device comprising an imaging unit having an image sensor in which pixels for capturing an image and specific pixels that output at least the occurrence of brightness changes and an address event signal indicating the location of such occurrences are arranged in a matrix on the same plane, and an imaging device capable of exposure control by analyzing the image captured by the imaging unit, An acquisition unit that acquires the aforementioned address event signal, An integration unit generates integrated information by accumulating the address event signals acquired by the acquisition unit, A slope calculation unit generates slope information indicating the slope of the change in the address event signal based on the integrated information generated by the integrated unit, An imaging device comprising: an exposure control unit that controls the imaging unit with an exposure determined based on the tilt information generated by the tilt calculation unit.

2. The imaging apparatus according to claim 1, further comprising an exposure determination unit that determines the exposure according to the magnitude of the tilt indicated by the tilt information, wherein the tilt information is information indicating a change in brightness.

3. The aforementioned calculation unit is The imaging device according to claim 1 or 2, characterized in that if the tilt information indicates an increase in brightness, 1 is added to the integrated information, and if the tilt information indicates a decrease in brightness, 1 is subtracted from the integrated information to generate the tilt information.

4. The imaging apparatus according to claim 1 or 2, further comprising a control unit that weights the address event signals output by specific pixels within a range set by the user.

5. The imaging apparatus according to claim 1 or 2, further comprising a control unit that weights address event signals output from specific pixels that have been determined to be continuously increasing or decreasing in brightness over a predetermined period of time.

6. The aforementioned slope calculation unit, The imaging device according to claim 1 or 2, characterized in that it generates the slope information by calculating the amount of change in the cumulative information per unit time.

7. The control unit further, The imaging device according to claim 5, characterized in that, if it is determined that the exposure has been controlled based on the tilt information, the weighting is removed.

8. The aforementioned accumulating unit further, The imaging apparatus according to claim 2, characterized in that the integrated information is changed to a predetermined value based on the instructions of the exposure determination unit.

9. The system further includes an image acquisition unit that acquires the image captured by the imaging unit, The exposure determination unit further, The imaging device according to claim 2, characterized in that it is possible to switch between determining the exposure by analyzing the acquired image in accordance with whether a predetermined standard is met, or determining the exposure according to the magnitude of the tilt.

10. The imaging apparatus according to claim 9, characterized in that the predetermined standard is that at least one of the number of frames of the image captured by the imaging unit and the tilt indicated by the tilt information is smaller than the corresponding standard value.

11. The aforementioned accumulating unit further, The imaging apparatus according to claim 1 or 2, characterized in that, if the integrated information includes the address event signal of the specific pixel, the integrated information is weighted to generate the integrated information such that the slope indicated by the slope information becomes larger, or to generate the integrated information such that the slope indicated by the slope information becomes smaller.

12. A control method for an imaging device comprising an imaging unit having an image sensor in which pixels for capturing an image and specific pixels that output at least the occurrence of brightness changes and an address event signal indicating the location of such occurrences are arranged in a matrix on the same plane, wherein exposure control is possible by analyzing the image captured by the imaging unit, The acquisition step of acquiring the aforementioned address event signal, An integration step generates integrated information by accumulating the address event signals acquired in the acquisition step, A slope calculation step generates slope information indicating the slope of the change in the address event signal based on the integrated information generated by the integration step, A control method for an imaging device, comprising: an exposure control step of controlling the imaging unit with an exposure determined based on the tilt information generated by the tilt calculation step.

13. A program that causes a computer to execute a control method for an imaging device, which includes an imaging unit having an image sensor in which pixels for capturing an image and specific pixels that output at least the occurrence of brightness changes and an address event signal indicating the location of such occurrence are arranged in a matrix on the same plane, and which can control exposure by analyzing an image captured by the imaging unit, The control method described above is The acquisition step of acquiring the aforementioned address event signal, An integration step generates integrated information by accumulating the address event signals acquired in the acquisition step, A slope calculation step generates slope information indicating the slope of the change in the address event signal based on the integrated information generated by the integration step, A program characterized by comprising: an exposure control step that controls the imaging unit with an exposure determined based on the tilt information generated by the tilt calculation step.

Images