sensor

Abstract

Reduce data size while maintaining image quality. [Solution] A sensor 300 is provided, comprising: a first substrate having a first pixel block including a first photoelectric conversion element that converts light into electric charge and a second photoelectric conversion element that converts light into electric charge, and a second pixel block including a third photoelectric conversion element that converts light into electric charge and a fourth photoelectric conversion element that converts light into electric charge; a second substrate laminated with the first substrate, having a first processing block including a first event detection unit that detects events using a signal based on the electric charge converted by the first photoelectric conversion element and a first control unit that performs exposure control of the second photoelectric conversion element based on the detection result of the first event detection unit; and a second processing block including a second event detection unit that detects events using a signal based on the electric charge converted by the third photoelectric conversion element and a second control unit that performs exposure control of the fourth photoelectric conversion element based on the detection result of the second event detection unit.

Description

Technical Field

[0001] The present invention relates to a sensor.

Background Art

[0002] Conventionally, an imaging device including an AD conversion unit has been known (see, for example, Patent Document 1). Patent Document 1: Japanese Patent Application Laid-Open No. 2008-259107

Summary of the Invention

[0003] It is preferable to reduce the data amount of an imaging device.

[0004] In a first aspect of the present invention, there is provided a control device that controls imaging conditions of a sensor including one or more pixels, the control device including: an event detection unit that detects, in one or more pixels, an event indicating that a luminance signal changes beyond a predetermined threshold value and outputs an event detection signal; and a control unit that controls the imaging conditions of the sensor based on the event detection signal.

[0005] In a second aspect of the present invention, there is provided a control method for controlling imaging conditions of a sensor including one or more pixels, the control method including: detecting, in one or more pixels, an event indicating that a luminance signal changes beyond a predetermined threshold value; and controlling the imaging conditions of the sensor based on the detection of the event.

[0006] In a third aspect of the present invention, there is provided a program for controlling imaging conditions of a sensor including one or more pixels, the program causing a computer to execute: detecting, in one or more pixels, an event in which a luminance signal changes beyond a predetermined threshold value; and controlling the imaging conditions of the sensor based on the detection of the event.

[0007] Note that the above summary of the invention does not list all the features of the present invention. Also, sub-combinations of these feature groups can also be inventions. [Brief explanation of the drawing]

[0008] [Figure 1] The configuration of sensor 300 is outlined below. [Figure 2] An example of an event pulse counting method for the control device 100 according to the embodiment is shown. [Figure 3] The configuration of sensor 500 in the comparative example is shown. [Figure 4] An example of a method for determining the ADC resolution of sensor 500 in a comparative example is shown. [Figure 5] An example of the operation flowchart of the control device 100 is shown. [Figure 6] An example of the operation flowchart of the control device 100 is shown. [Figure 7] A more specific example of the configuration of sensor 300 is shown below. [Figure 8] A more specific example of the configuration of the imaging control unit 30 is shown below. [Figure 9] Examples of a computer 1200 in which multiple aspects of the present invention may be embodied in whole or in part are shown. [Modes for carrying out the invention]

[0009] The present invention will be described below through embodiments of the invention, but these embodiments are not intended to limit the invention as defined in the claims. Furthermore, not all combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0010] Figure 1 shows an overview of the configuration of the sensor 300. The sensor 300 in this example comprises a pixel unit 10 and a control device 100. The sensor 300 in this example is a sensor that outputs a pixel signal from the part where the brightness change exceeds a certain level, and is also called an event-driven sensor.

[0011] The pixel unit 10 comprises one or more pixels 11. In this example, the pixel unit 10 comprises multiple pixels 11 arranged in two dimensions. In this example, the pixel unit 10 has M × N pixels 11 (where M and N are natural numbers). Each pixel 11 has at least one photoelectric conversion element. As a result, the pixel unit 10 outputs a pixel signal. When one pixel 11 has two or more photoelectric conversion elements, at least one photoelectric conversion element outputs a photocurrent as a pixel signal for event detection, and the remaining photoelectric conversion elements output the accumulated charge as a pixel signal. Also, when one pixel 11 has one photoelectric conversion element, within the pixel block 12 described later, at least one photoelectric conversion element corresponding to one pixel 11 outputs a photocurrent as a pixel signal for event detection, and the remaining photoelectric conversion elements corresponding to the remaining pixels 11 output the accumulated charge as a pixel signal.

[0012] Furthermore, the pixel section 10 includes a pixel block 12 composed of one or more pixels 11. The pixel block 12 is a region containing m × n pixels 11 (where m and n are natural numbers).

[0013] The control device 100 controls the imaging conditions of the sensor 300. In one example, the control device 100 is a control circuit that controls the imaging conditions of the sensor 300 according to the input pixel signal. The control device 100 includes an event detection unit 20, an imaging control unit 30, a pulse counting unit 40, an AD conversion unit 50, and a storage unit 60.

[0014] The event detection unit 20 detects events based on the pixel signals output by the pixel unit 10. The pixel signals of the pixel unit 10 include brightness signals S of the photocurrent detected by one or more pixels 11. In one example, the event detection unit 20 detects events based on a comparison between the brightness signal S and a predetermined event detection threshold.

[0015] Here, the event detection unit 20 may use either the absolute value of the luminance signal S or the change in the luminance signal S as the event detection threshold. For example, when using the absolute value of the luminance signal S, multiple event detection thresholds are provided for each predetermined luminance value, and the event detection unit 20 determines that an event has occurred when the luminance signal S exceeds the event detection threshold. Alternatively, when using the change in the luminance signal S, the event detection unit 20 determines that an event has occurred when the change in the luminance signal S exceeds the event detection threshold. Thus, an event refers to a change in the luminance signal S that exceeds the event detection threshold. The event detection unit 20 generates an event detection signal Sd to transmit the event detection to the imaging control unit 30 and the pulse count unit 40.

[0016] The event detection signal Sd is a signal that includes an event pulse for determining whether or not an event has occurred. The event detection unit 20 outputs the generated event detection signal Sd to the imaging control unit 30 and the pulse counting unit 40. In this example, the event detection unit 20 is provided in the control device 100, but it may also be provided in the pixel unit 10.

[0017] The imaging control unit 30 controls the imaging conditions of the sensor 300 based on the event detection signal Sd. In one example, the imaging control unit 30 controls the imaging conditions based on the number of event pulses generated by the event detection signal Sd. The imaging control unit 30 may also output a reset signal RST to the pixel unit 10 to reset the charge accumulated in the pixel 11 based on the event detection signal Sd. When the reset signal RST is input to the pixel unit 10, it resets the charge accumulation in the photoelectric conversion element and starts new charge accumulation (i.e., imaging).

[0018] The imaging conditions include at least one of the ADC resolution of the sensor 300, the exposure time, and the gain of the amplifier. That is, the imaging control unit 30 controls at least one of the ADC resolution of the sensor 300, the exposure time, and the gain of the amplifier based on the event detection signal Sd. In one example, the imaging control unit 30 controls the ADC resolution of the AD conversion unit 50 based on the number of generated event pulses of the event detection signal Sd. Note that the change in the ADC resolution may be executed for each single pixel 11. In this case, the control device 100 can drive each pixel 11 asynchronously.

[0019] The pulse counting unit 40 counts event pulses based on the event detection signal Sd and generates a count value Nc. The pulse counting unit 40 outputs the generated count value Nc to the imaging control unit 30. The count value Nc may include up - counting and down - counting. The count value Nc may be the cumulative value of up - counting and down - counting, or may be the integrated value of only up - counting. Note that up - counting is generated when the event detection signal Sd exceeds the event detection threshold. On the other hand, down - counting is generated when the event detection signal Sd is below the event detection threshold.

[0020] The AD conversion unit 50 AD - converts the pixel signals output by the pixel unit 10. The AD conversion unit 50 in this example operates with the ADC resolution output by the imaging control unit 30. The AD conversion unit 50 may perform AD conversion using different ADC resolutions for each one or more pixels 11 or for each block 12. The AD conversion unit 50 outputs the converted digital signals to the storage unit 60. Thus, in this example, the luminance signal S due to the photocurrent generated by the photoelectric conversion element included in the pixel 11 is input to the event detection unit 20 without passing through the AD conversion unit 50, and the pixel signals due to the charges accumulated by other photoelectric conversion elements are input to the AD conversion unit 50. Therefore, not only can events be detected quickly, but low power consumption can be achieved by AD - converting only the pixel signals of the pixels 11 with large luminance changes.

[0021] The memory unit 60 stores the digital signal from the AD conversion unit 50. In one example, the memory unit 60 outputs the memory value of a past frame as a feedback signal FB to the imaging control unit 30. In this case, the imaging control unit 30 may control the imaging conditions based on the count value Nc from the pulse counting unit 40 and the feedback signal FB from the memory unit 60. Thereby, the control device 100 can control the imaging conditions based on the past output signal of the AD conversion unit 50.

[0022] Here, a method for controlling the imaging conditions by the imaging control unit 30 will be described. The imaging control unit 30 counts the event pulses based on the count value Nc output by the pulse counting unit 40 and controls the imaging conditions.

[0023] In one example, the imaging control unit 30 controls the imaging conditions based on the current count value Nc and the pulse accumulation value obtained by accumulating the past count values Nc. In this case, the imaging control unit 30 holds the pulse accumulation value obtained by accumulating the past count values Nc. Then, the imaging control unit 30 reads out the held pulse accumulation value and determines the imaging conditions using the accumulated number obtained by adding the current count value Nc. Note that the imaging control unit 30 may consider the plus and minus of the count value Nc with respect to the accumulation of the count value Nc. That is, the imaging control unit 30 cancels out the plus count value Nc with the minus count value Nc. Since the luminance value per count is predetermined, the luminance value corresponding to the current count value Nc can be obtained by using the accumulated count value.

[0024] In another example, the imaging control unit 30 controls the imaging conditions based on the current brightness value and past brightness values ​​stored in the memory unit 60. In this case, the memory unit 60 holds brightness values ​​previously read from the AD conversion unit 50. In one example, the imaging control unit 30 controls the imaging conditions based on the brightness value output by the sensor 300 in response to the detection of the previous event and the event pulse output by the event detection unit 20 upon detection of a new event. For example, the imaging control unit 30 calculates the current brightness value by adding or subtracting the current brightness value per pulse to the brightness value stored in the memory unit 60. The imaging control unit 30 controls the imaging conditions based on the calculated current brightness value.

[0025] As described above, the control device 100 may control the imaging conditions for the current event based on both the brightness signal S of past events and the pulse count value Nc during imaging. This allows the imaging conditions to be updated to follow the change in environment, even if the environment of the current event changes rapidly from the environment of past events. Alternatively, the control device 100 may control the imaging conditions for the current event based only on the pulse count value Nc during imaging.

[0026] The control device 100 in this example can perform AD conversion with a resolution sufficient for the brightness value by changing the ADC resolution in response to event detection. Therefore, it is possible to limit the output of unnecessary bits and compress the amount of data. Furthermore, by compressing the amount of data, the control device 100 can reduce power consumption. By optimally adjusting the ADC resolution, the control device 100 can achieve a high dynamic range.

[0027] Figure 2 shows an example of an event pulse counting method for the control device 100 according to the embodiment. The vertical axis represents the luminance signal S, and the horizontal axis represents time T. In this example, the luminance signal S is used to detect events, but other signals may be used to detect events.

[0028] The event detection unit 20 generates an event pulse in response to a change in the luminance signal S. In one example, the event detection unit 20 sets one or more thresholds and generates an event pulse when the luminance signal S exceeds one of these thresholds. In this example, the event detection unit 20 sets thresholds Sth1 to Sth5 and generates an event pulse when the luminance signal S falls within one of these thresholds. That is, the event detection unit 20 generates an event pulse when the luminance signal S increases or decreases beyond any of the thresholds Sth1 to Sth5. Thresholds Sth1 to Sth5 are examples of event detection thresholds.

[0029] For example, the event detection unit 20 generates an up-count event pulse when any of the thresholds Sth1 to Sth5 are exceeded. Conversely, the event detection unit 20 generates a down-count event pulse when any of the thresholds Sth1 to Sth5 are fallen below. In this example, the thresholds Sth1 to Sth5 are set at equal intervals, but they do not have to be at equal intervals.

[0030] The pulse counting unit 40 generates a count value Nc by counting the event pulses generated by the event detection unit 20. The pulse counting unit 40 counts at least one of the up count and the down count. For example, the pulse counting unit 40 accumulates the up count and down count considering the plus or minus sign. This makes it possible to detect whether the brightness signal S has increased or decreased by a predetermined value. In this example, the imaging control unit 30 changes the imaging conditions when the count value Nc of the pulse counting unit 40 reaches "±3". Note that the criterion for determining the count value Nc may be a value other than ±3.

[0031] Events E1 to E7 represent event pulses generated during a predetermined period. The imaging control unit 30 may output a reset signal RST to the pixel unit 10 each time an event occurs to start imaging.

[0032] In event E1, an upcount event pulse is generated. This indicates that one of the thresholds Sth1 to Sth5 was exceeded once in event E1. In this example, if the count value Nc of the pulse count unit 40 becomes +1 upon detection of one upcount event pulse, the imaging conditions are not changed.

[0033] In event E2, a down-count event pulse is generated. This indicates that in event E2, one of the thresholds Sth1 to Sth5 was exceeded once. In this example, if the count value Nc of the pulse count unit 40 becomes -1 due to the detection of one down-count event pulse, the imaging conditions are not changed.

[0034] In events E3 to E5, three up-count event pulses are generated. That is, because events E3 to E5 exceed any of the thresholds Sth1 to Sth5 three times, the count value Nc of the pulse count unit 40 becomes +3. The imaging control unit 30 determines that the brightness of the object being imaged has changed significantly beyond a predetermined level because the count value Nc of the pulse count unit 40 has reached +3, and changes the imaging conditions. For example, the imaging control unit 30 lowers the ADC resolution when the count value Nc of the pulse count unit 40 becomes +3 or higher.

[0035] More specifically, when the count value Nc of the pulse count unit 40 exceeds +n (where n is a predetermined integer), AD conversion may be performed without using predetermined lower bits. When the image target is bright, the output of the lower bits is buried in noise, so even if the resolution of the brightness range detected by the sensor 300 is reduced, it will have almost no effect on the resulting pixel signal. The predetermined lower bits are, for example, the two bits from the least significant.

[0036] On the other hand, when the count value Nc of the pulse count unit 40 falls below -n, AD conversion may be performed without using the predetermined higher bits. If the image target is dark, the higher bits become 0 and are unnecessary in the first place. Therefore, not using the higher bits does not affect the resulting pixel signal. The predetermined higher bits are, for example, the two most significant bits.

[0037] In this example, the control device 100 controls the imaging conditions by detecting events corresponding to changes in the luminance signal S. This allows the control device 100 to update the imaging conditions in response to rapid changes in the luminance signal S.

[0038] Figure 3 shows the configuration of a sensor 500 in a comparative example. The sensor 500 in this example comprises a pixel unit 510, an imaging control unit 530, an AD conversion unit 550, and a storage unit 560.

[0039] The sensor 500 determines the ADC resolution of the AD conversion unit 550 based on the pixel signal output by the pixel unit 510. The sensor 500 may store the determined ADC resolution in the storage unit 560.

[0040] The memory unit 560 outputs imaging information of past frames as a feedback signal FB to the imaging control unit 530. The imaging control unit 530 outputs past imaging conditions to the AD conversion unit 550. The AD conversion unit 550 then controls the ADC resolution based on the past imaging conditions and the pixel signal from the pixel unit 510.

[0041] Figure 4 shows an example of a method for determining the ADC resolution of sensor 500 in a comparative example. In this example, sensor 500 sets the ADC resolution based on a comparison between a luminance signal S and a luminance threshold Sth.

[0042] Sensor 500 sets the ADC resolution in units of frames of a predetermined length. That is, sensor 500 compares the luminance signal S with the luminance threshold Sth for each frame. For example, if the luminance signal S is smaller than the luminance threshold Sth, sensor 500 performs AD conversion only on the lower bits, and if the luminance signal S is larger than the luminance threshold Sth, it performs AD conversion only on the upper bits. In other words, if the luminance signal S is smaller than the luminance threshold Sth, AD conversion of the upper bits is unnecessary, and if the luminance signal S is larger than the luminance threshold Sth, AD conversion of the lower bits is unnecessary.

[0043] Sensor 500 calculates the ADC resolution of the current frame based on the luminance signal of past frames. For example, sensor 500 sets the ADC resolution of the current frame, frame F3, based on at least one data point from past frames, frame F1 and frame F2. Therefore, if the luminance signal S increases rapidly between past and current frames, causing the ADC resolution to switch, it becomes difficult to set an appropriate ADC resolution. In this example, sensor 500 performs AD conversion only on the lower bits because the lower bits are set for past frames. Even when it would be better to select upper bit AD conversion for the current frame, the brightness of the current frame cannot be estimated from the data of past frames, resulting in AD conversion being performed only on the lower bits.

[0044] Thus, since the sensor 500 adjusts the ADC resolution of the current frame based on past frames, it is difficult to optimally update the imaging conditions to keep up with the changes in the environment if the environment of the current frame changes rapidly from the environment of past frames. In other words, the control of imaging conditions based on events is affected by the imaging frame.

[0045] On the other hand, the control device 100 acquires the change in the brightness signal S as the event pulse count and controls the imaging conditions. Therefore, the control device 100 can quickly control the imaging conditions by following the change in the brightness signal S, rather than relying on past imaging frames.

[0046] Figure 5 shows an example of a flowchart of the operation of the control device 100. The control device 100 may control the imaging conditions of the sensor 300 using the flowchart in this example. In the flowchart in this example, the pixel 11 is reset each time an event pulse is generated.

[0047] The control device 100 monitors the brightness signal S output by the pixel 11 (S100). The control device 100 determines whether the brightness signal S has changed to or above a predetermined event detection threshold (S102). Specifically, the event detection unit 20 detects an event in which the brightness signal S changes beyond the event detection threshold in one or more pixels 11. If the brightness signal S does not change to or above the event detection threshold, the control device 100 continues to monitor the brightness signal S (S100). On the other hand, if the brightness signal S changes to or above the event detection threshold, the control device 100 generates an event pulse and outputs it to the pulse count unit 40 (S104).

[0048] The imaging control unit 30 then counts event pulses from the event detection unit 20 (S106). Based on the detection of an event, the imaging control unit 30 controls the imaging conditions of the sensor 300 (S108). For example, when the counter value Nc calculated by the pulse counting unit 40 reaches a preset standard, the imaging control unit 30 controls the imaging conditions based on that counter value Nc. The imaging conditions controlled by the imaging control unit 30 may be any of the ADC resolution of the sensor 300, the exposure time, or the gain of the amplifier. The control device 100 resets the charge accumulated in the pixel 11 (S110). After controlling the imaging conditions, the control device 100 starts exposure by the sensor 300 (S112). That is, in this example, the control device 100 starts exposure after the imaging control unit 30 has updated the imaging conditions. If at least one of the exposure time and the gain of the amplifier has been updated in step S108, the updated imaging conditions are used.

[0049] In this example, if the event detection unit 20 detects an event during the exposure period, the imaging control unit 30 re-controls the imaging conditions, resets one or more pixels 11, and restarts exposure by the sensor 300. For example, if an event occurs during the exposure period (S114), the event detection unit 20 starts generating an event pulse (S104). On the other hand, if no event occurs during the exposure period (S114), the control device 100 terminates the exposure (S116).

[0050] After exposure is complete, the AD conversion unit 50 performs A / D conversion on the pixel signal output by the pixel unit 10 (S118). Here, if the imaging control unit 30 has updated the ADC resolution as an imaging condition in step S108, the pixel signal is converted A / D based on the updated imaging condition. Subsequently, the storage unit 60 stores the converted digital signal and may output the stored digital signal to the outside (S120).

[0051] Figure 6 shows an example of a flowchart of the operation of the control device 100. The control device 100 may control the imaging conditions of the sensor 300 using the flowchart in this example. The flowchart in this example differs from the flowchart in Figure 5 in that the pixels 11 are not reset between the time the imaging conditions are calculated and the exposure is started. That is, in this example, the control device 100 resets the pixels 11 once to update the imaging conditions, and then starts exposure without resetting the pixels 11 again.

[0052] Steps S200 and S202 correspond to steps S100 and S102 in Figure 5, respectively. In step S202, if the brightness signal S changes to a predetermined event detection threshold or higher, the event detection unit 20 starts generating event pulses (S204). The imaging control unit 30 then resets the pixels 11 (S206). The pulse count unit 40 also counts the generated event pulses (S208). Subsequently, when the count value Nc from the pulse count unit 40 reaches a preset standard, the imaging control unit 30 sets imaging conditions based on the count value Nc and controls the imaging of the control device 100 (S210).

[0053] The control device 100 starts exposure based on the updated imaging conditions (S212). That is, in this example, the control device 100 starts exposure after the imaging control unit 30 controls the imaging conditions based on the event pulse count value Nc and resets the pixel unit 10.

[0054] If an event occurs during the exposure period (S214), the event detection unit 20 starts generating event pulses (S216). Furthermore, the pulse count unit 40 counts the generated event pulses (S218). Subsequently, while continuing exposure by the sensor 300, the imaging control unit 30 controls the imaging conditions based on the count value Nc from the pulse count unit 40 when the count value Nc reaches a preset standard (S220).

[0055] Thus, in this example, the imaging control unit 30, when the event detection unit 20 detects an event during the exposure period by the sensor 300, re-controls the imaging conditions and continues exposure by the sensor 300. In other words, the control device 100 may update the imaging conditions before the start of exposure, or it may update the imaging conditions during exposure. On the other hand, if no event occurs during the exposure period after the start of exposure (S212) (S214), exposure by the sensor 300 is terminated (S222).

[0056] After exposure is complete, the AD conversion unit 50 performs A / D conversion on the pixel signals output by the pixel unit 10 (S224). Here, if the imaging control unit 30 has updated the ADC resolution as an imaging condition in steps S210 and S220, the pixel signals are converted A / D based on the latest ADC resolution. Subsequently, the storage unit 60 stores the converted digital signal and may output the stored digital signal to the outside (S226).

[0057] As described above, after resetting the pixels 11 and updating the imaging conditions, the control device 100 can start exposure without resetting the pixels 11 again. Furthermore, the update of imaging conditions by the control device 100 may occur before or during exposure. This allows the control device 100 to capture images with optimal imaging conditions in response to the occurrence of an event.

[0058] Figure 7 shows a more specific example of the configuration of the sensor 300. In this example, the sensor 300 comprises a first substrate 110, a second substrate 120, and a third substrate 130.

[0059] The first substrate 110 has a pixel block array 112. The pixel block array 112 has one or more pixels 11. In this example, the pixel block array 112 has M × N pixels 11. The pixels 11 have a first photoelectric conversion element PD1 and a second photoelectric conversion element PD2. For example, the first photoelectric conversion element PD1 is an event detection element that detects events. The second photoelectric conversion element PD2 is a light-receiving element that captures images.

[0060] The second substrate 120 has a processing block array 122. The processing block array 122 comprises M × N data processing units 13. The second substrate 120 is stacked on the first substrate 110.

[0061] The data processing unit 13 comprises an event detection unit 20, an imaging control unit 30, a pulse counting unit 40, and an AD conversion unit 50. The M × N data processing units 13 are provided corresponding to each of the M × N pixels 11. "Provided correspondingly" means that each of the M × N data processing units 13 is electrically connected to each of the M × N pixels 11. Alternatively, "provided correspondingly" may mean that each of the M × N data processing units 13 is provided facing each of the M × N pixels 11.

[0062] In this example, the data processing unit 13 performs data processing on the corresponding pixels 11. When the data processing unit 13 is located directly beneath the pixels 11, the wiring is shortened, improving processing speed. For example, at least one of the event detection unit 20 and the imaging control unit 30 is located on the second substrate 120, corresponding to a block consisting of one or more pixels 11. This allows for a larger pixel aperture for the pixels 11 located on the first substrate 110.

[0063] The third substrate 130 has an output block array 132. The output block array 132 includes a data holding unit 70 and a data input / output unit 80. The third substrate 130 is laminated on the second substrate 120. That is, the first substrate 110, the second substrate 120, and the third substrate 130 are laminated together.

[0064] The data holding unit 70 holds the imaging data from the sensor 300. For example, the data holding unit 70 temporarily holds the imaging data acquired by the processing block array 122. The imaging data may include data such as imaging conditions in addition to the image data acquired by the pixel unit 10. The data input / output unit 80 outputs the imaging data held by the data holding unit 70 to the outside. The data input / output unit 80 may also output the imaging data to the imaging control unit 30.

[0065] As described above, the sensor 300 in this example is equipped with M × N imaging control units 30, each corresponding to one of the M × N pixels 11. Therefore, the sensor 300 can control the imaging conditions for each of the M × N pixels 11. This allows the sensor 300 to set appropriate imaging conditions for each pixel 11 according to the brightness distribution within the image. For example, by setting the ADC resolution for each of the M × N pixels 11, it is possible to reduce unnecessary bits and suppress the data output bandwidth compared to when AD conversion is performed with a uniform ADC resolution for all pixels. The sensor 300 in this example can reduce the amount of data while maintaining image quality by feeding back the necessary and sufficient bit depth for each of the M × N pixels 11 to the AD conversion unit 50.

[0066] Figure 8 shows a more specific example of the configuration of the imaging control unit 30. The imaging control unit 30 comprises a reset control unit 32, a gain control unit 34, an exposure control unit 36, and an ADC control unit 38.

[0067] The reset control unit 32 generates a reset signal RST to reset the charge accumulated in the pixel 11. Based on the event detection signal Sd input from the event detection unit 20, the reset control unit 32 outputs the generated reset signal RST to the pixel unit 10. For example, when operating according to the flowchart in Figure 6, the reset control unit 32 outputs the reset signal RST to the pixel unit 10 when an event detection signal Sd indicating that an event has occurred is input.

[0068] The gain control unit 34 controls the gain of the amplifier in the sensor 300 based on the count value Nc input from the pulse count unit 40. For example, the gain control unit 34 decreases the gain from its previous value when the count value Nc exceeds a predetermined threshold, and increases the gain from its previous value when the count value Nc falls below a predetermined threshold.

[0069] The exposure control unit 36 ​​controls exposure conditions such as exposure time based on the count value Nc input from the pulse count unit 40. For example, the exposure control unit 36 ​​shortens the exposure time if the count value Nc exceeds a predetermined threshold, and lengthens the exposure time if the count value Nc falls below a predetermined threshold. Alternatively, the exposure control unit 36 ​​may lengthen the exposure time if the image brightness falls below a predetermined threshold, and shorten the exposure time if the image brightness exceeds a predetermined threshold.

[0070] The ADC control unit 38 controls the ADC resolution based on the count value Nc input from the pulse count unit 40. For example, the ADC control unit 38 lowers the ADC resolution to a lower level than the previous resolution when the count value Nc exceeds a predetermined threshold, and raises the ADC resolution to a higher level than the previous resolution when the count value Nc falls below a predetermined threshold. Alternatively, the ADC control unit 38 may increase the ADC resolution when the image brightness falls below a predetermined threshold, and decrease the ADC resolution when the image brightness exceeds a predetermined threshold. Note that the gain control by the gain control unit 34, the exposure time control by the exposure control unit 36, and the ADC resolution control by the ADC control unit 38 may be controlled in combination. In this case, the same count value Nc may be set as the threshold referenced by these control units, or different count values ​​Nc may be set for each unit.

[0071] Figure 9 shows an example of a computer 1200 in which multiple aspects of the present invention may be embodied in whole or in part. A program installed on the computer 1200 can cause the computer 1200 to function as an operation or one or more sections of an apparatus according to an embodiment of the present invention, or to execute such operation or one or more sections, and / or to cause the computer 1200 to execute a process or a stage of such process according to an embodiment of the present invention. Such a program may be executed by the CPU 1212 to cause the computer 1200 to perform a particular operation associated with some or all of the blocks in the flowcharts and block diagrams described herein.

[0072] The computer 1200 according to this embodiment includes a CPU 1212, RAM 1214, a graphics controller 1216, and a display device 1218, which are interconnected by a host controller 1210. The computer 1200 also includes input / output units such as a communication interface 1222, a hard disk drive 1224, a DVD-ROM drive 1226, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The computer also includes legacy input / output units such as a ROM 1230 and a keyboard 1242, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0073] The CPU 1212 operates according to programs stored in the ROM 1230 and RAM 1214, thereby controlling each unit. The graphics controller 1216 acquires image data generated by the CPU 1212 from a frame buffer provided in RAM 1214 or from itself, and displays the image data on the display device 1218.

[0074] The communication interface 1222 communicates with other electronic devices via a network. The hard disk drive 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD-ROM drive 1226 reads programs or data from the DVD-ROM 1201 and provides them to the hard disk drive 1224 via the RAM 1214. The IC card drive reads programs and data from the IC card and / or writes programs and data to the IC card.

[0075] The ROM 1230 stores boot programs and / or hardware-dependent programs of the computer 1200, which are executed by the computer 1200 upon activation. The input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via parallel ports, serial ports, keyboard ports, mouse ports, etc.

[0076] The program is provided on a computer-readable medium such as a DVD-ROM 1201 or an IC card. The program is read from the computer-readable medium and installed on a hard disk drive 1224, RAM 1214, or ROM 1230, which are also examples of computer-readable medium, and executed by the CPU 1212. The information processing described within these programs is read by the computer 1200, resulting in coordination between the program and the various types of hardware resources described above. The apparatus or method may be configured to realize the manipulation or processing of information in accordance with the use of the computer 1200.

[0077] For example, when communication is performed between a computer 1200 and an external device, the CPU 1212 may execute a communication program loaded into RAM 1214 and, based on the processing described in the communication program, instruct the communication interface 1222 to perform communication processing. Under the control of the CPU 1212, the communication interface 1222 reads transmission data stored in a transmission buffer processing area provided in a recording medium such as RAM 1214, a hard disk drive 1224, a DVD-ROM 1201, or an IC card, transmits the read transmission data to the network, or writes received data received from the network to a reception buffer processing area provided on the recording medium.

[0078] Furthermore, the CPU 1212 may read all or necessary parts of files or databases stored on external storage media such as the hard disk drive 1224, DVD-ROM drive 1226 (DVD-ROM 1201), or IC card into the RAM 1214, and perform various types of processing on the data in the RAM 1214. The CPU 1212 then writes the processed data back to the external storage media.

[0079] Various types of information, such as various types of programs, data, tables, and databases, may be stored on the recording medium and subjected to information processing. The CPU 1212 may perform various types of processing on the data read from RAM 1214, including various types of operations, information processing, conditional judgments, conditional branching, unconditional branching, information retrieval / replacement, etc., as described throughout this disclosure and specified by the program instruction sequence, and write the results back to RAM 1214. The CPU 1212 may also retrieve information in files, databases, etc., within the recording medium. For example, if multiple entries are stored in the recording medium, each having an attribute value of a first attribute associated with an attribute value of a second attribute, the CPU 1212 may search among the multiple entries for an entry that matches the condition for which the attribute value of the first attribute is specified, read the attribute value of the second attribute stored in that entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0080] The programs or software modules described above may be stored on or near computer 1200 on a computer-readable medium. Alternatively, recording media such as hard disks or RAM provided within a server system connected to a dedicated communication network or the Internet can be used as computer-readable media, thereby providing programs to computer 1200 via the network.

[0081] Furthermore, it should be noted that the execution order of operations, procedures, steps, and stages in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not explicitly stated as "before," "prior to," etc., and can be implemented in any order unless the output of a previous process is used in a later process. Even if the operation flow in the claims, specifications, and drawings is described using phrases such as "first," "next," etc. for convenience, it does not mean that it is essential to perform them in that order.

[0082] Although the present invention has been described above using embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various modifications or improvements can be made to the above embodiments. It will be clear from the claims that such modified or improved forms may also be included in the technical scope of the present invention. (Item 1) A control device for controlling the imaging conditions of a sensor having one or more pixels, An event detection unit detects an event in one or more pixels that indicates a change in brightness signal exceeding a predetermined threshold and outputs an event detection signal. A control unit controls the imaging conditions of the sensor based on the event detection signal, A control device equipped with the following features. (Item 2) The imaging conditions include at least one of the ADC resolution of the sensor, the exposure time, and the gain of the amplifier. The control device described in item 1. (Item 3) The system further includes a pulse counter that counts the event pulses output by the event detection unit upon detection of the aforementioned event. The control unit controls the imaging conditions based on the number of event pulses generated. The control device described in item 1 or 2. (Item 4) The control unit controls the imaging conditions based on the brightness value output by the sensor in response to the detection of the previous event and the event pulse output by the event detection unit upon detection of a new event. The control device described in item 1 or 2. (Item 5) The control unit, when the event detection unit detects an event during the exposure period by the sensor, controls the imaging conditions again, resets the charge accumulation on the one or more pixels, and restarts exposure by the sensor. A control device as described in any one of items 1 through 4. (Item 6) The control unit, when the event detection unit detects an event during the exposure period by the sensor, re-controls the imaging conditions and continues exposure by the sensor. A control device as described in any one of items 1 through 4. (Item 7) One or more pixels, A control device as described in any one of items 1 to 6, A sensor equipped with the following features. (Item 8) The one or more pixels include an event detection element for detecting the event and a light-receiving element for capturing an image. The sensor described in item 7. (Item 9) A first substrate on which the one or more pixels are provided, Other substrates stacked on the first substrate, Equipped with, At least one of the event detection unit and the control unit is provided on the other substrate in correspondence with the block consisting of one or more pixels. The sensor described in item 7 or 8. (Item 10) A first substrate on which the one or more pixels are provided, A second substrate is laminated on the first substrate and is provided with the event detection unit and the control unit, A third substrate is laminated on the second substrate and is provided with a data holding section for holding imaging data of the sensor, Equipped with, The sensor described in item 7 or 8. (Item 11) A control method for controlling the imaging conditions of a sensor having one or more pixels, In the aforementioned one or more pixels, an event is detected that indicates a change in the brightness signal exceeding a predetermined threshold, Based on the detection of the aforementioned event, the imaging conditions of the sensor are controlled, A control method including (Item 12) After controlling the imaging conditions, exposure by the sensor is started, If the sensor detects the event during the exposure period, the imaging conditions are controlled again. The charge accumulation on the sensor is reset, and exposure is restarted according to the newly controlled imaging conditions. The control method described in item 11, which includes the following. (Item 13) After controlling the imaging conditions, exposure by the sensor is started, If the event is detected during the exposure period by the sensor, the imaging conditions are controlled again while continuing the exposure by the sensor. The control method described in item 11, which includes the following. (Item 14) A program for controlling imaging conditions of a sensor having one or more pixels, In the aforementioned one or more pixels, an event is detected in which the brightness signal changes beyond a predetermined threshold, Based on the detection of the aforementioned event, the imaging conditions of the sensor are controlled, A control program that instructs a computer to execute a command. [Explanation of symbols]

[0083] 10...Pixel section, 11...Pixel, 12...Block, 13...Data processing section, 20...Event detection section, 30...Imaging control section, 32...Reset control section, 34...Gain control section, 36...Exposure control section, 38...ADC control section, 40...Pulse count section, 50...AD conversion section, 60...Storage section, 70...Data holding section, 80...Data input / output section, 100...Control device, 110...First substrate, 112...Pixel block array, 120...Second substrate, 122...Processing block array, 130...Third substrate, 132...Output block array, 300...Se Sensor, 500...Sensor, 510...Pixel unit, 530...Imaging control unit, 550...AD conversion unit, 560...Storage unit, 1200...Computer, 1201...DVD-ROM, 1210...Host controller, 1212...CPU, 1214...RAM, 1216...Graphics controller, 1218...Display device, 1220...Input / output controller, 1222...Communication interface, 1224...Hard disk drive, 1226...DVD-ROM drive, 1230...ROM, 1240...Input / output chip, 1242...Keyboard

Claims

[Claim 1] A first substrate having a first pixel block including a first photoelectric conversion element that converts light into electric charge and a second photoelectric conversion element that converts light into electric charge, and a second pixel block including a third photoelectric conversion element that converts light into electric charge and a fourth photoelectric conversion element that converts light into electric charge, A second substrate laminated with the first substrate, comprising: a first processing block including a first event detection unit that detects events using a signal based on the charge converted by the first photoelectric conversion element and a first control unit that performs exposure control of the second photoelectric conversion element based on the detection result of the first event detection unit; and a second processing block including a second event detection unit that detects events using a signal based on the charge converted by the third photoelectric conversion element and a second control unit that performs exposure control of the fourth photoelectric conversion element based on the detection result of the second event detection unit. A sensor equipped with the following features.

Images