Sensor

Abstract

To reduce the amount of data while keeping image quality.SOLUTION: A sensor 300 includes: a first substrate including a first pixel block having a first photoelectric conversion element to convert light into charges and a second photoelectric conversion element to convert light into charges, and a second pixel block having a third photoelectric conversion element to convert light into charges and a fourth photoelectric conversion element to convert light into charges; and a second substrate stacked on the first substrate. The second substrate includes a first process block including a first event detection unit to detect an event using a signal based on the charge converted in the first photoelectric conversion element and a first control unit to perform exposure control of the second photoelectric conversion element on the basis of the detection result of the first event detection unit, and a second process block including a second event detection unit to detect an event using a signal based on the charge converted in the third photoelectric conversion element and a second control unit to perform exposure control of the fourth photoelectric conversion element on the basis of the detection result of the second event detection unit.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a sensor. [Background technology]

[0002] 2. Description of the Related Art Conventionally, imaging devices equipped with an AD conversion unit are known (see, for example, Patent Document 1). Patent Document 1 Special Publication No. 2008-259107 Summary of the Invention

[0003] It is preferable to reduce the amount of data in an imaging device.

[0004] In a first aspect of the present invention, there is provided a control device that controls imaging conditions of a sensor having one or more pixels, the control device comprising: an event detection unit that detects an event indicating that a luminance signal changes beyond a predetermined threshold in one or more pixels and outputs an event detection signal; and a control unit that controls the imaging conditions of the sensor based on the event detection signal.

[0005] In a second aspect of the present invention, there is provided a control method for controlling imaging conditions of a sensor having one or more pixels, the control method including detecting an event indicating that a luminance signal changes beyond a predetermined threshold in one or more pixels, and controlling the imaging conditions of the sensor based on the detection of the event.

[0006] In a third aspect of the present invention, there is provided a program for controlling imaging conditions of a sensor having one or more pixels, the program being a control device that causes a computer to detect an event in which a luminance signal changes beyond a predetermined threshold in one or more pixels, and control the imaging conditions of the sensor based on the detection of the event.

[0007] The above summary of the invention does not list all of the features of the present invention. Also, subcombinations of these features may also be inventions. [Brief description of the drawings]

[0008] [Figure 1] 2 shows an outline of the configuration of the sensor 300. [Diagram 2] An example of a method for counting event pulses by the control device 100 according to the embodiment will be described. [Diagram 3] 2 shows the configuration of a sensor 500 according to a comparative example. [Figure 4] 13 shows an example of a method for determining the ADC resolution of the sensor 500 according to the comparative example. [Diagram 5] 2 shows an example of a flowchart of the operation of the control device 100. [Figure 6] 2 shows an example of a flowchart of the operation of the control device 100. [Figure 7] An example of a more specific configuration of the sensor 300 will be described. [Figure 8] An example of a more specific configuration of the imaging control unit 30 is shown. [Figure 9] An example of a computer 1200 in which aspects of the present invention may be embodied, in whole or in part, is shown. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0010] 1 shows an outline of the configuration of a sensor 300. The sensor 300 of this example includes a pixel unit 10 and a control device 100. The sensor 300 of this example is a sensor that outputs a pixel signal from a portion where there has been a luminance change of a certain level or more, and is also called an event-driven sensor.

[0011] The pixel unit 10 includes one or more pixels 11. The pixel unit 10 of this example includes a plurality of pixels 11 arranged two-dimensionally. The pixel unit 10 of this example includes M×N pixels 11 (M and N are natural numbers). The pixel 11 includes at least one photoelectric conversion element. This allows the pixel unit 10 to output a pixel signal. When one pixel 11 includes two or more photoelectric conversion elements, at least one photoelectric conversion element outputs a photocurrent as a pixel signal for event detection, and the remaining photoelectric conversion elements output accumulated charges as pixel signals. When one pixel 11 includes one photoelectric conversion element, in a pixel block 12 described later, a photoelectric conversion element corresponding to at least one pixel 11 outputs a photocurrent as a pixel signal for event detection, and the photoelectric conversion elements corresponding to the remaining pixels 11 output accumulated charges as pixel signals.

[0012] The pixel section 10 also includes a pixel block 12 made up of one or more pixels 11. The pixel block 12 is an area including m×n pixels 11 (m and n are natural numbers).

[0013] The control device 100 controls the imaging conditions of the sensor 300. In one example, the control device 100 is a control circuit that controls the imaging conditions of the sensor 300 in response to an input pixel signal. The control device 100 includes an event detection unit 20, an imaging control unit 30, a pulse count unit 40, an AD conversion unit 50, and a storage unit 60.

[0014] The event detection unit 20 detects an event based on a pixel signal output by the pixel unit 10. The pixel signal of the pixel unit 10 includes a luminance signal S of a photocurrent detected by one or more pixels 11. In one example, the event detection unit 20 detects an event based on a comparison between the luminance signal S and a predetermined event detection threshold.

[0015] Here, the event detection unit 20 may use the absolute value of the luminance signal S as the event detection threshold, or may use the amount of change in the luminance signal S. For example, when the absolute value of the luminance signal S is used, a plurality of event detection thresholds are provided for each predetermined luminance value, and the event detection unit 20 determines that an event has occurred when the luminance signal S exceeds the event detection threshold. When the amount of change in the luminance signal S is used, the event detection unit 20 determines that an event has occurred when the amount of change in the luminance signal S exceeds the event detection threshold. In this way, an event refers to a change in the luminance signal S that exceeds the event detection threshold. The event detection unit 20 generates an event detection signal Sd for transmitting the event detection to the imaging control unit 30 and the pulse count unit 40.

[0016] The event detection signal Sd is a signal including an event pulse for determining whether an event has occurred. The event detection unit 20 outputs the generated event detection signal Sd to the imaging control unit 30 and the pulse count unit 40. Note that the event detection unit 20 in this example is provided in the control device 100, but may be provided in the pixel unit 10.

[0017] The imaging control unit 30 controls the imaging conditions of the sensor 300 based on the event detection signal Sd. In one example, the imaging control unit 30 controls the imaging conditions based on the number of occurrences of event pulses of the event detection signal Sd. Note that the imaging control unit 30 may output a reset signal RST to the pixel unit 10 based on the event detection signal Sd to reset the charge accumulated in the pixel 11. When the reset signal RST is input, the pixel unit 10 resets the accumulation of charge in the photoelectric conversion element and starts a new accumulation of charge (i.e., imaging).

[0018] The imaging conditions include at least one of the ADC resolution, exposure time, and amplifier gain of the sensor 300. That is, the imaging control unit 30 controls at least one of the ADC resolution, exposure time, and amplifier gain of the sensor 300 based on the event detection signal Sd. In one example, the imaging control unit 30 controls the ADC resolution of the AD conversion unit 50 based on the number of occurrences of event pulses of the event detection signal Sd. Note that the change of the ADC resolution may be performed for each individual pixel 11. In this case, the control device 100 can drive each pixel 11 asynchronously.

[0019] The pulse count unit 40 counts the event pulses based on the event detection signal Sd to generate a count value Nc. The pulse count unit 40 outputs the generated count value Nc to the imaging control unit 30. The count value Nc may include an up-count and a down-count. The count value Nc may be an accumulated value of the up-count and the down-count, or may be an integrated value of only the up-count. The up-count is generated when the event detection signal Sd exceeds the event detection threshold. On the other hand, the down-count is generated when the event detection signal Sd falls below the event detection threshold.

[0020] The AD conversion unit 50 performs AD conversion on the pixel signal output by the pixel unit 10. The AD conversion unit 50 in this example operates with the ADC resolution output by the imaging control unit 30. The AD conversion unit 50 may perform AD conversion using different ADC resolutions for one or more pixels 11 or for each block 12. The AD conversion unit 50 outputs the converted digital signal to the storage unit 60. In this way, in this example, the luminance signal S due to the photocurrent generated in the photoelectric conversion element included in the pixel 11 is input to the event detection unit 20 without passing through the AD conversion unit 50, and the pixel signal due to the charge accumulated in the other photoelectric conversion elements is input to the AD conversion unit 50. Therefore, not only can the event be detected quickly, but also low power consumption can be achieved by AD converting only the pixel signal of the pixel 11 with a large luminance change.

[0021] The storage unit 60 stores the digital signal from the AD conversion unit 50. In one example, the storage unit 60 outputs a memory value of a past frame as a feedback signal FB to the imaging control unit 30. In this case, the imaging control unit 30 may control the imaging conditions based on the count value Nc from the pulse count unit 40 and the feedback signal FB from the storage unit 60. This allows the control device 100 to control the imaging conditions based on the past output signal of the AD conversion unit 50.

[0022] Here, there will be described a method for controlling the imaging conditions by the imaging control section 30. The imaging control section 30 counts event pulses based on the count value Nc output by the pulse count section 40, and controls the imaging conditions.

[0023] In one example, the imaging control unit 30 controls the imaging conditions based on the current count value Nc and a pulse cumulative value obtained by accumulating past count values ​​Nc. In this case, the imaging control unit 30 holds a pulse cumulative value obtained by accumulating past count values ​​Nc. Then, the imaging control unit 30 reads out the held pulse cumulative value, and determines the imaging conditions using the cumulative number obtained by adding the current count value Nc. Note that the imaging control unit 30 may take into account the positive and negative count values ​​Nc with respect to the accumulation of the count values ​​Nc. In other words, the imaging control unit 30 offsets a positive count value Nc with a negative count value Nc. Since the luminance value per count is predetermined, the luminance value corresponding to the current count value Nc can be obtained by using the accumulated count value.

[0024] In another example, the imaging control unit 30 controls the imaging conditions based on the current luminance value and the past luminance value stored in the storage unit 60. In this case, the storage unit 60 holds the luminance value previously read from the AD conversion unit 50. In one example, the imaging control unit 30 controls the imaging conditions based on the luminance value output by the sensor 300 in response to the detection of the previous event and the event pulse output by the event detection unit 20 in response to the detection of a new event. For example, the imaging control unit 30 calculates the current luminance value by adding or subtracting the current luminance value per pulse to or from the luminance value stored in the storage unit 60. The imaging control unit 30 controls the imaging conditions based on the calculated current luminance value.

[0025] As described above, the control device 100 may control the imaging conditions of the current event based on both the luminance signal S of the past event and the pulse count value Nc during imaging. This makes it possible to update the imaging conditions following the change in the environment even if the environment of the current event changes suddenly from the environment of the past event. Note that the control device 100 may control the imaging conditions of the current event based only on the pulse count value Nc during imaging.

[0026] The control device 100 of this example can perform AD conversion with a necessary and sufficient resolution for the luminance value by changing the ADC resolution in response to the detection of an event. Therefore, it is possible to limit the output of unnecessary bits and compress the amount of data. In addition, the control device 100 can reduce power consumption by compressing the amount of data. The control device 100 can achieve a high dynamic range by optimally adjusting the ADC resolution.

[0027] 2 shows an example of a method for counting event pulses by the control device 100 according to the embodiment. The vertical axis indicates the luminance signal S, and the horizontal axis indicates the time T. In this example, the luminance signal S is used to detect an event, but other signals may be used to detect an event.

[0028] The event detection unit 20 generates an event pulse in response to a change in the luminance signal S. In one example, the event detection unit 20 sets one or more thresholds, and generates an event pulse when the luminance signal S exceeds the threshold. In this example, the event detection unit 20 sets thresholds Sth1 to Sth5, and generates an event pulse when the luminance signal S becomes one of the thresholds Sth1 to Sth5. That is, the event detection unit 20 generates an event pulse when the luminance signal S increases or decreases beyond any of the thresholds Sth1 to Sth5. The thresholds Sth1 to Sth5 are examples of event detection thresholds.

[0029] For example, the event detection unit 20 generates an up-count event pulse when any of the thresholds Sth1 to Sth5 is exceeded. On the other hand, the event detection unit 20 generates a down-count event pulse when any of the thresholds Sth1 to Sth5 is fallen below. Note that, although the thresholds Sth1 to Sth5 in this example are set at equal intervals, they do not have to be at equal intervals.

[0030] The pulse counting section 40 generates a count value Nc by counting the event pulses generated by the event detection section 20. The pulse counting section 40 counts at least one of up-counting and down-counting. For example, the pulse counting section 40 accumulates the up-counting and down-counting while taking into account the plus and minus. This makes it possible to detect whether the luminance signal S has increased or decreased by a predetermined value. In this example, the imaging control section 30 changes the imaging conditions when the count value Nc of the pulse counting section 40 reaches "±3". Note that the criterion for the count value Nc may be a value other than ±3.

[0031] Events E1 to E7 indicate event pulses generated during a predetermined period. Note that the imaging control unit 30 may output a reset signal RST to the pixel unit 10 each time an event occurs to start imaging.

[0032] In the event E1, an up-count event pulse is generated. That is, this indicates that any of the thresholds Sth1 to Sth5 has been exceeded once in the event E1. In this example, when the count value Nc of the pulse count unit 40 becomes +1 due to the detection of one up-count event pulse, the imaging conditions are not changed.

[0033] In the event E2, a down-counting event pulse is generated. That is, this indicates that in the event E2, the threshold value falls below any one of the threshold values ​​Sth1 to Sth5 once. In this example, when the count value Nc of the pulse counting unit 40 becomes −1 due to the detection of one down-counting event pulse, the imaging conditions are not changed.

[0034] Three up-count event pulses are generated in the events E3 to E5. That is, the events E3 to E5 exceed any of the thresholds Sth1 to Sth5 three times, causing the count value Nc of the pulse count unit 40 to become +3. Since the count value Nc of the pulse count unit 40 has reached +3, the imaging control unit 30 determines that the brightness of the imaging target has changed more than a predetermined amount, and changes the imaging conditions. For example, when the count value Nc of the pulse count unit 40 becomes +3 or more, the imaging control unit 30 lowers the ADC resolution.

[0035] More specifically, when the count value Nc of the pulse count unit 40 exceeds +n (n is a predetermined integer), AD conversion may be performed without using the predetermined lower bits. If the object to be imaged is bright, the output of the lower bits is buried in noise, so there is almost no effect on the obtained pixel signal even if the resolution of the luminance range detected by the sensor 300 is reduced. The predetermined lower bits are, for example, the two least significant bits.

[0036] On the other hand, when the count value Nc of the pulse counting unit 40 falls below -n, AD conversion may be performed without using the predetermined upper bits. If the imaging subject is dark, the upper bits become 0 and are unnecessary in the first place. Therefore, even if the upper bits are not used, there is no effect on the obtained pixel signal. The predetermined upper bits are, for example, the two most significant bits.

[0037] The control device 100 of this embodiment controls the imaging conditions by detecting an event according to a change in the luminance signal S. In this way, the control device 100 can update the imaging conditions in response to a high-speed change in the luminance signal S.

[0038] 3 shows the configuration of a sensor 500 according to a comparative example. The sensor 500 of this example includes a pixel section 510, an imaging control section 530, an AD conversion section 550, and a storage section 560.

[0039] The sensor 500 determines the ADC resolution of the AD conversion section 550 based on the pixel signal output by the pixel section 510. The sensor 500 may store the determined ADC resolution in the storage section 560.

[0040] The storage unit 560 outputs imaging information of a past frame as a feedback signal FB to the imaging control unit 530. The imaging control unit 530 outputs past imaging conditions to the AD conversion unit 550. Then, the AD conversion unit 550 controls the ADC resolution based on the past imaging conditions and the pixel signal from the pixel unit 510.

[0041] 4 shows an example of a method for determining the ADC resolution of the sensor 500 according to the comparative example. The sensor 500 of this example sets the ADC resolution based on a comparison between the luminance signal S and a luminance threshold value Sth.

[0042] The sensor 500 sets the ADC resolution in frame units of a predetermined length. That is, the sensor 500 compares the luminance signal S with the luminance threshold Sth for each frame. For example, when the luminance signal S is smaller than the luminance threshold Sth, the sensor 500 performs AD conversion of only the lower bits, and when the luminance signal S is larger than the luminance threshold Sth, the sensor 500 performs AD conversion of only the upper bits. In other words, when the luminance signal S is smaller than the luminance threshold Sth, AD conversion of the upper bits is not necessary, and when the luminance signal S is larger than the luminance threshold Sth, AD conversion of the lower bits is not necessary.

[0043] The sensor 500 calculates the ADC resolution of the current frame based on the luminance signal of the past frame. For example, the sensor 500 sets the ADC resolution of the current frame, frame F3, based on at least one of data of the past frames, frames F1 and F2. Therefore, when the luminance signal S increases rapidly between the past frame and the current frame and the ADC resolution is switched, it is difficult to set an appropriate ADC resolution. In the sensor 500 of this example, since the AD conversion of the lower bits is set in the past frame, even if the AD conversion of the higher bits should be selected in the current frame, the brightness of the current frame cannot be estimated from the data of the past frame, so the AD conversion of only the lower bits is performed.

[0044] In this way, since the sensor 500 adjusts the ADC resolution of the current frame based on the past frames, if the environment of the current frame changes suddenly from the environment of the past frames, it is difficult to optimally update the imaging conditions in accordance with the change in the environment. In other words, the control of the imaging conditions based on the event is affected by the imaging frame.

[0045] On the other hand, the control device 100 obtains the change in the luminance signal S as the count number of event pulses and controls the imaging conditions. Therefore, the control device 100 can quickly control the imaging conditions by following the change in the luminance signal S, not from the past imaging frame.

[0046] 5 shows an example of a flowchart of the operation of the control device 100. The control device 100 may use the flowchart of this example to control the imaging conditions of the sensor 300. In the flowchart of this example, the pixel 11 is reset every time an event pulse is generated.

[0047] The control device 100 monitors the luminance signal S output by the pixel 11 (S100). The control device 100 determines whether the luminance signal S has changed to or above a predetermined event detection threshold (S102). Specifically, the event detection unit 20 detects an event in which the luminance signal S changes beyond the event detection threshold in one or more pixels 11. If the luminance signal S does not change to or above the event detection threshold, the control device 100 continues to monitor the luminance signal S (S100). On the other hand, if the luminance signal S changes to or above the event detection threshold, the control device 100 generates an event pulse and outputs it to the pulse count unit 40 (S104).

[0048] Then, the imaging control unit 30 counts the event pulses from the event detection unit 20 (S106). The imaging control unit 30 controls the imaging conditions of the sensor 300 based on the detection of the event (S108). For example, when the counter value Nc calculated by the pulse count unit 40 reaches a preset reference, the imaging control unit 30 controls the imaging conditions based on the counter value Nc. The imaging conditions controlled by the imaging control unit 30 may be any of the ADC resolution, exposure time, and amplifier gain of the sensor 300. The control device 100 resets the charge accumulated in the pixel 11 (S110). After controlling the imaging conditions, the control device 100 starts exposure by the sensor 300 (S112). That is, the control device 100 of this example starts exposure after the imaging control unit 30 updates the imaging conditions. If at least one of the exposure time and the amplifier gain has been updated in step S108, the updated imaging conditions are used.

[0049] In this embodiment, when the event detection unit 20 detects an event during the exposure period, the imaging control unit 30 controls the imaging conditions again, resets one or more pixels 11, and restarts exposure by the sensor 300. For example, when an event occurs during the exposure period (S114), the event detection unit 20 starts generating an event pulse (S104). On the other hand, when no event occurs during the exposure period (S114), the control device 100 ends the exposure (S116).

[0050] After the exposure is completed, the AD conversion unit 50 performs AD conversion on the pixel signal output by the pixel unit 10 (S118). Here, if the imaging control unit 30 has updated the ADC resolution as an imaging condition in step S108, the pixel signal is AD converted based on the updated imaging condition. Thereafter, the storage unit 60 may store the converted digital signal and output the stored digital signal to the outside (S120).

[0051] Fig. 6 shows an example of a flowchart of the operation of the control device 100. The control device 100 may use the flowchart of this example to control the imaging conditions of the sensor 300. The flowchart of this example differs from the flowchart of Fig. 5 in that the pixel 11 is not reset between the calculation of the imaging conditions and the start of exposure. That is, the control device 100 of this example resets the pixel 11 once to update the imaging conditions, and then starts exposure without resetting the pixel 11.

[0052] Steps S200 and S202 correspond to steps S100 and S102 in FIG. 5, respectively. In step S202, if the luminance signal S changes to a value equal to or greater than a predetermined event detection threshold, the event detection unit 20 starts generating an event pulse (S204). Then, the imaging control unit 30 resets the pixel 11 (S206). Also, the pulse count unit 40 counts the generated event pulses (S208). Thereafter, when the count value Nc from the pulse count unit 40 reaches a preset reference, the imaging control unit 30 sets imaging conditions based on the count value Nc and controls the imaging of the control device 100 (S210).

[0053] The control device 100 starts exposure based on the updated imaging conditions (S212). That is, in the control device 100 of this example, the imaging control unit 30 controls the imaging conditions based on the count value Nc of the event pulses, and resets the pixel unit 10 before starting exposure.

[0054] If an event occurs during the exposure period (S214), the event detection unit 20 starts generating an event pulse (S216). Furthermore, the pulse count unit 40 counts the generated event pulses (S218). Thereafter, while continuing exposure by the sensor 300, when the count value Nc from the pulse count unit 40 reaches a preset reference, the imaging control unit 30 controls the imaging conditions based on the count value Nc (S220).

[0055] In this way, when the event detection unit 20 detects an event during the exposure period by the sensor 300, the imaging control unit 30 of this example controls the imaging conditions again and continues the exposure by the sensor 300. That is, the control device 100 may update the imaging conditions before the start of exposure, or may update the imaging conditions during exposure. On the other hand, if no event occurs during the exposure period after the start of exposure (S212) (S214), the exposure by the sensor 300 is ended (S222).

[0056] After the exposure is completed, the AD conversion unit 50 performs AD conversion on the pixel signal output by the pixel unit 10 (S224). If the imaging control unit 30 has updated the ADC resolution as an imaging condition in steps S210 and S220, the pixel signal is AD converted based on the latest ADC resolution. Thereafter, the storage unit 60 may store the converted digital signal and output the stored digital signal to the outside (S226).

[0057] As described above, after the control device 100 resets the pixel 11 once to update the imaging conditions, it can start exposure without resetting the pixel 11. Furthermore, the control device 100 may update the imaging conditions before the start of exposure or during the exposure period. This allows the control device 100 to realize imaging under optimal imaging conditions in response to the occurrence of an event.

[0058] 7 shows an example of a more specific configuration of the sensor 300. The sensor 300 of this example includes a first substrate 110, a second substrate 120, and a third substrate .

[0059] The first substrate 110 has a pixel block array 112. The pixel block array 112 has one or more pixels 11. In this example, the pixel block array 112 has M×N pixels 11. The pixel 11 has a first photoelectric conversion element PD1 and a second photoelectric conversion element PD2. For example, the first photoelectric conversion element PD1 is an event detection element that detects an event. Also, the second photoelectric conversion element PD2 is a light receiving element that captures an image.

[0060] The second substrate 120 has a processing block array 122. The processing block array 122 includes M×N data processing units 13. The second substrate 120 is laminated on the first substrate 110.

[0061] The data processing unit 13 includes an event detection unit 20, an imaging control unit 30, a pulse count unit 40, and an AD conversion unit 50. The M×N data processing units 13 are provided corresponding to the M×N pixels 11, respectively. Provided correspondingly means that the M×N data processing units 13 are electrically connected to the M×N pixels 11, respectively. Also, provided correspondingly may mean that the M×N data processing units 13 are provided facing the M×N pixels 11, respectively.

[0062] The data processing units 13 in this example execute data processing of the pixels 11 provided corresponding to them. When the data processing units 13 are provided directly under the pixels 11, the wiring is shortened and the processing speed can be improved. For example, at least one of the event detection unit 20 and the imaging control unit 30 is provided on the second substrate 120 in correspondence with a block consisting of one or more pixels 11. This allows the pixel aperture of the pixels 11 provided on the first substrate 110 to be made larger.

[0063] The third substrate 130 has an output block array 132. The output block array 132 includes a data holding section 70 and a data input / output section 80. The third substrate 130 is stacked on the second substrate 120. That is, the first substrate 110, the second substrate 120, and the third substrate 130 are stacked.

[0064] The data storage unit 70 stores imaging data of the sensor 300. For example, the data storage unit 70 temporarily stores imaging data acquired by the processing block array 122. The imaging data may include data on the imaging conditions and the like in addition to data on the image acquired by the pixel unit 10. The data input / output unit 80 outputs the imaging data stored in the data storage unit 70 to the outside. The data input / output unit 80 may also output the imaging data to the imaging control unit 30.

[0065] In this way, the sensor 300 of this example is provided with M×N imaging control units 30 corresponding to each of the M×N pixels 11. Therefore, the sensor 300 can control the imaging conditions for each of the M×N pixels 11. This allows the sensor 300 to set appropriate imaging conditions for each pixel 11 according to the luminance distribution in the screen. For example, by setting the ADC resolution for each of the M×N pixels 11, it is possible to reduce unnecessary bits and suppress the data output band compared to the case where AD conversion is performed with a uniform ADC resolution for all pixels. The sensor 300 of this example can reduce the amount of data while maintaining image quality by feeding back a necessary and sufficient bit depth for each of the M×N pixels 11 to the AD conversion unit 50.

[0066] 8 shows an example of a more specific configuration of the imaging control unit 30. The imaging control unit 30 includes a reset control unit 32, a gain control unit , an exposure control unit , and an ADC control unit .

[0067] The reset control unit 32 generates a reset signal RST for resetting the charge accumulated in the pixel 11. The reset control unit 32 outputs the generated reset signal RST to the pixel unit 10 based on the event detection signal Sd input from the event detection unit 20. For example, when operating based on the flowchart of Fig. 6, the reset control unit 32 outputs the reset signal RST to the pixel unit 10 when an event detection signal Sd indicating that an event has occurred is input.

[0068] The gain control unit 34 controls the gain of the amplifier included in the sensor 300 based on the count value Nc input from the pulse count unit 40. For example, the gain control unit 34 reduces the gain from the previous value when the count value Nc exceeds a predetermined threshold value, and increases the gain from the previous value when the count value Nc falls below the predetermined threshold value.

[0069] The exposure control unit 36 ​​controls exposure conditions such as exposure time based on the count value Nc input from the pulse count unit 40. For example, the exposure control unit 36 ​​shortens the exposure time when the count value Nc exceeds a predetermined threshold, and lengthens the exposure time when the count value Nc falls below a predetermined threshold. The exposure control unit 36 ​​may also lengthen the exposure time when the brightness of the image falls below a predetermined threshold, and shorten the exposure time when the brightness of the image exceeds a predetermined threshold.

[0070] The ADC control unit 38 controls the ADC resolution based on the count value Nc input from the pulse count unit 40. For example, the ADC control unit 38 lowers the ADC resolution when the count value Nc exceeds a predetermined threshold, and raises the ADC resolution when the count value Nc falls below the predetermined threshold. The ADC control unit 38 may raise the ADC resolution when the luminance of the image falls below a predetermined threshold, and lower the ADC resolution when the luminance of the image exceeds the predetermined threshold. The gain control by the gain control unit 34, the exposure time control by the exposure control unit 36, and the ADC resolution control by the ADC control unit 38 may be controlled in combination. In this case, the same count value Nc may be set as a threshold value to be referred to by these control units, or different count values ​​Nc may be set.

[0071] 9 illustrates an example of a computer 1200 in which aspects of the present invention may be embodied in whole or in part. Programs installed on the computer 1200 may cause the computer 1200 to function as or perform operations associated with an apparatus or one or more sections of the apparatus according to an embodiment of the present invention, and / or to perform a process or steps of a process according to an embodiment of the present invention. Such programs may be executed by the CPU 1212 to cause the computer 1200 to perform specific operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

[0072] The computer 1200 according to this embodiment includes a CPU 1212, a RAM 1214, a graphic controller 1216, and a display device 1218, which are connected to each other by a host controller 1210. The computer 1200 also includes input / output units such as a communication interface 1222, a hard disk drive 1224, a DVD-ROM drive 1226, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The computer also includes legacy input / output units such as a ROM 1230 and a keyboard 1242, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0073] The CPU 1212 operates according to a program stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphic controller 1216 retrieves image data generated by the CPU 1212 into a frame buffer or the like provided in the RAM 1214 or into the graphic controller 1216 itself, and causes the image data to be displayed on the display device 1218.

[0074] The communication interface 1222 communicates with other electronic devices via a network. The hard disk drive 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD-ROM drive 1226 reads programs or data from the DVD-ROM 1201 and provides the programs or data to the hard disk drive 1224 via the RAM 1214. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0075] The ROM 1230 stores therein a boot program and the like that is executed by the computer 1200 upon activation, and / or programs that depend on the hardware of the computer 1200. The input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via a parallel port, a serial port, a keyboard port, a mouse port, and the like.

[0076] The programs are provided by a computer-readable medium such as a DVD-ROM 1201 or an IC card. The programs are read from the computer-readable medium, installed in the hard disk drive 1224, the RAM 1214, or the ROM 1230, which are also examples of computer-readable media, and executed by the CPU 1212. Information processing described in these programs is read by the computer 1200, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be constructed by realizing information manipulation or processing according to the use of the computer 1200.

[0077] For example, when communication is performed between the computer 1200 and an external device, the CPU 1212 may execute a communication program loaded in the RAM 1214 and instruct the communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of the CPU 1212, the communication interface 1222 reads transmission data stored in a transmission buffer processing area provided in the RAM 1214, the hard disk drive 1224, the DVD-ROM 1201, or a recording medium such as an IC card, and transmits the read transmission data to a network, or writes reception data received from the network to a reception buffer processing area or the like provided on the recording medium.

[0078] Furthermore, the CPU 1212 may cause all or a necessary portion of a file or database stored in an external recording medium such as a hard disk drive 1224, a DVD-ROM drive 1226 (DVD-ROM 1201), an IC card, etc. to be read into the RAM 1214, and perform various types of processing on the data on the RAM 1214. The CPU 1212 then writes back the processed data to the external recording medium.

[0079] Various types of information, such as various types of programs, data, tables, and databases, may be stored in the recording medium and undergo information processing. The CPU 1212 may perform various types of processing on the data read from the RAM 1214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., as described throughout this disclosure and specified by the instruction sequence of the program, and write back the results to the RAM 1214. The CPU 1212 may also search for information in a file, database, etc. in the recording medium. For example, when a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 1212 may search for an entry that matches a condition, in which the attribute value of the first attribute is specified, from among the plurality of entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0080] The above-described programs or software modules may be stored in a computer-readable medium on the computer 1200 or in the vicinity of the computer 1200. Also, a recording medium such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable medium, thereby providing the programs to the computer 1200 via the network.

[0081] It should be noted that the order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before," "prior to," etc., and may be realized in any order unless the output of a previous process is used in a later process. Even if the operational flow in the claims, specifications, and drawings is explained using "first," "next," etc. for convenience, it does not mean that it is essential to perform the process in this order.

[0082] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is clear to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the description of the claims that such modifications and improvements can also be included in the technical scope of the present invention. (Item 1) A control device for controlling an imaging condition of a sensor having one or more pixels, an event detection unit that detects an event indicating that a luminance signal changes beyond a predetermined threshold in the one or more pixels and outputs an event detection signal; a control unit that controls an imaging condition of the sensor based on the event detection signal; A control device comprising: (Item 2) The imaging conditions include at least one of an ADC resolution, an exposure time, and an amplifier gain of the sensor; Item 1. The control device according to item 1. (Item 3) a pulse counter that counts the event pulses output by the event detection unit upon detection of the event, The control unit controls the imaging conditions based on the number of occurrences of the event pulses. 3. The control device according to item 1 or 2. (Item 4) the control unit controls the imaging conditions based on a luminance value output by the sensor in response to a previous detection of the event and an event pulse output by the event detection unit in response to a new detection of the event. 3. The control device according to item 1 or 2. (Item 5) When the event detection unit detects an event during an exposure period by the sensor, the control unit controls the imaging conditions again to reset accumulation of charge in the one or more pixels and restart exposure by the sensor. 5. The control device according to any one of items 1 to 4. (Item 6) When the event detection unit detects an event during an exposure period by the sensor, the control unit controls the imaging conditions again and continues the exposure by the sensor. 5. The control device according to any one of items 1 to 4. (Item 7) One or more pixels; A control device according to any one of items 1 to 6; A sensor comprising: (Item 8) The one or more pixels include an event detection element that detects the event and a light receiving element that captures an image. Item 7. The sensor according to item 7. (Item 9) a first substrate on which the one or more pixels are provided; Another substrate laminated on the first substrate; Equipped with At least one of the event detection unit and the control unit is provided on the other substrate in correspondence with the block of one or more pixels. Item 9. The sensor according to item 7 or 8. (Item 10) a first substrate on which the one or more pixels are provided; a second substrate laminated on the first substrate, the second substrate having the event detection unit and the control unit provided thereon; a third substrate laminated on the second substrate and provided with a data storage unit that stores imaging data of the sensor; Equipped with Item 9. The sensor according to item 7 or 8. (Item 11) A method for controlling an imaging condition of a sensor having one or more pixels, comprising: detecting an event indicative of a luminance signal changing beyond a predetermined threshold at the one or more pixels; Controlling an imaging condition of the sensor based on the detection of the event; A control method comprising: (Item 12) starting exposure by the sensor after controlling the imaging conditions; re-controlling the imaging conditions when the event is detected during an exposure period by the sensor; resetting the accumulation of charge in the sensor and restarting exposure under the re-controlled imaging conditions; Item 12. The control method according to item 11, comprising: (Item 13) starting exposure by the sensor after controlling the imaging conditions; when the event is detected during an exposure period by the sensor, the imaging condition is controlled again while continuing the exposure by the sensor; Item 12. The control method according to item 11, comprising: (Item 14) A program for controlling imaging conditions of a sensor having one or more pixels, detecting an event in which a luminance signal changes beyond a predetermined threshold in the one or more pixels; Controlling an imaging condition of the sensor based on the detection of the event; A control device program that causes a computer to execute the above. [Explanation of symbols]

[0083] 10 pixel section, 11 pixel, 12 block, 13 data processing section, 20 event detection section, 30 imaging control section, 32 reset control section, 34 gain control section, 36 exposure control section, 38 ADC control section, 40 pulse count section, 50 AD conversion section, 60 memory section, 70 data retention section, 80 data input / output section, 100 control device, 110 first substrate, 112 pixel block array, 120 second substrate, 122 processing block array, 130 third substrate, 132 output block array, 300 cell sensor, 500, sensor, 510, pixel section, 530, imaging control section, 550, AD conversion section, 560, storage section, 1200, computer, 1201, DVD-ROM, 1210, host controller, 1212, CPU, 1214, RAM, 1216, graphic controller, 1218, display device, 1220, input / output controller, 1222, communication interface, 1224, hard disk drive, 1226, DVD-ROM drive, 1230, ROM, 1240, input / output chip, 1242, keyboard

Claims

1. a first substrate having a first pixel block including a first photoelectric conversion element that converts light into an electric charge and a second photoelectric conversion element that converts light into an electric charge, and a second pixel block including a third photoelectric conversion element that converts light into an electric charge and a fourth photoelectric conversion element that converts light into an electric charge; a second substrate laminated on the first substrate, the second substrate having a first processing block including a first event detection unit that detects an event using a signal based on the charge converted by the first photoelectric conversion element and a first control unit that performs exposure control of the second photoelectric conversion element based on the detection result of the first event detection unit, and a second processing block including a second event detection unit that detects an event using a signal based on the charge converted by the third photoelectric conversion element and a second control unit that performs exposure control of the fourth photoelectric conversion element based on the detection result of the second event detection unit; A sensor comprising:

2. 2. The sensor according to claim 1, the first control unit controls an exposure time of the second photoelectric conversion element based on a detection result of the first event detection unit; The second control unit controls an exposure time of the fourth photoelectric conversion element based on a detection result of the second event detection unit. Sensor.

3. The sensor according to claim 1 or 2, the first processing block includes a first counting unit that counts the events detected by the first event detection unit; the second processing block includes a second counting unit that counts the events detected by the second event detection unit; the first control unit performs exposure control of the second photoelectric conversion element based on a count result of the first count unit; The second control unit performs exposure control of the fourth photoelectric conversion element based on a count result of the second counting unit. Sensor.

4. The sensor according to any one of claims 1 to 3, the first processing block includes a first signal processing unit that performs signal processing on a signal based on the charge converted by the second photoelectric conversion element; the second processing block includes a second signal processing unit that performs signal processing on a signal based on the charge converted by the fourth photoelectric conversion element; Sensor.

5. 5. The sensor according to claim 4, the first signal processing unit includes a first conversion unit that converts a signal based on the charge converted by the second photoelectric conversion element into a digital signal; the second signal processing unit includes a second conversion unit that converts a signal based on the charge converted by the fourth photoelectric conversion element into a digital signal; Sensor.

6. 6. The sensor according to claim 5, A sensor comprising a third substrate laminated with the first substrate and having a holding portion that holds a first digital signal converted into a digital signal from a signal based on charges converted by the second photoelectric conversion element in the first conversion portion, and a second digital signal converted into a digital signal from a signal based on charges converted by the fourth photoelectric conversion element in the second conversion portion.

7. 7. The sensor according to claim 6, the third substrate has an output section that outputs the first digital signal held in the holding section and the second digital signal held in the holding section; Sensor.

8. The sensor according to any one of claims 4 to 7, the first control unit controls the first signal processing unit based on a detection result of the first event detection unit; The second control unit controls the second signal processing unit based on a detection result of the second event detection unit. Sensor.

9. The sensor according to any one of claims 1 to 8, the first processing block is disposed at a position facing the first pixel block in a stacking direction in which the first substrate and the second substrate are stacked, the second processing block is disposed at a position facing the second pixel block in the stacking direction. Sensor.

10. The sensor according to any one of claims 1 to 9, The second pixel block is arranged next to the first pixel block in the row direction. Sensor.

11. The sensor according to any one of claims 1 to 9, the second pixel block is arranged next to the first pixel block in the column direction; Sensor.

12. 2. The sensor according to claim 1, the first substrate has a third pixel block including a fifth photoelectric conversion element that converts light into an electric charge and a sixth photoelectric conversion element that converts light into an electric charge; the second substrate has a third processing block including a third event detection unit that detects an event by using a signal based on the charge converted by the fifth photoelectric conversion element, and a third control unit that performs exposure control of the sixth photoelectric conversion element based on a detection result of the third event detection unit; Sensor.

13. 13. The sensor of claim 12, the first control unit controls an exposure time of the second photoelectric conversion element based on a detection result of the first event detection unit; the second control unit controls an exposure time of the fourth photoelectric conversion element based on a detection result of the second event detection unit; the third control unit controls an exposure time of the sixth photoelectric conversion element based on a detection result of the third event detection unit. Sensor.

14. The sensor according to claim 12 or 13, the first processing block includes a first counting unit that counts the events detected by the first event detection unit; the second processing block includes a second counting unit that counts the events detected by the second event detection unit; the third processing block includes a third counting unit that counts the events detected by the third event detection unit; the first control unit performs exposure control of the second photoelectric conversion element based on a count result of the first count unit; the second control unit performs exposure control of the fourth photoelectric conversion element based on a count result of the second counting unit; The third control unit performs exposure control of the sixth photoelectric conversion element based on a count result of the third counting unit. Sensor.

15. The sensor according to any one of claims 12 to 14, the first processing block includes a first signal processing unit that performs signal processing on a signal based on the charge converted by the second photoelectric conversion element; the second processing block includes a second signal processing unit that performs signal processing on a signal based on the charge converted by the fourth photoelectric conversion element; the third processing block includes a third signal processing unit that performs signal processing on a signal based on the charge converted by the sixth photoelectric conversion element; Sensor.

16. 16. The sensor of claim 15, the first signal processing unit includes a first conversion unit that converts a signal based on the charge converted by the second photoelectric conversion element into a digital signal; the second signal processing unit includes a second conversion unit that converts a signal based on the charge converted by the fourth photoelectric conversion element into a digital signal; The third signal processing unit includes a third conversion unit that converts a signal based on the charge converted by the sixth photoelectric conversion element into a digital signal. Sensor.

17. 17. The sensor of claim 16, A sensor comprising a third substrate, the third substrate being a substrate stacked on the first substrate and having a holding portion for holding a first digital signal obtained by converting a signal based on charges converted by the second photoelectric conversion element in the first conversion unit into a digital signal, a second digital signal obtained by converting a signal based on charges converted by the fourth photoelectric conversion element in the second conversion unit into a digital signal, and a third digital signal obtained by converting a signal based on charges converted by the sixth photoelectric conversion element in the third conversion unit into a digital signal.

18. 18. The sensor of claim 17, the third substrate has an output section that outputs the first digital signal held in the holding section, the second digital signal held in the holding section, and the third digital signal held in the holding section. Sensor.

19. A sensor according to any one of claims 15 to 18, the first control unit controls the first signal processing unit based on a detection result of the first event detection unit; The second control unit controls the second signal processing unit based on a detection result of the second event detection unit, The third control unit controls the third signal processing unit based on a detection result of the third event detection unit. Sensor.

20. A sensor according to any one of claims 12 to 19, the first processing block is disposed at a position facing the first pixel block in a stacking direction in which the first substrate and the second substrate are stacked, the second processing block is disposed at a position facing the second pixel block in the stacking direction, the third processing block is disposed at a position facing the third pixel block in the stacking direction. Sensor.

21. A sensor according to any one of claims 12 to 20, The second pixel block is arranged next to the first pixel block in the row direction. Sensor.

22. 22. The sensor of claim 21, the third pixel block is arranged alongside the first pixel block in the row direction; Sensor.

23. 22. The sensor of claim 21, the third pixel block is arranged alongside the first pixel block in the column direction; Sensor.

24. A sensor according to any one of claims 12 to 20, the second pixel block is arranged next to the first pixel block in the column direction; Sensor.

25. 25. The sensor of claim 24, the third pixel block is arranged alongside the first pixel block in the column direction; Sensor.

Images