Photoelectric conversion device and equipment

Abstract

To provide a technique that can detect an event and obtain an image signal in parallel in time, and can suppress degradation of the image signal.SOLUTION: A photoelectric conversion device includes a plurality of pixel blocks arranged therein, each of which includes a plurality of first pixels, a determination unit, and an adder unit. Each of the plurality of first pixels has a first detection unit that detects light and outputs a detection signal, and a first counter that counts the detection signals output during a first time period and outputs a count value. The determination unit detects an event on the basis of the count value of the first counter, and the adder unit integrates the output of the first counter over a second time period and outputs the result as an image signal.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a photoelectric conversion device and an apparatus. [Background technology]

[0002] With the recent spread of IoT, AI, and autonomous driving, there is a demand for image sensors with lower power consumption and higher speed. One such sensor proposed is an asynchronous sensor called an event-based sensor or dynamic vision sensor. Patent Document 1 describes a solid-state imaging element in which pixels that detect address events and pixels that output pixel signals are arranged. Patent Document 2 describes a solid-state imaging device that reads out pixel signals based on the detection result of an address event. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2020-096347 [Patent Document 2] Patent Publication No. 2020-057949 Summary of the Invention [Problem to be solved by the invention]

[0004] In the configuration of Patent Document 1, it is not possible to output a pixel signal from a pixel that detects an address event, and in the configuration of Patent Document 2, the pixel signal is read out based on the result of detecting an address event, so it is not possible to detect an address event and read out a pixel signal at the same time.

[0005] An object of the present invention is to provide a technique that can detect an event and acquire an image signal in parallel in time and suppress degradation of the image signal. [Means for solving the problem]

[0006] The photoelectric conversion device of the present invention is characterized in that a plurality of pixel blocks are arranged, each of which includes a plurality of first pixels, a judgment unit, and an adder unit, and each of the plurality of first pixels has a first detection unit that detects light and outputs a detection signal, and a first counter that counts the detection signals output per first time period and outputs a count value, the judgment unit detects an event based on the count value of the first counter, and the adder unit accumulates the output of the first counter over a second time period and outputs the result as an image signal. Effect of the Invention

[0007] According to the present invention, it is possible to provide a technique that can detect an event and acquire an image signal in parallel in time and suppress degradation of the image signal. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of a photoelectric conversion device according to a first embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of the configuration of a pixel block according to the first embodiment. [Diagram 3] FIG. 4 is a diagram showing an example of the arrangement of color filters of pixels in a pixel block according to the first embodiment. [Figure 4] FIG. 2 is a diagram showing an example of pixel arrangement in a pixel block according to the first embodiment. [Diagram 5] FIG. 2 is a diagram showing an example of a circuit configuration of a pixel according to the first embodiment. [Figure 6] FIG. 4 is a diagram showing an example of memory allocation in a storage unit according to the first embodiment; [Figure 7] FIG. 4 is a timing chart showing the operation of the first embodiment. [Figure 8] FIG. 11 is a diagram showing an example of pixel arrangement according to the second embodiment. [Figure 9] FIG. 11 is a diagram showing an example of memory allocation in a storage unit according to a second embodiment. [Figure 10] Timing diagram showing the operation of the second embodiment [Figure 11] Flowchart diagram according to the third embodiment [Figure 12] FIG. 13 is a diagram showing an example of the configuration of a pixel block according to the fourth embodiment. [Figure 13] FIG. 13 is a diagram showing an example of a circuit configuration of a pixel according to Example 4. [Figure 14] FIG. 13 is a diagram showing an example of memory allocation in a storage unit according to a fourth embodiment. [Figure 15] FIG. 13 is a timing chart showing the operation of the fourth embodiment. [Figure 16] FIG. 13 is a diagram showing an example of the configuration of a pixel block according to the fifth embodiment. [Figure 17] 13 shows an example of application of the photoelectric conversion device according to the embodiment to equipment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0010] In the following embodiments, an image capture device capable of detecting an event will be described as an example of a photoelectric conversion device. However, the photoelectric conversion device in each embodiment is not limited to an image capture device, and may be applied to other photodetection devices based on photoelectric conversion. Examples of other photodetection devices include a distance measuring device and a focus detection device.

[0011] <Example 1> (Configuration of photoelectric conversion device) The configuration of the photoelectric conversion device 10 will be described with reference to FIG. 1. The photoelectric conversion device 10 includes a pixel array section 120, in which a plurality of pixel blocks 130 are arranged in a matrix. In the pixel block 130, pixels (denoted by P(i,j)) are arranged in a matrix. In this example, the pixels are arranged in a matrix of 4 rows by 4 columns in the pixel block 130, but this is not limiting. The pixels may be arranged in 16 rows by 16 columns, or 4 rows by 2 columns. The pixel control section 110 can scan the pixel array and control reading of signals from the pixels. The post-processing section 140 can rearrange the signals read from the pixels and perform signal correction processing. The output IF section 150 can output the post-processed signal to the outside of the photoelectric conversion device 10.

[0012] (Pixel block configuration) Next, the configuration of the pixel block 130 will be described with reference to Fig. 2. The pixel block 130 can include a pixel unit 210, a determination unit 200 that detects events, and an output interface (IF) unit 270. The determination unit 200 can include a memory unit 220, an adder unit 230, a subtracter unit 240, a threshold control unit 250, a comparator unit 260, and a memory management unit 280.

[0013] Each of the pixel units 210 shown in FIG. 2 represents one unit of a pixel. When pixels are arranged in a matrix of 4 rows and 4 columns in the pixel block 130, there are 16 pixel units 210 in the pixel block 130. There are two types of pixel units 210: pixels for generating frame images (hereinafter referred to as "F pixels") and pixels for generating frame images and detecting events (hereinafter referred to as "H pixels"). Event detection here includes determining whether the output value of a pixel according to the amount of incident light has changed more than a predetermined value. In event detection, for example, when the difference value between the amount of incident light and a reference value for event detection at a certain time exceeds a predetermined threshold, it is determined that an event has occurred and an event detection signal is generated. The F pixels and H pixels are provided with counters for acquiring signals for event detection and frame image generation, as described later. The counters of the H pixels and F pixels are controlled by an event counter reset signal RES_EV and an image counter reset signal RES_FL, and can output count values ​​according to the incident light according to the period of each reset signal. The count value output from the pixel unit 210 is input to the storage unit 220, the adder unit 230, the subtracter unit 240, and the IF unit 270, respectively.

[0014] Here, the configuration of the pixel section 210 will be described using an example of the circuit of the pixel section 210 shown in FIG. 5. A circuit for detecting light using a SPAD (Single Photon Avalanche Diode) is shown here. The photoelectric conversion element 510 to which light is incident is, for example, an avalanche photodiode (hereinafter referred to as "APD"). The photoelectric conversion element 510 is connected to a quenching element 520. In this example, a first power supply voltage Vd is applied to the cathode of the APD via the quenching element 520, and a second power supply voltage Vg is applied to the anode of the APD. A pixel control signal CLKB is input to a control terminal of the quenching element 520. Here, the APD is set to operate in Geiger mode. In this example, a PMOS transistor is used for the quenching element 520, and the quenching element is configured using the resistance component of the transistor.

[0015] In the Geiger mode, a predetermined bias voltage is applied to the gate of the PMOS transistor as the pixel control signal CLKB. When a photon is incident on the APD, which is the photoelectric conversion element 510, an avalanche current flows due to avalanche multiplication. When the avalanche current flows, the cathode voltage of the APD drops from the first power supply voltage due to the resistance component of the quenching element 520. When the cathode voltage drops and becomes equal to or lower than the breakdown voltage of the APD, the cathode voltage rises again. This change in the cathode voltage is input to the counter 530 as a light detection signal by the inverter 540.

[0016] The input of the inverter 540 is at high level when no photons are incident, but is at low level when a photon is incident, so that a photodetection signal corresponding to the incident photon is output from the inverter 540. The circuit that generates the photodetection signal, including the photoelectric conversion element 510, the quenching element 520, and the inverter 540, functions as a light detection unit.

[0017] The counter 530 counts the number of photodetection signals, thereby counting the number of photons incident on the APD within a set unit time. The counter 530 is controlled by a signal whose period is the unit time, and a count value corresponding to the number of photons incident per unit time is output from the pixel unit 210. The counter 530 is also reset by a signal whose period is the unit time, so that it can start the next count.

[0018] A signal for detecting an event and a signal for outputting an image signal can be supplied to the pixel as signals having a period of unit time. The signal for resetting the counter 530 when an event is detected is indicated as an event counter reset signal RES_EV. The signal for resetting the counter 530 when an image signal is output is indicated as an image counter reset signal RES_FL. The event counter reset signal RES_EV and the image counter reset signal RES_FL can have different periods.

[0019] Returning to FIG. 2, the pixel block 130 will now be described. The storage unit 220 has a memory area for saving a reference value for detecting an event and a memory area for generating a frame image. The adder 230 adds the count value counted and output by the counter 530 controlled by the event counter reset signal RES_EV to the data stored in the memory area for generating a frame image of the storage unit 220. The subtracter 240 subtracts the reference value for detecting an event stored in the storage unit 220 from the count value output from the counter 530 controlled by the event counter reset signal RES_EV and outputs the result. The occurrence of an event can be detected from the amount of change in the count value from the reference value.

[0020] The threshold control unit 250 can control the threshold value when determining the occurrence of an event. The threshold control unit 250 can control the threshold value according to the driving mode of the photoelectric conversion device, such as the amount of incident light, the time during which the reference value is stored, and the detection frequency. When the change in the signal obtained by subtraction by the subtraction unit 240 exceeds the threshold value, it is determined that an event has occurred, and this state is called an ON-event. When the change is below the threshold value, the event has not occurred, and this state is called an OFF-event. The threshold value can also be controlled according to the frequency of occurrence of the event or the number of occurrences. By providing the threshold control unit 250 for each pixel block 130, it is possible to appropriately control the threshold value and detect the occurrence of an event with high accuracy. The comparison unit 260 compares the output value of the subtraction unit 240 with the threshold value, and outputs the event detection signal EV_OUT when a change exceeding the threshold value is detected. The event detection signal EV_OUT and the image signal can be output to the outside of the pixel block 130 via the IF unit 270.

[0021] (Storage part) Next, the storage unit 220 will be described with reference to Fig. 6. Fig. 6 is a diagram showing memory allocation. The storage unit 220 is allocated with an event memory area MEM_EV that stores a reference value for an event when detecting an event, and an image memory area MEM_FL that stores a count value accumulated for generating an image signal. The event memory area MEM_EV and the image memory area MEM_FL can be provided in the storage unit 220 corresponding to each of the H pixels arranged in a pixel block.

[0022] The event memory area MEM_EV is a memory for storing a reference value for determining whether an event has been detected, and stores the count value of H pixels at a certain time. The reference value stored in the event memory area MEM_EV is updated under the control of the memory management unit 280 when an event is detected. The updating will be described later with reference to FIG. 7. The image memory area MEM_FL is a memory for generating data for a frame image based on the count value of the counter 530 controlled by the event counter reset signal RES_EV from the H pixels. In the image memory area MEM_FL, the count values ​​read out at the cycle of the event counter reset signal RES_EV during one cycle of the image counter reset signal RES_FL are sequentially accumulated. The data in the image memory area MEM_FL is output to the IF unit 270 at the cycle of the image counter reset signal RES_FL. When the data is output, the image memory area MEM_FL is reset. Note that the updating of the reference value by the memory management unit 280 is just one example. As another example, the reference value stored in the event memory area MEM_EV may be updated using the count value CNT_HPIX of H pixels and the event detection signal EV_OUT.

[0023] Here, the period for detecting an event can usually be made shorter than the period for acquiring an image signal, and the period for the event counter reset signal RES_EV can be made shorter than the period for the image counter reset signal RES_FL. Therefore, assuming that the bit length of the counter 530 in FIG. 5 is 11 bits, the data length of the output from the F pixel can be 11 bits, whereas the data length of the output from the H pixel can be 8 bits. Therefore, when the data length of the image signal of the F pixel is 11 bits, the data length of the image memory in the image memory area MEM_FL needs to be 11 bits, but the event memory area MEM_EV can be 8 bits. However, the ratio and capacity of the event memory area and image memory area in the storage unit can be changed according to the period for detecting an event, the period of the image signal, and the ratio of F pixels to H pixels in the pixel block.

[0024] (Pixel arrangement example) An example of arranging color filters in pixels in a pixel block 130 is shown in Fig. 3. In Fig. 3, R, G, and B indicate the red, green, and blue color filters, and the pixels in which these filters are arranged are pixels that mainly capture images. Fig. 3(A) shows an arrangement for detecting grayscale when the filters of all pixels are transparent. Fig. 3(B) shows an example of a Bayer array. Fig. 3(C) shows an example of an RCCB array in which the green G of the Bayer array is replaced with a transparent C.

[0025] Next, an example of the arrangement of F pixels and H pixels will be described with reference to Fig. 4. In this embodiment, the F pixels output count values ​​to the IF unit 270 for each image read cycle. The H pixels output count values ​​to the storage unit 220, the adder 230, and the subtracter 240 for each event detection cycle. The arrangement of F pixels and H pixels is not limited to that shown in Fig. 4. For example, the pixel block 130 may have one H pixel, or all pixels may be H pixels. Furthermore, the arrangement and number of F pixels and H pixels may be changed for each event detection, depending on changes in the amount of incident light, information on defective pixels, and changes in the driving method for the H pixels and F pixels.

[0026] (Timing chart) 7 is a timing chart showing the operation of the first embodiment. Time t700 is the start time of resetting the counter for event detection and the counter for generating a frame image. The resetting of the counter is controlled by the event counter reset signal RES_EV and the image counter reset signal RES_FL. At time t700, the event counter reset signal RES_EV and the image counter reset signal RES_FL go from low level to high level.

[0027] The count value of an H pixel is, for example, 8 bits long. The count value of an H pixel CNT_HPIX[8] is reset by an event counter reset signal. The count value of an F pixel is, for example, 11 bits long. The count value of an F pixel CNT_FPIX

[11] and the data in the image memory area MEM_FL

[11] for generating a frame image are reset by an image counter reset signal RES_FL. Note that the numbers in [ ] above are examples of data bit lengths.

[0028] At time t701, the event counter reset signal RES_EV and the image counter reset signal RES_FL go from high to low, the counter reset is released, and the event detection period and the period for generating a frame image are started. When photons are incident on the APD in the pixel unit 210, the counter 530 counts the light detection signal, and a count value according to the amount of light incident during the event detection period and the period for generating a frame image is output. In this way, the counter of the H pixel counts the number of photons during the event detection period controlled by the event counter reset signal RES_EV. Meanwhile, the counter of the F pixel counts the number of photons during the period for generating a frame image at a cycle controlled by the image counter reset signal RES_FL.

[0029] At time t702, the accumulation enable ADD_EN input to the adder 230 goes from low to high. As a result, the adder 230 adds the count value CNT_EV[8] of the H pixels to the data stored in the image memory area MEM_FL

[11] of the memory to perform accumulation, updates the data, and saves it in the image memory area MEM_FL

[11] . The count value CNT_EV[8] of the H pixels is accumulated in the image memory area MEM_FL

[11] for each event period.

[0030] It is determined whether an event has occurred at time t703. The H pixel count value at the reference time is held in advance in the event memory area MEM_EV[8] as a reference value. The subtraction unit 240 calculates the difference between the H pixel count value CNT_HPIX[8] and the event memory area MEM_EV[8] (i.e., CNT_HPIX[8] - MEM_EV[8]) and outputs it to the comparison unit 260. The comparison unit 260 compares the difference value with a threshold value output from the threshold control unit 250. If the difference value is greater than the threshold value, it is determined that an event has occurred as an ON event. If an event has occurred, the event detection signal EV_OUT switches from low level to high level, and the comparison unit 260 outputs the event detection signal EV_OUT (high level). When the event detection signal EV_OUT becomes high level, the H pixel count value CNT_HPIX[8] corresponding to the detection of the event is stored in the event memory area MEM_EV[8] as a new reference value. Note that positive and negative thresholds may be prepared, and an event may be detected when the value is greater than the positive or negative threshold.

[0031] Time t704 is a period in which the counter for detecting an event of the H pixels is reset again. The event counter reset signal RES_EV goes from low to high, the count value CNT_HPIX[8] of the H pixels is reset, and the H pixels move to the next period for detecting an event. Meanwhile, the count value of the H pixels output from the counter for each period for detecting an event is accumulated in the image memory area MEM_FL

[11] for generating an image signal. The accumulation is performed until time t705, when the image memory area MEM_FL

[11] is reset by the image counter reset signal RES_FL. The data stored in the image memory area is data corresponding to the count value counted by the counter of the F pixels during the same period. In other words, the image memory area MEM_FL

[11] holds data equivalent to the image signal of the F pixels.

[0032] At time t705, the count period of the counter for generating a frame image ends. The count value CNT_FPIX

[11] of the F pixel is input from the pixel unit 210 to the IF unit 270 as an image signal of the F pixel, and is output to the outside. In addition, data of the image memory area MEM_FL

[11] is also input to the IF unit 270, and is output to the outside as an image signal of the H pixel.

[0033] In this way, event detection and image signal acquisition at the H pixels can be performed in parallel at the same time as image signal acquisition at the F pixels. The image signal output from the pixel block 130 can include image information from not only the F pixels but also the H pixels. Since image signals can be acquired from the H pixels at the same time as event detection, there is no loss of image signals as occurs when pixels for event detection are arranged. This eliminates the need for pixel signal complementation processing. This can reduce the processing load and suppress deterioration of image quality. In addition, event detection can be performed even while counting image signals. This can prevent event detection from becoming discontinuous in time, improving event detection accuracy.

[0034] <Example 2> The following mainly describes the differences from the first embodiment, and omits a description of the parts that overlap with the first embodiment. In this embodiment, the allocation of the memory area of ​​the storage unit to the H pixels can be changed to increase the number of pixels for which event detection is performed in the pixel block 130. This makes it possible to increase the resolution of event detection.

[0035] Fig. 8 shows an example of pixel allocation within a pixel block in the second embodiment. Here, the pixels indicated by E are pixels for detecting an event, and the H pixels are allocated for event detection (hereinafter referred to as "E pixels"). In other words, Fig. 8 shows an example in which the H pixels and some of the F pixels are replaced with E pixels in the pixel arrangement shown in Fig. 4. In the example of Fig. 8, the number of pixels capable of detecting an event is doubled compared to the example shown in Fig. 4.

[0036] FIG. 9 is a diagram showing memory allocation of the storage unit 220 in the second embodiment. In addition to the event memory area MEM_EV1 that stores the reference value in the storage unit 220, the image memory area MEM_FL

[11] that was allocated for integrating the image signal is allocated to the additional event memory area MEM_EV2. The memory area allocation can be changed by the memory management unit 280. The event memory area MEM_EV1 and the additional event memory area MEM_EV2 are memory areas that store reference values ​​for different E pixels when performing event detection. If the two E pixels are called E1 pixels and E2 pixels, the event memory area MEM_EV1 and the additional event memory area MEM_EV2 can store the reference value of the E1 pixel and the reference value of the E2 pixel, respectively. When the occurrence of an event is detected, the event detection signal EV_OUT1 of the E1 pixel or the event detection signal EV_OUT2 of the E2 pixel is output. The data of the event memory area MEM_EV1 and the additional event memory area MEM_EV2 are updated according to the output of the event detection signal EV_OUT. The data length of the event memory area MEM_EV1 and the additional event memory area MEM_EV2 may be 8 bits, similar to the first embodiment.

[0037] 10 is a timing chart showing the operation in Example 2. The operation of the F pixel is omitted since it is the same as that in Example 1. In addition, the reset operation of the event detection counter of the E pixel from time t700 to t701 corresponds to the reset operation of the event detection counter of the H pixel in Example 1.

[0038] Time t1003 is the time when the E1 pixel event is detected. The count value CNT_E1PIX[8] of the E1 pixel at the reference time is held in advance as a reference value in the event memory area MEM_EV1[8]. The subtraction unit 240 calculates the difference value (CNT_E1PIX[8] - MEM_EV1[8]) between the count value CNT_E1PIX[8] and the reference value in the event memory area MEM_EV1[8] and outputs it to the comparison unit 260. The comparison unit 260 compares the difference value with the threshold value output from the threshold control unit 250, and if the difference value is greater than the threshold value, the event is detected as an ON-event. When an event is detected, the event detection signal EV_OUT1 switches from low level to high level. Accordingly, the E1 pixel count value CNT_E1PIX[8] at the time of the current event detection is stored as a new reference value in the event memory area MEM_EV1[8].

[0039] Time t1004 is the event detection time of the E2 pixel. The operation performed at t1003 for the E1 pixel is performed for the E2 pixel. If an event is detected, the event detection signal EV_OUT2 switches from low level to high level. The event detection signal EV_OUT2 is input to the IF unit 270. In addition, the reference value stored in the additional event memory area MEM_EV2 is updated.

[0040] While the image memory area MEM_FL

[11] for accumulating image signals is assigned to the event memory area MEM_EV2, the H pixel can be used as the E pixel for event detection. During this period, the E pixel can detect events but cannot acquire image signals. However, because event detection can be performed using a larger number of pixels than in the first embodiment, the resolution of event detection can be increased, and the accuracy of event detection can be improved.

[0041] <Example 3> This embodiment provides a photoelectric conversion device capable of dynamically switching between the operation of embodiment 1 and the operation of embodiment 2. Event detection can be performed at high resolution, and both image data acquisition and event detection can be achieved without pixel complementation processing.

[0042] The operation of the third embodiment will be described with reference to the flowchart shown in FIG. 11. First, as described in the second embodiment, the image memory area MEM_FL

[11] is operated as the event memory area MEM_EV1 to perform event detection using H pixels (S301). Event detection is repeated until an on-event is detected (S302, S303). If an on-event is detected, the operation is switched to using H pixels and F pixels as in the first embodiment (S304). Thereafter, the operation of simultaneously acquiring an event detection signal and an image signal is repeated (S305). When the difference value between the event detection signal and the reference value becomes equal to or less than a predetermined threshold, it is determined that an off-event has been detected (S306). At this time, the mode (S301) for performing event detection using H pixels as described in the second embodiment can be switched to. The operation of this embodiment may be set and operated individually for each pixel block 130.

[0043] In the mode described in the second embodiment, in which the number of pixels for detecting an event can be increased from the normal number, the event detection accuracy can be improved, and the occurrence of an event for which a frame image should be acquired (on-event) can be detected with high accuracy. In addition, when an event is detected, both the event detection using F pixels and H pixels and the acquisition of image data can be performed, so that image signal complementation processing is not required, and degradation of image quality can be suppressed.

[0044] <Example 4> The following mainly describes the differences from the first embodiment. In this embodiment, the acquisition of image data and the detection of an event can be performed simultaneously without using the adder 230.

[0045] This embodiment will be described with reference to Figures 12 and 13. Figure 12 shows the configuration of a pixel block 130. The pixel block 130 can include a pixel unit 210, a determination unit 200 that determines an event, an output interface unit 270, a selector 1210, and a counter unit 1211. The determination unit 200 can include a memory unit 220, a subtraction unit 240, a threshold control unit 250, and a comparison unit 260.

[0046] Here, this embodiment will be described taking an H pixel arranged in the pixel unit 210 as an example. In this embodiment, the H pixel and the F pixel can be arranged in the pixel block 130 as appropriate. When light is incident, a photodetection signal PD_OUT is output to the selector 1210 from a detection unit including a photoelectric conversion element 510, a quenching element 520, and an inverter 540. The photodetection signal PD_OUT is input to the selector 1210. The selector 1210 can select which pixel unit 210 outputs the photodetection signal PD_OUT. The selector 1210 can be controlled to select a pixel to be used for event detection according to information indicating a defect related to the pixel, the type of color filter provided in the pixel, and the frequency and accuracy of event detection. The photodetection signal PD_OUT signal from the pixel unit 210 selected by the selector 1210 is input to a counter unit 1211 that performs counting for event detection. The counter unit 1211 counts the number of photodetection signals PD_OUT. The data for event detection may be shorter than the image data, so for example, the image data may be 11 bits and the data length for event detection may be 8 bits. Therefore, the counter unit 1211 may be an 8-bit counter.

[0047] The circuit of the H pixel will be described with reference to FIG. 13. The bit length of the counter 530 of the H pixel is, for example, 11 bits, and the count value can be output as an image signal. When the count value of the counter 530 is used as an image signal, the counter 530 can be controlled by an image counter reset signal RES_FL. The counter 530 counts the light detection signal PD_OUT, which is the output of the inverter 540, and is reset by the counter reset signal RES_FL. The count value of the counter 530 can be directly input from the pixel unit 210 to the IF unit 270, as with the F pixel. In this embodiment, the image memory area MEM_FL

[11] for images is not required. Therefore, the memory capacity of the storage unit 220 can be reduced compared to the first embodiment. In addition, the output of the inverter 540 is input to the counter 530 and is input to the selector 1210 via the output terminal of the H pixel as the light detection signal PD_OUT and used for event detection.

[0048] 15 is a timing chart showing the operation in the fourth embodiment, corresponding to FIG. 7 in the first embodiment. From time t1500, a reset period begins for the counter unit 1211 and the counter 530 in the pixel unit 210. At time t1500, the event counter reset signal RES_EV and the image counter reset signal RES_FL change from low level to high level. The counter unit 1211 is reset by the event counter reset signal RES_EV. The counter 530 in the pixel unit 210 is reset by the image counter reset signal RES_FL.

[0049] At time t1501, the event counter reset signal RES_EV and the image counter reset signal RES_FL change from high level to low level, and an event detection period and a period for counting pixel signals are started. When photons are incident on the APD in the pixel unit 210, the counter 530 counts up. The counter 530 is controlled by the image counter reset signal RES_FL and counts until the end of the counting period (t1504), and a count value corresponding to an image signal according to the amount of incident light is output. At the same time, when photons are incident on the APD, a light detection signal PD_OUT is output and input to the counter unit 1211 via the selector 1210. The counter unit 1211 counts the light detection signal PD_OUT and outputs a count value for event detection. The selector 1210 can select a pixel to be assigned to event detection. There may be multiple pixels selected by the selector. In the pixel block 130, the assignment of H pixels and F pixels can be changed for each event frame, the amount of incident light, and information on defective pixels.

[0050] At time t1502, a determination is made as to whether an event has been detected. The operation here is performed in the same manner as for the count value CNT_HPIX[8] of the H pixels in the first embodiment. That is, the subtraction unit 240 calculates the difference between the count value CNT_EV[8] of the counter unit 1211 and a reference value, and the comparison unit 260 compares it with a threshold value to determine whether an event has occurred. When the occurrence of an event is detected, the reference value is updated.

[0051] Time t1503 is a period in which the event detection counter unit 1211 is reset. The event counter reset signal goes from low level to high level, the count value CNT_EV[8] of the counter unit 1211 is reset, and a transition to a cycle for detecting the next event occurs. Meanwhile, in order to acquire an image signal, the count value CNT_PIX

[11] by the counters for F pixels and H pixels continues to be counted.

[0052] At time t1504, the count period of the image signal ends. The count value CNT_PIX

[11] is output from the pixel unit 210 to the IF unit 270 as the image signals of the F pixel and H pixel, and is output to the outside of the pixel block.

[0053] In this embodiment, a selector 1210 and a counter unit 1211 are added, but the addition unit 230 is no longer necessary and the memory capacity of the storage unit 220 can be reduced by the amount of the image memory area, making it possible to simultaneously detect an event and obtain image data with a small configuration.

[0054] <Example 5> In this embodiment, event detection is performed by taking the logical sum of light detection signals PD_OUT signals from a plurality of pixels. The configuration of a pixel block 130 in this embodiment will be described with reference to FIG. 16. A logical sum circuit 1610 is provided instead of the selector 1210 provided in the fourth embodiment. The pixel block 130 can include a pixel unit 210, a determination unit 200, an output interface unit 270, a counter unit 1211, and the logical sum circuit 1610. The determination unit 200 can include a memory unit 220, a subtraction unit 240, a threshold control unit 250, and a comparison unit 260.

[0055] At low luminance, the effect of noise on the output signal of the pixel unit is large, and the S / N ratio is poor, so the accuracy of event detection is reduced. Therefore, the logical sum of the light detection signal PD_OUT signal output from the multiple pixel units 210 is taken by the logical sum circuit 1610, and the output of the logical sum circuit 1610 is counted by the counter unit 1211. Since the logical sum of the detection signals from the multiple pixels is taken as the count value, the amount of incident light appears to have increased, and the S / N ratio at low luminance is improved. The determination unit 200 determines the occurrence of an event based on the count value. The determination is made by subtracting a reference value from the count value by the subtraction unit 240. The event determination is made by comparing the subtracted value with a threshold value by the comparison unit 260. Since the number of photons incident on one pixel is small at low luminance, the data length of the counter unit may be kept at 8 bits without being lengthened. The number of pixels to be logically summed may be changed according to the luminance. This makes it possible to improve the accuracy of event detection, especially at low luminance.

[0056] <Examples of photoelectric conversion device applications> Hereinafter, an apparatus 1000 including a semiconductor device 1100 including a package 1020 on which a semiconductor chip 1110 including a semiconductor integrated circuit is mounted as shown in FIG. 17 will be described. The semiconductor chip 1110 is accommodated in the package 1020 and mounted on the apparatus 1000. In the configuration shown in FIG. 17, the semiconductor chip 1110 includes a photoelectric conversion device according to the above-mentioned embodiment. The semiconductor device 1100 can include a package 1020 including a base 1010 to which the semiconductor chip 1110 is fixed and a light-transmitting member 1030 such as glass facing the semiconductor chip 1110. In the package 1020, a bonding member such as a wire or a bump that connects an inner lead provided on the base 1010 to a terminal such as a pad electrode provided on the semiconductor chip 1110 can be arranged.

[0057] The device 1000 may include at least one of an optical device 1040, a control device 1050, a processing device 1060, a display device 1070, a storage device 1080, and a mechanical device 1090. The optical device 1040 is, for example, a lens, a shutter, or a mirror. The control device 1050 controls a semiconductor chip 1110. The control device 1050 is, for example, a semiconductor device such as an ASIC.

[0058] The processing device 1060 processes an output signal from a photoelectric conversion device included in the semiconductor chip 1110. The processing device 1060 is a semiconductor device such as a CPU or ASIC for configuring an AFE analog front end or a DFE digital front end. For example, an image may be generated based on an imaging signal when an event is detected. The display device 1070 is an EL display device or a liquid crystal display device that displays an information image obtained by the semiconductor chip 1110. The storage device 1080 is a magnetic device or a semiconductor device that stores the information image obtained by the semiconductor chip 1110. The storage device 1080 is a volatile memory such as an SRAM or a DRAM, or a non-volatile memory such as a flash memory or a hard disk drive.

[0059] The mechanical device 1090 has a moving part or a propulsion part such as a motor or an engine. In the device 1000, the signal output from the semiconductor chip 1110 is displayed on the display device 1070, or transmitted to the outside by a communication device (not shown) included in the device 1000. For this purpose, the device 1000 may further include a storage device 1080 and a processing device 1060 in addition to the memory circuit and arithmetic circuit included in the semiconductor chip 1110. The mechanical device 1090 may be controlled based on the signal output from the semiconductor chip 1110.

[0060] The device 1000 is suitable for electronic devices such as information terminals having a photographing function, such as smartphones, wearable terminals, and cameras, such as interchangeable lens cameras, compact cameras, video cameras, and surveillance cameras. The mechanical device 1090 in the camera may be a device capable of driving components of the optical device 1040 for zooming, focusing, and shutter operation. Alternatively, the mechanical device 1090 in the camera may be a device capable of moving the optical device 1040 for vibration isolation operation.

[0061] The device 1000 may be a transport device such as a vehicle, a ship, or an aircraft. The mechanical device 1090 in the transport device may be used as a moving device. The device 1000 as a transport device is suitable for transporting a semiconductor chip 1110 or for assisting and / or automating driving by using a photographing function. The processing device 1060 for assisting and / or automating driving can perform processing for operating the mechanical device 1090 as a moving device based on information obtained by the semiconductor chip 1110. Alternatively, the device 1000 may be a medical device such as an endoscope, a measuring device such as a distance measuring sensor, an analytical device such as an electron microscope, an office machine such as a copier, or an industrial device such as a robot.

[0062] (Other embodiments) The disclosure of this specification includes the following photoelectric conversion devices and instruments. (Item 1) A photoelectric conversion device in which a plurality of pixel blocks are arranged, each pixel block including a plurality of first pixels, a determination unit, and an addition unit, each of the plurality of first pixels includes a first detection unit that detects light and outputs a detection signal; and a first counter that counts the detection signals output per first time period and outputs a count value; the determination unit detects an event based on the count value of the first counter; The photoelectric conversion device, wherein the adding section integrates the output of the first counter over a second period of time and outputs the integrated output as an image signal. (Item 2) the pixel block further comprises a plurality of second pixels; The photoelectric conversion device described in item 1, characterized in that each of the multiple second pixels has a second detection unit that detects light and outputs a detection signal, and a second counter that counts the detection signals output per second time period and outputs the counted number of detection signals as an image signal. (Item 3) The photoelectric conversion device described in item 1 or 2, characterized in that the pixel block further includes a memory unit, the memory unit having a first area assigned to store a reference value for detecting the event for each of the plurality of first pixels and a second area assigned to store the integrated value for each of the plurality of first pixels. (Item 4) 4. The photoelectric conversion device according to item 3, wherein the reference value is updated every time the event is detected. (Item 5) 5. The photoelectric conversion device according to item 3 or 4, wherein at least a part of the area allocated to the second area of ​​the storage unit can be changed to the first area. (Item 6) The photoelectric conversion device described in any one of items 1 to 5, characterized in that the pixel block has a first mode in which the first pixel detects the event, and a second mode in which the first pixel detects the event and outputs the image signal, and the second mode is started in response to detection of the event in the first mode. (Item 7) 7. The photoelectric conversion device according to any one of items 1 to 6, wherein the first time period is shorter than the second time period. (Item 8) The photoelectric conversion device according to any one of items 1 to 7, characterized in that the first time during which the event is detected and the second time during which the accumulation by the adder unit is performed overlap in part. (Item 9) A photoelectric conversion device in which a plurality of pixel blocks are arranged, each pixel block including a plurality of pixels, a determination unit, a selector, and a first counter, each of the plurality of pixels has a detection unit that detects light and outputs a detection signal to the selector; The selector selects and outputs the detection signal; the first counter counts the output of the selector and outputs a count value; The photoelectric conversion device according to claim 1, wherein the determination unit detects an event based on the count value. (Item 10) 10. The photoelectric conversion device according to item 9, wherein each of the plurality of pixels further comprises a second counter, the second counter counting the detection signal and outputting the counted signal as an image signal. (Item 11) A photoelectric conversion device in which a plurality of pixel blocks are arranged, each pixel block including a plurality of pixels, a determination unit, a logical sum circuit, and a first counter, each of the plurality of pixels has a detection unit that detects light and outputs a detection signal to the logical sum circuit; the logical sum circuit outputs a logical sum of the detection signals; the first counter counts the output of the OR circuit and outputs a count value; The photoelectric conversion device according to claim 1, wherein the determination unit detects an event based on the count value. (Item 12) 12. The photoelectric conversion device according to item 11, wherein each of the plurality of pixels further comprises a second counter, the second counter counting the detection signal and outputting the counted signal as an image signal. (Item 13) 13. The photoelectric conversion device according to any one of items 1 to 12, further comprising a threshold control unit capable of storing and changing a threshold value for detecting the event. (Item 14) A photoelectric conversion device according to any one of items 1 to 13, and a processing device that processes an output signal from the photoelectric conversion device.

[0063] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0064] 130: pixel block, 210: pixel section, 220: memory section, 230: adder section, 240: subtracter section, 250: threshold control section, 260: comparator section, 270: interface (IF) section

Claims

1. A photoelectric conversion device in which a plurality of pixel blocks are arranged, each pixel block including a plurality of first pixels, a determination unit, and an addition unit, each of the plurality of first pixels includes a first detection unit that detects light and outputs a detection signal; and a first counter that counts the detection signal output per first time period and outputs a count value; the determination unit detects an event based on the count value of the first counter, The adder integrates the output of the first counter over a second period of time and outputs the integrated output as an image signal.

2. the pixel block further comprises a plurality of second pixels; The photoelectric conversion device according to claim 1, characterized in that each of the plurality of second pixels has a second detection unit that detects light and outputs a detection signal, and a second counter that counts the detection signals output per second time and outputs them as an image signal.

3. The photoelectric conversion device according to claim 1, characterized in that the pixel block further includes a memory unit, the memory unit having a first area assigned to store a reference value for detecting the event for each of the plurality of first pixels and a second area assigned to store the integrated value for each of the plurality of first pixels.

4. 4. The photoelectric conversion device according to claim 3, wherein the reference value is updated every time the event is detected.

5. The photoelectric conversion device according to claim 3 , wherein at least a part of the area allocated to the second area of ​​the storage unit can be changed to the first area.

6. The photoelectric conversion device according to claim 1, characterized in that the pixel block has a first mode in which the first pixel detects the event, and a second mode in which the first pixel detects the event and outputs the image signal, and starts the second mode in response to detection of the event in the first mode.

7. 2. The photoelectric conversion device according to claim 1, wherein the first time period is shorter than the second time period.

8. The photoelectric conversion device according to claim 1 , wherein the first time during which the event is detected and the second time during which the addition is performed by the adder overlap in part.

9. A photoelectric conversion device in which a plurality of pixel blocks are arranged, each pixel block including a plurality of pixels, a determination unit, a selector, and a first counter, each of the plurality of pixels has a detection unit that detects light and outputs a detection signal to the selector; The selector selects and outputs the detection signal; the first counter counts the output of the selector and outputs a count value; The photoelectric conversion device according to claim 1, wherein the determination unit detects an event based on the count value.

10. 10. The photoelectric conversion device according to claim 9, wherein each of the plurality of pixels further includes a second counter, the second counter counting the detection signal and outputting the counted signal as an image signal.

11. A photoelectric conversion device in which a plurality of pixel blocks are arranged, each pixel block including a plurality of pixels, a determination unit, a logical sum circuit, and a first counter, each of the plurality of pixels has a detection unit that detects light and outputs a detection signal to the logical sum circuit; the logical sum circuit outputs a logical sum of the detection signals; the first counter counts the output of the OR circuit and outputs a count value; The photoelectric conversion device according to claim 1, wherein the determination unit detects an event based on the count value.

12. 12. The photoelectric conversion device according to claim 11, wherein each of the plurality of pixels further comprises a second counter, the second counter counting the detection signal and outputting the counted signal as an image signal.

13. 2. The photoelectric conversion device according to claim 1, further comprising a threshold control unit capable of storing and changing a threshold value for detecting the event.

14. The photoelectric conversion device according to claim 1 , and a processing device that processes an output signal from the photoelectric conversion device.

Images