Photoelectric conversion device, photoelectric conversion system, moving body, control method for photoelectric conversion device, computer program product and non-transitory computer-readable storage medium

By dividing the frame time period into multiple sub-frame time periods in the photoelectric conversion device and using a counting unit and processor for correction processing, the image noise problem under low photon incidence conditions is solved, and the signal-to-noise ratio is improved and the dynamic range is expanded.

CN123123747APending Publication Date: 2026-11-17CANON KK
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202610636458.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2025-05-14
Filing Date
2026-05-11
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing photoelectric conversion devices are prone to image noise under low photon incidence conditions, and the pre-acquisition operation reduces the number of photons in the video signal, which easily generates noise.

Method used

The design employs multiple pixels and a timing controller. By dividing each frame into multiple sub-frame periods and generating multiple pulse signals and readout signals in each sub-frame period, the counting unit counts the photons after the first detection, and performs correction processing in the processor. The correction method is adjusted according to the timing of pulse signal generation and the length of the sub-frame period.

Benefits of technology

It effectively corrects image noise, improves the signal-to-noise ratio (SNR), and extends the dynamic range, thereby improving image quality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123123747A_ABST
    Figure CN123123747A_ABST
Patent Text Reader

Abstract

Disclosed are a photoelectric conversion device, a photoelectric conversion system, a moving body, a control method for the photoelectric conversion device, a computer program product, and a non-transitory computer-readable storage medium. A photoelectric conversion device including pixels, a timing controller, and a processor is provided. The timing controller generates pulse signals and readout signals defining a detection period for forming the subframe period during a subframe period that forms a frame period. Each pixel includes a counter configured to count the number of pulse signals and perform accumulation for each frame period. The timing controller supplies the readout signal at the end of the subframe period, and the counting result is read out. The processor performs correction based on the counting result and changes the correction method according to at least one of the generation timing of each pulse signal and the length of the subframe period from the start of the frame period to the readout of the count result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to photoelectric conversion devices, photoelectric conversion systems, mobile bodies, control methods for photoelectric conversion devices, computer programs, and non-transitory computer-readable storage media. Background Technology

[0002] Photoelectric conversion devices capable of detecting weak light at the single-photon level using avalanche multiplication are known. US-2017-0186798 discloses a photoelectric conversion device in which a sensor chip with multiple pixels and a circuit chip having circuitry for performing signal processing are stacked. Japanese Patent Publications Nos. 2025-009968 and 2025-009969 describe a weighted counting operation for weighting and counting signals detected by an avalanche photodiode. Japanese Patent Publication No. 2021-153346 discloses a pre-acquisition operation that pre-outputs some video signals before outputting a single frame of video signal.

[0003] Since the weighted counting operation weights the coefficients based on the timing of photon incidence, variations in the incident timing manifest as image noise when photon incidence is low. Furthermore, when using the pre-acquisition operation, the number of incident photons in the pre-output video signal is reduced compared to a single frame of video signal, thus easily generating image noise. Summary of the Invention

[0004] Some embodiments of this disclosure provide an advantageous technique for correcting image noise.

[0005] According to some embodiments, a photoelectric conversion device is provided, the photoelectric conversion device including a plurality of pixels, a timing controller, and a processor, each pixel including an avalanche photodiode, wherein the timing controller is configured to generate a plurality of pulse signals and a readout signal in a plurality of sub-frame periods forming each frame period, the plurality of pulse signals defining a plurality of detection periods forming each sub-frame period, the readout signal being used to read signals from the plurality of pixels, each of the plurality of pixels including a counting unit, the counting unit being configured to count the number of pulse signals after the first detection of photons in the plurality of pulse signals in each sub-frame period and to perform accumulation for each frame period, the timing controller being configured to supply the readout signal at the end of each sub-frame period, and the counting result counted by the counting unit being read out to the processor according to the readout signal, and the processor being configured to perform correction processing on an image signal based on the counting result, and to change the method of the correction processing according to at least one of the generation timing of each of the plurality of pulse signals in each sub-frame period and the period length of the sub-frame period from the beginning of the frame period to the time when the counting result is read out.

[0006] According to some other embodiments, a control method for a photoelectric conversion device is provided, the photoelectric conversion device including a plurality of pixels and a timing controller, each pixel including an avalanche photodiode, wherein the timing controller generates a plurality of pulse signals and a readout signal in a plurality of sub-frame periods forming each frame period, the plurality of pulse signals defining a plurality of detection periods forming each sub-frame period, the readout signal being used to read signals from the plurality of pixels, each of the plurality of pixels including a counting unit, the counting unit being configured to count the number of pulse signals after the first detection of photons in the plurality of pulse signals in each sub-frame period and to perform accumulation for each frame period, the timing controller being configured to supply the readout signal at the end of each sub-frame period, and the counting result counted by the counting unit being read out to the processor according to the readout signal, the control method including performing correction processing on an image signal based on the counting result, and in the execution, the method of correction processing being changed according to at least one of the generation timing of each of the plurality of pulse signals in each sub-frame period and the period length of the sub-frame period from the beginning of the frame period to the time when the counting result is read out.

[0007] The features of this disclosure will become clear from the following description of embodiments with reference to the accompanying drawings. The following description of embodiments is given by way of example. Attached Figure Description

[0008] Figure 1This is a schematic view illustrating an example configuration of a photoelectric conversion device according to an embodiment;

[0009] Figure 2 This is a schematic view showing the configuration of the sensor substrate of the photoelectric conversion device according to an embodiment;

[0010] Figure 3 This is a schematic view showing the configuration of the circuit board of the photoelectric conversion device according to an embodiment;

[0011] Figure 4 This is a view showing an example of the configuration of pixels according to an embodiment;

[0012] Figure 5A and Figure 5B This is a view used to illustrate the photon detection operation;

[0013] Figure 6A and Figure 6B This is a view used to illustrate weighted counting;

[0014] Figure 7 This is a view showing an example of the configuration of pixels according to an embodiment;

[0015] Figure 8 This is a timing diagram illustrating an example of the operation of a pixel and a timing controller according to an embodiment;

[0016] Figure 9 This is a view showing an example of the configuration of pixels according to an embodiment;

[0017] Figure 10 This is a timing diagram illustrating an example of the operation of a pixel and a timing controller according to an embodiment;

[0018] Figure 11 This is a block diagram illustrating an example configuration of a photoelectric conversion device and a moving body according to an embodiment;

[0019] Figure 12 This is a view used to illustrate the photoelectric conversion method of the camera controller according to an embodiment;

[0020] Figure 13 This is a view showing an example of multiple frame images according to an embodiment;

[0021] Figure 14 This is a view illustrating the relationship between the memory circuitry and the buffer according to an embodiment;

[0022] Figure 15 This is a flowchart illustrating an example of driving a photoelectric conversion device according to an embodiment;

[0023] Figure 16This is a flowchart illustrating an example of driving a photoelectric conversion device according to an embodiment;

[0024] Figure 17 This is a block diagram illustrating an example configuration of a photoelectric conversion device according to an embodiment;

[0025] Figure 18 This is a block diagram illustrating the functions in a processor according to an embodiment;

[0026] Figure 19 This is a view showing the relationship between the generation timing of multiple clock signals TCLK and the weighting coefficients;

[0027] Figure 20A and Figure 20B This is a view illustrating an example of digital gain adjustment in a gain adjuster according to an embodiment;

[0028] Figure 21 This is a view illustrating an example of digital gain adjustment in a gain adjuster according to an embodiment;

[0029] Figure 22 This is a flowchart of gain adjustment according to an embodiment;

[0030] Figure 23A and Figure 23B This is a view used to illustrate the effects of the embodiments;

[0031] Figure 24 This is a view showing an example of an image signal that produces luminance noise in dark areas;

[0032] Figure 25 This is a view illustrating an example of changes in a reference pixel in a white adjuster according to an embodiment;

[0033] Figure 26 This is a flowchart of white balance adjustment according to an embodiment;

[0034] Figure 27 This is a view used to illustrate the effects of the embodiments;

[0035] Figure 28 This is a view showing the differences between images due to the difference in the duration of the exposure time of the subframes;

[0036] Figure 29 This is a view illustrating an example of the variation in correction intensity in a noise corrector according to an embodiment;

[0037] Figure 30 This is a flowchart of noise correction according to an embodiment;

[0038] Figure 31A and Figure 31BThis is a view illustrating an example of a variation of the correction table in a gamma corrector according to an embodiment;

[0039] Figure 32 This is a flowchart of gamma correction according to an embodiment;

[0040] Figure 33A and Figure 33B This is a view illustrating an example of an image signal input to a compressor according to an embodiment;

[0041] Figure 34 This is a view illustrating an example of a variation in the compression process in a compressor according to an embodiment;

[0042] Figure 35 This is a flowchart of the compression process according to an embodiment;

[0043] Figure 36A and Figure 36B This is a view illustrating an example of a photoelectric conversion system using a photoelectric conversion device according to an embodiment;

[0044] Figure 37 This is a view illustrating an example of a photoelectric conversion system using a photoelectric conversion device according to an embodiment;

[0045] Figure 38 This is a view illustrating an example of a photoelectric conversion system using a photoelectric conversion device according to an embodiment;

[0046] Figure 39A and Figure 39B This is a view illustrating an example of a photoelectric conversion system using a photoelectric conversion device according to an embodiment;

[0047] Figure 40A and Figure 40B This is a view illustrating an example of a photoelectric conversion system using a photoelectric conversion device according to an embodiment; and

[0048] Figure 41 This is a view showing an example of a photoelectric conversion system using a photoelectric conversion device according to an embodiment. Detailed Implementation

[0049] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. It should be noted that the following embodiments are not intended to limit the scope of the claims. Several features are described in the embodiments, but not all such features are necessary, and multiple such features can be appropriately combined. Furthermore, in the drawings, the same reference numerals are given the same or similar configuration, and redundant descriptions are omitted.

[0050] Embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. Note that in the following description, terms indicating specific directions or positions (e.g., "up," "down," "right," "left," and other terms including these terms) are used as needed. These terms are used to facilitate understanding of the embodiments with reference to the accompanying drawings, and the meaning of the terms does not limit the technical scope of the present disclosure.

[0051] In this specification, plan views are taken from a direction perpendicular to the light-incident surface of the semiconductor layer. Cross-sectional views indicate a plane in a direction perpendicular to the light-incident surface of the semiconductor layer. Note that if the light-incident surface of the semiconductor layer is microscopically rough, the plan view is defined with reference to the light-incident surface of the semiconductor layer during macroscopic observation.

[0052] In the example described below, the detector for detecting photons is formed by an avalanche photodiode (APD). A fixed potential is applied to the anode of the APD, and the signal is extracted from the cathode side. Therefore, the semiconductor region containing charges of the same polarity as the signal charge as the majority carriers is an n-type semiconductor region, and the semiconductor region containing charges of a different polarity than the signal charge as the majority carriers is a p-type semiconductor region. Note that a fixed potential can be applied to the cathode of the APD, and the signal can be extracted from the anode side. In this case, the semiconductor region containing charges of the same polarity as the signal charge as the majority carriers is a p-type semiconductor region, and the semiconductor region containing charges of a different polarity than the signal charge as the majority carriers is an n-type semiconductor region. The following description will depict the case where one node of the APD is set to a fixed potential, but the potentials of both nodes can vary.

[0053] In this specification, if the term "impurity concentration" is used simply, it refers to the net impurity concentration obtained by subtracting the impurity concentration compensated by impurities of the opposite conductivity type. That is, "impurity concentration" indicates the net (NET) doping concentration. A region where the concentration of added p-type impurities is higher than the concentration of added n-type impurities is a p-type semiconductor region. Conversely, a region where the concentration of added n-type impurities is higher than the concentration of added p-type impurities is an n-type semiconductor region.

[0054] Figure 1An example configuration of a photoelectric conversion device 100 used in a photoelectric conversion apparatus according to an embodiment of the present disclosure is illustrated schematically. The photoelectric conversion device 100 may have a structure, for example, in which a sensor substrate 11 and a circuit substrate 21 are stacked. The sensor substrate 11 may include, for example, a photoelectric conversion element array 12 comprising a plurality of photoelectric conversion elements (APDs). The circuit substrate 21 may include processing circuitry 22 for processing signals output from the plurality of photoelectric conversion elements.

[0055] Figure 2 An example configuration of the sensor substrate 11 is schematically shown. The photoelectric conversion element array 12 of the sensor substrate 11 may include a plurality of photoelectric conversion elements 102 arranged to form multiple rows and columns. Each photoelectric conversion element 102 is a constituent element of a detector for detecting photons and also a constituent element of a pixel 101. In the following description, the photoelectric conversion device 100 has one aspect as part of an image capturing device. However, the photoelectric conversion device 100 may be formed as part of another device. For example, the photoelectric conversion device 100 may be formed as a ranging device (e.g., a focus detection device or a ranging device using time of flight (TOF), a photometer (e.g., a device for measuring the amount of incident light), etc. Note that the plurality of pixels 101 may be arranged linearly (one-dimensionally), and in this case, the photoelectric conversion device 100 may form a line sensor.

[0056] Figure 3 The configuration of the processing circuitry 22 on the circuit board 21 is schematically shown. The processing circuitry 22 may include, for example, a pixel circuit array 120 including multiple pixel circuits 103, a readout circuit 112, a controller 115, a horizontal scanning circuit 111, multiple signal lines 113, a vertical scanning circuit 110, an output circuit 114, a setter 130, etc. Each pixel circuit 103, together with its corresponding photoelectric conversion element 102, forms a pixel 101 and processes the signal output from the photoelectric conversion element 102. One pixel circuit 103 may be provided corresponding to one photoelectric conversion element 102, or one pixel circuit 103 may be provided corresponding to two or more photoelectric conversion elements 102. Even in the latter case, each pixel 101 can be considered to include both a photoelectric conversion element 102 and a pixel circuit 103. In the example, a photoelectric conversion element 102 and a pixel circuit 103 may be electrically connected via a connector. The photoelectric conversion element array 120 and the pixel circuit array 120 form a pixel array. The setting device 130 can set the operating mode of the photoelectric conversion device 100 according to, for example, user operation or instructions from a controller (not shown).

[0057] The vertical scanning circuit 110 can sequentially select multiple rows in the photoelectric conversion element array 12 according to control signals supplied, for example, from the controller 115. The vertical scanning circuit 110 may include at least one of, for example, a shift register and an address decoder. The readout circuit 112 reads signals output from pixel circuits 103 corresponding to pixels 101 in the rows selected by the vertical scanning circuit 110 via multiple signal lines 113. The horizontal scanning circuit 111 supplies signals from, for example, a row read out by the readout circuit 112 to the output circuit 114 in a predetermined order.

[0058] Figure 4 An example configuration of a pixel 101 formed by a photoelectric conversion element 102 and a pixel circuit 103 is shown. Figure 4 In the example shown, a pixel circuit 103 is assigned to a photoelectric conversion element 102. The photoelectric conversion element 102 includes an APD 201. At least one of the plurality of constituent elements of the pixel circuit 103 may be arranged on the sensor substrate 11. Alternatively, all of the plurality of constituent elements of the pixel circuit 103 may be arranged on the sensor substrate 11.

[0059] The APD 201 generates charge pairs based on incident light through photoelectric conversion. A voltage VH can be applied to the cathode of the APD 201. A voltage VL is applied to the anode of the APD 201, and the potential of VH is higher than that of VL. The potential difference between VH and VL is applied between the anode and cathode of the APD 201. This potential difference is the reverse bias voltage that enables the APD 201 to perform avalanche multiplication operation. The charge generated by photons entering the APD 201 causes avalanche multiplication, thereby generating an avalanche current. The mode in which a voltage higher than the breakdown voltage of the APD 201 is applied between the anode and cathode is called Geiger mode. The mode in which a voltage near or below the breakdown voltage is applied between the anode and cathode of the APD 201 is called linear mode. An APD operating in Geiger mode is called a SPAD. In the example, the voltage VH is 1V and the voltage VL is -30V.

[0060] The pixel circuit 103 may include a quenching element 202 connected between a terminal supplied with voltage VH and the cathode of the APD 201. The quenching element 202 can be understood as an element that supplies voltage VH to the cathode of the APD 201. The quenching element 202 has the function of converting changes in the avalanche current generated in the APD 201 into a voltage signal. The quenching element 202 acts as a load circuit (quenching circuit) during signal multiplication via avalanche multiplication and is used to suppress avalanche multiplication by suppressing the voltage applied to the APD 201. This is referred to as quenching operation.

[0061] The pixel circuit 103 may additionally include at least one of, for example, a waveform shaping circuit 210, a signal processing circuit 211, and a selection circuit 212. The waveform shaping circuit 210 can output a pulse signal by shaping the potential change of the cathode of the APD 201 during photon detection. The waveform shaping circuit 210 may include, for example, an inverter circuit. Figure 4 In the example shown, the waveform shaping circuit 210 is formed by an inverter, but the waveform shaping circuit 210 can be formed by connecting multiple inverters in series or by another circuit that has a waveform shaping effect.

[0062] Signal processing circuit 211 can be configured to generate a count signal indicating the number of incident photons based on a pulse signal output from waveform shaping circuit 210, and maintain the count value. The count signal maintained in signal processing circuit 211 can be reset when a first control pulse of an effective level is supplied from vertical scanning circuit 110 via drive line 213. When a second control pulse of an effective level is supplied from vertical scanning circuit 110 via drive line 214, selection circuit 212 can electrically connect signal processing circuit 211 and signal line 113. Selection circuit 212 may include, for example, a buffer circuit.

[0063] Figure 5A and Figure 5B This is a view used to illustrate the photon detection operation performed by the photoelectric conversion device 100. Figure 5A It shows Figure 4 An excerpted view of APD 201, quenching element 202, and waveform shaping circuit 210. The input side of waveform shaping circuit 210 is node A, and the output side is node B. Figure 5B The upper side shows Figure 5A The waveform at node A in the diagram, and Figure 5B The lower side shows Figure 5A The waveform at node B is shown. From time t0 to time t1, the potential difference between VH and VL is applied to APD 201. When a photon enters at time t1, avalanche multiplication occurs in APD 201. This causes an avalanche multiplication current to flow into quenching element 202, and the voltage (potential) at node A drops. When the voltage drop increases and the potential difference applied to APD 201 decreases, avalanche multiplication in APD 201 stops. Thereafter, current from voltage VL to compensate for the voltage drop flows into node A, and node A stabilizes at its original potential level at time t3. At this time, the portion of the output waveform at node A that exceeds a given threshold is shaped by waveform shaping circuit 210, and a pulse signal appears at node B.

[0064] Reference Figure 6A and Figure 6B The principle of the weighted counting operation according to this disclosure is described here. Figure 6A This is a view showing an example of the detection probability of the first photon relative to time under each light intensity condition, as well as the weighting coefficients in each time period. Figure 6B The top view is a graph showing the probability of photon detection relative to the amount of light. Figure 6B The lower view is a graph showing the number of photon detection signals relative to the amount of light. Here, the first photon detected in each detection period is referred to as the first photon. Note that in this specification, the amount of light means the amount of light entering the photoelectric conversion device 100 (pixel 101) per unit time.

[0065] like Figure 6A As shown, under high light intensity conditions, the probability of detecting the first photon is higher in the early part of the detection period and decreases over time. If the detection period is divided into a first period and a second period following the first period, the probability of detecting the first photon is high during the first period under high light intensity conditions. Similarly, under medium light intensity conditions, the photon detection probability decreases over time. However, compared to the high light intensity case, the time correlation of the first photon detection probability is low, and the first photon can be detected even during the second period. Under low light intensity conditions, the detection probability of the first photon depends little on time.

[0066] exist Figure 6B On the upper side, dashed line A indicates the probability of detecting the first photon during the first time period under each light intensity, and alternating long and short dashed lines B indicate the probability of detecting the first photon during the second time period under each light intensity. The solid line (A+B), which is the sum of dashed line A and alternating long and short dashed lines B, indicates the probability of detecting the first photon during the detection time period under each light intensity. As indicated by dashed line A, the greater the light intensity, the higher the probability of detecting the first photon during the first time period, and under high light intensity conditions, the probability saturates. Specifically, under low to medium light intensity conditions, the probability of detecting the first photon during the first time period increases as a power function with respect to the increase of light intensity. On the other hand, as indicated by alternating long and short dashed lines B, under low to medium light intensity conditions, the greater the light intensity, the higher the probability of detecting the first photon during the second time period, and under medium to high light intensity conditions, the probability decreases. This is because, under high light intensity conditions, in many cases, the first photon is detected during the first time period, while in the second time period, it is a photon that enters after the detected first photon. Regarding the probability of detecting the first photon under each light intensity, as indicated by the solid line (A+B) which is the sum of the dashed line A and the alternating long and short dashed lines B, the probability of photon detection is higher under low to medium light intensity conditions, and converges to a predetermined value under medium to high light intensity conditions.

[0067] exist Figure 6B The lower side shows a graph of the count values ​​for each light intensity. Like Figure 6B The detection probability of the first photon shown on the upper side is the same as that of the second and subsequent photons entering the APD. That is, under low to medium light intensity conditions, the greater the light intensity, the greater the number of photons counted. Under medium to high light intensity conditions, the count value saturates with increasing light intensity. Under high light intensity conditions, the increase in the count value and the increase in the number of incident photons are not one-to-one, and it can be said that the weight of the count value increases with each count. In other words, it can be said that under low to medium light intensity conditions, the expected number of detected photons depends on the light intensity, while under medium to high light intensity conditions, the weight of the count value depends on the light intensity.

[0068] In the embodiments described below, a counting operation is performed while the weighting is changed according to the timing of the detected photons. In this embodiment, each frame period is divided into multiple sub-frame periods, and each sub-frame period is further divided into multiple detection periods. Each sub-frame period is also referred to as a recharge period. A frame is a still image, and a frame period is the period for acquiring a still image. For example, when 30 frames of still images are continuously reproduced, a one-second moving image is generated. By appropriately setting the number of sub-frame periods forming a frame period (in other words, the number of divisions of a frame period) and the weighting coefficients, improvements in signal-to-noise ratio (SNR) and expansion of dynamic range can be achieved simultaneously.

[0069] In this embodiment, multiple recharge periods (corresponding to the exposure periods of subframes, also called subframe periods), obtained by dividing the exposure period corresponding to a frame (also called a frame period), are divided into, for example, three types, and weighting coefficients for the three types are provided. This disclosure is not limited to this, and two or more types of recharge periods are sufficient. Here, a frame is a still image; for example, when 30 frames of still images are continuously reproduced, a one-second moving image is formed. By appropriately setting the number of frame period divisions and the weighting coefficients, both improved SNR and expanded dynamic range can be achieved simultaneously.

[0070] Figure 7 An example of the configuration of a pixel 101 according to this embodiment is shown. Figure 7In the example shown, the quenching element 202 is formed by a PMOS transistor, and the waveform shaping circuit 210 is formed by an inverter. The APD 201, quenching element 202, and waveform shaping circuit 210 can form a detector 510 for detecting photons. The signal processing circuit 211 can form a counting unit 520 that adds a cumulative value to a detection period in each of the plurality of subframe periods that form each frame period, based on a weight value of the detection period in which the detector 510 detected photons. As described later, the counting unit 520 counts the number of pulse signals following the first photon detection among the plurality of pulse signals output from the waveform shaping circuit 210 in each subframe period, and accumulates this number based on the frame period. The counting unit 520 can include a counter 502 and a counter controller 501 that controls the counter 502. The counter controller 501 can be formed, for example, by an AND circuit. The output signal from the waveform shaping circuit 210 is input to one of the two input terminals of the AND circuit that forms the counter controller 501, and the clock signal TCLK can be input to the other input terminal.

[0071] In addition to the clock signal TCLK, the vertical scan circuit 110 can also supply the enable signal EN, the reset signal RES, and the select signal SEL to the pixel 101. The clock signal TCLK, the enable signal EN, and the reset signal RES can be supplied from the controller 115. Figure 3 As shown, the vertical scanning circuit 110 and the controller 115 can form a timing controller TC, which generates timing for controlling the operation of multiple pixels 101. The timing controller TC can drive the multiple pixels 101 using either a global shutter method or a rolling shutter method. In the global shutter method, the exposure period for detecting photons is shared across the multiple pixels 101, while in the rolling shutter method, the exposure period for measuring the number of photons entering each pixel 101 varies between rows.

[0072] The timing controller TC can cause each of the clock signal TCLK, enable signal EN, reset signal RES, and selection signal SEL to change at a timing point according to the operating mode set by the setter 130. In addition, the timing controller TC can determine, according to the set operating mode, one or both of the number of subframe periods forming each frame period and the weight value of the detection period in which photons are detected among multiple detection periods.

[0073] Figure 8An example of the operation of pixel 101 and timing controller TC (vertical scan circuit 110 and controller 115) according to this embodiment is shown. Hereinafter, a low logic level of a signal is represented as L, and a high logic level of a signal is represented as H. As described above, each frame period includes multiple subframe periods (recharge periods). For example, in Figure 8 In this context, the time interval from time t0 to time t5 is a subframe interval (recharge interval). The timing controller TC defines multiple detection intervals that form each subframe interval using the clock signal TCLK (in... Figure 8 In the example, there are three detection periods. The clock signal TCLK has one aspect that includes a signal comprising multiple pulse signals defining multiple detection periods, and in each subframe, the intervals between the multiple pulse signals can be unequal and can, for example, monotonically decrease. The first detection period is the period where the weighting coefficient is represented as "+3", the second detection period is the period where the weighting coefficient is represented as "+2", and the third detection period is the period where the weighting coefficient is represented as "+1". The clock signal TCLK is also the signal that causes the counter 502 to perform a counting operation during the period when the voltage Vc at the output node of the APD 201 is at L (the period when the output of the waveform shaping circuit 210 is at H).

[0074] If a photon is detected by detector 510 during the first detection period ("+3"), Vc changes to L (the output of waveform shaping circuit 210 changes to H), and thereafter, the clock signal TCLK changes to H three times. In response, counter 502 performs three counting operations, thus adding the value "3" based on the first detection period to the count value Count of counting unit 520 (which is immediately reset to 0 by the reset signal RES). If a photon is detected by detector 510 during the second detection period ("+2"), Vc changes to L (the output of waveform shaping circuit 210 changes to H), and thereafter, the clock signal TCLK changes to H twice. In response, counter 502 performs two counting operations, thus adding the value "2" based on the second detection period to the count value Count of counting unit 520. If a photon is detected by detector 510 during the third detection period ("+1"), Vc changes to L (the output of waveform shaping circuit 210 changes to H), and thereafter, the clock signal TCLK changes to H only once. In response, counter 502 performs a counting operation only once, thus adding the value "1" based on the third detection period to the count value Count of counting unit 520. As described above, the method of adding the weighted values ​​of the detection periods in each subframe period based on the photon detected by detector 510 in multiple detection periods is called weighted addition.

[0075] The operation starting from time t0 will now be described exemplarily. The timing of the photon's entry is denoted as "incident photon," and... Figure 8 The operation illustrated depends on the timing of photon entry. At time t0, the logic level of the enable signal EN changes from L to H. Therefore, the exposure period in the k-th frame begins. Additionally, at time t0, the logic level of the clock signal VR changes to L and then immediately returns to H. Therefore, the quenching element 202 is turned on and Vc is reset (recharged). At time t1, the logic level of the clock signal TCLK changes to H. The clock signal TCLK is the signal that defines the timing of changing the weighting coefficients (in other words, the detection period) and is also the signal that causes the counter 502 to perform a counting operation during the period when Vc is at L (the period when the output of the waveform shaping circuit 210 is at H).

[0076] At time t1, since Vc is at H (the output of waveform shaping circuit 210 is at L), counter 502 does not perform a counting operation even if the clock signal TCLK changes to H. At time t2, a photon enters APD 201. Therefore, the level of output signal Vc changes to L, and the logic level of the output of waveform shaping circuit 210 changes to H. At time t3, if the level of clock signal TCLK changes to H, the logic level of the output of the AND circuit used as counter controller 501 changes to H. Therefore, counter 502 performs a counting operation and adds 1 to the count value as count value Count. At time t4, the level of clock signal TCLK changes to H again, and the logic level of the output of the AND circuit used as counter controller 501 thus changes to H. Therefore, counter 502 performs a counting operation and adds 1 to the count value as count value Count. That is, during the period from time t1 to time t4, 2 is added as a weight value to the count value as count value Count. In other words, during the subframe period, 2 is added as a weight value to the count value Count.

[0077] During the period from time t4 to time t5, the logic level of the enable signal EN changes to L. Therefore, counter 502 is controlled to be disabled. No photons entering APD 201 are detected during this period. The period from time t1 to time t5 is a subframe period, and a new subframe period begins at time t5.

[0078] During the subframe period starting at time t5, a photon enters APD 201 at time t6. In response, the level of the output signal Vc changes to L, and the logic level of the output of waveform shaping circuit 210 changes to H. At times t7, t9, and t11, the level of clock signal TCLK changes to H, and counter 502 performs a counting operation each time, adding a total of 3 to the count value Count. That is, during the period from time t5 to time t11, 3 is added to the count value Count. In other words, during the subframe period, 3 is added as a weight value to the count value Count.

[0079] During the period from time t16 to time t17, the selection signal SEL changes to H. Therefore, the selection circuit 212 outputs the accumulated value held in the signal processing circuit 211 to the output line. At time t18, photons enter the APD 201, but since the logic level of the enable signal EN is L, no photon-based signal is counted. During the period from time t19 to time t20, the level of the reset signal RES changes to H, and the signal held in the signal processing circuit 211 is reset. At time t21, the enable signal EN changes from L to H, and the exposure period in the (k+1)th frame begins.

[0080] As described above, the counting unit 520 adds the counted value (count value) to the weighted value of the detection period in each of the multiple subframe time periods, based on the photon detected by the detector in the multiple detection time periods. In the example above, the addition of the weighted values ​​of the detection periods in the multiple detection time periods is given by the number of clock signals TCLK, which are pulse signals. However, alternatively, for example, the weighted values ​​of the detection periods in the multiple detection time periods can be provided to the counting unit 520 from the timing controller TC, and the counting unit 520 can add the counted value to the weighted value. The number of detection periods forming each subframe is not limited to three, and can be any number of two or more. The weighted value of the detection period in the multiple detection time periods can be any value.

[0081] Figure 9 It was shown as Figure 7 The example shown is a modified example of the configuration of pixel 101. Figure 9 In the configuration shown, a control circuit 322 has been added, and it is connected to... Figure 7 Compared to the configuration shown, the components of the counter controller 501 have changed. Figure 9 In the configuration shown, the counter controller 501 includes an AND circuit 321, a D flip-flop 323, and a multiplexer 324. The AND circuit 321 has an AND... Figure 7 The counter controller 501 shown has the same function. The D flip-flop 323 holds information indicating whether a photon was detected in the immediately preceding subframe period (recharge period). The multiplexer 324 selects either the output of the AND circuit 321 or the output of the waveform shaping circuit 210 based on the signal STAT indicating the information held by the D flip-flop 323, and provides it to the counter 502. If the D flip-flop 323 indicates that no photon was detected in the immediately preceding subframe period, then in the current subframe period, even if Vc changes to L at photon incidence, no weighted summation is performed, and only 1 is always added. With this configuration, the possibility of counting noise as a photon detection result is reduced. Therefore, statistical deviations in the accumulated value can be suppressed, particularly improving the SNR under low light conditions.

[0082] Here, the clock signal BCLK is used to generate clock signals VR and TCLK. Additionally, the trigger signal TRIG is a pulse signal used to selectively generate clock signals VR and TCLK from the clock signal BCLK. The clock signal BCLK and the trigger signal TRIG can be generated by the timing controller TC. If a pulse of the clock signal BCLK is input when the logic level of the trigger signal TRIG is H, then the clock signal VR is generated as a negative logic pulse. If a pulse of the clock signal BCLK is input when the logic level of the trigger signal TRIG is L, then the clock signal TCLK is generated as a positive logic pulse.

[0083] Clock signals VR and TCLK can be generated using a shared clock signal BCLK. In this case, the effects of skew and other factors can be reduced compared to the case where clock signals VR and TCLK are transmitted via different signal lines. Specifically, even with a large number of pixels (e.g., 1,000,000 or more), image quality degradation is less likely to occur.

[0084] The control circuit 322 can be formed, for example, by a combination of NAND and AND circuits. The clock signal BCLK and the trigger signal TRIG can be input to the input terminals of the NAND circuit. Additionally, the inverted signals of the clock signal BCLK and the trigger signal TRIG can be input to the input terminals of the AND circuit.

[0085] Figure 10 It shows having Figure 9The following is an example of the operation of the pixel 101 and timing controller TC (vertical scan circuit 110 and controller 115) configured as shown. If the D flip-flop 323 detects a photon in the immediately preceding subframe period, the logic level of the signal STAT is set to H; otherwise, the logic level of the signal STAT is set to L. If the logic level of the signal STAT is L and a photon is detected in the current subframe, a counting operation is performed only once in that subframe period, regardless of which of the multiple detection periods includes the detection period to which the timing belongs, and only 1 is added to the count value (count value). If the logic level of the signal STAT is H and a photon is detected in the current subframe, one of 1, 2, and 3 is added as a weight value to the count value (count value) according to which of the multiple detection periods to which the timing belongs.

[0086] At time t0, the logic level of the enable signal EN changes from L to H. Therefore, the exposure period in the k-th frame begins. During the period from time t0 to time t4, no photons are detected. In the subframe period starting from time t4, the signal STAT changes to L. At time t5, a photon enters. Since the logic level of the signal STAT is L, only one counting operation is performed simultaneously with the photon detection event, and 1 is added as a weight to the count value (count value).

[0087] During the subframe period from time t9 to time t14, the signal STAT changes to H because a photon was detected in the immediately preceding subframe period (from time t4 to time t9). At time t11, a photon enters. Since the logic level of the signal STAT is H, a counting operation is performed at times t12 and t13 when the clock signal TCLK rises, and during the subframe period, 2 is added as a weight value to the count value (count).

[0088] During the subframe period from time t14 to time t18, the logic level of the signal STAT remains H because a photon was detected in the immediately preceding subframe period (from time t9 to time t14). From this point onward, in each subframe period, if a photon is detected in the immediately preceding subframe period, the logic level of the signal STAT remains H; otherwise, if no photon is detected in the immediately preceding subframe period, the logic level of the signal STAT changes to L.

[0089] At time t23, the enable signal EN changes from L to H. Therefore, the exposure period in the (k+1)th frame period begins. Depending on the frame period switch, the logic level of the signal STAT can be forced to L. The level of the signal STAT in the first sub-frame period of the (k+1)th frame period can be determined based on the photon detection results in the last sub-frame period of the kth frame period.

[0090] In this embodiment, the weighted count is controlled based on whether a photon is detected in the immediately preceding subframe period, but this disclosure is not limited thereto. For example, the weighted count can be controlled based on whether a photon is detected in a subframe period two or more periods prior. The weighted count can also be controlled based on whether a photon is detected in multiple past subframe periods.

[0091] Here, the control circuit 322 can be provided for each pixel, or it can be provided for multiple pixels together.

[0092] This section describes a pre-acquisition operation that pre-outputs some video signals before outputting a single frame of video signal. Figure 11 This is a functional block diagram of the photoelectric conversion device 700 and the moving body 710 according to this embodiment. Figure 11 Some of the functional blocks shown can be implemented by having a computer (not shown) installed in the photoelectric conversion device 700 and the moving body 710 execute a computer program stored in a memory (not shown) that serves as a storage medium.

[0093] However, some or all of these can be implemented in hardware. As hardware, this can be achieved using dedicated application-specific integrated circuits (ASICs) or processors (reconfigurable processors or DSPs). Additionally, Figure 11 The functional blocks shown may not be contained in the same housing and may be formed by separate devices connected via signal paths.

[0094] The photoelectric conversion device 700 may include the aforementioned photoelectric conversion equipment 100, imaging optical system 701, processor 703, recognizer 704, camera controller 705, storage unit 706, communicator 707, etc. As described above, the pixel 101 arranged in the photoelectric conversion device 100 is configured to include an APD for photoelectric conversion of optical images.

[0095] According to this embodiment, the photoelectric conversion device 700 is installed in the moving body 710, and the camera unit, which includes a set of imaging optical systems 701 and photoelectric conversion devices 100, is configured to capture at least one direction, such as the front side, rear side, and lateral side of the moving body. Multiple camera units may be provided in the moving body 710.

[0096] Processor 703 performs correction processing on the image signal based on the counting result from the counting unit 520 of photoelectric conversion device 100. For example, the correction processing can be image processing such as black level correction, gamma curve adjustment, noise reduction, digital gain adjustment, demosaicing, and data compression. Therefore, processor 703 can generate final image signal data. If photoelectric conversion device 100 includes an on-chip color filter such as an RGB filter, processor 703 can perform processing such as white balance correction and color conversion.

[0097] The output of processor 703 is supplied to recognizer 704, electronic control unit (ECU) 711 of moving body 710, and camera controller 705. Recognizer 704 performs image recognition based on image signal data, thereby identifying people and vehicles in the vicinity and generating warnings when necessary.

[0098] In this embodiment, the example of a car will be used to describe the mobile body 710. However, this disclosure is not limited thereto, and the mobile body 710 can be any of an airplane, train, ship, drone, AGV, and robot, if it is capable of movement.

[0099] The camera controller 705 includes a CPU that functions as a computer and a memory that stores computer programs. The CPU executes the computer programs stored in the memory, thereby controlling the various units of the photoelectric conversion device 700.

[0100] Note that the camera controller 705 has control functions and controls the length of the exposure period of each frame of the photoelectric conversion device 100 or the timing of the control signal CLK via, for example, the control pulse generator of the photoelectric conversion device 100.

[0101] Storage unit 706 includes, for example, a recording medium such as a memory card or hard disk, and can store and read image signals. Communicator 707 includes a wireless or wired interface and outputs the generated image signals to the outside of photoelectric conversion device 700 and receives various types of signals from the outside.

[0102] The ECU 711 includes a CPU that functions as a computer and a memory that stores computer programs. In the ECU 711, the CPU executes the computer programs stored in the memory, thereby controlling the various units of the moving body 710.

[0103] The output of ECU 711 is supplied to vehicle controller 712 and display unit 713. Vehicle controller 712 performs drive, stop, and direction control of the vehicle as a moving body based on the output of ECU 711. Display unit 713 includes, for example, display elements such as liquid crystal devices or organic EL devices, and is mounted in moving body 710.

[0104] Based on the output of ECU 711, display unit 713 uses, for example, a GUI to display to the operator of mobile body 710 images acquired using photoelectric conversion device 100 or various kinds of information related to the vehicle's driving status.

[0105] Figure 11 The processor 703 and identifier 704 shown may not be installed in the mobile body 710. For example, the processor 703, identifier 704, etc., may be located in an external terminal, which is set up independently of the mobile body 710 and configured to remotely control the mobile body 710 or monitor the movement of the mobile body 710.

[0106] Figure 12 This is a view used to illustrate the photoelectric conversion method performed by the camera controller 705 according to this embodiment. In this embodiment, a frame with a length of 33.3 ms is divided into four parts. That is, as... Figure 12 As shown, frame 1 is divided into frames 1_1, 1_2, 1_3 and 1_4 with equal time intervals (8.33 ms).

[0107] Frame 1_1 has a cumulative time period from the start time T0 of frame 1 to time T1, and frame 1_2 has a cumulative time period from time T0 to time T2. Frame 1_3 has a cumulative time period from time T0 to time T3, and frame 1_4 has a cumulative time period from time T0 to time T4.

[0108] The timing controller TC (vertical scanning circuit 110 and controller 115) resets the signal processing circuit 211 at time T0. In addition, the timing controller TC obtains count values ​​C1_1, C1_2, C1_3 and C1_4 from the signal processing circuit 211 at times T1 to T4 respectively.

[0109] The count values ​​C1_1, C1_2, C1_3, and C1_4 are temporarily stored in a memory circuit provided in the pixel circuit 103. The memory circuit can be arranged, for example, in the selection circuit 212. The signal of a row temporarily stored in the memory circuit is sequentially output from the photoelectric conversion device 100 via the buffer of the readout circuit 112. The timing controller TC is configured to supply a readout signal for reading signals from the multiple pixels 101 (pixel circuit 103) at the end of each subframe period, and to read the counting result counted by the counting unit 520 to the processor 703 according to the readout signal.

[0110] As described above, according to this embodiment, the accumulated count results during the time period of frame 1_1 are read from time T1 to time T2 and processed quickly by the recognizer 704. Therefore, image recognition can be performed quickly. Similarly, the signals accumulated during the time periods of frames 1_2, 1_3, and 1_4 are read sequentially from time T2 to time T3, from time T3 to time T4, and from time T4 to time T1, respectively, and image recognition can be performed repeatedly.

[0111] Figure 13 This is a view showing an example of a divided image composed of multiple frames. (e.g.) Figure 13 As shown, because the accumulation period is short, the image in frame 1_1 is dark, but the subject blur of a suddenly appearing person is minimal. On the other hand, since the accumulation period increases sequentially from frames 1_2, 1_3, to 1_4, subject blur of suddenly appearing people is more likely to occur. In this case, subject blur rarely occurs for stopped vehicles or white lines, and the longer the accumulation period, the easier it is to improve contrast.

[0112] As described above, in this embodiment, a frame has a first accumulation period (the period of frame 1_1) and a second accumulation period (the period of frame 1_4). At this time, the first accumulation period is shorter than the second accumulation period, and the signal generated in the first accumulation period is output during the time from the end of the first accumulation period to the end of the second accumulation period.

[0113] In this embodiment, the first and second cumulative periods include an overlapping period, and the first and second cumulative periods begin simultaneously. Furthermore, the end of the second cumulative period is timed to coincide with the end of a subframe. In this embodiment, the second cumulative period is an integer multiple of the first cumulative period, but it does not necessarily need to be an integer multiple of the first cumulative period. It is sufficient that the second cumulative period is longer than the first cumulative period (the first cumulative period is shorter than the second cumulative period) and that the end of the second cumulative period is later than the end of the first cumulative period.

[0114] Images with short and long accumulation periods are generated. A timing for ending the short accumulation period is set before the timing for ending the long accumulation period. Once the short accumulation period ends, image signal data is output from the photoelectric conversion device 100 and sent to the subsequent recognizer 704. Subject recognition is then performed based at least on the signal generated during the first accumulation period. Recognizer 704 identifies the subject based at least on the signal generated during the first accumulation period.

[0115] Therefore, if image signal data is output for each frame, image recognition can be performed based solely on the frame duration. However, in this embodiment, image recognition can be performed after at least 1 / 4 of a frame duration. Therefore, for example, if the moving body 710 is moving at high speed, obstacles can be quickly identified. For example, braking can be applied quickly. Additionally, obstacles can be avoided in earlier stages, for example.

[0116] In images with a long accumulation period, contrast can be improved. For this reason, the image can be used as a display image to be displayed on the display unit 713. That is, images with a short accumulation period are suitable for rapid subject recognition, while images with a long accumulation period are suitable for display.

[0117] Furthermore, in this embodiment, since the APD 201 is used as a photoelectric conversion element 102 arranged in the pixel 101, unlike a CMOS sensor, the accumulated charge is not degraded by the readout signal. Therefore, as described above, the accumulation periods can overlap. Since there is no readout noise originating from the analog signal, the original signal is not degraded even if signal readout is performed multiple times within a frame.

[0118] Figure 14 This is a view illustrating the relationship between the memory circuitry and the buffer according to this embodiment. Figure 14 The memory circuitry arranged in N rows × M columns is shown. Figure 3 The states in the pixel circuit 103 are shown. The memory circuit is represented as memory 1-1 to NM. Additionally, Figure 14 The buffers 1 to M shown are indicated Figure 3 The buffer is arranged in the readout circuit 112 shown. Figure 14 The output circuit 114 shown corresponds to Figure 3 The input circuit 114 is shown.

[0119] Figure 15 and Figure 16 This is a flowchart illustrating details of an example of driving the photoelectric conversion device 700 according to this embodiment. For example, the CPU of the computer in the camera controller 705 executes a computer program stored in memory. Therefore, Figure 15 and Figure 16 The steps in the flowchart shown are executed sequentially.

[0120] exist Figure 15 In step S101, i is set to 1. Next, in step S102, the count value Count in the signal processing circuit 211 at time Ti is output to the memory circuit arranged in the pixel circuit 103. At this time, output is made to all memory circuits simultaneously. This operation corresponds to... Figure 12 The operation at time T1 is shown.

[0121] Next, in step S103, j is set to 1, and in step S104, Figure 14 the count value Count(j-k-i) in the memory circuit j-k-i shown in the figure is output to the buffer. At this time, the count values in the first column to the M-th column are simultaneously output to the buffer. This operation means the operation of receiving the count value of the first row in Figure 14 into the buffer.

[0122] Next, in step S105, k is set to 1, and in step S106, the count value Count(j-k-i) in the buffer k is output to the output circuit 114. This operation corresponds to the operation of reading the signal in the leftmost buffer in Figure 14 from the output circuit.

[0123] Next, the process proceeds to step S107 shown in Figure 16 , and in step S107, it is determined whether k<M. If "No" in step S107, then in step S108, k=k+1 is set to increment k by 1, and the process returns to step S106. This operation corresponds to the operation of reading the signal in the buffer of the second column from the left in Figure 14 from the output circuit.

[0124] If "No" in step S107, that is, if k=M, this means that the reading of the signal in the buffer of the M-th column in Figure 14 from the output circuit is completed. Next, the process proceeds to step S109 to determine whether j<N. If "Yes" in step S109, then in step S110, j=j+1 is set to increment j by 1, and the process returns to step S104. This corresponds to the operation of starting to read the next row.

[0125] If "No" in step S109, this means that the reading of all rows is completed. Therefore, the process proceeds to step S111 to determine whether j<4. If "Yes" in step S111, then i=i+1 is set to increment i by 1, and the process returns to step S102. This operation corresponds to the operation of starting the next read at time T2.

[0126] If "No" in step S111, this means that the reading at time T4 is completed. Therefore, the process proceeds to step S113, and the signal processing circuit 211 is reset by a reset signal. This operation corresponds to the reset operation of the signal processing circuit 211 at time T4 in Figure 12 . In the manner described above, the signals accumulated in the photoelectric conversion apparatus 100 can be sequentially read out.

[0127] In this embodiment, the count value is read out within the cumulative period corresponding to the shortest 1 / 4 frame period in a frame time interval. However, this disclosure is not limited to this, and the length of the shortest cumulative period can be changed to, for example, 1 / 5 frame period or 1 / 3 frame period, depending on the recognition accuracy of the recognizer 704. Furthermore, for example, the length of the shortest cumulative period can be changed according to the brightness of the subject.

[0128] Furthermore, even if the readout period is set to every 1 / 4 frame, it is possible to determine the accuracy of image recognition based on the subject's brightness or image recognition precision. Figure 12 The counter 502 is reset midway through the accumulation period of frame 1_1, making the actual accumulation period shorter than 1 / 4 frame period. Additionally, for example, counter 502 can... Figure 12 The time T1 and other times in the time frame are temporarily reset. Therefore, the count value read at time T4 can be adjusted.

[0129] Next, the photoelectric conversion device 1000 according to this embodiment will be described. The photoelectric conversion device 1000 is configured to perform the weighted counting operation or the pre-acquisition operation described above, and to change the method of performing correction processing on the image signal based on the counting result counted by the counting unit 520. Figure 17 This is a block diagram illustrating an example of a schematic configuration of the photoelectric conversion device 1000 according to this embodiment. The photoelectric conversion device 1000 can be applied to various photoelectric conversion systems. Examples of suitable photoelectric conversion systems are digital still cameras, digital video cameras, monitor cameras, copiers, fax machines, mobile phones, vehicle-mounted cameras, and observation satellites, and application examples will be described later. A camera module including an optical system such as a lens and an image capture device is also included in the photoelectric conversion device 1000 according to this embodiment. As an example, Figure 17 A block diagram of a digital still camera is shown.

[0130] Figure 17 The photoelectric conversion device 1000 shown may include an image capturing device 1004 and a lens 1002 that forms an optical image of a subject on the image capturing device 1004. The photoelectric conversion device 1000 may also include an aperture 1003 capable of changing the amount of light passing through the lens 1002 and a baffle 1001 for protecting the lens 1002. The lens 1002 and the aperture 1003 form an optical system that converges light onto the image capturing device 1004. The image capturing device 1004 converts the optical image formed by the lens 1002 into an electrical signal. The image capturing device 1004 uses the aforementioned photoelectric conversion device 100, which includes multiple pixels 101 and a timing controller TC (vertical scanning circuit 110 and controller 115).

[0131] The photoelectric conversion device 1000 includes a processor 1007, which is an image generator configured to generate image signal data by processing image signals output from the image capture device 1004. The processor 1007 performs various types of corrections or compressions as needed and outputs image data. The processor 1007 may be formed in a semiconductor layer on which the image capture device 1004 is disposed. That is, the image capture device 1004 and the processor 1007 may be formed in the same semiconductor layer. However, this disclosure is not limited thereto, and the processor 1007 may be formed in a different semiconductor layer than the image capture device 1004.

[0132] The photoelectric conversion device 1000 may further include a memory 1010 configured to temporarily store image signal data and an external interface (I / F) 1013 configured to communicate with, for example, a computer disposed outside the photoelectric conversion device 1000. Furthermore, the photoelectric conversion device 1000 may include a recording medium 1012, such as a semiconductor memory, configured to record or read image capture data, and a recording medium control interface (I / F) 1011 configured to perform recording in or reading from the recording medium 1012. The recording medium 1012 may be included within the photoelectric conversion device 1000 or may be removable.

[0133] The photoelectric conversion device 1000 may include a general control / arithmetic unit 1009 for controlling various types of arithmetic processing and the entire digital still camera, and a timing generator 1008 for outputting various types of timing signals to the image capture device 1004 and the processor 1007. Here, some or all of the timing signals output from the timing generator 1008 may be input externally to the photoelectric conversion device 1000. In other words, the photoelectric conversion device 1000 includes at least an image capture device 1004, as an example of the photoelectric conversion device 100 described above, and a processor 1007 for processing image signals based on the counting results performed by the counting unit 520 of the image capture device 1004.

[0134] The image capture device 1004 outputs an image signal based on the counting result to the processor 1007. The processor 1007 performs predetermined signal processing on the image signal output from the image capture device 1004 and outputs image signal data for generating an image.

[0135] Figure 18 This is a block diagram illustrating the functions in processor 1007. Video signal acquisition unit 1800 receives the image signal output from image capture device 1004 and converts it into an image data format that can be processed in subsequent stages. The converted image signal undergoes de-mosaic processing in developer 1801 and is then supplied to corrector 1802.

[0136] The corrector 1802 may include a gain adjuster 1810, a white balance adjuster 1812, a noise corrector 1813, a gamma corrector 1814, and a compressor 1815. The functions included in the corrector 1802 are not limited to those shown, and another function may be added. Additionally, any of the functions included in the corrector 1802 may be removed. It can be said that the corrector 1802 only needs to include at least one of the functions of the gain adjuster 1810, white balance adjuster 1812, noise corrector 1813, gamma corrector 1814, and compressor 1815.

[0137] The image signal data generated by the corrector 1802 can be sent to the memory 1010 configured to temporarily store the image signal data. Alternatively, the image signal data generated by the corrector 1802 can be sent to an external I / F 1013 configured to communicate with, for example, a computer outside the photoelectric conversion device 1000.

[0138] In the corrector 1802, a drive mode determiner 1820 and a setter 1821 are arranged. The photoelectric conversion device 1000 according to this embodiment can operate in multiple drive modes. The multiple drive modes include a reference... Figure 12 The driving modes described above acquire the counting results of at least two or more subframes with different exposure times within a single frame (driving modes that perform pre-acquisition operations: sometimes referred to as pre-acquisition modes below). Additionally, several driving modes include reference... Figure 8 The mode described above adds a photon count value relative to the clock signal TCLK at the photon incident timing (the drive mode for performing weighted counting operations; sometimes referred to as the weighted counting mode below). The pre-acquisition mode and the weighted counting mode can be driven simultaneously. The drive mode determiner 1820 determines which mode to enable and supplies the mode determination result to the functions included in the corrector 1802. The setter 1821 sets the functions of the corrector 1802 to perform correction processing based on the determination result of the drive mode determiner 1820. Therefore, as will be described in detail later, the processor 1007 can change the method of correction processing based on at least one of the generation timing of each of the plurality of pulse signals (clock signal TCLK) in each subframe period (weighted counting mode) and the duration of the subframe period from the beginning of the frame period to the reading of the count result (pre-acquisition mode). Here, the method of changing the correction processing method differs among the functions included in the corrector 1802. The change of the correction processing method in each function will be described later for each function.

[0139] Figure 19This is a view illustrating the relationship between the generation timing and weighting coefficients of multiple clock signals TCLK in each subframe under weighted counting mode. In the first pulse signal generation mode, the timing controller TC generates many clock signals TCLK in regions with large weighting coefficients and small step sizes for the weighting coefficients in these regions. Therefore, an image that more richly expresses the tones in high-brightness regions can be generated. For example, in the first pulse signal generation mode, the timing controller TC can generate clock signals TCLK such that the time interval increases with the progression of the subframe period. The interval of clock signals TCLK in each subframe time interval can be set to a logarithmic compression of the actual time. On the other hand, in the second pulse signal generation mode, the timing controller TC generates many clock signals TCLK in regions with small weighting coefficients and small step sizes for the weighting coefficients in these regions. Therefore, an image that more richly expresses the tones in low-brightness regions can be generated. It can be said that the second pulse signal generation mode is a mode where low-brightness tones are more pronounced than in the first pulse signal generation mode. For example, in the second pulse signal generation mode, the timing controller TC can generate clock signals TCLK at equal intervals. As described above, the timing controller TC is configured to operate in multiple pulse signal generation modes with different generation timings for multiple clock signals TCLK, including a first pulse signal generation mode and a second pulse signal generation mode.

[0140] Next, each process in the corrector 1802 of the processor 1007 will be described. As described above, the correction process in the corrector 1802 is executed as a correction process. Figure 18 At least one of the functions shown is sufficient.

[0141] Gain adjuster 1810 performs gain adjustment on the image signal output from developer 1801. Gain adjuster 1810 has the function of digitally increasing / decreasing the signal level of the image signal output from each pixel 101. That is, gain adjuster 1810 has the function of controlling the brightness of the image.

[0142] Figure 20A and Figure 20B This is a view illustrating an example of digital gain adjustment in the gain adjuster 1810. A commonly known method for determining the brightness of an image is to combine shutter speed with the aperture and adjust the digital gain. In this embodiment, the shutter speed is determined by the exposure time of the image capturing device 1004, and the longer the exposure time, the lower the shutter speed. The aperture is determined by the aperture stop 1003, and the larger the aperture, the smaller the f-number.

[0143] Figure 20A It is a view showing which digital gain to set to maintain the brightness of the image when the f-value is fixed and the shutter speed is increased. Figure 20AEach dashed line in the diagram indicates the f-number, and any point on the same dashed line indicates the same f-number. For example, with an f-number of F1.4 and a shutter speed (SS) increasing from 1 second to 1 / 8 second, the digital gain needs to be increased from ×2 to ×16 to maintain the brightness of the image.

[0144] Figure 20B A digital gain adjustment method for the gain adjuster 1810 according to this embodiment is shown. For example, the digital gain is not set to ×16, but is limited to ×8, and luminance is maintained by, for example, changing the f-number from F1.4 to F1.

[0145] Figure 21 It shows Figure 20B The diagram illustrates how the maximum digital gain adjustment (sometimes referred to below as the limit) changes based on the difference in exposure duration of each subframe in the pre-acquisition mode and the difference in pulse signal generation modes in the weighted counting mode. First, if the subframe's exposure duration is short, the digital gain limit is set to a low value in either pulse signal generation mode. Additionally, if the timing controller TC operates in a first pulse signal generation mode where high-brightness hues are richly expressed, the limit is set lower than when the timing controller TC operates in a second pulse signal generation mode.

[0146] Figure 22 It is executed by the general-purpose control / arithmetic unit 1009. Figure 21 The flowchart illustrates the setting of the gain adjustment limit factor. In step S2200, the drive mode determiner 1820 determines the drive mode. Next, in step S2201, the setter 1821 obtains the digital gain limit factor based on the determined drive mode. The limit factor can be obtained by using an input / output conversion such as a preset table. For example, the general control / arithmetic unit 1009 reads a table stored in the memory 1010. Next, in step S2202, the setter 1821 sets the digital gain limit factor obtained in step S2201 to the gain adjuster 1810.

[0147] Figure 23A and Figure 23B This is a view used to illustrate the application effect of this embodiment. In weighted counting mode, the amount of additional photon count is determined based on the timing of the photon's entry during the pulse generation of the clock signal TCLK. Therefore, if the photon incident count is not large enough, even under the same illuminance, for example, depending on the photon incident timing, a result such as... Figure 23AThe image shown exhibits uneven brightness. This phenomenon will be referred to as shadow brightness noise in the following text. This phenomenon is particularly noticeable in low-brightness areas with low photon incidence or in situations with short exposure times. For this reason, the shorter the exposure time of a subframe in the pre-acquisition mode, the more pronounced the shadow brightness noise. Furthermore, shadow brightness noise is significant in the weighted counting mode, particularly in the first pulse signal generation mode where the tone is coarse in low-brightness areas.

[0148] like Figure 20B and Figure 21 As shown, the limiting magnification in the gain adjuster 1810 changes. That is, when the method of correction processing used to correct the brightness of the image signal changes, even if the exposure time of a subframe in the pre-acquisition mode is short, the photon incident count is increased by making the f-number smaller. Therefore, as... Figure 23B As shown, dark area luminance noise can be reduced, thus minimizing image noise generation and its impact on appearance. Similarly, when the timing controller TC operates in the first pulse signal generation mode, the photon incident count is increased by reducing the f-number. Therefore, dark area luminance noise can be reduced, thus minimizing image noise generation and its impact on appearance.

[0149] Next, the change in the correction processing method in the white adjuster 1812 will be described. The white adjuster 1812 adjusts the white balance of the image. Therefore, the color of the subject in the image is represented by a color close to that of the subject in real space. Since this function calculates the reference color for color adjustment of the image from the color of the subject in the image, it is necessary to capture the subject without color noise. Typically, white balance adjustment is achieved by the following steps: calculating the color average of all pixels in the target image, and changing the magnification to multiply the color components included in each image signal so that the balance of red, blue, and green falls within a predetermined range.

[0150] Figure 24 This is a view showing an example of an image signal that generates dark area luminance noise in an image signal supplied to a video signal acquisition unit 1800. Figure 24Each cell in the image indicates a pixel, and the shading indicates brightness. Black indicates a dark state, and white indicates a bright state. If you observe the image signal pixel by pixel within a four-pixel RGGB block—that is, after de-mosaicing—the pixels will have different colors even at the same brightness. This means that color noise has been introduced. For example, if the four pixels in the top right corner are de-mosaiced, the pixel values ​​are R=12, G=3, and B=12. However, if the four pixels in the bottom right corner are de-mosaiced, the pixel values ​​are R=24, G=12, and B=12. This difference is particularly noticeable when photons enter few low-brightness areas or when the exposure time is short. If such color noise is present, white balance adjustments may not be performed correctly.

[0151] Use such as Figure 25 The control shown describes how processor 1007 modifies a reference pixel among multiple pixels to be referenced for white balance adjustment. More specifically, it removes color noise-generating regions from the reference region used to perform white balance adjustment. The pixel brightness value on the vertical axis is a threshold value for the brightness value of each pixel in the image data input to white balance adjuster 1812, and pixels with brightness values ​​less than the threshold are excluded from the reference pixel used to calculate white balance. (See reference...) Figure 24 As mentioned above, image noise is particularly noticeable when photons enter very few low-brightness areas or when the exposure time is short. Therefore, when the brightness value is set to a threshold, pixels arranged in the image noise generation area can be excluded from the reference pixels used for white balance adjustment.

[0152] Among multiple pixels, pixels with image signals not less than a predetermined threshold are selected as reference pixels. For example... Figure 25 As shown, the shorter the exposure time of a subframe in the pre-acquisition mode, the higher the threshold is set. Pixels with higher brightness values ​​than the time interval are excluded from the reference pixel. Additionally, if the timing controller TC operates in the first pulse signal generation mode, a higher threshold is set than if it operates in the second pulse signal generation mode.

[0153] Figure 26 This is a flowchart illustrating a detailed example of white balance adjustment control performed by the general control / arithmetic unit 1009. (The details regarding the control method will be omitted.) Figure 22 The same processing is described in the text.

[0154] In step S2600, the setting factor of the digital gain set in the gain adjuster 1810 is obtained. Next, in step S2601, a threshold for excluding color noise pixels from the white balance calculation area is obtained. The threshold can be obtained by using an input / output conversion of a preset table. For example, the general control / arithmetic unit 1009 reads the table stored in the memory 1010, and the setter 1821 applies the table to the white adjuster 1812. In this case, the setting factor of the digital gain obtained in step S2600 can be divided to eliminate the influence of the digital gain setting factor.

[0155] Subsequently, in step S2602, the area range of the reference region set for white balance is obtained. This setting is performed by the general control / arithmetic unit 1009 via the setter 1821.

[0156] Next, in step S2603, regarding the image signal input to the white balance adjuster 1812, it is determined whether the brightness of each pixel is less than a threshold. If the pixel brightness is less than the threshold, the process proceeds to step S2604; otherwise, the process ends. In step S2604, pixels with brightness less than the threshold are excluded from the reference pixels used for white balance adjustment.

[0157] Figure 27 This is a view used to illustrate the application effect of this embodiment. Each cell indicates a pixel, and Figure 27 This illustrates the state of dark area luminance noise generated when photons enter all pixels with the same brightness. Figure 27 In the eight cells on the right, the change in incident timing is significant. As the de-mosaiced brightness is observed pixel by pixel, Y=12 for the two right pixels and Y=36 for the two left pixels. Here, color noise is generated in the two right pixels, but the brightness value is smaller compared to the two left pixels. Therefore, when applying a threshold to the de-mosaiced brightness, the white balance reference area can be set only for the two left pixels, which do not generate color noise.

[0158] As described above, color noise-generating areas are excluded from the white balance reference area. In other words, the method of correction processing used for white balance adjustment is changed. This prevents white balance adjustment from malfunctioning due to shadow brightness noise. As a result, the color variation of the subject between real space and the image is reduced.

[0159] Next, a change in the correction processing method in the noise corrector 1813 will be described. The noise corrector 1813 corrects image noise. Image noise correction is performed by replacing the value of the image signal of the target pixel with information from the image signals of the surrounding pixels, or by multiplying it by a coefficient. Image noise may be luminance noise generated by the photoelectric conversion element 102 itself. Typically, pixel value replacement based on information from the surrounding pixels is performed based on a range of 3×3 pixels or 5×5 pixels around the target pixel. The range of the surrounding pixels can be changed depending on the desired correction intensity or whether the correction is for achieving high contrast or for moderating contrast. Various methods exist for obtaining information from the surrounding pixels, including median filters with a distribution suitable for removing impulse noise and Gaussian filters relatively suitable for retaining edge information. To increase the correction intensity, different methods are used for each correction method. For example, in a median filter, the correction intensity can be increased by expanding the range of the surrounding pixels to be referenced relative to the target pixel as the center. In a Gaussian filter, the correction intensity can be increased by making the coefficient multiplied with the surrounding pixels larger. The noise corrector 1813 can be implemented not as a single correction, but as a collection of functions for performing multiple correction processes. For example, correction can be performed by applying a mean-mode filter followed by a Gaussian filter. The correction described above is a process performed within a single frame of the image. However, correction processing can be performed in different frame periods for the target pixel; for example, correction processing for supplementing information based on differences between frames can be performed. In this case, the correction strength can be increased by increasing the number of frame periods to be referenced.

[0160] Figure 28 This view illustrates the difference in appearance between images due to variations in the exposure duration of subframes in pre-capture mode. Because the exposure duration in the left image is shorter than in the right image, the photon incident count is lower, resulting in significant shadow noise. Furthermore, in weighted counting mode, the hue on the lower brightness side is coarser, generating more shadow noise. Therefore, noise correction processing needs to be performed based on the exposure duration of the subframes in pre-capture mode and the generation timing of the pulse signal for each clock signal TCLK in weighted counting mode.

[0161] Figure 29This is a schematic illustration of a method for changing the correction intensity of noise correction processing based on the exposure time duration of a subframe in pre-acquisition mode and the pulse signal generation timing in weighted counting mode. The shorter the exposure time duration of a subframe, the more shadow noise is generated. For this reason, in pre-acquisition mode, control is executed to increase the correction intensity of the noise correction processing, so that noise is more strongly suppressed as the exposure time duration shortens. This achieves the ability to change the intensity of noise correction processing according to the noise generation situation. Furthermore, shadow noise is more likely to be generated in the first pulse signal generation mode, which is concentrated on the high brightness side, compared to the second pulse signal generation mode where the pulse generation position is concentrated on the low brightness side. Therefore, control is executed to make the correction intensity higher when the timing controller TC operates in the first pulse signal generation mode compared to when it operates in the second pulse signal generation mode. This achieves the ability to change the correction intensity of noise correction processing according to the noise generation situation.

[0162] Figure 30 This is a flowchart illustrating details of an example of control performed by the general control / arithmetic unit 1009 to change the correction method of the noise corrector 1813. (The remaining text is omitted.) Figure 22 The same processing is described in the text.

[0163] In step S3000, the driving mode determined in step S2200 is input, and the correction intensity to be set for the noise corrector 1813 is determined. The correction intensity can be determined by using an input / output conversion of a preset table. For example, the general control / arithmetic unit 1009 reads a table stored in memory 1010, and the setter 1821 applies this table to the noise corrector 1813. For correction functions such as median filters or Gaussian filters, the same type of filter can always be used, or filters can be replaced or added / removed. In this case, in step S3001, in addition to the driving mode, a two-input table can be prepared by adding the type of correction function as an input value to the table. It is sufficient to transfer the type of correction function from the general control / arithmetic unit 1009 to the setter 1821.

[0164] As described above, the correction intensity in the noise corrector 1813 can be changed according to the exposure time of the subframe in the pre-acquisition mode and the generation timing of the pulse signal in the weighted counting mode. By changing the correction processing method, the generation of image noise or its impact on appearance can be reduced in the image signal data output from the processor 1007.

[0165] Next, the changes to the correction method in the gamma corrector 1814 will be described. The gamma corrector 1814 performs gamma correction to correct the brightness distribution of the image. The gamma corrector 1814 uses a correction table to convert the brightness (pixel value) of the input image signal and output it. Correction is performed for each pixel, and in the case of an RGB image, correction is performed individually for each color pixel. This correction is typically performed based on the spectral characteristics of the human eye and is performed for the purpose of making the image appearance closer to the appearance of the subject actually observed by the human eye by enhancing the dark areas.

[0166] In this scenario, if the exposure time of a subframe in the pre-acquisition mode is short, and the generation timing of the pulse signal in the weighted counting mode is located at a position with coarse tones in a low-brightness region, then dark-area luminance noise with brightness variations in the dark areas of the image is generated. Therefore, if the luminance level of the dark areas of the image is increased by gamma correction, the noise is amplified, and the image noise is significant in the dark areas. In this embodiment, the gamma correction processing method is changed to make the image noise in the dark areas insignificant.

[0167] Figure 31A and Figure 31B This is a view showing the gamma correction table in the gamma corrector 1814. A table is used to convert the brightness value (pixel value) of the image signal input to the gamma corrector 1814 for each input brightness. In gamma correction, the brightness level of the image signal is converted according to a predetermined conversion factor, and the output brightness level corresponding to the input brightness level is determined by the gamma correction table. Here, the conversion table is set so that the converted brightness values ​​are larger in dark areas. If a consistent table is used regardless of the driving mode, as mentioned above, a large amount of image noise is generated in the dark areas of the image.

[0168] Therefore, different gamma correction tables are applied in the gamma corrector 1814 based on the exposure duration of the subframe in the pre-acquisition mode and the generation timing of the pulse signal in the weighted counting mode. This suppresses the enhancement of image noise in the dark areas of the image.

[0169] Figure 31A The difference in gamma correction tables under weighted counting mode is illustrated. The gamma corrector 1814 of processor 1007 sets the conversion factor to be smaller, at least in a portion of the range where the input brightness is below a predetermined threshold, compared to the case where the timing controller operates in the first pulse signal generation mode. In other words, the conversion factor in the dark areas of the low-brightness, hue-coarse first pulse signal generation mode is set smaller than that in the low-brightness, hue-rich second pulse signal generation mode. This suppresses image noise in the dark areas of the image.

[0170] Figure 31B The differences in the gamma correction table are shown based on the differences in the exposure time duration of subframes in the pre-acquisition mode. More specifically, the shorter the exposure time duration of a subframe, the smaller the conversion factor set by the gamma corrector 1814 of the processor 1007 in the dark areas. Figure 31B This demonstrates changing the conversion factor at least within a certain range where the input brightness is below a predetermined threshold. This suppresses image noise in the darker areas of the image.

[0171] Figure 32 This is a flowchart illustrating details of an example of control performed by the general control / arithmetic unit 1009 to change the correction method of the noise corrector 1814. (The remaining text is omitted.) Figure 22 The same processing is described in the text.

[0172] In step S3200, the gamma correction table is read out based on the driving mode determined in step S2200. For example, the gamma correction table can be pre-stored in memory 1010, read out by general control / arithmetic unit 1009, and supplied to gamma corrector 1814. Here, Figure 31A and Figure 31B The table values ​​are shown in weighted counting mode and pre-fetch mode, but a table can also be prepared for situations where both driving modes are enabled simultaneously. In step S3201, the readout table is set in gamma corrector 1814 via setter 1821.

[0173] As described above, the gamma correction table in the gamma corrector 1814 can be changed according to the exposure time of the subframe in the pre-acquisition mode and the generation timing of the pulse signal in the weighted counting mode. By changing the correction processing method, the image noise level in the dark areas of the image can be reduced in the image output from the processor 1007.

[0174] Next, a change in the compression processing method for reducing the impact of luminance noise in dark areas in compressor 1815 will be described. The image signal input to compressor 1815 has a large data volume and may significantly compress memory capacity. Typically, compression processing is performed on the image signal to reduce the amount of data, and the image signal is compressed into image signal data in, for example, JPEG format.

[0175] For example, in JPEG compression, the human eye's insensitivity to high-frequency components is utilized. A discrete cosine transform (DCT) is used to perform frequency conversion on the image signal, removing high-frequency components, and the image signal is then regenerated. Therefore, the amount of information is reduced, making the reduction imperceptible to the human eye. The compression ratio can be used to control the high-frequency components to be removed. The higher the compression ratio, the lower the number of frequency components to be removed. The lower the frequency to be removed, the easier it is for the human eye to perceive the lack of information.

[0176] As described above, if the exposure time of a subframe in the pre-acquisition mode is short, and the pulse signal generation timing in the weighted counting mode is at a low-brightness, coarse-toned location, then dark-area luminance noise, characterized by brightness variations in the dark areas of the image, is generated. This brightness variation refers to an increase in high-frequency components, which is not considered a compression target, and image noise in the dark areas tends to remain in the compressed image. For this reason, the capacity of the compressed image data is easily increased.

[0177] Figure 33A An example of an image generated from an image signal input to compressor 1815 is shown, where shaded areas indicate dark areas with low brightness, and grid lines define blocks that are each units of DCT transform. Figure 33B It shows the Figure 33A The image shown is a histogram obtained by performing a DCT transform. Here, in typical JPEG compression, for example, M[Hz] is used as a threshold to remove frequencies of M[Hz] or higher, and the image signal data is regenerated through an inverse transform. On the other hand, in this embodiment, Figure 33A The image shown is labeled into at least two or more regions according to brightness level, and a different frequency is used as a threshold in each region to remove frequency components. For example, in Figure 33A In dark areas with low brightness, use Figure 33B The N[Hz] threshold shown is used to remove frequency components, and in the remaining region, M[Hz] is used as the threshold to remove frequency components.

[0178] Figure 34 This is a schematic diagram illustrating a method for changing the compression processing based on the exposure time of the subframe in pre-acquisition mode and the generation timing of the pulse signal in weighted counting mode. The value after DCT transformation on the vertical axis indicates that the larger the value, the lower the frequency component.

[0179] If the exposure period of a subframe is short, a large amount of image noise is generated. For this reason, when the exposure period is short, the threshold for the DCT-transformed value is higher. That is, the threshold is set such that if the exposure period of a subframe is short, the frequency components to be removed include low-frequency components. Furthermore, compared to the second pulse signal generation mode where the pulse generation position is concentrated on the low-brightness side, the first pulse signal generation mode in the weighted counting drive mode, where the pulse generation position is concentrated on the high-brightness side, is more prone to image noise. Therefore, in the weighted counting drive mode where the pulse generation position is concentrated on the low-brightness side, the threshold for the DCT-transformed value is larger. That is, if the timing controller TC operates in the first pulse signal generation mode, the threshold is set such that, compared to the case where the timing controller operates in the second pulse signal generation mode, the frequency components to be removed include low-frequency components.

[0180] Figure 35 This is a flowchart illustrating a detailed example of control performed by the general control / arithmetic unit 1009 to adjust the compression ratio of the compressor 1815 for each region. (The remaining text is omitted.) Figure 22 The same processing is described in the text.

[0181] In step S3500, a region-determined threshold is obtained for the brightness value (pixel value) of the image signal supplied to compressor 1815. For example, the region-determined threshold can be pre-stored in memory 1010. General purpose control / arithmetic unit 1009 reads the threshold and applies it to compressor 1815 via setter 1821. Furthermore, since the brightness value varies according to the digital gain, it is possible to perform... Figure 26 The same process as step S2600.

[0182] In step S3501, the aforementioned threshold for the frequency component used for compression is obtained. Similarly, for example, the value pre-stored in memory 1010 can be read and applied by the general control / arithmetic unit 1009 for the threshold of the frequency component.

[0183] Next, in step S3502, based on the threshold determined by the region obtained in step S3500, the image signal input to compressor 1815 is marked into two or more regions. In the case of a region where the brightness value (pixel value) is less than the threshold, the process proceeds to step S3503, and that region is marked as a first region. In the case of a region where the brightness value is equal to or greater than the threshold, the process proceeds to step S3504, and that region is marked as a second region. In this embodiment, an example of marking the first and second regions has been described. However, this disclosure is not limited thereto, and three or more regions can be marked. In this case, step S3506 branches to three or more destinations.

[0184] In step S3505, a DCT transform is performed on the image signal. In step S3506, it is determined whether the DCT-transformed image signal is marked in a first region. If it is in the first region, the process proceeds to step S3507, and values ​​not greater than a threshold are converted to 0 using a first threshold for the frequency component. If it is in the second region, the process proceeds to step S3508, and values ​​not greater than a threshold are converted to 0 using a second threshold for the frequency component.

[0185] Next, in step S3509, the DCT-transformed image signal is converted back into an image signal (image signal data).

[0186] Here, the threshold for the frequency components read in step S3501 is a threshold corresponding to the number of regions read, and different thresholds are used for the first and second regions. Additionally, in Figure 35 In the flowchart shown, the first region is the dark area of ​​the image, and the first threshold of the frequency component is greater than the second threshold.

[0187] As described above, the threshold setting of the frequency component of the compressor 1815 for each region can be changed according to the exposure time of the subframe in the pre-acquisition mode and the generation timing of the pulse signal in the weighted counting mode. By changing the compression method, an image with reduced image noise in the dark areas of the image can be provided in the image output from the processor 1007.

[0188] Below, an example of applying the above-described photoelectric conversion device 1000 to a photoelectric conversion system or a moving body will be described.

[0189] Figure 36A and Figure 36B This is a view showing the configuration of the photoelectric conversion system 1300 and the moving body 1301 according to a first application example. Figure 36AAn example of a photoelectric conversion system related to an in-vehicle camera is shown. The photoelectric conversion system 1300 includes an image capture device 1310. The image capture device 1310 is the aforementioned photoelectric conversion device 1000. The photoelectric conversion system 1300 includes an image processor 1312 that performs image processing on multiple image data acquired by the image capture device 1310. The photoelectric conversion system 1300 also includes a distance acquirer 1316 for calculating the distance to a target object, and a collision determiner 1318 for determining the possibility of a collision based on the calculated distance. Here, the distance acquirer 1316 can acquire distance information to the target object using a time-of-flight (ToF) method, or it can acquire distance information using parallax information, etc. That is, the distance information is information related to parallax, defocus, distance to the target object, etc. The collision determiner 1318 can use one of multiple pieces of distance information to determine the possibility of a collision. The distance acquirer 1316 can be implemented by specially designed hardware or by a software module. The distance acquisition unit 1316 can be implemented using a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or the like. Alternatively, the distance acquisition unit 1316 can be implemented using a combination of these.

[0190] The photoelectric conversion system 1300 is connected to the vehicle information acquisition device 1320 and can acquire vehicle information such as vehicle speed, yaw rate, and steering angle. The photoelectric conversion system 1300 is also connected to the ECU 1330, which is a control device configured to output a control signal for generating braking force on the vehicle based on the determination result of the collision determiner 1318. Furthermore, the photoelectric conversion system 1300 is connected to an alarm device 1340, which generates an alarm to the driver based on the determination result of the collision determiner 1318. For example, if the collision determiner 1318 determines that a collision is highly likely, the ECU 1330 controls the drive unit (mechanical device) 1360 to perform braking, release the accelerator pedal, or suppress engine output, thereby controlling the vehicle to avoid a collision and reduce damage. The alarm device 1340 issues an alarm sound, displays alarm information on the screen of a car navigation system, or applies vibration to the seat belt or steering wheel, thereby alerting the user.

[0191] In this application example, the perimeter (e.g., front or rear) of the vehicle (moving body 1301) is captured by the photoelectric conversion system 1300. Figure 36B The photoelectric conversion system is shown when capturing the front side of a vehicle (image capture range 1350). The vehicle information acquisition device 1320 sends commands to the photoelectric conversion system 1300 or the image capture device 1310. This configuration can further improve the accuracy of ranging.

[0192] The example of implementing control to avoid collision with another vehicle has been described above. The photoelectric conversion system 1300 can also be applied to control autonomous driving while following another vehicle or without deviating from its lane. Furthermore, the photoelectric conversion system 1300 can be applied not only to vehicles such as automobiles, but also to mobile bodies (mobile devices) such as ships, aircraft, or industrial robots. The mobile body includes one or both of a drive force generator and a rotating body, where the drive force generator generates the driving force primarily used to move the mobile body, and the rotating body primarily uses the force to move the mobile body. The drive force generator can be an engine, motor, etc. The rotating body can be a tire, wheel, marine propeller, propeller of the mobile body, etc. Additionally, the photoelectric conversion system can be applied not only to mobile bodies, but also to devices that widely use object recognition, such as Intelligent Transportation Systems (ITS).

[0193] Figure 37 This is a block diagram illustrating an example configuration of the distance image sensor 1401 in a photoelectric conversion system according to this embodiment. Figure 37 As shown, the distance image sensor 1401 includes an optical system 1402, a photoelectric conversion device 1403, an image processing circuit 1404, a monitor 1405, and a memory 1406. The distance image sensor 1401 can then receive light (modulated light or pulsed light) projected from the light source device 1411 toward the subject and reflected by the surface of the subject, thereby acquiring a distance image corresponding to the distance to the subject.

[0194] The optical system 1402 is formed by including one or more lenses and guides the image light (incident light) from the subject to the photoelectric conversion device 1403, and forms an image on the light receiving surface (sensor portion) of the photoelectric conversion device 1403.

[0195] As photoelectric conversion device 1403, the aforementioned photoelectric conversion device 1000 is applied, and a distance signal indicating the distance obtained from the light receiving signal output from photoelectric conversion device 1403 is supplied to image processing circuit 1404.

[0196] The image processing circuit 1404 performs image processing to form a distance image based on the distance signal supplied from the photoelectric conversion device 1403. Then, the distance image (image data) obtained by image processing is supplied to the monitor 1405 and displayed thereon, and is supplied and stored (recorded) in the memory 1406.

[0197] By applying the photoelectric conversion device 1000 described above, the distance image sensor 1401 with this configuration can, for example, acquire a more accurate distance image as the characteristics of the pixels improve.

[0198] Figure 38 This is a view showing an example of a schematic configuration of an endoscopic surgical system 1250 as a photoelectric conversion system. Figure 38 This illustration shows an operator (doctor) 1231 using an endoscopic surgical system 1250 to operate on a patient 1232 on a bed 1233. Figure 38 As shown, the endoscopic surgical system 1250 consists of an endoscope 1200, surgical instruments 1210, and a trolley 1234 equipped with various devices for endoscopic surgery.

[0199] Endoscope 1200 includes a tube 1201 and a camera head 1202. The tube 1201 includes a region of a predetermined length from the distal end that is inserted into a body cavity of a patient 1232. The camera head 1202 is connected to the proximal end of the tube 1201. Figure 38 The example shown illustrates an endoscope 1200 configured as a so-called rigid endoscope including a rigid endoscope tube 1201, but the endoscope 1200 can be configured as a so-called flexible endoscope including a flexible endoscope tube.

[0200] An opening for embedding an objective lens is provided at the distal end of the endoscope tube 1201. A light source device 1203 is connected to the endoscope 1200, and light generated by the light source device 1203 is guided to the distal end of the endoscope tube 1201 via a light guide extending inside the endoscope tube 1201, and emitted via the objective lens towards the target for observation within the body cavity of the patient 1232. Note that the endoscope 1200 can be a forward-looking endoscope, or a forward-oblique-looking endoscope, or a lateral-looking endoscope.

[0201] An optical system and a photoelectric conversion device are disposed in the camera head 1202, and reflected light (observation light) from the observed target is focused by the optical system onto the photoelectric conversion device. The observation light is photoelectrically converted by the photoelectric conversion device to generate an electrical signal corresponding to the observation light, that is, an image signal corresponding to the observed image. The photoelectric conversion device 1000 described above can be used as the photoelectric conversion device. The image signal is transmitted as RAW data to the camera control unit (CCU) 1235.

[0202] The CCU 1235 comprises a central processing unit (CPU), a graphics processing unit (GPU), and other components, and centrally controls the operation of the endoscope 1200 and the display device 1236. Furthermore, the CCU 1235 receives image signals from the camera head 1202 and performs various types of image processing on the image signals, such as developing (de-mosaicing), to display an image based on the image signals.

[0203] Under the control of CCU 1235, display device 1236 displays an image based on the image signal that has undergone image processing by CCU 1235.

[0204] The light source device 1203 is formed by a light source such as a light-emitting diode (LED) and supplies illumination light to the endoscope 1200 when imaging the operating parts, etc.

[0205] Input device 1237 is an input interface for endoscopic surgical system 1250. Users can input various types of information or commands into endoscopic surgical system 1250 via input device 1237.

[0206] The treatment tool control device 1238 controls the drive of the energy treatment tool 1212 to perform tissue ablation or cutting, blood vessel sealing, etc.

[0207] The light source device 1203 supplying illumination light to the endoscope 1200 during imaging of the operating section can be formed, for example, a white light source formed by an LED, a laser light source, or a combination thereof. If the white light source is formed by a combination of RGB laser light sources, the output intensity and timing of each color (each wavelength) can be precisely controlled, thus the light source device 1203 can adjust the white balance of the captured image. In this case, the observed target is illuminated by laser beams from the RGB laser light sources in a time-division manner, and the driving of the image sensor of the camera head 1202 is controlled synchronously with the illumination timing, thereby enabling the capture of images corresponding to R, G, and B respectively in a time-division manner. In this way, color images can be obtained without providing color filters in the image sensor.

[0208] The drive of the light source device 1203 can be controlled to change the intensity of the light to be output for each predetermined time. The drive of the image sensor of the camera head 1202 can be controlled in a time-division manner to acquire images in sync with the timing of the change in light intensity, and the images can be combined to produce a high dynamic range image without so-called loss of shadow detail or highlight detail.

[0209] The light source device 1203 can be configured to supply light in a predetermined wavelength band corresponding to special light observation. In special light observation, for example, the wavelength dependence of light absorption in body tissue is utilized. More specifically, by performing illumination with light in a narrow band, predetermined tissues, such as blood vessels in the mucosal surface, are captured with high contrast compared to the illumination light used in normal observation (i.e., white light). Alternatively, in special light observation, fluorescence observation can be performed to obtain an image using fluorescence generated by performing radiation with excitation light. In fluorescence observation, for example, body tissue can be irradiated with excitation light and fluorescence from the body tissue can be observed, or a reagent such as indocyanine green (ICG) can be locally injected into the body tissue while irradiating it with excitation light corresponding to the fluorescence wavelength of the reagent, thereby obtaining a fluorescence image. The light source device 1203 can be configured to supply narrow band light and / or excitation light corresponding to this special light observation.

[0210] Figure 39A Glasses 1600 (smart glasses) are described as a photoelectric conversion system. Glasses 1600 includes a photoelectric conversion device 1602. The photoelectric conversion device 1602 is the photoelectric conversion device 1000 described in each of the above embodiments. A display device including a light-emitting device such as an OLED or LED can be disposed on the rear surface side of the lens 1601. One or more photoelectric conversion devices 1602 can be provided. Alternatively, various types of photoelectric conversion devices can be combined. The arrangement position of the photoelectric conversion device 1602 is not limited to... Figure 39A The layout is shown in the diagram.

[0211] The glasses 1600 also include a control device 1603. The control device 1603 serves as a power source for supplying power to the photoelectric conversion device 1602 and the aforementioned display device. Additionally, the control device 1603 controls the operation of the photoelectric conversion device 1602 and the display device. An optical system configured to focus light onto the photoelectric conversion device 1602 is formed on the lens 1601.

[0212] Figure 39BGlasses 1610 (smart glasses) according to an application example are described. Glasses 1610 includes a control device 1612, and a photoelectric conversion device corresponding to a photoelectric conversion device 1602 and a display device are mounted on the control device 1612. The photoelectric conversion device in the control device 1612 and an optical system configured to project light emitted from the display device are formed in a lens 1611, and an image is projected onto the lens 1611. The control device 1612 serves as a power source for supplying power to the photoelectric conversion device and the display device, and controls the operation of the photoelectric conversion device and the display device. The control device may include a gaze detector for detecting the wearer's gaze. Gaze detection can be accomplished using infrared light. An infrared emitter emits infrared light towards the eyeball of a user who is looking at a displayed image. An image capture unit including a light receiving element detects the reflected light from the emitted infrared light from the eyeball, thereby obtaining a captured image of the eyeball. A reducer is provided for reducing the light from the infrared emitter to the display unit in a planar image, thereby reducing image quality degradation.

[0213] This method detects a user's gaze at a displayed image from a captured image of the eye, obtained by capturing infrared light. Any known method can be applied to gaze detection using captured images of the eye. As an example, a gaze detection method based on Purkinje images obtained by reflecting light through the cornea can be used.

[0214] More specifically, gaze detection processing based on central pupillary corneal reflection is performed. Using central pupillary corneal reflection, a gaze vector representing the direction (rotation angle) of the eyeball is calculated based on the Purkinje image and the pupil image included in the captured image of the eyeball, thereby detecting the user's gaze.

[0215] The display device according to this application example may include a photoelectric conversion device, which includes a light receiving element, and controls the display image of the display device based on the user's gaze information from the photoelectric conversion device.

[0216] More specifically, the display device determines a first field of view and a second field of view (excluding the first field of view) based on gaze information. The first and second field of view can be determined by the display device's control unit, or they can receive those areas determined by an external control unit. Within the display area of ​​the display device, the display resolution of the first field of view can be controlled to be higher than that of the second field of view. That is, the resolution of the second field of view may be lower than that of the first field of view.

[0217] Furthermore, the display area includes a first display area and a second display area different from the first display area, and a higher priority area can be determined from the first and second display areas based on viewing information. The first and second field of view areas can be determined by the control device of the display device, or they can receive those areas determined by an external control device. The resolution of the higher priority area can be controlled to be higher than the resolution of the areas other than the higher priority area. That is, the resolution of the relatively low priority area may be low.

[0218] Note that AI can be used to determine the primary field of view or higher priority areas. The AI ​​can be a model configured to estimate the angle of the gaze and the distance to a target object in front of the gaze from an image of the eye, using images of the eye and the actual direction of the eye's observation in the image as supervisory data. The AI ​​program can be held by a display device, a photoelectric conversion device, or an external device. If an external device holds the AI ​​program, it is transmitted to the display device via communication.

[0219] When display control is performed based on gaze detection, smart glasses that also include a photoelectric conversion device configured to capture external images can be applied. The smart glasses can display the captured external image information in real time.

[0220] The aforementioned photoelectric conversion device and photoelectric conversion system can be applied to electronic devices such as so-called smartphones or tablets. Figure 40A and Figure 40B This is a view showing an example of an electronic device 1500 equipped with a photoelectric conversion device. Figure 40A The front surface side of the electronic equipment 1500 is shown, and Figure 40B The rear surface side of the electronic equipment 1500 is shown.

[0221] like Figure 40A As shown, the display 1510 for displaying images is arranged at the center of the front surface of the electronic device 1500. Then, along the upper side of the front surface of the electronic device 1500, a front-facing camera 1521 and 1522 using the aforementioned photoelectric conversion device 1000, an IR light source 1530 emitting infrared light, and a visible light source 1540 emitting visible light are arranged.

[0222] like Figure 40B As shown, a rear camera 1551 and 1552 using the aforementioned photoelectric conversion device 1000, an IR light source 1560 emitting infrared light, and a visible light source 1570 emitting visible light are arranged along the upper side of the rear surface of the electronic equipment 1500.

[0223] By applying the photoelectric conversion device 1000 described above, the electronic equipment 1500 with the above configuration can capture, for example, higher quality images. It should be noted that the photoelectric conversion device can be applied to electronic equipment such as infrared sensors, rangefinders using active infrared sources, security cameras, or personal or biometric authentication cameras. This can improve the accuracy and performance of the electronic equipment.

[0224] Figure 41 This is a block diagram of an X-ray CT apparatus. The photoelectric conversion device 1000 described above is suitable for the detector of an X-ray CT apparatus. According to this application example, the X-ray CT apparatus 30 includes an X-ray generator 310, a wedge 316, a collimator 318, an X-ray detector 320, a top plate 330, a rotating frame 340, and a high-voltage generating device 350. The X-ray CT apparatus 30 also includes a data acquisition system (DAS) 351, a signal processor 352, a display unit 353, and a controller 354.

[0225] The X-ray generator 310 is formed, for example, a vacuum tube that generates X-rays. The vacuum tube of the X-ray generator 310 is supplied with high voltage and filament current from the high voltage generating device 350. X-rays are generated when thermionic electrons are emitted from the cathode (filament) to the anode (target).

[0226] The wedge 316 is a filter that adjusts the amount of X-rays emitted from the X-ray generator 310. The wedge 316 attenuates the amount of X-rays, resulting in a predetermined distribution of X-rays emitted from the X-ray generator 310 onto the subject. The collimator 318 is formed of a lead plate that narrows the irradiation range of the X-rays that have passed through the wedge 316. The X-rays generated by the X-ray generator 310 are shaped into a cone beam by the collimator 318, and the subject on the top plate 330 is irradiated by the X-rays.

[0227] The X-ray detector 320 is formed using the photoelectric conversion device 1000 described above. The X-ray detector 320 detects X-rays passing through the subject from the X-ray generator 310 and outputs a signal corresponding to the amount of X-rays to the DAS 351.

[0228] The rotating frame 340 is annular and configured to be rotatable. The X-ray generator 310 (wedge 316 and collimator 318) and the X-ray detector 320 are arranged to face each other within the rotating frame 340. The X-ray generator 310 and the X-ray detector 320 can rotate together with the rotating frame 340.

[0229] The high-voltage generating device 350 includes a boost circuit and outputs high voltage to the X-ray generator 310. The DAS 351 includes an amplifier circuit and an A / D conversion circuit, and outputs the signal from the X-ray detector 320 as digital data to the signal processor 352.

[0230] The signal processor 352 includes a central processing unit (CPU), read-only memory (ROM), and random access memory (RAM), and can perform image processing on digital data. The display unit 353 includes a flat panel display device, and can display X-ray images. The controller 354 includes a CPU, ROM, RAM, etc., and controls the operation of the entire X-ray CT apparatus 30.

[0231] Other embodiments

[0232] One or more embodiments of the present invention can also be implemented by a computer that reads and executes computer-executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more fully referred to as a "non-transitory computer-readable storage medium") to perform the functions of one or more embodiments described above and / or includes one or more circuits (e.g., application-specific integrated circuits (ASICs)) for performing the functions of one or more embodiments described above, and by a method executed by a computer of a system or device, for example, by reading and executing computer-executable instructions from a storage medium to perform the functions of one or more embodiments described above and / or controlling one or more circuits to perform the functions of one or more embodiments described above. The computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessor unit (MPU)) and may include a network of individual computers or individual processors to read and execute the computer-executable instructions. The computer-executable instructions may be provided to the computer, for example, from a network or storage medium. Storage media may include one or more of the following: hard disk, random access memory (RAM), read-only memory (ROM), storage device for distributed computing systems, optical disc (such as compact disc (CD), digital versatile disc (DVD) or Blu-ray disc (BD)™), flash memory device, memory card, etc.

[0233] Other embodiments

[0234] The embodiments of the present invention can also be implemented by the following method: providing software (including computer program products of computer programs) that performs the functions of the above embodiments to a system or device via a network or various storage media, and the computer (central processing unit (CPU) or microprocessor unit (MPU) of the system or device) reads and executes the computer program.

[0235] According to this disclosure, a technique that is beneficial for correcting image noise can be provided.

[0236] While this disclosure has been described with reference to embodiments, it is to be understood that this disclosure is not limited to the disclosed embodiments. The scope of the appended claims is to be given the broadest description so as to cover all such modifications as well as equivalent structures and functions.

Claims

1. A photoelectric conversion device, the photoelectric conversion device comprising a plurality of pixels, a timing controller and a processor, wherein each pixel comprises an avalanche photodiode, wherein, The timing controller is configured to generate multiple pulse signals and readout signals in multiple sub-frame periods that form each frame period, the multiple pulse signals defining multiple detection periods that form each sub-frame period, and the readout signals used to read signals from the multiple pixels. Each of the plurality of pixels includes a counting unit configured to count the number of pulse signals following the first detection of photons among the plurality of pulse signals in each subframe period and to perform accumulation for each frame period. The timing controller is configured to supply the readout signal at the end of each subframe period, and the counting result performed by the counting unit is read out to the processor according to the readout signal. The processor is configured to perform correction processing on the image signal based on the counting result, and to change the method of the correction processing according to at least one of the generation timing of each of the plurality of pulse signals in each subframe period, and the duration of the subframe period from the start of the frame period to the reading of the counting result in each frame period.

2. The photoelectric conversion device according to claim 1, wherein, The correction process includes gain adjustment, and The processor is configured to change the maximum gain adjustment factor based on at least one of the generation timing and the time period length.

3. The photoelectric conversion device according to claim 2, wherein, When the time period is a first length, the processor is configured to set the maximum gain adjustment factor to be lower than a second length, which is longer than the first length.

4. The photoelectric conversion device according to claim 2, wherein, The timing controller is configured to operate under multiple pulse signal generation modes with different generation timings, including a first pulse signal generation mode and a second pulse signal generation mode, wherein the low-brightness hue is higher in the second pulse signal generation mode than in the first pulse signal generation mode, and When the timing controller operates in the first pulse signal generation mode, the processor is configured to set the maximum gain adjustment factor to be lower than when the timing controller operates in the second pulse signal generation mode.

5. The photoelectric conversion device according to claim 1, wherein, The correction process includes white balance adjustment, and The processor is configured to change, based on at least one of the generation timing and the duration, a reference pixel among the plurality of pixels to be referenced for the white balance adjustment.

6. The photoelectric conversion device according to claim 5, wherein, The processor is configured to Among the plurality of pixels, a pixel whose image signal is not less than a predetermined threshold is selected as the reference pixel, and When the time period length is a first length, the threshold is set to be higher than a second length, where the second length is longer than the first length.

7. The photoelectric conversion device according to claim 5, wherein, The timing controller is configured to operate under multiple pulse signal generation modes with different generation timings, including a first pulse signal generation mode and a second pulse signal generation mode, wherein the low-brightness hue is higher in the second pulse signal generation mode than in the first pulse signal generation mode, and The processor is configured to Among the plurality of pixels, a pixel whose image signal is not less than a predetermined threshold is selected as the reference pixel, and When the timing controller operates in the first pulse signal generation mode, the threshold is set to be higher than when the timing controller operates in the second pulse signal generation mode.

8. The photoelectric conversion device according to claim 1, wherein, The correction process includes noise correction, and The processor is configured to change the correction length of the noise correction based on at least one of the generation timing and the time period length.

9. The photoelectric conversion device according to claim 8, wherein, The noise correction is performed based on at least one of the image signals of the surrounding pixels arranged around the target pixel and the image signals of the target pixel in different frame periods. The change in correction intensity includes at least one of the following: a change in the number of surrounding pixels, a change in the coefficients applied to the surrounding pixels, and a change in the number of different frame periods. The correction intensity increases when the number of surrounding pixels increases, when the coefficient applied to the surrounding pixels becomes higher, and when the number of different frame periods increases.

10. The photoelectric conversion device according to claim 8, wherein, When the time period is a first length, the processor is configured to set the correction intensity to a higher length than a second length, where the second length is longer than the first length.

11. The photoelectric conversion device according to claim 8, wherein, The timing controller is configured to operate under multiple pulse signal generation modes with different generation timings, including a first pulse signal generation mode and a second pulse signal generation mode, wherein the low-brightness hue is higher in the second pulse signal generation mode than in the first pulse signal generation mode, and When the timing controller operates in the first pulse signal generation mode, the processor is configured to set the correction intensity to be higher than when the timing controller operates in the second pulse signal generation mode.

12. The photoelectric conversion device according to claim 1, wherein, The correction process includes gamma correction, and The processor is configured to change the conversion factor of the gamma correction based on at least one of the generation timing and the time period length.

13. The photoelectric conversion device according to claim 12, wherein, When the time period length is a first length, the processor is configured to set the conversion factor to be less than a second length, which is longer than the first length, at least in a portion of the input brightness range below a predetermined threshold.

14. The photoelectric conversion device according to claim 12, wherein, The timing controller is configured to operate under multiple pulse signal generation modes with different generation timings, including a first pulse signal generation mode and a second pulse signal generation mode, wherein the low-brightness hue is higher in the second pulse signal generation mode than in the first pulse signal generation mode, and When the timing controller operates in the first pulse signal generation mode, the processor is configured to set the conversion factor to be less than when the timing controller operates in the second pulse signal generation mode, at least in a portion of the range where the input brightness is below a predetermined threshold.

15. The photoelectric conversion device according to claim 1, wherein, The correction process includes a compression process used to reduce the amount of information in the image data formed by the image signals output from the plurality of pixels, and The processor is configured to For image data that has undergone frequency conversion, remove frequency components that are not less than a predetermined threshold, and The threshold is changed based on at least one of the generation timing and the time period length.

16. The photoelectric conversion device according to claim 15, wherein, When the time period length is a first length, the processor is configured to set the threshold such that, compared to a second length that is longer than the first length, the frequency components to be removed include low-frequency components.

17. The photoelectric conversion device according to claim 15, wherein, The timing controller is configured to operate under multiple pulse signal generation modes with different generation timings, including a first pulse signal generation mode and a second pulse signal generation mode, wherein the low-brightness hue is higher in the second pulse signal generation mode than in the first pulse signal generation mode, and When the timing controller operates in the first pulse signal generation mode, the processor is configured to set the threshold such that, compared to when the timing controller operates in the second pulse signal generation mode, the frequency components to be removed include low-frequency components.

18. A photoelectric conversion system, comprising: The photoelectric conversion device according to any one of claims 1 to 17; as well as An image processing circuit configured to generate an image using a signal output from the photoelectric conversion device.

19. A movable body comprising the photoelectric conversion device according to any one of claims 1 to 17, comprising: A timing controller configured to control the movement of the moving body using a signal output from the photoelectric conversion device.

20. A control method for a photoelectric conversion device, the photoelectric conversion device comprising a plurality of pixels and a timing controller, each pixel comprising an avalanche photodiode, wherein, The timing controller generates multiple pulse signals and readout signals in multiple sub-frame periods that form each frame period. The multiple pulse signals define multiple detection periods that form each sub-frame period, and the readout signals are used to read signals from the multiple pixels. Each of the plurality of pixels includes a counting unit configured to count the number of pulse signals following the first detection of photons among the plurality of pulse signals in each subframe period and to perform accumulation for each frame period. The timing controller is configured to supply the readout signal at the end of each subframe period, and the counting result performed by the counting unit is read out to the processor based on the readout signal. The control method includes performing correction processing on the image signal based on the counting results, and In the execution, the method of correction processing is changed according to at least one of the generation timing of each of the plurality of pulse signals in each subframe period and the duration of the subframe period from the start of the frame period to the reading of the counting result in each frame period.

21. A computer program product comprising a program for causing a computer to perform the control method according to claim 20.

22. A non-transitory computer-readable storage medium storing a program for causing a computer to perform the control method according to claim 20.

Citation Information

Patent Citations

  • Solid state image sensor and imaging apparatus

    JP2021153346A

  • Photoelectric conversion device and photoelectric conversion system having the photoelectric conversion device

    JP2025009968A

  • Photoelectric conversion device and photoelectric conversion system having photoelectric conversion device

    JP2025009969A

  • Stacked SPAD image sensor

    US20170186798A1