solid-state imaging device

Abstract

To obtain good image quality characteristics under imaging conditions ranging from low illumination to high illumination. [Solution] The solid-state imaging device includes a pixel array 120 having PD pixels 20a and SPAD pixels 20b, and a pixel signal collaborative calculation circuit 27 that performs a noise compression calculation according to the illuminance based on the output signals of the PD pixels 20a and the output signals of the SPAD pixels 20b associated with the PD pixels 20a, and generates a corrected output signal for each of the PD pixels 20a.

Description

[Technical Field]

[0001] The present invention relates to a solid-state imaging device. [Background technology]

[0002] In recent years, solid-state imaging devices that use single-photon avalanche diodes (SPADs) as photoelectric conversion elements have been attracting attention. SPADs output the number of incident photons as a count, providing high sensitivity even in low-light environments. Although there are limitations on integration, increasing the size of the counter allows for imaging without saturation even in high-light environments, providing a wide dynamic range.

[0003] However, it is difficult to achieve a high pixel count with a solid-state imaging element using a SPAD. To address this issue, Patent Document 1 discloses an imaging device that combines a first imaging means using a SPAD sensor with a second imaging means using a CMOS sensor, which facilitates high image quality. Patent Document 1 addresses the issue that "SPAD sensors avalanche-amplify charge generated not only in normal pixels but also in defective pixels, affecting surrounding pixels and resulting in a group of defective pixels containing multiple adjacent defective pixels." Patent Document 1 discloses a technology for correcting defective pixels in a first imaging element using a SPAD sensor with pixels in a second imaging element using a CMOS sensor.

[0004] Furthermore, Example 4 discloses "CCD and CMOS sensors have more noise than SPAD sensors, especially in low-light environments. Hereinafter, a correlation value calculation method that reduces the influence of noise even when the correlation value is calculated based on the output of an imaging element such as a noisy CCD or CMOS sensor." In Example 4, correction of defective pixels of a SPAD sensor is performed using a reliability α corresponding to the luminance value. Specifically, when the luminance value is equal to or less than a threshold value Th1, the reliability α is 0.0, and the pixel signal of the CMOS sensor is deemed unreliable. Correction is performed using signals from pixels of the SPAD sensor surrounding the defective pixel of the SPAD sensor, rather than using pixels of the CCMOS sensor. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2023-90043 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the imaging device of Patent Document 1 corrects defective pixels occurring in a first imaging means using a SPAD sensor with pixel signals from a second imaging means using a CMOS sensor, which requires the device to have two separate imaging means, making the configuration complicated. Furthermore, Patent Document 1 corrects defective image quality occurring in the SPAD sensor, and does not address the issue of image quality overall.

[0007] The present invention has been made to solve the above-mentioned problems, and therefore, a main object of the present invention is to provide a solid-state imaging device that has good image quality characteristics under imaging conditions ranging from low illuminance to high illuminance. [Means for solving the problem]

[0008] The above object of the present invention can be achieved by the following.

[0009] a pixel array having PD pixels and SPAD pixels; a pixel signal collaborative calculation circuit that performs a noise compression calculation according to illuminance based on the output signals of the PD pixels and the output signals of the SPAD pixels associated with the PD pixels, and generates a corrected output signal for each of the PD pixels; A solid-state imaging element comprising: [Effects of the Invention]

[0010] According to the present invention, it is possible to provide a solid-state imaging device that has good image quality characteristics under imaging conditions ranging from low illuminance to high illuminance. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a schematic diagram illustrating the configuration of a solid-state imaging element according to a first embodiment. [Figure 2A] FIG. 2 is a schematic diagram illustrating the configuration of a pixel array according to the first embodiment. [Figure 2B] FIG. 2 is a schematic diagram showing the configuration of a pixel block. [Figure 3] 1 is an overall block diagram showing the configuration of a solid-state imaging device; [Figure 4A] FIG. 1 is a schematic diagram illustrating the output characteristics of a PD pixel. [Figure 4B] FIG. 10 is a schematic diagram illustrating the range in which the noise compression coefficient is used from low illuminance to high illuminance. [Figure 5] FIG. 10 is a schematic diagram illustrating a noise compression calculation process. [Figure 6] 10 is an example of mapping information showing the relationship between the illuminance of the calculation parameter and the noise compression coefficient. [Figure 7] 10A and 10B are schematic diagrams showing the relationship between the output signal of each pixel and noise compression calculation in a low-illumination shooting environment. [Figure 8] 10A and 10B are diagrams showing specific examples of output signals in a low-light shooting environment. [Figure 9] FIG. 10 is a schematic diagram showing a noise compression calculation process in a low-light shooting environment. [Figure 10]10A and 10B are schematic diagrams showing the relationship between the output signal of each pixel and noise compression calculation in a high-illumination shooting environment. [Figure 11] FIG. 10 is a diagram for explaining a specific example of a noise compression calculation. [Figure 12] FIG. 10 is a diagram for explaining a specific example of a noise compression calculation. [Figure 13] FIG. 10 is a diagram for explaining a specific example of a noise compression calculation. [Figure 14] FIG. 10 is a diagram for explaining a specific example of a noise compression calculation. [Figure 15] FIG. 10 is a diagram for explaining a specific example of a noise compression calculation. [Figure 16] FIG. 10 is a diagram for explaining a specific example of a noise compression calculation. [Figure 17] FIG. 10 is a diagram for explaining a specific example of a noise compression calculation. [Figure 18] FIG. 10 is a diagram for explaining a specific example of a noise compression calculation. [Figure 19] FIG. 10 is a diagram for explaining a specific example of a noise compression calculation. [Figure 20] FIG. 10 is an overall block diagram showing the configuration of a solid-state imaging device according to a second embodiment. [Figure 21] FIG. 10 is a schematic diagram for explaining an integrated output signal that is integrated using the ratio of the distance to a SPAD pixel. [Figure 22] 10A to 10C are diagrams showing various modified examples of pixel arrays. [Figure 23] 10A to 10C are diagrams showing various modified examples of pixel arrays. [Figure 24] 10A to 10C are diagrams showing various modified examples of pixel arrays. [Figure 25] 10A to 10C are diagrams showing various modified examples of pixel arrays. [Figure 26] 10A to 10C are diagrams showing various modified examples of pixel arrays. [Figure 27] 10A to 10C are diagrams showing various modified examples of pixel arrays. [Figure 28] 10A to 10C are diagrams showing various modified examples of pixel arrays. [Figure 29] 10A to 10C are diagrams showing various modified examples of pixel arrays. [Figure 30] 10A to 10C are diagrams showing various modified examples of pixel arrays. [Figure 31] 10A to 10C are diagrams showing various modified examples of pixel arrays. [Figure 32] 10A to 10C are diagrams showing various modified examples of pixel arrays. [Figure 33] 10A to 10C are diagrams showing various modified examples of pixel arrays. [Figure 34] FIG. 10 is an explanatory diagram showing processing in a solid-state imaging device according to a third embodiment. [Figure 35] FIG. 10 is an explanatory diagram showing processing in a solid-state imaging device according to a fourth embodiment. [Figure 36] FIG. 10 is a diagram showing a pixel array in a solid-state imaging device according to a fifth embodiment. [Figure 37] FIG. 13 is a schematic diagram showing an image creation process in the fifth embodiment. [Figure 38] FIG. 10 is a schematic diagram for explaining the usage ratio of various pixels at each illuminance level. [Figure 39] FIG. 13 is a schematic diagram showing the configuration of a pixel array of a solid-state imaging device according to a sixth embodiment. [Figure 40] FIG. 20 is a schematic diagram for explaining the usage ratio of each pixel used for motion detection and update rate determination according to the sixth embodiment. [Figure 41] 13 is a flowchart showing a process according to the sixth embodiment. [Figure 42] FIG. 10 is a schematic diagram for explaining a process of varying the frame rate upon event detection. [Figure 43] FIG. 13 is a schematic diagram for explaining the configuration of a solid-state imaging device according to a seventh embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the scope of the present invention is not limited to the disclosed embodiments. In the following drawings, the same reference numerals refer to the same components, and the dimensions of each component in the drawings are not drawn to scale for clarity and convenience of explanation. Meanwhile, the embodiments described below are merely examples, and various modifications are possible from such embodiments.

[0013] In the following, the terms "upper" and "above" may include not only something that is directly above in contact with something, but also something that is above without contacting something.

[0014] An element expressed in the singular includes a plurality of elements unless the context clearly dictates otherwise. Furthermore, when a part "includes" or "has" a certain element, it does not mean excluding other elements, but means that other elements may also be included, unless specifically stated to the contrary.

[0015] Also, use of the term "said" and similar referents applies to both the singular and the plural.

[0016] The use of any examples or exemplary terms (e.g., etc.) is intended merely to illustrate the technical idea, and the scope is not limited by said examples or exemplary terms, except as limited by the claims.

[0017] (First embodiment) <Configuration example of solid-state imaging device 100> FIG. 1 is a schematic diagram illustrating the configuration of a solid-state imaging device 100 according to the first embodiment.

[0018] As shown in FIG. 1, the solid-state imaging device 100 includes a pixel array 120, a vertical scanning unit 130, a horizontal scanning unit 140, and a signal output unit 150. The pixel array 120 includes a plurality of pixels 20 arranged in an array in row and column directions (hereinafter also referred to as the X direction or horizontal direction and the Y direction or vertical direction, respectively). The pixel array 120 includes all of the pixels 20 of the solid-state imaging device 100 (e.g., several million or more). The pixel array 120 also includes SPAD (Single-photon avalanche diode (first type of photoelectric conversion element)) pixels and CMOS-type PD (Photo Diode (second type of photoelectric conversion element)) pixels, which are general pixels different from SPAD pixels, as described below. In this paper, pixels formed by conventional photodiodes such as CMOS-types other than SPAD-types are simply referred to as PD pixels. PD pixels have different noise characteristics from SPAD pixels.

[0019] A synchronization signal is input to each of the vertical scanning unit 130 and the horizontal scanning unit 140, and an exposure control signal is input to the pixel array 120.

[0020] The vertical scanning unit 130 selects rows (Y) in order, and the horizontal scanning unit 140 selects columns (X) in order, thereby selecting each pixel 20 in order using an XY addressing method, and the pixel signals (count signals) of the selected pixels 20 are output via signal lines to the signal output unit 150. The signal output unit 150 compiles the pixel signals of each pixel output to the signal lines and outputs them as image data to an external recording unit or signal processing unit.

[0021] (pixel array 120) FIG. 2A is a schematic diagram showing the configuration of a pixel array in the first embodiment. FIG. 2B is a schematic diagram showing some of the pixel blocks in FIG. 2A. As shown in FIG. 2A, the pixel array 120 includes a plurality of pixel blocks 200 arranged in a two-dimensional lattice pattern (row and column directions) when viewed in a plan view perpendicular to the substrate surface of the solid-state imaging device 100. Note that FIG. 2A only shows some of the pixel blocks 200 and pixels 20. The pixel array 120 may be composed of, for example, several million or more pixels 20, but FIG. 2A shows only some of the pixels 20 (the same applies to the following FIGS. 21 and 22). Each pixel 20 is divided into pixel blocks 200 of a predetermined size. In the example shown in FIGS. 2A and 2B, one pixel block 200 includes one SPAD pixel 20b-1 and eight CMOS PD pixels 20a-1 to 20a-8 surrounding it. Hereinafter, when the PD pixels 20a-1 to 20a-8 are not distinguished from one another, they will be collectively referred to as PD pixels 20a (the same applies to SPAD pixels 20b). When the PD pixels 20a (hereinafter simply referred to as pixel 20a) and the SPAD pixels 20b (hereinafter simply referred to as pixel 20b) are not distinguished from one another, they will be collectively referred to as pixel 20. In the first embodiment, each PD pixel 20a in one pixel block 200 corresponds to one SPAD pixel 20b. In the example shown in FIG. 2A, each SPAD pixel 20b corresponds to (is covered by) one of the R, G, and B color filters. In FIGS. 2A and 2B (as well as in FIG. 21, etc., described later), the PD pixels 20a are simply denoted as R, G, and B, corresponding to the respective color filters, as shown in the legend.

[0022] FIG. 3 is an overall block diagram showing the configuration of the solid-state imaging device 100. As shown in FIG.

[0023] (pixel 20a) Each of the pixels 20a-1 to 20a-n includes a PD 201, an AMP 202, a comparator 203, a counter 204, a latch 205, and a PD control circuit 206. The PD 201 is a CMOS-type photoelectric conversion element. A constant current source 122 is connected to each AMP 202. A RAMP waveform generation circuit 123 is connected to the comparator 203. The RAMP waveform generation circuit 123 outputs a ramp signal that increases or decreases steadily over a specific period of time under predetermined conditions (initial value, slope).

[0024] The comparator 203 compares the pixel signal and reset signal from the AMP 202 with the ramp signal, and counts with the counter 204 until the signal is inverted. Correlated double sampling is performed based on the count values ​​of the pixel signal and reset signal, and the differential count value is output as a digital signal. The latch 205 holds the digital signal. This latch 205 outputs the held digital signal V101 to the pixel-signal collaborative calculation circuit 27 under the control of the PD control circuit 206. The PD control circuit 206 generates control signals for the respective functional units 122, 123, 201 to 205. Note that some of the functions of the PD control circuit 206 are performed by the vertical scanning unit 130, horizontal scanning unit 140, and signal output unit 150 in FIG. 1.

[0025] (Pixel 20b) The pixel 20b-1 has a SPAD 211, an ADC 212, a counter 214, a latch 215, and a SPAD control circuit 216. The SPAD 201 is a SPAD-type photoelectric conversion element. The ADC 212 is, for example, an inverter, and converts the output of the SPAD 211 into a pulse signal. The counter 214 and the latch 215 have the same functions as the counter 204 and latch 205 described above. The latch 215 outputs the held digital signal V201 to the pixel-signal collaborative calculation circuit 27 under the control of the SPAD control circuit 216.

[0026] The SPAD control circuit 216 generates control signals to the respective functional units 211 to 215. Note that some of the functions of the PD control circuit 206 are performed by the vertical scanning unit 130, horizontal scanning unit 140, and signal output unit 150 in FIG.

[0027] The pixel signal collaborative calculation circuit 27 is a circuit that calculates the digital signals output from the PD pixel 20a and the SPAD pixel 20b. For example, the pixel signal collaborative calculation circuit 27 performs a noise compression calculation on the pixel signal V101 based on the pixel signal V101 of the pixel 20a-1 and the pixel signal V201 of the SPAD pixel 20b-1 associated with the pixel 20a-1, to generate a corrected output signal V501 for the pixel signal V101. The pixel signal collaborative calculation circuit 27 performs a noise compression calculation on the pixel signal V102 based on the pixel signal V102 of the pixel 20a-2 and the pixel signal V201 of the SPAD pixel 20b-1 associated with the pixel 20a-2, to generate a corrected output signal V502 for the pixel signal V102. In addition, the pixel signal collaborative calculation circuit 27 performs a noise compression calculation on the pixel signal V10n based on the pixel signal V10n of the pixel 20a-n and the pixel signal V201 of the SPAD pixel 20b-1 associated with the pixel 20a-n, to generate a corrected output signal V50n of the pixel signal V10n.

[0028] The pixel signal collaborative calculation circuit 27 has a noise compression coefficient setting circuit 271a (see FIG. 5 described later) and calculation parameters 271b. The function of the pixel signal collaborative calculation circuit 27 will be described later.

[0029] The pixel cooperation control circuit 28 outputs control signals to the PD control circuit 206 , the SPAD control circuit 216 , and the pixel signal cooperation calculation circuit 27 .

[0030] (Noise compression calculation) The noise compression calculation will be described with reference to Fig. 4A to Fig. 16. As described above, the solid-state imaging device 100 according to this embodiment has a pixel array 120 in which SPAD pixels 20b and PD pixels 20a are arranged in a mixed manner.

[0031] First, with reference to FIG. 4A, the characteristics of each type of photoelectric conversion element will be described before explaining the noise compression calculation.

[0032] (Low illumination ~ High illumination) FIG. 4A is a schematic diagram showing the characteristics of a PD pixel. In imaging control using the PD pixel 20a, noise becomes the dominant term under low illumination conditions, resulting in an image that is overwhelmed with noise. Examples of noise include circuit noise and optical shot noise. These noises have a particularly large effect under low illumination conditions. Circuit noise includes dark noise and read noise. Dark noise occurs even when no light is irradiated. Dark noise is nearly constant and does not depend on the incident illumination. Read noise is electronic noise generated during the signal readout process. Optical shot noise is noise caused by the random arrival of photons according to a Poisson distribution. Sensitivity is the magnitude of the PD pixel output relative to the incident illumination on the PD pixel 20a, and is the slope of an approximately linear function that shows the relationship between the incident illumination and the PD pixel output signal.

[0033] Furthermore, at high illuminance, there is a charge saturation (limit) due to the floating diffusion (FD) capacitance. At illuminance greater than the charge saturation point, the output saturates.

[0034] In the PD pixel 20a, noise becomes dominant in the output signal at a predetermined low illuminance. In this embodiment, the illuminance lower than the intersection point p1 between the optical shot noise and the dark noise is called the low illuminance range. The region higher than the inflection point p2 at which saturation occurs is called the high illuminance region. The region between the low illuminance region and the high illuminance region is called the medium illuminance region. The intersection point p1 and the inflection point p2 that divide the low, medium, and high illuminances vary depending on the characteristics of the PD pixel 20a used, but the intersection point p1 is set to, for example, an illuminance of 10 to 10 2 In other words, the range of the predetermined low illuminance is, for example, illuminance 0 or more, 10 to 10 2 For example, the inflection point p2 is 10 3 ~10 4 In other words, the specified high illuminance is in the range of 10 3 ~10 4 The range is equal to or greater than either of the values ​​in [lux]. The predetermined medium illuminance is in the middle range between low illuminance and high illuminance.

[0035] To summarize the above, PD pixels and SPAD pixels have the following advantages and disadvantages respectively.

[0036] <PD pixel> (1) In the low illuminance region, for PD pixels, noises such as dark noise, read noise, and optical shot noise become dominant terms, resulting in an image buried in noise. (2) Also, in the high illuminance region, PD pixels have charge saturation (limit) due to the floating diffusion capacitance, and cannot increase the dynamic range. (3) On the other hand, PD pixels, including the control circuit, can reduce the circuit size and achieve high resolution.

[0037] <SPAD pixel> In contrast, SPAD pixels have the following advantages and disadvantages. (1) When one photon enters the pixel, an electrical pulse signal can be output by avalanche-like multiplication. By using the technology of obtaining illuminance by counting the number of times (referred to as PCS: Photon Counting Sensor), low noise can be achieved in low illuminance, and a high dynamic range can be realized by sufficiently expanding the number of bits of the counter circuit even in high illuminance. (2) On the other hand, the circuit size required for its count control becomes large, and high resolution cannot be achieved. (3) Also, along with the individual count control for each pixel, the power consumption also increases, and there is a problem that it is difficult to use in mobile applications powered by a battery.

[0038] In this embodiment, the problem is solved by mixing PD pixels and SPAD pixels and performing cooperative control that utilizes the characteristics of each pixel. The details will be described later, but the outline is as follows. Figure 4B is a schematic diagram for explaining an example of the noise compression coefficient rn and its usage range from low illuminance to high illuminance. The noise compression coefficient rn will be described later (Figure 5, etc.).

[0039] (a) In low-illumination areas, SPAD pixel information is given priority. Specifically, as shown in Figure 4B, by using a noise compression coefficient rn of less than 0.5, the output signals of SPAD pixels are used at a higher ratio than the output signals of PD pixels. For example, in extremely low-illumination areas, the captured image is formed using only SPAD pixel information, without using PD pixel information. This results in lower resolution, but image information with a higher SNR (signal-to-noise ratio) is obtained.

[0040] (b) In the medium illumination range, information from both PD pixels and SPAD pixels is used in conjunction depending on the illumination level. Specifically, in low to medium illumination, as the illumination level increases, PD pixel information is given higher priority. Specifically, by using a noise compression coefficient rn exceeding 0.5, the output signal from PD pixels is used at a higher rate than the output signal from SPAD pixels. This allows for high-resolution image information to be obtained.

[0041] (c) In high-illumination areas where PD pixels are saturated or close to saturated, SPAD pixel information is given priority. Specifically, by using a noise compression coefficient rn of less than 0.5, the output signals of SPAD pixels are used at a higher ratio than the output signals of PD pixels. This results in a captured image that is formed with image information of a high dynamic range, although at a lower resolution.

[0042] (Outline of noise compression calculation) FIG. 5 is a schematic diagram outlining the noise compression calculation process. FIG. 6 shows an example of mapping information indicating the relationship between the illuminance of the calculation parameters and the noise compression coefficient. The calculation parameters 271b include a gain coefficient gr and mapping information. The gain coefficient gr is a coefficient for adjusting the scale (sensitivity) of the output signal between the PD pixel 20a and the SPAD pixel 20b, and is determined by the characteristics (aperture ratio, material, lens shape, wavelength dependency, temperature characteristics) of each type of optical conversion element. The mapping information describes the relationship between the illuminance and the noise compression coefficient rn. The mapping information can be in the form of a table shown in FIG. 6(a) or the form of a relational expression shown in FIG. 6(b), and either may be used. Note that in this embodiment, the output value (count value) of the SPAD pixel 20b is used as the illuminance value. The noise compression coefficient rn can take a value in the range of 0 to 1. 6(a), the noise compression coefficient rn can be set as a negative multiplier of 2, such as 1, 0.5, 0.25, 0.125, or 0.0625, allowing the pixel signal collaborative calculation circuit 27 to perform noise compression calculations using bit shift calculations. In the first embodiment shown in FIGS. 2B and 3, the count value of one SPAD pixel 20b in the pixel block 200 is used for this SPAD pixel 20b.

[0043] The calculation parameters 271b, such as the gain coefficient gr and mapping information, may be configured so that they can be freely set by the user through register settings from an external I / F. In this case, the calculation parameters 271b may be subdivided and set according to factors such as illuminance, temperature, and filters (R, G, B). FIGS. 6(a) and 6(b) are merely examples, and the count value (illuminance) and coefficients can be set as appropriate. For example, although the relational expression in FIG. 6(b) shows a linear expression, it may also be a quadratic expression, an N-th order expression, or a function using a root.

[0044] The pixel signal collaborative calculation circuit 27 calculates the corrected output signal V50 according to equation (1).

[0045] As shown in FIG. 5 and the following equation (1), the pixel signal collaborative calculation circuit 27 first multiplies the PD pixel output signal V10 by the gain coefficient gr. Then, the SPAD pixel output signal V20 is subtracted from this. This is then multiplied by the noise compression coefficient rn. This noise compression coefficient rn is set according to the illuminance (SPAD count value) as shown in the mapping information in FIGS. 6(a) and 6(b). Then, the SPAD pixel output signal V20 is added to obtain the corrected output signal V50. V50=((V10×gr-V20)×rn+V20) / gr (Formula 1) (Example of calculation output in low light) FIG. 7 is a schematic diagram showing the relationship between the output signals of PD pixels and SPAD pixels and the noise compression calculation in a low-illuminance shooting environment. The horizontal axis represents time, and the vertical axis represents the output signal (illuminance count) of the PD pixel 20a or the SPAD pixel 20b. Note that in FIG. 7 (as well as in FIGS. 34 and 35 described below), the output signals of the PD pixel 20a and the SPAD pixel 20b are aligned along the vertical axis (scale adjustment using the gain coefficient gr). The example shown in FIG. 7 shows an example in which the amount of incident light gradually increases over time in the latter half of the image. FIG. 7(a) shows the output signal of the PD pixel a, and FIG. 7(b) shows the output signal of the SPAD pixel 20b in the same pixel block 200 at the same timing of the incident light amount as in FIG. 7(a) (the same applies to FIG. 8 described below).

[0046] As shown in Figure 7(a), at low illumination, the PD pixel output signal has high noise, and the output signal is buried in noise. On the other hand, as shown in Figure 7(b), even at the same low illumination, the SPAD pixel output signal has the advantage of low noise and a high SNR. Figure 7(c) shows the corrected output signal obtained by the noise compression calculation using equation (1) shown in Figure 5. As shown in Figure 7(c), the corrected output signal V50 maintains the absolute value of the signal and has reduced variation (noise) compared to the output signal V10 before correction.

[0047] Figure 8 shows a specific example of an output signal in a low-light shooting environment. Figure 8 shows the transition of the output signal in a situation where the incident light is low for about 20 frames and then gradually becomes brighter. As shown in Figure 8(a), under low light conditions, the PD pixel output signal is noisy and the output signal is buried in noise. On the other hand, as shown in Figure 8(b), even under the same low light conditions, the SPAD pixel output signal has low noise and can be read out with a high SNR (signal-to-noise ratio). The SPAD pixel output signal shown in Figure 8(b) is before scale correction. The scale (sensitivity) of the PD pixel output signal is adjusted by multiplying the PD pixel output signal by the gain coefficient gr (see Figure 5). Figure 8(c) shows the corrected output signal obtained by noise compression calculation using Equation (1) shown in Figure 5. As shown in Figure 8(c), the corrected output signal V50 maintains the absolute value of the signal and has reduced variance (noise) compared to the output signal V10 before correction.

[0048] Fig. 9 shows a specific example of an output signal in a low-illuminance shooting environment. (d1) to (d7) in Fig. 9 respectively show the output signals of each pixel at two levels of irradiated light intensity. (d1) to (d7) in Fig. 9 correspond to (d1) to (d7) in Fig. 5, respectively. That is, (d1) in Fig. 9 is the PD pixel output signal, and (d7) is the SPAD pixel output signal.

[0049] In (d1) of Figure 9, the PD pixel output signal has a large amount of noise, such as shot noise and circuit noise. In (d1), the output signals corresponding to two levels of irradiated light intensity are "10" and "30." The noise is -8, +7, -18, and +17, which is about the same magnitude as the output signal. In (d1), the PD pixel output signal is dominated by circuit noise. In (d7), which shows the SPAD pixel output signal, the output signals corresponding to two levels of irradiated light intensity are "100" and "300." The noise is -14, +12, -28, and +26, which is significantly smaller than the output signal. In (d7), the noise in the SPAD pixel output signal is dominated by shot noise.

[0050] (d2) is the output signal obtained by multiplying (d1) by the gain coefficient gr. The gain coefficient gr may be changed by register settings as described above.

[0051] (d3) is the output signal obtained by subtracting (d7). In (d3), only the noise component remains (at low illumination), or the noise component contains the signal component (at low to medium illumination).

[0052] Next, this is multiplied by the noise compression coefficient rn to obtain (d4). The noise compression coefficient rn is set according to the illuminance (see Figure 6). In Figure 9, for ease of understanding, "0.1" is used as the noise compression coefficient rn. When the illuminance is low, the noise compression coefficient rn is reduced. For example, while "0.1" is used as the noise compression coefficient rn in Figure 9, it can be set to "0" when the illuminance (count value) of the SPAD pixel is 200 or less, as shown in Figure 6(a), and "0.25" can be set when the illuminance (count value) is 300 or more.

[0053] Next, (d7) is added to this to obtain (d5), which has the noise of (d7) (-14, +12, etc.) added to it.

[0054] (d6) is the final corrected output signal V50 obtained by multiplying this by 1 / gr. Note that this multiplication by 1 / gr is done as a last resort to match the final output signal to the PD output vibration, but if the scale is matched to the SPAD signal side, this final stage is omitted. Compared to the initial PD pixel output signal d(d1), the final corrected output signal V50 maintains the absolute value of the signal and has reduced variation (noise) compared to the uncorrected output signal V10.

[0055] (Example of calculation output in case of high illuminance) FIG. 10 is a schematic diagram showing the relationship between the output signals of the PD pixel and the SPAD pixel and the noise compression calculation in a high-illuminance shooting environment.

[0056] As shown in Figure 10(a), at high illuminance, the PD pixel output signal saturates at count y2. Count y2 corresponds to inflection point p2 in Figure 4A. On the other hand, as shown in Figure 10(b), even at the same high illuminance, the SPAD pixel output signal does not saturate and continues counting if the bit width of the count circuit is sufficient. Figure 10(c) shows the corrected output signal obtained by the noise compression calculation using equation (1) shown in Figure 5. As shown in Figure 10(c), the corrected output signal V50 does not saturate and illuminance information can be obtained even at high illuminance (count y2 or higher) compared to the output signal V10 before correction.

[0057] 11 to 16 are diagrams showing specific examples of noise compression calculations when low-illuminance light is incident on the pixel array 120. All of FIGS. 11 to 16 show output signals and the like from each pixel 20 in the pixel array 120, which is configured with 12 × 12 pixels 20. Similarly to the pixel blocks 200 shown in FIGS. 2A and 2B, the pixel blocks 200 are 3 × 3, with a SPAD pixel 20b in the center surrounded by eight PD pixels 20a. Only in FIGS. 11 and 12 are some of the PD pixels 20a, SPAD pixels 20b, and pixel blocks 200 designated by reference numerals. The numerical values ​​shown in FIGS. 11 to 16 are scaled to match the output signal of the PD pixel 20a. For example, in FIG. 12, if there is a 10-fold difference in scale between the output signals of the SPAD pixel 20b and the PD pixel for the same input illuminance value, the output signal (illuminance information value) of the SPAD pixel is scaled down to one-tenth.

[0058] Fig. 11 shows the amount of incident light. Fig. 12 shows pixel output signals corresponding to the amount of incident light in Fig. 11. Furthermore, noise components are added in Fig. 12, but the noise is smaller in the SPAD pixel 20b and larger in the PD pixel 20a.

[0059] 13 to 16 show the corrected output signal V50 of each pixel 20 when the level of the noise compression coefficient rn is changed. Note that for the SPAD pixel 20b at the center of each pixel block 200, the output signal V20 of the SPAD pixel 20b is output as is.

[0060] 13 shows the corrected output signal V50 when the noise compression coefficient rn=0. In this case, the output signal of the PD pixel 20a is not used, but is replaced with the output signal of the PD pixel 20a belonging to the pixel block 200 and the output signal of the SPAD pixel 20b located at the center of the pixel block. In this case, the noise component of the output signal is reduced, but the resolution (dpi) is reduced to one-third.

[0061] Figure 14 shows the corrected output signal V50 when the noise compression coefficient rn=0.125. Figure 15 shows the corrected output signal V50 when the noise compression coefficient rn=0.25. Figure 16 shows the corrected output signal V50 when the noise compression coefficient rn=0.375. As the noise compression coefficient rn increases, the noise component of the output signal increases, but the resolution gradually improves.

[0062] In the examples of the noise compression coefficient rn in FIGS. 13 to 16, the noise compression coefficient rn is uniformly used for all pixel blocks 200 regardless of the illuminance at the position of each pixel block 200 (as a predetermined low-illuminance band). However, a more preferable example would be to use a different noise compression coefficient rn for each pixel block 200 depending on the illuminance of each pixel block 200. Using a different noise compression coefficient rn for each pixel block 200 depending on the illuminance of each pixel block 200 achieves effective noise compression while maintaining high resolution and high contrast, resulting in an image close to the ideal image (noise-free image) of FIG. 11. An example of this is shown in FIGS. 17 to 19. FIG. 17 shows the amount of incident light. In the example shown in FIG. 17, the amount of light is greater in the lower right quarter region than in the other regions. FIG. 18 shows the pixel output signal corresponding to the amount of incident light in FIG. 17. Furthermore, although noise components are added in FIG. 18, the noise is smaller in the SPAD pixel 20b and larger in the PD pixel 20a. Figure 19 shows the corrected output signal V50 obtained by applying a noise compression coefficient corresponding to illuminance in the relational expression method of Figure 6(b). Here, a linear expression (rn ranges from 0 to 1) is used in which α is 0.025, β is 0, and n is 0 in the relational expression of Figure 6(b), and a different noise compression coefficient rn is used depending on the count value of each SPAD pixel, i.e., the illuminance of each pixel block. According to this embodiment, as shown in Figures 18 and 19, noise in extremely low illuminance areas is reliably suppressed, while achieving both high resolution and high contrast.

[0063] As described above, the solid-state imaging device according to the first embodiment includes a pixel array having PD pixels and SPAD pixels, and a pixel signal collaborative calculation circuit that performs noise compression calculation according to illuminance based on output signals of the PD pixels and output signals of SPAD pixels associated with the PD pixels to generate corrected output signals for the PD pixels. With this configuration, good image quality characteristics can be obtained when capturing images under shooting conditions ranging from low illuminance to high illuminance.

[0064] (Second embodiment) Next, a solid-state imaging device 100 according to a second embodiment will be described with reference to FIGS. 20 and 21. In the first embodiment described above, the PD pixel 20a to be corrected is associated with one SPAD pixel 20b in the same pixel block 200, and the output signal of this SPAD pixel 20b is used to perform a noise compression calculation for the PD pixel 20a. In contrast, in the second embodiment described below, the PD pixel 20a to be corrected is associated with multiple surrounding SPAD pixels 20b, and an integrated output signal is calculated by integrating the output signals of these PD pixels 20a. The calculated integrated output signal is then used to perform noise compression for the PD pixel 20a.

[0065] Fig. 20 is an overall block diagram showing the configuration of a solid-state imaging device 100 according to the second embodiment. Fig. 21 is a schematic diagram for explaining an integrated output signal that is integrated using the ratio of the distance to the SPAD pixel 20b. Fig. 20 corresponds to Fig. 3. As shown in Fig. 20, in the second embodiment, correction of each PD pixel 20a1-n is performed using an integrated output signal that is obtained by integrating output signals from multiple SPAD pixels 20b1-20bn.

[0066] FIG. 21 is a diagram illustrating a portion of the pixel array 120x. The arrangement of the PD pixels 20a and SPAD pixels 20b throughout the pixel array 120x is similar to the example shown in FIG. 2A, but the arrangement of the color filters is different. The pixel signal collaborative calculation circuit 27 generates an integrated output signal by integrating the output signals of multiple SPAD pixels using a distance ratio based on position information of the PD pixel to be corrected and the position information of each of the multiple PD pixels. The upper part of FIG. 21 shows the signal of one pixel block (red color) within the pixel array 120x, surrounded by a dashed line. For example, as shown in FIG. 21, the integrated output signal d11 used to correct the PD pixel 20a11 is calculated by integrating the output signals of four SPAD pixels 20b1 to 20b4 that are adjacent in the vertical, horizontal, and vertical directions around the pixel 20a11 using a distance ratio. For example, by performing bilinear interpolation in both the vertical and horizontal directions using output signals "2, 5, 5, 8" of four adjacent SPAD pixels 20b, "6" can be obtained as the integrated output signal d11 for the PD pixel 20a11. Note that the integrated output signal for the outermost PD pixel 20a for which linear interpolation (interpolation) is not possible may be obtained by directly using the output signal of one adjacent SPAD pixel 20b, or may be calculated by extrapolation as shown in FIG. 21.

[0067] Thereafter, the pixel signal collaborative calculation circuit 27 uses the integrated output signal at the position corresponding to the PD pixel 20a to be corrected to generate a corrected output signal for each PD pixel 20a through processing similar to that of the first embodiment. In this way, the solid-state imaging device according to the second embodiment performs noise compression calculation according to illuminance based on the output signal of the target PD pixel and the integrated output signal to generate a corrected output signal for each PD pixel. In this way, better image quality characteristics can be obtained under shooting conditions ranging from low illuminance to high illuminance.

[0068] (Various modified examples) 22 to 33 are diagrams showing various modified examples of the pixel array 120 used in the solid-state imaging device 100. These modified examples can be applied to the first and second embodiments. The configuration other than the pixel array is the same as that of the first or second embodiment, and therefore a description thereof will be omitted. The above-described pixel array 120 and the pixel arrays 120a to 120s shown in FIGS. 22 to 33 have in common that SPAD pixels 20b are arranged adjacent to PD pixels 20a at equal intervals throughout the entire pixel array.

[0069] In the pixel arrays 120a to 120e shown in Figures 22 to 26, the notations are as shown in the legend in Figure 22. In the pixel arrays 120a to 120c shown in Figures 23 to 25, the SPAD pixels 20b are large, and are four times (2 × 2) the size of the PD pixels 20a. The pixel arrays 120d and 120e shown in Figures 25 and 26 are examples in which the arrangement ratio of the SPAD pixels 20b is increased.

[0070] 27 to 33 are diagrams showing various modified examples of the pixel array 120 used in the solid-state imaging device 100. In the pixel arrays 120m to 120r shown in FIGS. 27 to 33, the notations are as shown in the legend of FIG. 27. In these modified examples, the SPAD pixels 20b are not provided with color filters. In these modified examples, pixel information at the position of the SPAD pixels 20b is created from pixel information of the surrounding PD pixels 20a, as will be described below.

[0071] In the modified example shown in FIG. 27, the SPAD pixel 20b12 is divided into four pixels d1 to d4 based on its position information and size information, and pixel information for each of the pixels d1 to d4 is generated by binning (averaging) the pixel information of the surrounding PD pixels 20a. For example, pixel d1 is replaced with pixel information obtained by averaging the pixel information (output signals) of the three adjacent R PD pixels 20a after noise compression. Similarly, pixel d2 is replaced with pixel information obtained by averaging the pixel information (output signals) of the three adjacent G PD pixels 20a after noise compression. Similarly, pixels d3 and d4 are replaced with G and B pixel information obtained by averaging the pixel information of the surrounding pixels, respectively. This allows an image to be generated without significantly reducing the perceived resolution, even if a colorless SPAD pixel 20b is disposed. Note that while an example in which the pixel information of the three PD pixels 20a is simply averaged is shown here, in an example such as FIG. 28, the pixel information may be weighted according to the distance ratio and binned.

[0072] (Third embodiment) Next, a solid-state imaging device according to a third embodiment will be described with reference to FIG. 34. In the third embodiment, illuminance is detected based on the output signal of the SPAD pixel 20b, and when the illuminance is low, the exposure time of the PD pixel 20a is set longer than normal compared to when the illuminance is not low. The exposure time (frame rate) of the SPAD pixel 20b is constant. In addition, when the illuminance is medium to high, other than low, the exposure times of the SPAD pixel 20b and the PD pixel 20a are the same.

[0073] As shown in FIG. 34(a), the exposure time is extended when the illuminance is low. For example, the exposure time is extended by several times, e.g., two to four times, compared to normal exposure. This reduces the effect of noise from the PD pixel 20a under low illuminance conditions. As in the first and second embodiments, the pixel signal collaborative calculation circuit 27 performs noise compression calculation according to the illuminance using the obtained output signal V10 from the PD pixel 20a and the output signal V20 from the associated SPAD pixel 20b (FIG. 34(b)). Here, the noise compression calculation can be performed using the output signal V20 of the frame from the SPAD pixel 20b, whose exposure starts at the same time as the start of the exposure period of the PD pixel 20a. However, the output signal V20 of another frame within the exposure period of the PD pixel 20a may also be used. Thereafter, the output signal from the PD pixel 20a that has been subjected to the noise compression calculation is subjected to frame interpolation processing with reference to the output signal V20 of the SPAD pixel 20b to generate a corrected output signal V50 (FIG. 34(c)). By performing frame interpolation processing using the output signals V20 of SPAD pixels with high temporal resolution, it is possible to perform frame interpolation processing with higher accuracy than frame interpolation processing using only output signals of PD pixels.

[0074] As described above, in the third embodiment, the influence of noise from the PD pixel 20a can be reduced by extending the exposure time in low illumination. Therefore, in the third embodiment, the noise compression coefficient rn in low illumination can be increased compared to the first and second embodiments. By doing so, the weighting of the PD pixel 20a in low illumination can be increased compared to the first and second embodiments, and a decrease in resolution can be suppressed even in low illumination. Furthermore, by lowering the readout rate or frame rate of the PD pixel 20, power consumption can be reduced. Furthermore, even if the frame rate of the PD pixel is reduced, the frame rate of the SPAD pixel information is not reduced, so the temporal resolution can be maintained.

[0075] (Fourth embodiment) Next, a solid-state imaging device according to a fourth embodiment will be described with reference to Fig. 35. In the fourth embodiment, illuminance is detected based on the output signals of the SPAD pixels 20b, and in the case of low illuminance, the output signals of the PD pixels 20a are averaged using a moving average.

[0076] In the example shown in Figure 35(a), the moving average is calculated over a section count of n = 4. That is, the signal for the fifth frame uses data obtained by averaging the output signals for the first to fourth consecutive frames immediately before it. The signal for the sixth frame uses data obtained by averaging the output signals for the second to fifth consecutive frames immediately before it.

[0077] In the fourth embodiment, by using data averaged by moving average in low illumination, the responsiveness to movement, etc., decreases, but the influence of noise on the PD pixel 20a can be reduced. Therefore, in the fourth embodiment, as in the third embodiment, the noise compression coefficient rn in low illumination can be made larger compared to the first and second embodiments. This allows the weighting of the PD pixel 20a in low illumination to be increased compared to the first and second embodiments, and reduces the degradation of resolution even in low illumination.

[0078] (Fifth embodiment) Next, a solid-state imaging device 100 according to a fifth embodiment will be described with reference to FIGS. 36 to 39. FIG. 36 is a diagram showing a pixel array 120e of the solid-state imaging device 100 according to the fifth embodiment, in which the same number of PD pixels 20a and SPAD pixels 20b are laid out alternately. The pixel array 120e shown in FIG. 36 is the same as the pixel array 120e shown in FIG. 26, and shows a partial range of pixels 20. FIG. 37 is a schematic diagram for explaining the usage ratio of various pixels at each illuminance. FIG. 38 is a schematic diagram for explaining the usage ratio of SPAD pixels and PD pixels at each illuminance.

[0079] (Extremely low illumination) In the fifth embodiment, during extremely low illuminance, the pixel signal collaborative calculation circuit 27 does not use pixel information from the PD pixels 20a, but uses pixel information from only the SPAD pixels 20b. Note that Fig. 37 shows only the R color, and the B and G pixels 20 are not shown, but similar processing is performed for the B and G colors. The extremely low illuminance here refers to, for example, the first half of the low illuminance range, or the range in which the noise compression coefficient rn is set to 0.0 (see Fig. 6(a)).

[0080] As shown in Figure 37[A], in the case of extremely low illuminance, the pixel signal collaborative calculation circuit 27 performs binning processing on the output signal of the SPAD pixel 20b, thereby generating one R output signal. As a result, one piece of pixel information is output from 16 pixels (4 x 4). In other words, the resolution is low. In addition, in this case, noise compression processing (equivalent to noise compression calculation) is not performed. In Figure 38, the horizontal axis represents illuminance, and the vertical axis represents the weighting ratio. The weighting ratio corresponds to the noise compression coefficient rn. As shown in Figure 38, in the case of extremely low illuminance, the output signals of the PD pixels (denoted as RGB) are not used, and only the output signals of the SPAD pixels are used.

[0081] (Low to medium illuminance) As shown in Figure 37[B], when the illuminance is low to medium (higher than extremely low illuminance), the pixel signal collaborative calculation circuit 27 performs noise compression processing on the binned output signals of the SPAD pixels 20b and the PD pixel 20a. The range of the SPAD pixel binning processing is not limited to eight pixels, and the range can be changed depending on the illuminance (four pixels, two pixels). The binned output signals of the SPAD pixels 20b can also be used as image information. As shown in Figure 38, when the illuminance is low to medium, the weighting ratio of the output signal of the PD pixel 20a gradually increases as the illuminance increases. This gradually increases the resolution.

[0082] (high illuminance) In the case of high illuminance, as shown in Figure 37[C], the pixel signal collaborative calculation circuit 27 performs noise reduction processing on the binned output signals of the SPAD pixels 20b and the output signals of the PD pixels 20a. Furthermore, the ratio of blending pixel information of the binned SPAD pixels 20b with pixel information of the non-binned SPAD pixels 20b is increased, and the blended output signals of each SPAD pixel 20b can also be used as image information. This enables higher resolution image processing.

[0083] (Ultra high illuminance) As shown in Figure 37[D], in the case of extremely high illuminance, the pixel signal collaborative calculation circuit 27 creates an image using only the SPAD pixels 20b. Here, extremely high illuminance refers to the range in which the PD pixels are saturated, or the range in which the noise compression coefficient rn is set to 0.0 on the high illuminance side (see Figure 6(a)). This allows the dynamic range to be expanded to areas where illuminance information could not be obtained using the PD pixels 20a alone due to charge saturation.

[0084] (Sixth embodiment) Next, a solid-state imaging device according to a sixth embodiment will be described with reference to FIGS. 39 to 42. In the sixth embodiment, a pixel array 120t includes PD pixels 20a for R, G, and B, SPAD pixels 21b, and PD pixels 20a used as a DVS sensor. The PD pixel 20a used as a DVS (Dynamic Vision Sensor) sensor uses a circuit configuration to detect events due to changes in incident light over time, i.e., the movement of a subject, and output the detected signals. For example, the difference between the output signals of the previous frame and the current frame is calculated, and if the difference is equal to or greater than a predetermined value, an event detection signal is output.

[0085] Furthermore, the update rate (frame rate) of the DVS sensor is generally set higher than the update rate of the RGB PD pixels 20a. In the sixth embodiment described below, the update rate of the PD pixels 20a used as the DVS sensor is also set higher than the update rate of the RGB PD pixels 20a when the illuminance is medium to low or higher (see FIG. 40(b1) described later). Furthermore, when the illuminance is low, the DVS PD pixels 20a are not used as the DVS sensor, and the SPAD pixels 20b take over that function. The SPAD pixels 20b can be used not only for motion detection but also for image processing and noise compression processing of the PD pixels 20a.

[0086] 39 is a schematic diagram showing the configuration of a pixel array 120t of a solid-state imaging device according to a sixth embodiment. In addition to RGB PD pixels 20a and PD pixels 20a, the pixel array 120t also includes DVS PD pixels 20c (hereinafter simply referred to as DVS pixels, denoted by "D" in the figure). In the pixel array 120t, the DVS pixels are arranged adjacent to the RGB PD pixels at equal intervals throughout the pixel array. Furthermore, the range of influence of an event detected by the DVS pixels 20c, which will be described below, is reflected in the exposure control of the PD pixels 20a belonging to the same pixel block.

[0087] Fig. 40 is a schematic diagram for explaining the usage ratio of each pixel used for motion detection and update rate determination according to the sixth embodiment. Fig. 41 is a flowchart showing processing according to the sixth embodiment.

[0088] (Processing based on the output signal of the SPAD pixel 20b) (Steps S11 and S12) Steps S11-S13 are processes performed based on the output signals of the SPAD pixels 20b. As shown in Figure 41, the pixel signal collaborative calculation circuit 27 first determines the illuminance by detecting the SPAD pixels 20b. Specifically, the illuminance of the SPAD pixels 20b is classified into low, medium, and high illuminance. The predetermined low, medium, and high illuminances are the same as the ranges described in Figure 4A, but are not necessarily limited to this and may be set to a second predetermined range different from this.

[0089] (Step S13) The pixel signal collaborative calculation circuit 27 performs settings for motion detection, update rate, and pixel (compression rate) generation according to the illuminance.

[0090] (Steps S131-S133) In this process, as shown in FIG. 40(a), the pixel signal collaborative calculation circuit 27 performs motion detection using the signal from the SPAD pixel 20b more preferentially than the signal from the DVS pixel 20c at low illuminance. At medium illuminance, the pixel signal collaborative calculation circuit 27 performs motion detection using the DVS pixel 20c more preferentially than the SPAD pixel 20b. At high illuminance, the pixel signal collaborative calculation circuit 27 performs motion detection using the DVS pixel 20c more preferentially than the SPAD pixel 20b. The result of this determination is handed over to the process in step S21.

[0091] (Steps S134-S136) Furthermore, with regard to image creation, as shown in FIG. 40(c), the pixel signal collaborative calculation circuit 27 prioritizes the SPAD pixel 20b over the RGB pixel 20a when detecting signals at low illuminance (the noise compression coefficient rn is set to less than 0.5). At medium illuminance, the pixel signal collaborative calculation circuit 27 prioritizes the RGB pixel 20a over the SPAD pixel 20b when detecting signals (for example, the noise compression coefficient rn is set to more than 0.5). At high illuminance, the pixel signal collaborative calculation circuit 27 prioritizes the SPAD pixel 20b over the RGB pixel 20a when detecting signals (for example, the noise compression coefficient rn is set to less than 0.5). The processing here is similar to the noise compression processing of the first embodiment shown in FIGS. 3 to 6.

[0092] (Processing based on the output signal of the DVS pixel 20c) Steps S21 to S23 are processes that are performed mainly based on the output signal of the DVS pixel 20c.

[0093] (Step S21) The pixel cooperation control circuit 28 executes motion detection cooperation control, in which, as shown in Fig. 40(a), an event is detected using signals from one or both of the DVS pixels 20c and the SPAD pixels 20b depending on the illuminance.

[0094] (Step S22) Depending on the result of the motion detection, the pixel cooperation control circuit 28 advances the process to exposure control in step S23, where exposure of the RGB pixels is controlled.

[0095] (Step S231) The pixel cooperation control circuit 28 can reduce power consumption by stopping exposure of the RGB pixels 20a or slowing down the frame rate depending on the motion detection stationary determination result. The processing here is as shown in Figure 40(b2).

[0096] (Step S232) The pixel cooperation control circuit 28 exposes the RGB pixels 20a depending on the event detection, i.e., the determination result of a moving object (motion). The processing here is as shown in FIG. 40(b1). In the case of medium illuminance, the RGB pixels 20a are exposed at a normal frame rate, and in the case of low or high illuminance, the RGB pixels 20a are exposed at a lower frame rate or the exposure is stopped. Since the SPAD pixels and RGB pixels operate cooperatively only in the motion detection area of ​​the DVS pixel 20c, power saving can be achieved.

[0097] (Processing based on output signals of RGB pixels 20a) In step S31, the output signals of the RGB pixels 20a are subjected to noise compression processing similar to that in embodiment 1. The pixel cooperation control circuit 28 controls the exposure of the RGB pixels 20a according to the motion detection and update rate determination results in steps S13 and S23. This processing is as shown in Figures 40(b1), (b2), and (c).

[0098] (Variation) In the sixth embodiment described with reference to FIGS. 39 to 41, exposure of the RGB pixels 20a is stopped when there is no motion, but the frame rate may be reduced as in the modified example shown in FIG. 42. In the example shown in FIG. 42, when the event detection result by the DVS pixel 20c determines that there is "motion," exposure and readout are performed at a normal frame rate (e.g., 60 fps). On the other hand, when there is no motion and the pixel is determined to be "still," the frame rate is set to a rate several times lower than normal (e.g., 15 or 12 fps). This allows power consumption to be reduced without affecting image quality.

[0099] (Seventh embodiment) Next, a solid-state imaging device according to the seventh embodiment will be described with reference to FIG. 43. In the seventh embodiment, the arrangement of the PD pixels 20a and the SPAD pixels 20b in the pixel array 120 is the same as in the first and second embodiments. For example, the arrangement is as shown in FIG. 43(a). FIG. 43(b) is an overall block diagram showing the configuration of a solid-state imaging device according to the seventh embodiment. In the pixel array 120, all of the PD pixels 20a are imaged by the vertical scanning unit 130 (see FIG. 1) using a rolling shutter method in which exposure and signal charge reading are performed row by row to sequentially collect output signals. Meanwhile, all of the SPAD pixels 20b are imaged by a global shutter method in which the start and end of the exposure period are the same. Furthermore, for each of the PD pixels 20a, a pixel signal collaborative calculation circuit 27 performs noise compression processing using the output signals of the SPAD pixels 20b described in the first embodiment.

[0100] In the seventh embodiment, the PD pixel data memory stores first image data generated by the rolling shutter method using output signals from the PD pixels 20a1-n after noise compression processing correction, and the SPAD pixel data memory stores second image data generated by the global shutter method using output signals from the SPAD pixels 20b1-n.

[0101] The application corrects the first image data (hereinafter referred to as "rolling shutter correction") based on information about a pair of first and second image data obtained by corresponding simultaneous capture, so that the first image data is equivalent to that captured using a global shutter method, and generates corrected image data. The application also calculates the velocity and acceleration of an object included in the image based on the rolling shutter speed of the PD pixel 20a and information about the first and second image data. The calculated information is output as object velocity and acceleration information. While the seventh embodiment illustrates an example in which noise compression processing is performed by the pixel signal collaborative calculation circuit 27, this may be omitted. Specifically, the application may perform rolling shutter correction using first image data generated from the output signal of the PD pixel 20a that is not subjected to noise compression processing, and second image data generated from the output signal of the SPAD pixel 20b.

[0102] In this way, in the seventh embodiment, first image data is generated from the SPAD pixels using the global shutter method, second image data is generated from the PD pixels using the rolling shutter method, and the first and second image data are simultaneously captured and rolling shutter correction is performed on the second image data, thereby correcting distortion of moving objects, etc., that occurs with the rolling shutter method.

[0103] This embodiment has the following advantages due to the following configuration.

[0104] (1) The solid-state imaging device according to this embodiment includes a pixel signal collaboration circuit that performs noise compression calculations according to illuminance based on the output signals of PD pixels and SPAD pixels associated with the PD pixels, and generates corrected output signals for each PD pixel. This allows for good image quality characteristics to be obtained when capturing images under a wide range of illumination conditions, from low to high.

[0105] (2) In the solid-state imaging device according to this embodiment, each PD pixel is associated with a plurality of the SPAD pixels, and the pixel signal collaborative calculation circuit performs noise compression calculation according to illuminance based on the output signals of the plurality of SPAD pixels associated with the PD pixel, to generate a corrected output signal for each of the PD pixels. The pixel signal collaborative calculation circuit also generates an integrated output signal by integrating the output signals of the plurality of SPAD pixels using a distance ratio based on positional information of the PD pixel to be corrected and positional information of each of the plurality of SPAD pixels. Then, based on the output signal of the target PD pixel and the integrated output signal, it performs noise compression calculation according to illuminance to generate a corrected output signal for each PD pixel. This enables noise compression processing with higher accuracy, resulting in better image quality characteristics.

[0106] (3) In the solid-state imaging device according to this embodiment, when the illuminance is low, the pixel signal collaborative calculation circuit generates a corrected output signal for the PD pixel by using the output signal of the SPAD pixel at a higher ratio than the output signal of the PD pixel, thereby obtaining image information with reduced noise and a high SNR.

[0107] (4) In the solid-state imaging device according to this embodiment, the pixel signal collaborative calculation circuit generates a corrected output signal for the PD pixel by using the output signal of the PD pixel at a higher ratio than the output signal of the SPAD pixel when the illuminance is a predetermined medium level. This allows high-resolution image information to be obtained.

[0108] (5) In addition, in the solid-state imaging device according to this embodiment, when the illuminance is high, the pixel signal collaborative calculation circuit generates a corrected output signal for the PD pixel by using the output signal of the SPAD pixel at a higher ratio than the output signal of the PD pixel, thereby obtaining image information with a high dynamic range.

[0109] (6) In addition, in the solid-state imaging device according to this embodiment, the pixel signal collaborative calculation circuit extends the exposure time of the PD pixels to be longer than the exposure time of the SPAD pixels when the illuminance is a predetermined low level. This reduces noise in the PD pixels themselves in low illuminance, thereby further increasing the SNR. Furthermore, if the increased SNR is allocated to improving resolution and the noise compression rate is relaxed, a decrease in resolution can be suppressed.

[0110] (7) In the solid-state imaging device according to this embodiment, the pixel signal collaborative calculation circuit uses a moving average of the PD pixel over a number of consecutive frames as the output signal of the PD pixel, thereby achieving the same effect as in (6) above.

[0111] (8) In addition, in the solid-state imaging device according to this embodiment, when an event is detected, the pixel signal collaborative calculation circuit performs a noise compression calculation according to the illuminance based on the output signals of the PD pixels and the output signals of the SPAD pixels associated with the PD pixels, and generates a corrected output signal for each PD pixel. This reduces power consumption without affecting image quality.

[0112] The above-described configuration of the solid-state imaging device 100 etc. has been described as a main configuration for explaining the features of the above-described embodiment, but is not limited to the above configuration and may be modified in various ways within the scope of the claims. Furthermore, configurations included in general solid-state imaging devices 100 etc. are not excluded. [Explanation of symbols]

[0113] 100 solid-state image sensor 120 pixel array 200 pixel blocks 20 pixels 20a PD pixel (RGB pixel) 20b SPAD pixel 20c PD pixel (DVS pixel) 27 Pixel signal cooperative calculation circuit 271a Noise compression coefficient setting circuit 271b Calculation parameters 28 Pixel coordination control circuit

Claims

1. a pixel array having PD pixels and SPAD pixels; a pixel signal collaborative calculation circuit that performs a noise compression calculation according to illuminance based on the output signals of the PD pixels and the output signals of the SPAD pixels associated with the PD pixels, and generates a corrected output signal for each of the PD pixels; A solid-state imaging element comprising:

2. A plurality of the SPAD pixels are associated with each of the PD pixels, 2. The solid-state imaging device according to claim 1, wherein the pixel signal collaborative calculation circuit performs a noise compression calculation according to illuminance based on output signals of the plurality of SPAD pixels associated with the PD pixel, and generates a corrected output signal for each of the PD pixels.

3. 2. The solid-state imaging device according to claim 1, wherein when the illuminance is a predetermined low illuminance, the pixel signal collaborative calculation circuit uses the output signal of the SPAD pixel at a higher ratio than the output signal of the PD pixel to generate the corrected output signal of the PD pixel.

4. 4. The solid-state imaging device according to claim 3, wherein when the illuminance is a predetermined medium illuminance that is higher than the low illuminance, the pixel signal collaborative calculation circuit uses the output signal of the PD pixel at a higher ratio than the output signal of the SPAD pixel to generate the corrected output signal of the PD pixel.

5. 5. The solid-state imaging device according to claim 4, wherein when the illuminance is a predetermined high illuminance that is higher than the medium illuminance, the pixel signal collaborative calculation circuit uses the output signal of the SPAD pixel at a higher ratio than the output signal of the PD pixel to generate the corrected output signal of the PD pixel.

6. The pixel signal collaborative calculation circuit calculates a corrected output signal of the PD pixel by the following formula 1 using a noise compression coefficient rn according to illuminance:

2. The solid-state imaging device according to claim 1, wherein the noise compression coefficient rn is set to be lowest at low illuminance and high illuminance, and highest at medium illuminance between the low illuminance and the high illuminance, and is set to be gradually higher values ​​as the illuminance increases from the low illuminance to the medium illuminance, and is set to be gradually lower values ​​as the illuminance increases from the medium illuminance to the high illuminance. V50=((V10×gr-V20)×rn+V20) / gr (Formula 1) (where V50 is the corrected output signal, V10 is the output signal value of the PD pixel, V20 is the output signal value of the SPAD pixel, and gr is the gain coefficient for adjusting the scale of the output signal)

7. 2. The solid-state imaging device according to claim 1, wherein said pixel signal collaborative calculation circuit makes the exposure time of said PD pixels longer than the exposure time of said SPAD pixels when said illuminance is a predetermined low illuminance.

8. 2. The solid-state imaging device according to claim 1, wherein the pixel signal collaborative calculation circuit uses a moving average of the PD pixel over a plurality of consecutive frames as the output signal of the PD pixel when the illuminance is a predetermined low illuminance.

9. 9. The solid-state imaging device according to claim 3, wherein the predetermined low illuminance is in the range of 10 to 100 lux.

10. the pixel signal collaborative calculation circuit generates an integrated output signal by integrating output signals of the plurality of SPAD pixels using a distance ratio based on position information of the PD pixel to be corrected and position information of each of the plurality of SPAD pixels; 3. The solid-state imaging device according to claim 2, wherein a noise compression calculation is performed according to illuminance based on the output signal of the target PD pixel and the integrated output signal, to generate a corrected output signal for each of the PD pixels.

11. The noise compression coefficient rn can be set by the user according to the illuminance, 2. The solid-state imaging device according to claim 1, wherein the pixel signal collaborative calculation circuit calculates the corrected output signal of the PD pixel by the following formula 1 using the noise compression coefficient rn according to a set illuminance. V50=((V10×gr-V20)×rn+V20) / gr (Formula 1) (where V50 is the corrected output signal, V10 is the output signal value of the PD pixel, V20 is the output signal value of the SPAD pixel, and gr is the gain coefficient for adjusting the scale of the output signal)

12. 2. The solid-state imaging device according to claim 1, wherein the noise compression operation is realized by a bit shift operation.

13. A plurality of the SPAD pixels are associated with each of the PD pixels, 2. The solid-state imaging device according to claim 1, wherein the pixel signal collaborative calculation circuit performs a noise compression calculation according to illuminance based on binning output signals of the plurality of SPAD pixels associated with the PD pixel, and generates a corrected output signal for each of the PD pixels.

14. 14. The solid-state imaging device according to claim 13, wherein the output signals of the binned SPAD pixels are blended with the output signals of the non-binned SPAD pixels, and the blended output is output as pixel information.

15. the pixel array further comprises a DVS pixel for event detection; 2. The solid-state imaging device according to claim 1, wherein, when an event is detected, the pixel signal collaborative calculation circuit performs a noise compression calculation according to illuminance based on the output signal of the PD pixel and the output signal of the SPAD pixel associated with the PD pixel, to generate a corrected output signal for each of the PD pixels.

16. The solid-state imaging device according to claim 15 , wherein the event is detected using the output signal of the SPAD pixel and the output signal of the DVS pixel at a ratio according to illuminance.

17. 17. The solid-state imaging device according to claim 16, wherein when the illuminance is a second predetermined low illuminance, the event is detected using a higher ratio of the output signal of the SPAD pixel than the output signal of the DVS pixel.

18. 18. The solid-state imaging device of claim 17, wherein when the illuminance is higher than the second predetermined low illuminance, the event is detected using an output signal of the DVS pixel at a higher ratio than an output signal of the SPAD pixel, or using an output signal of only the DVS pixel.

19. No color filter is disposed at the position of the SPAD pixel of the pixel array, 2. The solid-state imaging device according to claim 1, wherein an output signal of a color corresponding to the position of the SPAD pixel is generated from the PD pixel provided with a color filter around the SPAD pixel.

20. generating first image data from the SPAD pixels in a global shutter system; generating second image data from the PD pixels of a rolling shutter system; 2. The solid-state imaging device according to claim 1, wherein rolling shutter correction is performed on the second image data from the first and second image data captured simultaneously.

21. 2. The solid-state imaging device according to claim 1, wherein the SPAD pixels are arranged adjacent to the PD pixels at equal intervals across the entire pixel array.

22. 16. The solid-state imaging device according to claim 15, wherein the DVS pixels are arranged adjacent to the PD pixels at equal intervals throughout the pixel array.

Images