Photoelectric conversion device, control device, and photoelectric conversion system

Abstract

To provide an advantageous technique for improving quality of images acquired in devices that capture images of different wavelength ranges.SOLUTION: A photoelectric conversion device includes: an imaging element having a plurality of pixels that output digital data according to the number of incident photons; and a control part that controls a projection part. The plurality of pixels include: a first pixel having sensitivity to light in a first wavelength range; and a second pixel having sensitivity to light in a second wavelength range different from the first wavelength range. The projection part generates a projection light including the second wavelength range. The control part controls the projection part such that an effective light emission period in which an intensity of the projection light is larger than a predetermined intensity and a non-effective light emission period in which the intensity of the projection light is smaller than the predetermined intensity are defined, and a first imaging period in which the first pixel performs imaging includes at least a part of the non-effective light emission period. A second imaging period that does not include the effective light emission period and in which the second pixel performs imaging includes at least a part of the effective light emission period and at least a part of the non-effective light emission period.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a photoelectric conversion device, a control device, and a photoelectric conversion system. [Background technology]

[0002] In mixed reality, in which virtual space information is superimposed on real space in real time and displayed to the user, it is necessary to estimate the three-dimensional shape of objects existing in real space. In particular, a higher level of mixed reality can be achieved by estimating the three-dimensional shape of a specific subject in real space that is superimposed on virtual space with high accuracy and speed. To achieve this, there is a method for simultaneously capturing an image of real space and estimating its three-dimensional shape. Patent Document 1 describes a range image sensor that uses a solid-state imaging device capable of capturing images with visible light and infrared light to capture images with visible light and infrared light every frame scanning period while irradiating the space to be captured with IR pulses every other frame scanning period. This range image sensor generates a range image that eliminates the influence of the infrared component in external light by subtracting the IR pixel signal obtained by capturing images without IR pulse irradiation from the IR pixel signal obtained by capturing images with IR pulse irradiation. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2008-8700 Summary of the Invention [Problem to be solved by the invention]

[0004] In a device that captures images in different wavelength ranges, such as the visible light range and the infrared range, if the imaging periods for the two types of pixels for capturing images in the different wavelength ranges are determined exclusively, it may be difficult to obtain images with quality appropriate for the respective purposes of the two types of pixels.

[0005] An object of the present invention is to provide an advantageous technique for improving the quality of images acquired in an apparatus that captures images in different wavelength ranges. [Means for solving the problem]

[0006] One aspect of the present invention relates to a photoelectric conversion device including an imaging element having a plurality of pixels that output digital data corresponding to the number of incident photons, and a control unit that controls a projection unit, wherein in the photoelectric conversion device, the plurality of pixels include first pixels that are sensitive to light in a first wavelength range and second pixels that are sensitive to light in a second wavelength range different from the first wavelength range, the projection unit generates projection light that includes the second wavelength range, the control unit controls the projection unit to define effective light emission periods during which the intensity of the projection light is greater than a predetermined intensity and non-effective light emission periods during which the intensity of the projection light is less than the predetermined intensity, the first imaging period during which the first pixels perform imaging includes at least a portion of the non-effective light emission period and does not include the effective light emission period, and the second imaging period during which the second pixels perform imaging includes at least a portion of the effective light emission period and at least a portion of the non-effective light emission period. [Effects of the Invention]

[0007] According to the present invention, an advantageous technique is provided for improving the quality of images acquired in a device that captures images in different wavelength ranges. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing the configuration of an imaging system according to a first embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of an arrangement of a plurality of pixels in the image sensor of the first embodiment. [Figure 3] 1 is a diagram showing a configuration of an image sensor according to a first embodiment and a detailed configuration example of each pixel included in the image sensor; [Figure 4] FIG. 2 is a diagram illustrating an example of the operation of the imaging system according to the first embodiment. [Figure 5] 3A and 3B are diagrams schematically showing a first image and a second image captured by the imaging system of the first embodiment. [Figure 6] FIG. 10 is a diagram illustrating an example of the operation of the imaging system according to the second embodiment. [Figure 7] FIG. 10 is a diagram illustrating an example of the operation of the imaging system according to the second embodiment. [Figure 8] FIG. 10 is a diagram illustrating an example of the operation of the imaging system according to the third embodiment. [Figure 9] FIG. 10 is a diagram illustrating the operation of the imaging system according to the fourth embodiment. [Figure 10] FIG. 10 is a diagram showing the configuration of an imaging system according to a fifth embodiment. [Figure 11] FIG. 10 is a diagram showing an example of the configuration of an image sensor according to a fifth embodiment and a detailed configuration of each pixel included in the image sensor. [Figure 12] FIG. 11 is a diagram illustrating an example of the operation of the imaging system according to the fifth embodiment. [Figure 13] FIG. 13 is a diagram showing the configuration of an imaging system according to a sixth embodiment. [Figure 14] FIG. 13 is a diagram showing a configuration of an image sensor according to a sixth embodiment and a detailed configuration example of each pixel included in the image sensor. [Figure 15] FIG. 11 is a diagram illustrating an example of the operation of the imaging system according to the fifth embodiment. [Figure 16] FIG. 13 is a diagram showing the configuration of an imaging system according to a seventh embodiment. [Figure 17] FIG. 13 is a diagram showing an example of the configuration of an image sensor according to a seventh embodiment and a detailed configuration of each pixel included in the image sensor. [Figure 18] FIG. 13 is a diagram illustrating the operation of the imaging system according to the seventh embodiment. [Figure 19] FIG. 13 is a diagram illustrating an example of the operation of the imaging system according to the eighth embodiment. [Figure 20] FIG. 13 is a diagram showing an example of the configuration of an image sensor according to an eighth embodiment and a detailed configuration of each pixel included in the image sensor. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] The imaging system described below is an application example of the photoelectric conversion device or photoelectric conversion system of the present invention.

[0011] (First embodiment) An imaging system 100 according to a first embodiment will be described with reference to FIGS. 1, 2, 3, 4, 5, and 6. FIG. 1 illustrates the configuration of the imaging system 100 according to the first embodiment. The imaging system 100 may include an imaging device 101, a projection unit 103, and a control unit 104. The imaging device 101 includes an imaging element 102. The imaging element 102 may include a plurality of pixels that output digital data (pixel values) corresponding to the number of incident photons. In one example, these pixels may include avalanche photodiodes (APDs). The plurality of pixels may include a first pixel sensitive to light in a first wavelength range and a second pixel sensitive to light in a second wavelength range different from the first wavelength range. Here, an example will be described in which the first wavelength range is the wavelength range of visible light and the second wavelength range is the wavelength range of infrared light (infrared light). However, this is merely an example, and the two may be different wavelength ranges in the wavelength range of visible light, for example.

[0012] The imaging device 101 may include an optical system that forms an optical image of an object to be imaged on an imaging plane of the imaging element 102. The projection unit 103 projects projection light in a second wavelength range (in this example, the wavelength range of infrared light) onto an area covering the angle of view of the imaging device 101. The projection light may be, for example, patterned light (pattern light). The pattern of the pattern light is preferably a random pattern in which no similar portions appear within the projected area, but is not limited thereto and may be, for example, a dot pattern or a grid pattern. The projection light may also be uniform light. The control unit 104 may be configured to control the projection unit 103. The control unit 104 may also be configured to control the imaging device 101. Alternatively, the control unit 104 may be incorporated into the imaging device 101. The imaging system 100 may be configured to capture a first image (here, a visible light image) using a plurality of first pixels and a second image (here, a visible light and infrared light image) using a plurality of second pixels.

[0013] FIG. 2 illustrates an example of an arrangement of multiple pixels in the image sensor 102. The multiple pixels in the image sensor 102 include, for example, three types of pixels sensitive to visible light, specifically, R pixels 201, G pixels 202, and B pixels 203, and one type of pixel sensitive to infrared light, IR pixels 204. However, when obtaining a monochrome image of visible light, only one type of pixel sensitive to visible light may be used. The R pixels 201, G pixels 202, and B pixels 203 are examples of first pixels sensitive to light in a first wavelength range, and the IR pixels 204 are examples of second pixels sensitive to light in a second wavelength range different from the first wavelength range. The arrangement of the multiple pixels is not limited to the example illustrated in FIG. 2. For example, the first row may alternately include R pixels and B pixels, and the second row may alternately include G pixels and IR pixels.

[0014] FIG. 3 illustrates the configuration of the image sensor 102 and a detailed example of the configuration of each pixel included in the image sensor 102. The R pixel 201, G pixel 202, and B pixel 203, which are pixels sensitive to visible light, and the IR pixel 204, which is a pixel sensitive to infrared light, may have the same circuit configuration. The image sensor 102 may include, for example, an image capture control unit 301, a readout unit 302, and a vertical scanning circuit 311. FIG. 3 illustrates only four pixels: the R pixel 201, the G pixel 202, the B pixel 203, and the IR pixel 204. The image capture control unit 301 controls each pixel via the vertical scanning circuit 311. The pixel values ​​(digital data) obtained by each pixel are read out by the readout unit 302. The image capture control unit 301 sends various signals for controlling image capture and readout to the vertical scanning circuit 311. The image capture control unit 301 also sends a signal for controlling readout to the readout unit 302.

[0015] The vertical scanning circuit 311 has an ENABLE signal, a RESET signal, a READn signal, and a LATCH signal as signals for controlling imaging and readout of each pixel. The ENABLE signal is a signal that enables the count operation of the counter 306 of each pixel when asserted (activated). The RESET signal is a signal that resets the count value (pixel value) of the counter 306 of each pixel when asserted. The LATCH signal is a signal that latches the count value of the counter 306 in a pixel when it transitions (rising in this case). The READn signal is a signal provided for each row, and when asserted, causes each pixel in the corresponding row to output pixel values ​​(digital data) to the readout unit 302. The readout unit 302 sequentially outputs the pixel values ​​(digital data) read out from each pixel to the outside of the image sensor 102.

[0016] Next, each pixel will be described in detail. Each pixel may include, for example, an APD 303, a quench element 304, a waveform shaping unit 305, a counter 306, a latch circuit 309, and a selection transistor 310. The APD 303 is a photoelectric conversion unit. When a photon is incident on the APD 303, a charge pair corresponding to the incident light is generated by photoelectric conversion. A first voltage is supplied to the anode of the APD 303, and a second voltage higher than the first voltage may be supplied to the cathode of the APD 303. The charge pairs generated by photoelectric conversion cause avalanche multiplication, generating an avalanche current. The quench element 304 connects the APD 303 to a power supply that supplies the second voltage and converts changes in the avalanche current generated in the APD 303 into a voltage signal. The quench element 304 also functions as a load circuit during signal multiplication by avalanche multiplication, suppressing the voltage supplied to the APD 303 to stop avalanche multiplication.

[0017] The waveform shaping unit 305 generates a pulse signal by shaping a change in the potential of the cathode of the APD 303 when a photon is incident on the APD 303. The counter 306 can operate by receiving an ENABLE signal (enable signal) and a RESET signal (reset signal) supplied from the vertical scanning circuit 311, as well as a pulse signal supplied from the waveform shaping unit 305. The counter 306 can include an AND circuit 308 as an enable circuit, and a counter circuit 307. When the ENABLE signal is asserted and the RESET signal is deasserted, if a pulse is supplied from the waveform shaping unit 305, the count value of the counter circuit 307 is incremented. When the RESET signal is asserted, the count value of the counter circuit 307 is reset to a predetermined value.

[0018] The latch circuit 309 is a circuit that latches the count value of the counter 306 (counter circuit 307) in response to the rising edge of a LATCH signal (latch signal). The value latched by the latch circuit 309 is transmitted to a downstream selection transistor 310. When a READ signal (read signal) is asserted, the selection transistor 310 outputs the count value as a pixel value (digital data) to the readout unit 302 via a read line. Each pixel maintains its pixel value even after the pixel value (digital data) is read out by the readout unit 302.

[0019] FIG. 4 illustrates the operation of the imaging system 100. The control unit 104 outputs an imaging start signal, a readout start signal, and a projection signal, and the vertical scanning circuit 311 outputs an ENABLE signal, a RESET signal, a LATCH signal, and a READ signal. The control unit 104 can control the projection unit 103 using the projection signal to define an effective light-emitting period during which the intensity of the projection light is greater than a predetermined intensity and a non-effective light-emitting period during which the intensity of the projection light is less than the predetermined intensity. In this example, the period during which the projection signal is asserted is the effective light-emitting period. The predetermined intensity is, for example, a target minimum intensity. Even immediately after the current supply to a light-emitting element of the projection unit 103 is cut off, the light-emitting element may still emit some light, but the period during which the light is less than the predetermined intensity is the non-effective light-emitting period.

[0020] The control unit 104 or the vertical scanning circuit 311 controlled by the control unit 104 defines a first imaging period during which the R pixel 201, G pixel 202, and B pixel 203 serving as first pixels capture images. The control unit 104 or the vertical scanning circuit 311 controlled by the control unit 104 defines a second imaging period during which the IR pixel 204 serving as second pixels capture images. The imaging period during which the pixels capture images can be understood as a period during which the number of incident photons on the pixels can be reflected in digital data as an output value. The first imaging period during which the R pixel 201, G pixel 202, and B pixel 203 serving as first pixels capture images is a period that includes at least a portion of the non-effective light emission period but does not include the effective light emission period. The second imaging period during which the IR pixel serving as second pixels captures images is a period that includes at least a portion of the effective light emission period and at least a portion of the non-effective light emission period.

[0021] 4, the control unit 104 defines the effective light-emitting period and the non-effective light-emitting period so that one imaging cycle includes a non-effective light-emitting period and an effective light-emitting period following the non-effective light-emitting period. In the example shown in Fig. 4, the first imaging period is the period from the start to the end of the non-effective light-emitting period, and the second imaging period is the period from the start of the non-effective light-emitting period to the end of the effective light-emitting period.

[0022] The latch circuit 309 of each of the multiple pixels (first pixel, second pixel) may perform a latch operation at the end of the first imaging period and perform a latch operation again at the end of the second imaging period. The counter 306 of each of the multiple pixels may be released from reset at the start of the first imaging period and the second imaging period, and the count operation may be enabled throughout the second imaging period. The readout unit 302 reads digital data from the latch circuits 309 of the R pixel 201, G pixel 202, and B pixel 203 as the first pixel after the end of the first imaging period, and reads digital data from the latch circuit 309 of the IR pixel 204 as the second pixel after the end of the second imaging period.

[0023] A more specific example of the operation of the imaging system 100 will be described below. At time t401, the control unit 104 raises the imaging start signal. This rise indicates the start and end of an imaging cycle. Therefore, the imaging cycle that started at time t401 ends with the rise of the imaging start signal at time t404. Time t401 also marks the start of the first imaging period, second imaging period, and non-emission period within the imaging cycle. In response to the rise of the imaging start signal, the imaging control unit 301 transmits the imaging start signal to the vertical scanning circuit 311. As a result, the vertical scanning circuit 311 asserts the ENABLE signal to start imaging at each pixel.

[0024] At time t402, the control unit 104 raises the read start signal. Here, the control unit 104 raises the read start signal so as to end the first imaging period after the rise of the imaging start signal has elapsed a first predetermined time. This rise causes the imaging control unit 301 to transmit the read start signal to the vertical scanning circuit 311 and the readout unit 302. This causes the vertical scanning circuit 311 to raise the LATCH signal and begin asserting the READn signals for multiple rows sequentially. When the LATCH signal rises, the pixel value (digital data), which is the count value of the counter 306, is latched into the latch circuit 309 for each pixel, and the pixel value is read from the pixel in the corresponding row by asserting the READn signal. At this time, the pixel value latched by the latch circuit 309 is maintained. In this state, when the waveform shaping unit 305 generates a pulse signal for each pixel, the counter 306 increments its count value upon receiving the pulse signal. The vertical scanning circuit 311 asserts the READn signals for multiple rows in a predetermined order, causing the pixel values ​​of multiple rows to be read out sequentially by the readout unit 302. In response to the rising edge of the readout start signal, the control unit 104 asserts a projection signal, and in response to this, the projection unit 103 starts projecting projection light. Time t402 may be understood as the start time of the effective light emission period.

[0025] At time t403, the control unit 104 again raises the read start signal. Here, the control unit 104 raises the read start signal so as to end the second imaging period in response to the passage of a second predetermined time after the rise of the read start signal at time t402. This rise causes the projection signal to be deasserted, and the projection unit 103 ends the projection of projection light at time t403. The imaging control unit 301 also transmits the read start signal to the vertical scanning circuit 311 and the readout unit 302. This causes the vertical scanning circuit 311 to raise the LATCH signal and begin an operation of sequentially asserting the READn signals of multiple rows in a predetermined order. When the LATCH signal rises, the pixel value (digital data), which is the count value of the counter 306, is latched in the latch circuit 309 of each pixel, and the pixel value is read out when the READn signal is asserted. The rising edge of the LATCH signal and the subsequent readout of pixel values ​​by asserting the READn signal need only be completed by time t405 when the next readout starts, and may therefore be performed during the period from time t403 to time t405.

[0026] At time t404, the control unit 104 again raises the imaging start signal. Here, the control unit 104 raises the imaging start signal in response to the passage of a third predetermined time from the raising of the readout start signal at time t403, so as to end one imaging cycle that started at time t401. As described above, this raising marks the end of one imaging cycle and simultaneously starts the next imaging cycle.

[0027] The subsequent operations from time t404 to time t407 are a repetition of the operations from time t401 to time t404.

[0028] The period from time t401 to time t402, the period from time t402 to time t403, and the period from time t403 to time t404 can be determined based on the setting of the imaging cycle from time t401 to time t404. For example, the imaging cycle from time t401 to time t404 may be set to 16.666 ms, the period from time t401 to time t402 may be set to 15 ms, the period from time t402 to time t403 may be set to 1 ms, and the period from time t403 to time t404 may be set to 0.666 ms.

[0029] Furthermore, the above-mentioned periods may be adjusted for each imaging cycle depending on the imaging environment and / or the implementation of the imaging system. For example, projection light may be projected before the visible light image is read out as long as it does not affect the visible light image, or imaging by the IR pixel 204 may continue even after the effective light emission period has elapsed as long as the S / N ratio required for the infrared light image can be maintained.

[0030] The imaging control unit 301 may be understood as a control device. The control unit 1904 may be configured to control an imaging element having a plurality of pixels arranged to form a plurality of rows and a plurality of columns. Here, the plurality of pixels may include a first pixel (e.g., a B pixel 203) and a second pixel (e.g., an IR pixel 204) belonging to the same row, where the first pixel may be sensitive to light in a first wavelength range and the second pixel may be sensitive to light in a second wavelength range different from the first wavelength range. The imaging control unit 301 may be configured to generate a first signal defining the imaging period of the first pixel and a second signal defining the imaging period of the second pixel during a period in which signals from the first pixel and the second pixel are read out. In the example shown in FIG. 4 , the first signal is a readout start signal that transitions at time t402, and the second signal is a readout start signal that transitions at time t403. The first signal may define the end of the imaging period of the first pixel, and the second signal may define the end of the imaging period of the second pixel.

[0031] FIG. 5 illustrates a visible light image (first image) and an infrared light image (second image) captured in a certain environment. FIG. 5(A) schematically illustrates a visible light image (first image) captured by first pixels (R pixels 201, G pixels 202, and B pixels 203) without projecting infrared light (projected light). FIG. 5(B) schematically illustrates an infrared light image (second image) captured by projecting infrared light (projected light). When there is sufficient brightness in the environment but little infrared light, the infrared light image will be darker than the visible light image, as in the image in FIG. 5(B). However, by projecting infrared light, the brightness and features of the infrared light image can be increased. Note that FIG. 5(B) schematically illustrates an example in which the projected light includes a dot pattern, and the shape of an object present within the angle of view (field of view) is not taken into consideration. In reality, the dot pattern is arranged in the image according to the shape of the object.

[0032] According to the first embodiment, the imaging system 100 performs imaging using the first pixels without projecting projection light to obtain a first image (e.g., a visible light image). The imaging system 100 also performs imaging using the second pixels with and without projecting projection light to obtain a second image (e.g., a visible light and infrared light image). Therefore, the influence of the projection light is suppressed in the first image, and the S / N ratio can be improved in the second image due to the increased brightness caused by the projection of the projection light.

[0033] (Second embodiment) The second embodiment will be described below with reference to Figures 6 and 7, but matters not mentioned in the second embodiment may follow the first embodiment. The imaging system 600 of the second embodiment is similar to the imaging system 100 of the first embodiment in terms of the configuration and operation described with reference to Figures 2 to 5.

[0034] The imaging system 600 of the second embodiment calculates parallax by block matching based on captured images. Generally, block matching is advantageous for calculating parallax with higher accuracy for an image with many features compared to an image with few features. Therefore, in the second embodiment, parallax is calculated based on a second image (e.g., an image of visible light and infrared light) whose features have been increased by projecting projection light.

[0035] 6 shows the configuration of an imaging system 600 according to the second embodiment. In addition to the control unit 104 and the projection unit 103, the imaging system 600 according to the second embodiment includes an imaging device 601 constituting a stereo camera, and a parallax calculation unit 605 that calculates the parallax of stereo images captured by the imaging device 601.

[0036] The imaging device 601 is a stereo camera in which two or more imaging units are arranged at a predetermined distance from each other, and each imaging unit includes an imaging element 102 and an optical system (not shown) that forms an optical image of an object to be imaged on the imaging surface of the imaging element 102. The imaging element 102 may have a configuration similar to that of the imaging element 102 described in the first embodiment. The two or more imaging elements 102 basically operate in the same manner in synchronization, but do not necessarily always operate in the same manner.

[0037] The disparity calculation unit 605 calculates disparity based on images output from two or more image sensors 102. The calculated disparity can be output to the outside of the imaging system 600 as a disparity image having disparity values ​​at corresponding pixel positions. As a calculation method, for example, SAD (Sum of Absolute Difference), SSD (Sum of Squared Difference), semi-global matching, or the like can be adopted.

[0038] FIG. 7 is a flowchart illustrating a series of steps performed by the parallax calculation unit 605 in the imaging system 600 of the third embodiment to calculate parallax. In S700, the parallax calculation unit 605 waits for the end of the second imaging period. In S701, the parallax calculation unit 605 reads the first and second images read by the control unit 104 from the multiple first pixels and multiple second pixels of each imaging element 102, respectively. In S702, the parallax calculation unit 605 performs demosaic processing and monochromatization processing on the first and second images read in S701. Since the imaging element 102 has R pixels 201, G pixels 202, and B pixels 203 as first pixels and IR pixels 204 as second pixels, it is desirable to perform demosaic processing and monochromatization using coefficients that take into account the sensitivity of each wavelength range. In S703, the parallax calculation unit 605 calculates parallax for the pixel values ​​monochromatized in S702. In S704, the parallax calculation unit 605 outputs the parallax calculated in S703 to the outside of the imaging system 600. The parallax calculation unit 605 may generate a parallax image based on any one of the two or more imaging elements 102, or may generate a parallax image based on each of the two or more imaging elements 102.

[0039] According to the second embodiment, by applying the imaging system of the first embodiment to a stereo camera, it is possible to calculate parallax using a first image in which the influence of projected light is suppressed and a second image in which the S / N ratio is improved. Therefore, the second embodiment is advantageous for measuring distance with high accuracy. The imaging system 600 may be configured with a single imaging element capable of acquiring image plane phase difference.

[0040] (Third embodiment) The third embodiment is a modified example of the second embodiment. Matters not mentioned in the third embodiment follow the second embodiment. The imaging system 600 shown in FIG. 6 is used in the third embodiment. The imaging system 600 of the third embodiment calculates parallax based only on the second image.

[0041] 8 is a flowchart illustrating a series of steps in which the parallax calculation unit 605 calculates parallax in the imaging system 600 of the third embodiment. In S800, the parallax calculation unit 605 waits for the end of the second imaging period. In S801, the parallax calculation unit 605 reads the second image read from the second pixel of each imaging element 102 by the control unit 104. In S802, the parallax calculation unit 605 calculates parallax based on the second image read in S801. In S803, the parallax calculation unit 605 outputs the parallax calculated in S802 to the outside of the imaging system 600.

[0042] According to the third embodiment, by calculating the parallax based only on the second image read from the plurality of second pixels, it is possible to appropriately calculate the parallax by increasing the feature amount in the second wavelength range even for a subject with a small feature amount in the first wavelength range. Furthermore, whether to use the parallax calculation method of the second embodiment or the parallax calculation method of the third embodiment may be determined based on the feature amount of the captured first image. For example, if the feature amount of the first image is greater than a predetermined amount, the parallax calculation method of the second embodiment may be used, and if the feature amount of the first image is less than the predetermined amount, the parallax calculation method of the third embodiment may be used.

[0043] (Fourth embodiment) The fourth embodiment is a modified example of the second embodiment. Matters not mentioned in the fourth embodiment follow the second embodiment. The imaging system 600 shown in FIG. 6 is used in the fourth embodiment. The imaging system 600 of the third embodiment calculates parallax based on each of the first image and the second image.

[0044] FIG. 9 is a flowchart illustrating a series of steps performed by the parallax calculation unit 605 to calculate parallax in the imaging system 600 of the fourth embodiment. In S900, the parallax calculation unit 605 waits for the end of the second imaging period. In S901, the parallax calculation unit 605 reads the first image and the second image read by the control unit 104 from the multiple first pixels and the multiple second pixels of each imaging element 102, respectively. In S902, the parallax calculation unit 605 performs demosaic processing and monochromatization processing on the first image read in S901. In S903, the parallax calculation unit 605 calculates the parallax between the imaging elements based on the second image monochromatized in S902. In S904, the parallax calculation unit 605 performs demosaic processing on the second image read in S901. In S905, the parallax calculation unit 605 calculates the parallax between the imaging elements based on the second image demosaic processed in S904. In S906, the disparity calculation unit 605 outputs the disparity value calculated in S903 and S905. The process consisting of S902 and S903 and the process consisting of S904 and S905 may be executed in any order, or may be executed in parallel.

[0045] In the fourth embodiment, the parallax is calculated based on the first image and the second image. As a result, similar to the second and third embodiments, the influence of light in the second wavelength range on the first image is suppressed, and the parallax can be calculated with high accuracy by adding features to a subject with few features using light in the second wavelength range. Furthermore, the parallax image based on the first image sensor and the parallax image based on the second image sensor may be compared from the parallax image based on the first image and the parallax image based on the second image output from the parallax calculation unit 706, and the value of the parallax image with the smaller parallax difference may be selected to generate a single parallax image. This allows for the generation of parallax images with higher accuracy than those calculated in the second and third embodiments.

[0046] (Fifth embodiment) 10, 11, and 12, the fifth embodiment will be described below, but matters not mentioned in the fifth embodiment may follow those of the first to fourth embodiments. The image sensor 1002 in the imaging system 1000 of the fifth embodiment can reset the first pixel and the second pixel individually.

[0047] 10 shows the configuration of an imaging system 1000 according to the fifth embodiment. In addition to a control unit 104 and a projection unit 103, the imaging system 1000 according to the fifth embodiment includes an imaging device 1001 constituting a stereo camera, and a parallax calculation unit 605 that calculates the parallax of stereo images captured by the imaging device 1001. The imaging device 1001 is a stereo camera in which two or more imaging units are arranged at a predetermined distance, and each imaging unit includes an imaging element 1002 and an optical system (not shown) that forms an optical image of an object to be imaged on the imaging plane of the imaging element 1002. However, when applied to an application in which parallax detection is not required, the imaging element 1002 is composed of a single imaging unit, and the parallax calculation unit 605 can be omitted.

[0048] 11 shows a configuration of the image sensor 1002 and a detailed configuration example of each pixel included in the image sensor 1002. The multiple pixels of the image sensor 1002 include, for example, three types of pixels that are sensitive to visible light, specifically, R pixels 1101, G pixels 1102, and B pixels 1103, and one type of IR pixel 1104 that is sensitive to infrared light. However, when obtaining a monochrome image of visible light, only one type of pixel that is sensitive to visible light may be used. The R pixels 1101, G pixels 1102, and B pixels 1103 are examples of first pixels that are sensitive to light in a first wavelength range, and the IR pixel 1104 is an example of a second pixel that is sensitive to light in a second wavelength range different from the first wavelength range.

[0049] The R pixel 1101, G pixel 1102, and B pixel 1103, which are pixels sensitive to visible light, and the IR pixel 1104, which is a pixel sensitive to infrared light, may have the same circuit configuration. The image sensor 1002 may include, for example, an image capture control unit 1110, a readout unit 302, and a vertical scanning circuit 1111. In FIG. 11, only four pixels, the R pixel 1101, the G pixel 1102, the B pixel 1103, and the IR pixel 1104, are shown as the multiple pixels. The image capture control unit 1110 controls each pixel via the vertical scanning circuit 1111. The pixel values ​​(digital data) obtained at each pixel are read out by the readout unit 302. The image capture control unit 1110 sends various signals for controlling image capture and readout to the vertical scanning circuit 1111. The image capture control unit 1110 also sends a signal for controlling readout to the readout unit 302.

[0050] The vertical scanning circuit 1111 has an ENABLE signal, a RESET_RGB signal, a RESET_IR signal, and a READn signal as signals for controlling imaging and readout of each pixel. The ENABLE signal is a signal that enables the count operation of the counter 1105 of each pixel when asserted (activated). The RESET_RGB signal is a signal that resets the count values ​​of the counters 1105 of the R pixel 1101, G pixel 1102, and B pixel 1103 serving as first pixels when asserted. The RESET_IR signal is a signal that resets the count value of the counter 1105 of the IR pixel 1104 serving as a second pixel when asserted. Note that RESET signals corresponding to the R pixel 1101, G pixel 1102, and B pixel 1103 may be provided, respectively, so that the counters 1105 of the R pixel 1101, G pixel 1102, and B pixel 1103 serving as first pixels can be individually reset. The READn signal is a signal provided for each row, and when asserted, causes pixel values ​​(digital data) from each pixel in the corresponding row to be output to the readout unit 302. The readout unit 302 sequentially outputs the pixel values ​​(digital data) read from each pixel to the outside of the image sensor 102.

[0051] Next, each pixel will be described in detail. Each pixel may include, for example, an APD 303, a quenching element 304, a waveform shaping unit 305, a counter 1105, and a selection transistor 310. The counter 1105 may include an AND circuit 308 as an enable circuit and a counter circuit 307. A RESET_RGB signal is supplied to the counter circuit 307 of the R pixel 1101, G pixel 1102, and B pixel 1103, which serve as first pixels, and a RESET_IR signal is supplied to the counter circuit 307 of the IR pixel 1104, which serves as a second pixel.

[0052] In the counter circuits 307 of the R pixel 1101, G pixel 1102, and B pixel 1103, when the ENABLE signal is asserted and the RESET_RGB signal is deasserted, the count value is incremented when a pulse is supplied from the waveform shaping unit 305. When the RESET_RGB signal is asserted, the count value of the counter circuit 307 is reset to a predetermined value. In the counter circuit 307 of the IR pixel 1104, when the ENABLE signal is asserted and the RESET_IR signal is deasserted, the count value of the counter circuit 307 is incremented when a pulse is supplied from the waveform shaping unit 305. When the RESET_IR signal is asserted, the count value of the counter circuit 307 is reset to a predetermined value.

[0053] 12 illustrates the operation of the imaging system 1000. The control unit 104 outputs an imaging start signal, a readout start signal, and a projection signal, and the vertical scanning circuit 311 outputs an ENABLE signal, a RESET_RGB signal, a RESET_IR signal, and a READn signal.

[0054] The control unit 104 or the vertical scanning circuit 1111 controlled by the control unit 104 defines a first imaging period during which the R pixel 1101, G pixel 1102, and B pixel 1103 serving as first pixels capture images. The control unit 104 or the vertical scanning circuit 1111 controlled by the control unit 104 defines a second imaging period during which the IR pixel 1104 serving as second pixels capture images. The imaging period during which the pixels capture images can be understood as a period during which the pixels can generate digital data corresponding to the number of photons. The first imaging period during which the R pixel 1101, G pixel 1102, and B pixel 1103 serving as first pixels capture images is a period that includes at least a portion of the non-effective light emission period but does not include the effective light emission period. The second imaging period during which the IR pixel serving as second pixels captures images is a period that includes at least a portion of the effective light emission period and at least a portion of the non-effective light emission period.

[0055] 12, the control unit 104 defines the effective light-emitting period and the non-effective light-emitting period so that one imaging cycle includes an effective light-emitting period and a non-effective light-emitting period following the effective light-emitting period. In the example shown in Fig. 12, the first imaging period starts at the start of the non-effective light-emitting period and ends in the non-effective light-emitting period, and the second imaging period starts at the start of the effective light-emitting period and ends in the non-effective light-emitting period.

[0056] The counter 1105 of each of the multiple pixels (first pixel and second pixel) may have its counting operation enabled at the start of the second imaging period and its counting operation disabled at the end of the second imaging period. The counter 1105 of the first pixel (R pixel 1101, G pixel 1102, B pixel 1103) may be released from reset at the start of the first imaging period. The counter 1105 of the first pixel (R pixel 1101, G pixel 1102, B pixel 1103) may be reset after reading of the count values ​​from all the first pixels has been completed. The counter 1105 of the second pixel (IR pixel 1104) may be released from reset at the start of the second imaging period. The counter 1105 of the second pixel (IR pixel 1104) may be reset after reading of the count values ​​from all the second pixels has been completed. After the first and second imaging periods have ended, the readout unit 302 reads out count values ​​(digital data) that are pixel values ​​from the counters 1105 of the first and second pixels.

[0057] A more specific example of the operation of the imaging system 1000 will be described below. At time t1201, the control unit 104 raises the imaging start signal. This rise indicates the start and end of the imaging cycle. Therefore, the imaging cycle that started at time t1201 ends with the rise of the imaging start signal at time t1204. Time t1201 also marks the start of a second imaging period within the imaging cycle. Therefore, in synchronization with the rise of the imaging start signal, the control unit 104 asserts a projection signal, and in response, the projection unit 103 starts projecting projection light. Time t1201 may be understood as the start time of an effective light-emission period. During the effective light-emission period, the imaging device 1001 captures images using the IR pixel 1104 as the second pixel, but does not capture images using the R pixel 1101, G pixel 1102, or B pixel 1103 as the first pixel. Therefore, at time t1201, the vertical scanning circuit 1111 deasserts the RESET_IR signal but asserts the reset signal RESET_RGB.

[0058] At time t1202, the control unit 104 deasserts the projection signal. Here, after raising the imaging start signal at time t1201, the control unit 104 deasserts the projection signal after a first predetermined time has elapsed. In response to this, the projection unit 103 ends the projection of projection light. Furthermore, in order to start imaging by the first pixel in response to the end of projection of projection light, the vertical scanning circuit 1111 deasserts the reset RESET_RGB signal of the counter 1105 for the R pixel 1101, G pixel 1102, and B pixel 1103 as the first pixel.

[0059] At time t1203, the control unit 104 raises the read start signal. Here, after deasserting the projection signal at time t1302, the control unit 104 raises the read start signal so as to end the first imaging period and the second imaging period in response to the passage of a second predetermined time. This rise causes the vertical scanning circuit 1111 to deassert the ENABLE signal to end imaging and to start an operation of sequentially asserting the READn signals for multiple rows. The assertion of the READn signal reads pixel values ​​from the pixels in the corresponding row. The vertical scanning circuit 1111 also asserts the RESET_IR signal at least once during the period from time t1203 to time t1204.

[0060] At time t1204, the control unit 104 again raises the read start signal. Here, after raising the read start signal at time t1203, the control unit 104 raises the imaging start signal so as to end the imaging cycle in response to the passage of a third predetermined time.

[0061] The subsequent period from time t1204 to time t1207 is a repetition of the operations from time t1201 to time t1204.

[0062] In the fifth embodiment, the reset of the first pixel and the reset of the second pixel are individually released. More specifically, the reset of the second pixel is released and imaging is performed by the second pixel while projecting projection light. Thereafter, the projection of the projection light is stopped, and the reset of the first pixel is released, thereby starting imaging by the first pixel. Therefore, the influence of the projection light is suppressed in the first image obtained by imaging by the first pixel, and the S / N ratio can be improved in the second image obtained by imaging by the second pixel due to the increased brightness caused by the projection of the projection light. Furthermore, pixel values ​​can be read out from the first pixel and the second pixel in parallel, enabling high speed and low power consumption.

[0063] (Sixth embodiment) 13, 14, and 15, the sixth embodiment will be described below. However, matters not mentioned in the sixth embodiment may be the same as those in the first to fifth embodiments. The imaging element 1302 in the imaging system 1300 of the sixth embodiment can enable the first pixel and the second pixel individually.

[0064] 13 shows the configuration of an imaging system 1300 according to the sixth embodiment. In addition to the control unit 104 and the projection unit 103, the imaging system 1300 according to the sixth embodiment includes an imaging device 1301 constituting a stereo camera, and a parallax calculation unit 605 that calculates the parallax of stereo images captured by the imaging device 1301. The imaging device 1301 is a stereo camera in which two or more imaging units are arranged at a predetermined distance, and each imaging unit includes an imaging element 1302 and an optical system (not shown) that forms an optical image of an object to be imaged on the imaging plane of the imaging element 1302. However, when applied to an application in which parallax detection is not required, the imaging element 1302 is composed of a single imaging unit, and the parallax calculation unit 605 can be omitted.

[0065] 14 shows a configuration of the image sensor 1302 and a detailed configuration example of each pixel included in the image sensor 1302. The multiple pixels of the image sensor 1302 include, for example, three types of pixels that are sensitive to visible light, specifically, R pixels 1101, G pixels 1102, and B pixels 1103, and one type of IR pixel 1104 that is sensitive to infrared light. However, when obtaining a monochrome image of visible light, only one type of pixel that is sensitive to visible light may be used. The R pixels 1101, G pixels 1102, and B pixels 1103 are examples of first pixels that are sensitive to light in a first wavelength range, and the IR pixels 1104 are an example of second pixels that are sensitive to light in a second wavelength range different from the first wavelength range.

[0066] The R pixel 1101, G pixel 1102, and B pixel 1103, which are pixels sensitive to visible light, and the IR pixel 1104, which is a pixel sensitive to infrared light, may have the same circuit configuration. The image sensor 1302 may include, for example, an image capture control unit 1401, a readout unit 1402, and a vertical scanning circuit 1411. In FIG. 14, only four pixels, the R pixel 1101, the G pixel 1102, the B pixel 1103, and the IR pixel 1104, are shown as the multiple pixels. The image capture control unit 1410 controls each pixel via the vertical scanning circuit 1411. The pixel values ​​(digital data) obtained at each pixel are read out by the readout unit 1402. The image capture control unit 1401 sends various signals for controlling image capture and readout to the vertical scanning circuit 1411. The image capture control unit 1401 also sends a signal for controlling readout to the readout unit 1402.

[0067] The vertical scanning circuit 1411 has an ENABLE_RGB signal, an ENABLE_IR signal, a RESET signal, and a READn signal as signals for controlling imaging and readout of each pixel. The ENABLE_RGB signal is a signal that enables the counting operation of the counters 1105 of the R pixel 1101, G pixel 1102, and B pixel 1103 serving as first pixels when asserted (activated). The ENABLE_IR signal is a signal that enables the counting operation of the counter 1105 of the IR pixel 1104 serving as a second pixel when asserted (activated). The RESET signal is a signal that resets the count value of the counter 1105 of each pixel when asserted. Note that an ENABLE signal may be provided for each of the R pixel 1101, G pixel 1102, and B pixel 1103 so that the counter operations of the counters 1105 of the R pixel 1101, G pixel 1102, and B pixel 1103 serving as first pixels can be individually enabled. The READn signal is a signal provided for each row, and when asserted, causes pixel values ​​(digital data) from each pixel in the corresponding row to be output to the readout unit 302. The readout unit 302 sequentially outputs the pixel values ​​(digital data) read from each pixel to the outside of the image sensor 102.

[0068] The vertical scanning circuit 1411 may assert only the READ signal for a row including a specific pixel in response to a signal during readout from the imaging control unit 1401. The readout unit 1402 may output only the pixel value of a column including a specific pixel in response to a signal during readout from the imaging control unit 1401.

[0069] 15 illustrates the operation of the imaging system 1300. The control unit 104 outputs an imaging start signal, a readout start signal, and a projection signal, and the vertical scanning circuit 311 outputs an ENABLE_RGB signal, an ENABLE_IR signal, a RESET signal, and a READn signal.

[0070] The control unit 104 or the vertical scanning circuit 1411 controlled by the control unit 104 defines a first imaging period during which the R pixel 1101, G pixel 1102, and B pixel 1103 serving as first pixels capture images. The control unit 104 or the vertical scanning circuit 1411 controlled by the control unit 104 defines a second imaging period during which the IR pixel 1104 serving as second pixels capture images. The imaging period during which the pixels capture images can be understood as a period during which the pixels can generate digital data corresponding to the number of photons. The first imaging period during which the R pixel 1101, G pixel 1102, and B pixel 1103 serving as first pixels capture images is a period that includes at least a portion of the non-effective light emission period but does not include the effective light emission period. The second imaging period during which the IR pixel 1104 serving as second pixels captures images is a period that includes at least a portion of the effective light emission period and at least a portion of the non-effective light emission period.

[0071] 15, the control unit 104 defines the effective light-emitting period and the non-effective light-emitting period so that one imaging cycle includes a non-effective light-emitting period and an effective light-emitting period following the non-effective light-emitting period. Also, in the example shown in Fig. 15, the first imaging period is the period from the start to the end of the non-effective light-emitting period, and the second imaging period starts at the start of the effective light-emitting period and ends at the end of the effective light-emitting period.

[0072] The counter 1105 of each of the multiple pixels (first pixel and second pixel) may be released from reset at the start of the second imaging period. The counter 1105 of the first pixel (R pixel 1101, G pixel 1102, B pixel 1103) may have its counting operation enabled at the start of the first imaging period and its counting operation disabled at the end of the first imaging period. The counter 1105 of the second pixel (IR pixel 1104) may have its counting operation enabled at the start of the second imaging period and its counting operation disabled at the end of the second imaging period. The readout unit 1402 reads the count value, which is the pixel value, from the counter 1105 of the first pixel after the end of the first imaging period, reads the digital data, and reads the count value, which is the pixel value, from the counter 1105 of the second pixel after the end of the second imaging period.

[0073] A more specific example of the operation of the imaging system 1300 will be described below. At time t1501, the control unit 104 raises the imaging start signal. This rise indicates the start and end of an imaging cycle. Therefore, the imaging cycle that started at time t1501 ends with the rise of the imaging start signal at time t1504. Time t1501 also marks the start of the first and second imaging periods within the imaging cycle. In response to the rise of the imaging start signal, the imaging control unit 1401 transmits the imaging start signal to the vertical scanning circuit 1411. As a result, the vertical scanning circuit 1411 asserts the ENABLE_RGB signal and the ENABLE_IR signal to start imaging at the first and second pixels.

[0074] At time t1502, the control unit 104 raises the read signal. After raising the imaging start signal, the control unit 104 raises the read start signal so as to end the first imaging period in response to the passage of a first predetermined time. In response to this rise, the imaging control unit 1401 transmits the read start signal to the vertical scanning circuit 1411 and the read unit 1402. As a result, the vertical scanning circuit 1411 deasserts the ENABLE_RGB signal and starts an operation of sequentially asserting the READn signals for multiple rows. The operation of sequentially asserting the READn signals for multiple rows starts. When the READn signal is asserted, pixel values ​​are read from the pixels in the corresponding row, but the count values ​​of the counters 1106 for the R pixels 1101, G pixels 1102, and B pixels 1103, which are the first pixels, are maintained. Meanwhile, because the ENABLE_IR signal remains asserted, imaging by the IR pixel 1104, which is the second pixel, continues. The vertical scanning circuit 1411 reads out pixel values ​​of each row by asserting the READ signal for each row in a predetermined order. At this time, the vertical scanning circuit 1411 may read out only rows that include specific pixels in response to a signal from the imaging control unit 1401. Furthermore, the readout unit 1402 may read out only columns that include specific pixels in response to a signal from the imaging control unit 1401.

[0075] Furthermore, at time t1502, the control unit 104 asserts the projection signal after raising the read start signal, and in response, the projection unit 103 starts projecting projection light. At time t1503, the control unit 104 again raises the read start signal. Here, after raising the read start signal at time t1502, the control unit 104 raises the read start signal so that the second imaging period ends as a second predetermined time elapses. This rise causes the vertical scanning circuit 1411 to deassert the ENABLE_IR signal to end imaging by the second pixel and to start an operation of sequentially asserting the READn signals for multiple rows. The assertion of the READn signals reads pixel values ​​from the pixels in the corresponding rows. The vertical scanning circuit 1411 asserts the READ signals for each row in a predetermined order to read the pixel values ​​of each row. At this time, the vertical scanning circuit 1411 may read only rows containing specific pixels in response to a signal from the imaging control unit 1401. Furthermore, the readout unit 1402 may read out only columns including specific pixels in response to a signal from the imaging control unit 1401. Furthermore, at time t1503, the control unit 104 deasserts the projection signal, and in response, the projection unit 103 stops projecting projection light.

[0076] At time t1504, the control unit 104 again raises the imaging start signal. The control unit 104 raises the imaging start signal so as to end one imaging cycle that started at time t1501 in response to the passage of a third predetermined time from the rise of the readout start signal at time t1503. As described above, this rise marks the end of one imaging cycle and simultaneously starts the next imaging cycle.

[0077] The subsequent operations from time t1504 to time t1507 are a repetition of the operations from time t1501 to time t1504.

[0078] According to the sixth embodiment, the counting operation of the first pixel and the counting operation of the second pixel are individually enabled, thereby making it possible to obtain the first image and the second image without requiring a latch circuit. Furthermore, to obtain the first image, pixel values ​​can be read only from the first pixel by the vertical scanning circuit 1411 and the readout unit 1402, and / or to obtain the second image, pixel values ​​can be read only from the second pixel by the vertical scanning circuit 1411 and the readout unit 1402. This allows for higher speeds and lower power consumption.

[0079] (Seventh embodiment) The seventh embodiment will be described below with reference to Figures 16, 17, and 18, but matters not mentioned in the seventh embodiment may follow those of the first to sixth embodiments. In an imaging system 1600 of the seventh embodiment, an imaging element that controls pixels on a row-by-row basis is employed instead of the imaging element in the imaging systems of the first, fifth, and sixth embodiments.

[0080] 16 shows the configuration of an imaging system 1600 according to the seventh embodiment. In addition to the control unit 104 and the projection unit 103, the imaging system 1600 according to the seventh embodiment includes an imaging device 1601 constituting a stereo camera, and a parallax calculation unit 605 that calculates the parallax of stereo images captured by the imaging device 1601. The imaging device 1601 is a stereo camera in which two or more imaging units are arranged at a predetermined distance, and each imaging unit includes an imaging element 1602 and an optical system (not shown) that forms an optical image of an object to be imaged on the imaging plane of the imaging element 1602. However, when applied to an application in which parallax detection is not required, the imaging element 1602 is composed of a single imaging unit, and the parallax calculation unit 605 can be omitted.

[0081] 17 shows a configuration of the image sensor 1602 and a detailed configuration example of each pixel included in the image sensor 1602. The multiple pixels of the image sensor 1602 include, for example, three types of pixels that are sensitive to visible light, specifically, R pixels 1101, G pixels 1102, and B pixels 1103, and one type of IR pixel 1104 that is sensitive to infrared light. However, when obtaining a monochrome image of visible light, only one type of pixel that is sensitive to visible light may be used. The R pixels 1101, G pixels 1102, and B pixels 1103 are examples of first pixels that are sensitive to light in a first wavelength range, and the IR pixels 1104 are an example of second pixels that are sensitive to light in a second wavelength range different from the first wavelength range.

[0082] The R pixel 1101, G pixel 1102, and B pixel 1103, which are pixels sensitive to visible light, and the IR pixel 1104, which is a pixel sensitive to infrared light, may have the same circuit configuration. The image sensor 1602 may include, for example, an image capture control unit 1701, a readout unit 1402, and a vertical scanning circuit 1711. FIG. 17 shows only four pixels: the R pixel 1101, the G pixel 1102, the B pixel 1103, and the IR pixel 110. The image capture control unit 1701 controls each pixel via the vertical scanning circuit 1711. The pixel values ​​(digital data) obtained at each pixel are read out by the readout unit 1402. The image capture control unit 1701 sends various signals for controlling image capture and readout to the vertical scanning circuit 1711. The image capture control unit 1701 also sends a signal for controlling readout to the readout unit 1402.

[0083] The vertical scanning circuit 1711 has an ENABLEn signal, a RESETn signal, and a READn signal as signals for controlling imaging and readout of each pixel. The ENABLEn signal, when asserted (activated), enables the count operation of the counter 1105 of each pixel in the nth row to which it is supplied. The RESETn signal, when asserted, resets the count value of the counter 1105 of each pixel in the nth row to which it is supplied. The READn signal, when asserted, causes each pixel in the nth row to output pixel values ​​(digital data) to the readout unit 1402. The readout unit 1402 sequentially outputs the pixel values ​​(digital data) read out from each pixel to the outside of the image sensor 1602.

[0084] 18 illustrates the operation of the imaging system 1600. The control unit 104 outputs an imaging start signal, a readout start signal, and a projection signal, and the vertical scanning circuit 311 outputs an ENABLEn signal, a RESETn signal, and a READn signal.

[0085] The control unit 104 or the vertical scanning circuit 1711 controlled by the control unit 104 defines a first imaging period during which the R pixel 1101, G pixel 1102, and B pixel 1103, which serve as first pixels, capture images. The control unit 104 or the vertical scanning circuit 1711 controlled by the control unit 104 defines a second imaging period during which the IR pixel 1104, which serves as a second pixel, captures images. The imaging period during which the pixels capture images can be understood as a period during which the pixels can generate digital data corresponding to the number of photons.

[0086] 18, the control unit 104 defines the effective light-emitting period and the non-effective light-emitting period so that one imaging cycle includes a non-effective light-emitting period and an effective light-emitting period following the non-effective light-emitting period. The first imaging period in which the first pixels (R pixel 1101, G pixel 1102, B pixel 1103) perform imaging is the period from when the counters 1105 of the first pixels are enabled to when readout of pixel values ​​(digital data) from the counters 1105 of the first pixels begins. The second imaging period in which the second pixels (IR pixel 1104) perform imaging is the period from when the counters 1105 of the second pixels are enabled to when readout of pixel values ​​(digital data) from the counters 1105 of the second pixels begins. The effective light-emitting period starts after the pixel values ​​of the multiple first pixels (all first pixels to be read) are read out from the counters 1105 and ends before the pixel values ​​of the multiple second pixels are read out from the counters 1105.

[0087] A more specific example of the operation of the imaging system 1600 will be described below. At time t1801, the control unit 104 raises the imaging start signal. This rise indicates the start and end of an imaging cycle. Therefore, the imaging cycle that started at time t1801 ends with the rise of the imaging start signal at time t1804. Time t1801 also marks the start of the first imaging period and the second imaging period within the imaging cycle. In response to the rise of the imaging start signal, the imaging control unit 1701 transmits the imaging start signal to the vertical scanning circuit 1711. As a result, the vertical scanning circuit 1411 begins an operation of sequentially asserting the ENABLEn signals of multiple rows in a predetermined order to start imaging at each pixel. The assertion of the ENABLEn signals of multiple rows does not necessarily have to be performed at equal intervals, but it is desirable to perform it at equal intervals.

[0088] At time t1802, the control unit 104 raises the read start signal. Here, the control unit 104 raises the read start signal so as to end the first imaging period after the rise of the imaging start signal has elapsed a first predetermined time. This rise causes the imaging control unit 1701 to transmit the read start signal to the vertical scanning circuit 1711 and the readout unit 1402. As a result, the vertical scanning circuit 1711 starts an operation of sequentially asserting the READn signals for multiple rows in a predetermined order. Pixel values ​​are read out by asserting the READn signals, but the value of the counter 1106 is maintained. In this state, when the waveform shaping unit 305 generates a pulse signal for each pixel, the counter 1105 increments its count value upon receiving the signal. The vertical scanning circuit 1711 sequentially asserts the READn signals for multiple rows in a predetermined order, causing the readout unit 1402 to sequentially read out pixel values ​​for multiple rows.

[0089] It is desirable to assert the READn signals for the multiple rows in the same order as the assertion of the ENABLEn signals that specifies the start of imaging of the multiple rows, which began at time t1801, but this is not necessarily the case. After completing the readout of the pixels in the multiple rows in response to the rising edge of the readout start signal at time t1802, the control unit 104 asserts the projection signal, and the projection unit 103 projects projection light.

[0090] At time t1803, the control unit 104 again raises the read start signal. Here, after the rise of the read start signal at time t802, the control unit 104 raises the read start signal so as to end the second imaging period in response to the passage of a second predetermined time. This rise causes the imaging control unit 1701 to transmit the read start signal to the vertical scanning circuit 1711 and the readout unit 1402. As a result, the vertical scanning circuit 1711 begins an operation of sequentially asserting the READn signals of multiple rows in a predetermined order. Furthermore, in response to the transmission of the read start signal, the vertical scanning circuit 1711 sequentially deasserts the ENABLEn signals of multiple rows and asserts and deasserts (i.e., resets) the RESETn signals of multiple rows.

[0091] The vertical scanning circuit 1711 may read out only rows including specific pixels in response to a signal from the imaging control unit 1701. Furthermore, the readout unit 1402 may read out pixel values ​​of only columns including specific pixels in response to a signal from the imaging control unit 1401. Furthermore, the deassertion of the RESET signal for each row only needs to be completed by the time the ENABLE signal for each row is asserted after time t1804, and the RESET signal may be deasserted after time t1804.

[0092] At time t1804, the control unit 104 again raises the imaging start signal. In response to the passage of a third predetermined time from the rise of the readout start signal at time t1803, the control unit 104 raises the imaging start signal so as to end one imaging cycle that started at time t1801. As described above, this rise ends one imaging cycle and simultaneously starts the next imaging cycle.

[0093] The subsequent operations from time t1804 to time t1807 are a repetition of the operations from time t1801 to time t1804.

[0094] According to the seventh embodiment, in a method of controlling a plurality of pixels of an image sensor row by row, it is possible to obtain the same effects as the image sensing systems of the first, fifth, and sixth embodiments. Also, to obtain a first image, pixel values ​​can be read out only from first pixels by the vertical scanning circuit 1711 and the readout unit 1702, and / or to obtain a second image, pixel values ​​can be read out only from second pixels by the vertical scanning circuit 1711 and the readout unit 1702. This enables high speed and low power consumption.

[0095] (Eighth embodiment) 19 and 20, the eighth embodiment will be described below, but matters not mentioned in the eighth embodiment may follow those of the first to seventh embodiments. In the imaging system 1900 of the eighth embodiment, the imaging control unit in the imaging element in the imaging systems of the first, fifth, sixth, and seventh embodiments is provided outside the imaging element.

[0096] 19 shows the configuration of an imaging system 1900 according to the eighth embodiment. In addition to a control unit 1904 and a projection unit 103, the imaging system 1900 according to the eighth embodiment includes an imaging device 1901 constituting a stereo camera, and a parallax calculation unit 605 that calculates the parallax of stereo images captured by the imaging device 1901. The imaging device 1901 is a stereo camera in which two or more imaging units are arranged at a predetermined distance, and each imaging unit includes an imaging element 1902 and an optical system (not shown) that forms an optical image of an object to be imaged on the imaging plane of the imaging element 1902. However, when applied to an application in which parallax detection is not required, the imaging device 1901 is configured with a single imaging unit, and the parallax calculation unit 605 can be omitted.

[0097] 20 shows a configuration of the image sensor 1902 and a detailed configuration example of each pixel included in the image sensor 1902. The multiple pixels of the image sensor 1902 include, for example, three types of pixels that are sensitive to visible light, specifically, R pixels 201, G pixels 202, and B pixels 203, and one type of IR pixel 204 that is sensitive to infrared light. However, when obtaining a monochrome image of visible light, only one type of pixel that is sensitive to visible light may be used. The R pixels 201, G pixels 202, and B pixels 203 are examples of first pixels that are sensitive to light in a first wavelength range, and the IR pixel 204 is an example of a second pixel that is sensitive to light in a second wavelength range different from the first wavelength range.

[0098] The R pixel 201, G pixel 202, and B pixel 203, which are pixels sensitive to visible light, and the IR pixel 204, which is a pixel sensitive to infrared light, may have the same circuit configuration. The image sensor 1902 may include, for example, a readout unit 2002 and a vertical scanning circuit 2011. In FIG. 20 , only four pixels, the R pixel 201, the G pixel 202, the B pixel 203, and the IR pixel 204, are shown as the plurality of pixels.

[0099] A control unit 1904 disposed outside the image sensor 1902 may have the functions of the control unit 104 described above as well as the function of controlling a vertical scanning circuit 1911 and a readout unit 2002. The vertical scanning circuit 2011 has, for example, an ENABLEn signal, a RESETn signal, a READn signal, and a LATCHn signal as signals for controlling imaging and readout of each pixel. While Fig. 20 shows an example having a configuration similar to that of the pixels 201 to 204 of the first embodiment, pixel configurations of other embodiments may also be adopted.

[0100] The control unit 1904 may be understood as a control device. The control unit 1904 may be configured to control an imaging element having a plurality of pixels arranged to form a plurality of rows and a plurality of columns. Here, the plurality of pixels may include a first pixel (e.g., a B pixel 203) and a second pixel (e.g., an IR pixel 204) belonging to the same row, where the first pixel may be sensitive to light in a first wavelength range and the second pixel may be sensitive to light in a second wavelength range different from the first wavelength range. The control unit 1904 may be configured to generate a first signal defining an imaging period for the first pixel and a second signal defining an imaging period for the second pixel during a period in which signals from the first pixel and the second pixel are read out. In the example shown in FIG. 4 , the first signal is a readout start signal that transitions at time t402, and the second signal is a readout start signal that transitions at time t403. The first signal may define the end of the imaging period for the first pixel, and the second signal may define the end of the imaging period for the second pixel.

[0101] (Other embodiments) The present invention is not limited to the above-described embodiments and can be embodied in various other forms. For example, in the first to eighth embodiments, the first wavelength range or the second wavelength range may include the wavelength range of ultraviolet light, or both the first wavelength range and the second wavelength range may be wavelength ranges within the wavelength range of visible light, or the first wavelength range and the second wavelength range may be other wavelength ranges.

[0102] Furthermore, the imaging device in the imaging systems according to the second to eighth embodiments may be configured by an imaging element having image surface phase difference pixels for distance measurement, instead of a stereo camera.

[0103] Furthermore, the control unit in the first to eighth embodiments and the disparity calculation unit in the second to eighth embodiments may be configured with a hardware circuit that executes various flows, or may be configured with a program that executes the flow and a processor that executes that program.

[0104] Furthermore, the present invention may be configured as an imaging system that performs self-location estimation or spatial recognition based on two types of captured images, rather than as an imaging system that performs distance measurement using a stereo camera as in the second to eighth embodiments. For example, a self-location estimation method that uses images is Visual SLAM (Simultaneous Localization and Mapping). Visual SLAM extracts feature points from multiple images and estimates three-dimensional information about the surroundings and the position and orientation of the camera. Therefore, even in an environment where the feature amounts required for estimation cannot be obtained using visible light alone, applying the present invention makes it possible to obtain images with many feature amounts and perform more accurate estimation.

[0105] (others) This specification and the accompanying drawings include the following disclosure: (Item 1) A photoelectric conversion device including an image sensor having a plurality of pixels that outputs digital data according to the number of incident photons, and a control unit that controls a projection unit, the plurality of pixels include a first pixel sensitive to light in a first wavelength range and a second pixel sensitive to light in a second wavelength range different from the first wavelength range, the projection unit generates projection light including the second wavelength range, the control unit controls the projection unit so that an effective light-emitting period in which the intensity of the projection light is greater than a predetermined intensity and a non-effective light-emitting period in which the intensity of the projection light is less than the predetermined intensity are defined; a first imaging period in which the first pixel captures an image includes at least a part of the non-effective light-emitting period and does not include the effective light-emitting period; a second imaging period in which the second pixel performs imaging includes at least a part of the effective light-emitting period and at least a part of the non-effective light-emitting period; A photoelectric conversion device characterized by: (Item 2) the first wavelength range includes a wavelength range of visible light, The second wavelength range includes an infrared wavelength range. 2. The photoelectric conversion device according to item 1, (Item 3) the control unit defines the effective light-emitting period and the non-effective light-emitting period so that one imaging cycle includes the effective light-emitting period and the non-effective light-emitting period; 3. The photoelectric conversion device according to item 1 or 2, (Item 4) the control unit defines the effective light-emitting period and the non-effective light-emitting period such that one imaging cycle includes the non-effective light-emitting period and the effective light-emitting period following the non-effective light-emitting period; the first imaging period is a period from the start to the end of the non-effective light emission period, The second imaging period is a period from the start of the non-effective light-emitting period to the end of the effective light-emitting period. 3. The photoelectric conversion device according to item 1 or 2, (Item 5) Each of the plurality of pixels includes a counter that generates digital data according to the number of incident photons, and a latch circuit that performs a latch operation to latch an output of the counter, the latch circuit of each of the plurality of pixels performs the latch operation at the end of the first imaging period, and performs the latch operation again at the end of the second imaging period; 5. The photoelectric conversion device according to item 4, (Item 6) the counter of each of the plurality of pixels is released from reset at the start of the first imaging period and the second imaging period, and a count operation is enabled throughout the second imaging period; 6. The photoelectric conversion device according to item 5, (Item 7) a readout unit that reads out digital data from the latch circuit of the first pixel after the first imaging period has ended, and reads out digital data from the latch circuit of the second pixel after the second imaging period has ended, 7. The photoelectric conversion device according to item 6, (Item 8) the control unit defines the effective light-emitting period and the non-effective light-emitting period so that one imaging cycle includes the effective light-emitting period and the non-effective light-emitting period subsequent to the effective light-emitting period; the first imaging period starts at the start of the non-effective light-emitting period and ends during the non-effective light-emitting period; the second imaging period starts at the start of the effective light-emitting period and ends at the non-effective light-emitting period; 3. The photoelectric conversion device according to item 1 or 2, (Item 9) each of the plurality of pixels includes a counter that generates digital data according to the number of incident photons; a counting operation of the counter of each of the plurality of pixels is enabled at the start of the second imaging period and disabled at the end of the second imaging period; the counter of the first pixel is released from reset at the start of the first imaging period; The counter of the second pixel is released from reset at the start of the second imaging period. 9. The photoelectric conversion device according to item 8, wherein (Item 10) a readout unit that reads out digital data from the counters of the first pixel and the second pixel after the first imaging period and the second imaging period have ended, 10. The photoelectric conversion device according to item 9, (Item 11) the control unit defines the effective light-emitting period and the non-effective light-emitting period so that one imaging cycle includes the non-effective light-emitting period and the effective light-emitting period following the non-effective light-emitting period; the first imaging period is a period from the start to the end of the non-effective light emission period, the second imaging period starts at the start of the non-effective light-emitting period and ends at the end of the effective light-emitting period; 3. The photoelectric conversion device according to item 1 or 2, (Item 12) each of the plurality of pixels includes a counter that generates digital data according to the number of incident photons; the counter of each of the plurality of pixels is released from reset at the start of the second imaging period; a counting operation of the counter of the first pixel is enabled at the start of the first imaging period and disabled at the end of the first imaging period; a counting operation of the counter of the second pixel is enabled at the start of the second imaging period and disabled at the end of the second imaging period; Item 12. The photoelectric conversion device according to item 11. (Item 13) a readout unit that reads out digital data from the counter of the first pixel after the first imaging period has ended, and reads out digital data from the counter of the second pixel after the second imaging period has ended, 13. The photoelectric conversion device according to item 12, (Item 14) the control unit defines the effective light-emitting period and the non-effective light-emitting period so that one imaging cycle includes the non-effective light-emitting period and the effective light-emitting period following the non-effective light-emitting period; each of the plurality of pixels includes a counter that generates digital data according to the number of incident photons; the first imaging period is a period from when the counter of the first pixel is enabled to when readout of digital data from the counter of the first pixel starts; the second imaging period is a period from when the counter of the second pixel is enabled to when readout of digital data from the counter of the second pixel starts; the effective light-emitting period starts after the digital data of the plurality of first pixels is read out from the counters and ends before the digital data of the plurality of second pixels is read out from the counters. 3. The photoelectric conversion device according to item 1 or 2, (Item 15) A control device for controlling an image sensor having a plurality of pixels arranged to form a plurality of rows and a plurality of columns, the plurality of pixels including a first pixel and a second pixel belonging to the same row, the first pixel having sensitivity to light in a first wavelength range, and the second pixel having sensitivity to light in a second wavelength range different from the first wavelength range; the control device generates a first signal that defines an imaging period of the first pixel and a second signal that defines an imaging period of the second pixel during a period in which signals from the first pixel and the second pixel are read out; A control device characterized by: (Item 16) the first signal defines an end of an imaging period for the first pixel, and the second signal defines an end of an imaging period for the second pixel; Item 16. The control device according to item 15. (Item 17) A photoelectric conversion system comprising two or more image pickup devices and a parallax calculation unit that calculates parallax based on outputs of the two or more image pickup devices, Each of the two or more imaging devices is the photoelectric conversion device according to any one of items 1 to 14. A photoelectric conversion system comprising:

[0106] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0107] 100: Imaging system (photoelectric conversion device), 101: Imaging device, 102: Imaging element, 103: Projection unit, 104: Control unit

Claims

1. A photoelectric conversion device including an image sensor having a plurality of pixels that outputs digital data according to the number of incident photons, and a control unit that controls a projection unit, the plurality of pixels include a first pixel sensitive to light in a first wavelength range and a second pixel sensitive to light in a second wavelength range different from the first wavelength range, the projection unit generates projection light including the second wavelength range, the control unit controls the projection unit so that an effective light-emitting period in which the intensity of the projection light is greater than a predetermined intensity and a non-effective light-emitting period in which the intensity of the projection light is less than the predetermined intensity are defined; a first imaging period in which the first pixel performs imaging includes at least a part of the non-effective light-emitting period and does not include the effective light-emitting period; a second imaging period in which the second pixel performs imaging includes at least a part of the effective light-emitting period and at least a part of the non-effective light-emitting period; A photoelectric conversion device characterized by:

2. the first wavelength range includes a wavelength range of visible light, The second wavelength range includes an infrared wavelength range.

2. The photoelectric conversion device according to claim 1.

3. the control unit defines the effective light-emitting period and the non-effective light-emitting period so that one imaging cycle includes the effective light-emitting period and the non-effective light-emitting period; 2. The photoelectric conversion device according to claim 1.

4. the control unit defines the effective light-emitting period and the non-effective light-emitting period so that one imaging cycle includes the non-effective light-emitting period and the effective light-emitting period following the non-effective light-emitting period; the first imaging period is a period from the start to the end of the non-effective light emission period, the second imaging period is a period from the start of the non-effective light-emitting period to the end of the effective light-emitting period; 2. The photoelectric conversion device according to claim 1.

5. Each of the plurality of pixels includes a counter that generates digital data according to the number of incident photons, and a latch circuit that performs a latch operation to latch an output of the counter, the latch circuit of each of the plurality of pixels performs the latch operation at the end of the first imaging period, and performs the latch operation again at the end of the second imaging period; 5. The photoelectric conversion device according to claim 4.

6. the counter of each of the plurality of pixels is released from reset at the start of the first imaging period and the second imaging period, and a count operation is enabled throughout the second imaging period; 6. The photoelectric conversion device according to claim 5.

7. a readout unit that reads out digital data from the latch circuit of the first pixel after the first imaging period has ended, and reads out digital data from the latch circuit of the second pixel after the second imaging period has ended, 7. The photoelectric conversion device according to claim 6.

8. the control unit defines the effective light-emitting period and the non-effective light-emitting period so that one imaging cycle includes the effective light-emitting period and the non-effective light-emitting period subsequent to the effective light-emitting period; the first imaging period starts at the start of the non-effective light-emitting period and ends during the non-effective light-emitting period; the second imaging period starts at the start of the effective light-emitting period and ends at the non-effective light-emitting period; 2. The photoelectric conversion device according to claim 1.

9. each of the plurality of pixels includes a counter that generates digital data according to the number of incident photons; a counting operation of the counter of each of the plurality of pixels is enabled at the start of the second imaging period and disabled at the end of the second imaging period; the counter of the first pixel is released from reset at the start of the first imaging period; the counter of the second pixel is released from reset at the start of the second imaging period; 9. The photoelectric conversion device according to claim 8.

10. a readout unit that reads out digital data from the counters of the first pixel and the second pixel after the first imaging period and the second imaging period have ended, 10. The photoelectric conversion device according to claim 9.

11. the control unit defines the effective light-emitting period and the non-effective light-emitting period so that one imaging cycle includes the non-effective light-emitting period and the effective light-emitting period following the non-effective light-emitting period; the first imaging period is a period from the start to the end of the non-effective light emission period, the second imaging period starts at the start of the non-effective light-emitting period and ends at the end of the effective light-emitting period; 2. The photoelectric conversion device according to claim 1.

12. each of the plurality of pixels includes a counter that generates digital data according to the number of incident photons; the counter of each of the plurality of pixels is released from reset at the start of the second imaging period; a counting operation of the counter of the first pixel is enabled at the start of the first imaging period and disabled at the end of the first imaging period; a counting operation of the counter of the second pixel is enabled at the start of the second imaging period and disabled at the end of the second imaging period; 12. The photoelectric conversion device according to claim 11.

13. a readout unit that reads out digital data from the counter of the first pixel after the first imaging period has ended, and reads out digital data from the counter of the second pixel after the second imaging period has ended, 13. The photoelectric conversion device according to claim 12.

14. the control unit defines the effective light-emitting period and the non-effective light-emitting period so that one imaging cycle includes the non-effective light-emitting period and the effective light-emitting period following the non-effective light-emitting period; each of the plurality of pixels includes a counter that generates digital data according to the number of incident photons; the first imaging period is a period from when the counter of the first pixel is enabled to when readout of digital data from the counter of the first pixel starts; the second imaging period is a period from when the counter of the second pixel is enabled to when readout of digital data from the counter of the second pixel starts; the effective light-emitting period starts after the digital data of the plurality of first pixels is read out from the counters and ends before the digital data of the plurality of second pixels is read out from the counters.

2. The photoelectric conversion device according to claim 1.

15. A control device for controlling an image sensor having a plurality of pixels arranged to form a plurality of rows and a plurality of columns, the plurality of pixels including a first pixel and a second pixel belonging to the same row, the first pixel having sensitivity to light in a first wavelength range, and the second pixel having sensitivity to light in a second wavelength range different from the first wavelength range, the control device generates a first signal that defines an imaging period of the first pixel and a second signal that defines an imaging period of the second pixel during a period in which signals from the first pixel and the second pixel are read out; A control device characterized by:

16. the first signal defines an end of an imaging period for the first pixel, and the second signal defines an end of an imaging period for the second pixel; 16. The control device according to claim 15.

17. A photoelectric conversion system including two or more image pickup devices and a parallax calculation unit that calculates parallax based on outputs of the two or more image pickup devices, Each of the two or more imaging devices is a photoelectric conversion device according to any one of claims 1 to 14. A photoelectric conversion system comprising:

Images