Imaging device and imaging element

The imaging device optimizes structured light pattern readout by combining high and low-resolution areas with controlled exposure times, enhancing accuracy and reducing computational and power demands in distance measurement.

JP2024160740APending Publication Date: 2024-11-15NIKON CORP
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023075991
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-02
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Existing imaging devices face challenges in achieving accurate distance measurement while optimizing calculation cost, processing time, and power consumption due to varying granularities of structured light patterns, which affect the number of pixels read and the complexity of pattern matching.

Method used

The imaging device employs a combination of high-resolution and low-resolution readout areas for structured light patterns, optimizing pixel signal readout based on exposure times and pattern types, reducing data and power consumption while maintaining accuracy.

Benefits of technology

This approach improves distance measurement accuracy for main subjects while reducing calculation costs and processing time by selectively reading pixel signals in high-resolution areas and minimizing data and power consumption in low-resolution background areas.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024160740000001_ABST
    Figure 2024160740000001_ABST
Patent Text Reader

Abstract

To improve distance measurement accuracy and reduce calculation costs and processing time.SOLUTION: An imaging device includes a light source that performs irradiation with structured light including a plurality of pattern lights having different emission patterns between a first period and a second period that is shorter than the first period, a pixel unit having one or more pixels including a photoelectric conversion unit that converts incident light resulting from the irradiation with the structured light from the light source into an electric charge, and an exposure control unit that controls an accumulation time for accumulating electric charge in the photoelectric conversion unit to the first period or the second period on the basis of a pixel signal read out from the pixel.SELECTED DRAWING: Figure 5
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an imaging device and an imaging element. [Background technology]

[0002] 2. Description of the Related Art Imaging devices equipped with a plurality of camera modules are known (for example, see Patent Document 1). Conventionally, there has been a demand for improving the accuracy of distance measurement performance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2007-295113 A Summary of the Invention

[0004] The imaging device of the first disclosed technique includes a light source that irradiates structured light including a plurality of pattern lights having different emission patterns between a first period and a second period shorter than the first period, a pixel unit having one or more pixels including a photoelectric conversion unit that converts incident light resulting from the irradiation of the structured light from the light source into an electric charge, and an exposure control unit that controls an accumulation time for accumulating electric charge in the photoelectric conversion unit to the first period or the second period based on a pixel signal read out from the pixel.

[0005] The imaging element of the second disclosed technology has a pixel unit having one or more pixels including a photoelectric conversion unit that converts incident light, which is a result of irradiation of structural light from a light source including a plurality of pattern lights having different emission patterns between a first period and a second period shorter than the first period, into electric charges, and an exposure control unit that controls an accumulation time for accumulating electric charges in the photoelectric conversion unit to the first period or the second period based on a pixel signal read out from the pixel. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is an explanatory diagram showing an example of structured light distance measurement. [Diagram 2] FIG. 2 is an explanatory diagram showing an example of separation of a projection pattern of structured light. [Diagram 3] FIG. 3 is an explanatory diagram showing an example of separation by time modulation of the projection pattern of structured light when the exposure time is 1 msec. [Figure 4] FIG. 4 is an explanatory diagram showing an example of separation by time modulation of the projection pattern of structured light when the exposure time is 1 / 2 msec. [Diagram 5] FIG. 5 is a block diagram of an example of the configuration of the imaging device according to the first embodiment. [Figure 6] FIG. 6 is an explanatory diagram illustrating an example of a pixel block and a control block. [Figure 7] FIG. 7 is a circuit diagram showing an example of a circuit configuration of a pixel. [Figure 8] FIG. 8 is a timing chart showing an example of an imaging operation of the imaging element. [Figure 9] FIG. 9 is a block diagram showing an example of the operation of a pixel block. [Figure 10] FIG. 10 is an explanatory diagram showing an example of the operation of the imaging device. [Figure 11] FIG. 11 is a block diagram of an example of a configuration of an imaging apparatus according to the second embodiment. [Figure 12] FIG. 12 is an explanatory diagram showing an example of the operation of the imaging device. [Figure 13] FIG. 13 is an explanatory diagram showing an update example 1 of the exposure time map in the digital signal processor. [Figure 14] FIG. 14 is an explanatory diagram showing an update example 2 of the exposure time map in the digital signal processor. [Figure 15] FIG. 15 is an explanatory diagram showing an example of a spatial pattern of structured light. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS EXAMPLES

[0007] <Structured light distance measurement> Fig. 1 is an explanatory diagram showing an example of structured light distance measurement. Structured light distance measurement is a technology for measuring the distance to an object OB by irradiating the object OB with structured light and receiving the reflected light with an imaging element. Structured light is a pattern light with a different spatial light intensity distribution.

[0008] (A) to (C) show examples of projection patterns of structured light onto an object OB. Projection pattern Pa in (A) is a structured light pattern with finer granularity than projection pattern Pb in (B).

[0009] Since the granularity of the projection pattern Pa in (A) is finer than that of the projection pattern Pb, the number of pixels from which pixel signals are read out by the image sensor 101 is greater than in the case of (B). In this case, if all-pixel reading of pixel signals is performed by the image sensor 101, the calculation cost and processing time during pattern matching will be greater in (A) than in (B). On the other hand, if thinned-out reading of pixel signals is performed by the image sensor 101, the calculation cost and processing time during pattern matching will be reduced, but the number of pixels from which pixel signals are read out by the image sensor 101 will be reduced, resulting in reduced ranging accuracy compared to all-pixel reading.

[0010] Since the granularity of the projection pattern Pb in (B) is coarser than that of the projection pattern Pa, the number of pixels from which pixel signals are read out by the image sensor 101 may be smaller than in (A). In this case, (B) has lower computational costs and processing time during pattern matching than (A), but the number of pixels from which pixel signals are read out by the image sensor 101 is smaller, resulting in lower distance measurement accuracy than (A).

[0011] In contrast, in (C), a projection pattern Pa is used in the subject region surrounding the subject OB, and a projection pattern Pb is used in the background region around the projection pattern Pa. Structured light is irradiated with a projection pattern Pc that is a combination of the projection patterns Pa and Pb. The image sensor 101 then reads out pixel signals from a region Ra corresponding to the projection pattern Pa in the subject region as a full pixel readout region, and reads out pixel signals from a region Rb corresponding to the projection pattern Pb in the background region as a thinned readout region.

[0012] As a result, a high-resolution projection pattern Pa is obtained in the subject area, and a low-resolution projection pattern Pb is obtained in the background area, thereby improving the accuracy of measuring the distance to the subject OB and reducing the calculation cost and processing time.

[0013] Moreover, the image sensor 101 reads out pixel signals from the region Ra corresponding to the projection pattern Pa as a full pixel readout region, and reads out pixel signals from the region Rb corresponding to the projection pattern Pb as a thinned readout region, thereby reducing the amount of readout data and power consumption.

[0014] Hereinafter, the control of the structured light and the image sensor 101 for realizing the optimization of the projection pattern and the optimization of the pixel signal readout in (C) above will be specifically described.

[0015] <Separation of structured light projection patterns> Fig. 2 is an explanatory diagram showing an example of separation of a projection pattern of a structured light. The structured light 200 is a pattern light in which a plurality of types of global patterns are randomly arranged in a spatial direction perpendicular to the irradiation direction. In Fig. 2, as an example, two types of global patterns PA and PB (indicated as "A" and "B" in the figure) are randomly arranged in two rows and two columns.

[0016] The global pattern PA includes local patterns p0 and p1 (in the figure, these are labeled "0" and "1"). The global pattern PB includes local patterns p2 and p3 (in the figure, these are labeled "2" and "3"). That is, the projection pattern of the structured light 200 is composed of local patterns p0 to p4. The local patterns p0 to p4 are patterns with different light emission periods.

[0017] Focusing on the upper right 2-row, 2-column pattern region 201 of structured light 200, the spatial distribution of light in pattern region 201 differs depending on the exposure time length. Spatial distribution 211 is an example of the spatial distribution of structured light 200 during exposure of image sensor 101 with an exposure time of 1 msec, and spatial distribution 212 is an example of the spatial distribution of light of structured light 200 during exposure of image sensor 101 with an exposure time of 1 / 2 msec.

[0018] Fig. 3 is an explanatory diagram showing an example of separation by time modulation of the projection pattern of structured light 200 when the exposure time is 1 msec. One cycle is 1 msec, 1 / 4 sec is the light emission unit period, and the periods obtained by dividing 1 msec into four are light emission periods LT1 to LT4 (similar to Fig. 4). The brightness of local patterns p0 to p3 depends on the number of light emission periods; the more the number of light emission periods, the brighter the patterns are, and the fewer the number of light emission periods, the darker the patterns are.

[0019] The local patterns p0 and p1 are global patterns PA, and emit light for light emission periods LT2 and LT3 of 1 / 4 msec of the 1 msec exposure time. The light emission periods of the local patterns p0 and p1 are different. The local patterns p2 and p3 are global patterns PB, and emit light for light emission periods [LT1, LT2] and [LT1, LT3] of 1 / 2 msec of the 1 msec exposure time. The light emission periods of the local patterns p2 and p3 are different, but may overlap partially (in this example, there is overlap at LT1).

[0020] When the imaging element 101 is exposed for 1 msec, the spatial distribution of the structured light 200 becomes as shown in spatial distribution 211. Specifically, for example, the pixels of the imaging element 101 that receive the local patterns p0 and p1 are darker than the pixels that receive the local patterns p2 and p3. On the other hand, the pixels of the imaging element 101 that receive the local patterns p2 and p3 are brighter than the pixels that receive the local patterns p0 and p1. In this case, the local patterns p0 and p1 are global patterns PA and have the same brightness, so they are not distinguished. The local patterns p2 and p3 are also global patterns PB and have the same brightness, so they are not distinguished.

[0021] 4 is an explanatory diagram showing an example of separation by time modulation of the projection pattern of structured light 200 when the exposure time is 1 / 2 msec. The exposure time of 1 / 2 msec corresponds to light emission periods LT3 and LT4. The light emission cycle of local patterns p0 to p3 is the same as in FIG.

[0022] The local pattern p0 is a global pattern PA and does not emit light within the 1 / 2 msec exposure time, but the local pattern p1 emits light during the light emission period LT3. The local pattern p2 is a global pattern PB and does not emit light within the 1 / 2 msec exposure time, but the local pattern p3 emits light during the light emission period LT3.

[0023] When the imaging element 101 is exposed for 1 / 2 msec, the spatial distribution of the structured light 200 becomes as shown in spatial distribution 212. Specifically, for example, the pixels of the imaging element 101 that receive the local patterns p0 and p2 are darker than the pixels that receive the local patterns p1 and p3. On the other hand, the pixels of the imaging element 101 that receive the local patterns p1 and p3 are brighter than the pixels that receive the local patterns p0 and p2. Note that in this case, the local patterns p0 and p1 are global patterns PA, but are distinguished because their brightnesses are different. The local patterns p2 and p3 are also global patterns PB, but are distinguished because their brightnesses are different.

[0024] <Configuration example of imaging device> 5 is a block diagram illustrating an example of the configuration of the imaging device according to Example 1. The imaging device 500 includes a light source 510, an imaging element 101, and an external processing device 520.

[0025] A light source 510 emits structured light 200 to a subject in response to a synchronization signal from an external processing device 520 .

[0026] The imaging element 101 has a pixel block array 501 and a peripheral circuit section 502. The pixel block array 501 is a collection of multiple pixel blocks arranged in rows and columns, and is a pixel section that receives structured light 200 reflected by a subject. The number of pixel blocks in the row direction is N (N is an integer equal to or greater than 1), and the number of pixel blocks in the column direction is M (M is an integer equal to or greater than 1). Each pixel block includes multiple pixels arranged in rows and columns. The imaging device 500 is capable of controlling the exposure time for each pixel block. Note that a pixel block may be composed of one pixel.

[0027] The pixel block array 501 photoelectrically converts the structured light 200 reflected by the subject based on the driving triggers of the RESET pulse and the READ pulse from the peripheral circuit unit 502 and the exposure time map, and outputs a pixel signal to the peripheral circuit unit 502. Details of the pixel blocks will be described later with reference to FIG.

[0028] The peripheral circuit unit 502 has a synchronization signal processing unit 503, a digital signal processing unit 504, an output interface (IF) 505, and a global memory 506. The synchronization signal processing unit 503 receives a synchronization signal from an external processing device 520, generates drive triggers for a RESET pulse and a READ pulse, and outputs them to the pixel block array 501.

[0029] The digital signal processor 504 receives the image signal from the pixel block array 501 and executes signal processing such as CDS processing. The image signal is a collection of pixel signals from each pixel in the pixel block array 501.

[0030] The output IF 505 outputs the image signal processed by the digital signal processor 504 to the external processing device 520 in pixel block units.

[0031] The global memory 506 stores an exposure time map from the external processing device 520. The exposure time map is divided and allocated for each pixel block.

[0032] The external processing device 520 has a synchronization signal generation unit 521 and an image signal processing unit 522. The synchronization signal generation unit 521 generates a synchronization signal and outputs it to the light source 510 and the synchronization signal processing unit 503 of the peripheral circuit unit 502. This allows the imaging device 500 to synchronize the readout timing of the image signal in the imaging element 101 and the irradiation timing of the structured light 200 from the light source 510 by the synchronization signal.

[0033] The image signal processing unit 522 processes the image signal for each pixel block. The image signal processing unit 522 executes a distance image calculation 523, an object detection / movement information calculation 524, and an exposure time map generation 525.

[0034] The distance image calculation 523 is a process for calculating the distance to the subject in pixel block units (or pixel units) based on the image signal and generating a distance image. For example, the distance image calculation 523 is a process for generating a distance image by TOF (Time Of Flight).

[0035] Subject detection / movement information calculation 524 is a process for detecting the background and main subjects that are not the background among the subjects based on the distance image obtained in distance image calculation 523, and for calculating the direction and amount of movement of the main subject.

[0036] Specifically, for example, based on the flight time from when synchronization signal generation unit 521 outputs a synchronization signal to light source 510 (i.e., the timing of irradiating structured light 200) to when image signal processing unit 522 acquires an image signal from imaging element 101 (i.e., the timing of receiving incident light), distance image calculation 523 and subject detection / movement information calculation 524 are executed. That is, image signal processing unit 522 functions as a detection unit that detects the main subject and the background.

[0037] Exposure time map generation 525 is a process for generating an exposure time map based on the luminance value, which is the pixel signal of each pixel. Specifically, for example, in exposure time map generation 525, the image signal processing unit 522 calculates, for each pixel block, a statistical value such as the average value, median value, or mode value for the luminance value, which is the pixel signal from each pixel in the pixel block. In exposure time map generation 525, the image signal processing unit 522 converts the statistical value of the luminance value calculated for each pixel block into an exposure value for the pixel block.

[0038] For example, when the luminance statistical value is equal to or greater than the first threshold and less than the second threshold, the image signal processor 522 maintains the previous exposure value for the pixel block. When the luminance statistical value is less than the first threshold, the image signal processor 522 sets the exposure value for the pixel block so that the exposure time is longer than the previous exposure value.

[0039] If the statistical value of the brightness value is equal to or greater than the second threshold value, the image signal processor 522 sets the exposure value for the pixel block so that the exposure time is shorter than the previous exposure value. The exposure time map is composed of the brightness values ​​of each pixel block arranged in the row and column directions. The initial value of the exposure time map is set externally.

[0040] Although not shown, the image signal processing unit 522 may execute image processing such as color interpolation, white balance adjustment, edge enhancement, gamma correction, and gradation conversion, as well as encoding processing, decoding processing, compression / expansion processing, etc., on a frame image formed by an image signal. Furthermore, the image signal processing unit 522 may display the frame image on a display screen.

[0041] <Pixel Block and Control Block> 6 is an explanatory diagram showing an example of a pixel block and a control block. In the pixel block array 501, the pixel blocks 600 are arranged in M ​​rows and N columns. Like the pixel blocks 600, the control blocks 601 are also arranged in M ​​rows and N columns. For example, in a first semiconductor substrate and a second semiconductor substrate that are stacked, the pixel block 600 is disposed on the first semiconductor substrate that is stacked on the second semiconductor substrate, and the control block 601 in the same region as the pixel block 600 is disposed on the second semiconductor substrate directly below the pixel block 600.

[0042] The pixel block 600 is a collection of pixels 610 arranged in rows and columns. The pixel block 600 is also a pixel unit. The number of pixels in the row direction is n (n is an integer equal to or greater than 1), and the number of pixels in the column direction is m (m is an integer equal to or greater than 1). Therefore, the image sensor 101 has N×n×M×m pixels 610 in its entirety.

[0043] The control block 601 includes a row readout selection circuit 602 , an exposure timing control circuit 603 , a local memory 604 , a column readout selection circuit 605 , an ADC (Analog to Digital Converter) column 606 , and an SRAM (Static Random Access Memory) column 607 .

[0044] The row readout selection circuit 602 is a circuit that selects a row of the pixel block 600. Pixel signals of pixels 610 in a row selected by the row readout selection circuit 602 are read out. The exposure timing control circuit 603 is an exposure control unit that controls the exposure timing of the pixel block 600 based on the drive triggers of a RESET pulse and a READ pulse. A local memory 604 stores the exposure value of the pixel block in the exposure time map.

[0045] The column readout selection circuit 605 is a circuit that selects a column of the pixel block 600. Pixel signals are read out from a pixel group in a row selected by the row readout selection circuit 602 and a column selected by the column readout selection circuit 605. The ADC column 606 is a set of ADCs provided for each pixel column of the pixel block 600.

[0046] Each ADC is connected to the vertical signal line of the pixel column, and converts analog pixel signals from each pixel 610 in the pixel column into digital pixel signals. The SRAM column 607 is a set of SRAMs provided for each pixel column of the pixel block 600. Each SRAM is connected to each ADC, and stores the digital pixel signals from the ADC.

[0047] <Pixel 610> 7 is a circuit diagram showing an example of a circuit configuration of a pixel 610. The pixel 610 includes a photoelectric conversion unit 700 and a readout unit 710. The readout unit 710 has a transfer unit 701, a discharge unit 702, an FD (floating diffusion) 703, a reset unit 704, and a pixel output unit 705, and reads out a pixel signal based on the charge converted by the photoelectric conversion unit 700 to a vertical signal line 720.

[0048] The pixel output unit 705 has an amplifier unit 751 and a selection unit 752. The transfer unit 701, the discharge unit 702, the FD 703, the reset unit 704, the amplifier unit 751, and the selection unit 752 are referred to as a readout unit 710. The readout unit 710 will be described as an N-channel FET, but the type of transistor is not limited to this.

[0049] The photoelectric conversion unit 700 has a photoelectric conversion function of converting light into electric charges. The photoelectric conversion unit 700 accumulates the electric charges generated by photoelectric conversion. The photoelectric conversion unit 700 is formed of, for example, a photodiode.

[0050] The transfer unit 701 transfers the charge of the photoelectric conversion unit 700 to the FD 703. The transfer unit 701 controls the electrical connection between the photoelectric conversion unit 700 and the FD 703. The transfer unit 701 is configured by, for example, a transistor. The transfer unit 701 may also be an element that has at least a gate terminal and configures a part of a transistor with a part of the photoelectric conversion unit 700 as a source terminal and a part of the FD 703 as a drain terminal. The gate terminal of the transfer unit 701 is connected to a transfer control line 711 for inputting a transfer control signal φTX. The transfer control line 711 will be described later.

[0051] The discharge unit 702 discharges the charge accumulated in the photoelectric conversion unit 700 to a power supply wiring to which a power supply voltage VDD is supplied. The discharge unit 702 controls the connection between the photoelectric conversion unit 700 and the power supply wiring. The discharge unit 702 is formed of, for example, a transistor. The discharge unit 702 may also be an element that has at least a gate terminal and forms part of a transistor with a part of the photoelectric conversion unit 700 as a source terminal and a part of a diffusion region connected to the power supply wiring as a drain terminal.

[0052] A gate terminal of the discharge unit 702 is connected to a discharge control line 712 for inputting a discharge control signal φPDRST. Note that, although the discharge unit 702 has been described as discharging the charge of the photoelectric conversion unit 700 to a power supply wiring to which a power supply voltage VDD is supplied, the discharge unit 702 may be discharged to a power supply wiring to which a power supply voltage different from the power supply voltage VDD is supplied.

[0053] The FD 703 receives charges transferred from the photoelectric conversion unit 700 by the transfer unit 701. The FD 703 accumulates the charges transferred from the photoelectric conversion unit 700.

[0054] The reset unit 704 discharges the charge accumulated in the FD 703 to a power supply wiring to which a power supply voltage VDD is supplied. The reset unit 704 resets the potential of the FD 703 to the power supply voltage VDD, which is a reference potential. The reset unit 704 controls the electrical connection between the FD 703 and the power supply wiring.

[0055] The reset unit 704 is composed of, for example, a transistor. The reset unit 704 may be an element that has at least a gate terminal and that configures a part of a transistor with a part of the FD 703 as a source terminal and a part of a diffusion region connected to a power supply wiring as a drain terminal. The gate terminal of the reset unit 704 is connected to a reset control line 713 for inputting a reset control signal φRST. The reset control line 713 will be described later.

[0056] The pixel output unit 705 outputs a pixel signal based on the potential of the FD 703 to the vertical signal line 720. The pixel output unit 705 has an amplifier unit 751 and a selection unit 752. The amplifier unit 751 is configured with a transistor. The amplifier unit 751 has a gate terminal connected to the FD 703, a drain terminal connected to a power supply wiring to which a power supply voltage VDD is supplied, and a source terminal connected to the drain terminal of the selection unit 752.

[0057] The selection unit 752 controls the electrical connection between the pixel 610 and the vertical signal line 720. When the selection unit 752 electrically connects the pixel 610 and the vertical signal line 720, a pixel signal is output from the pixel 610 to the vertical signal line 720. The selection unit 752 is configured of a transistor.

[0058] The selection unit 752 may be an element that has at least a gate terminal and constitutes part of a transistor with a part of the amplifier unit 751 as a source terminal and a part of a diffusion region connected to the vertical signal line 720 as a drain terminal. The gate terminal of the selection unit 752 is connected to a selection control line 714 across the multiple pixel blocks 600 for inputting a selection control signal φSEL. The source terminal of the selection unit 752 is connected to the load current source 706.

[0059] The load current source 706 is connected to the vertical signal line 720, and supplies a current for reading out a pixel signal from the pixel 610. This makes it possible to stabilize the operation of the amplifier unit 751. The load current source 706 is also connected to the vertical signal line 720.

[0060] The FD 703 and the pixel output unit 705 may be shared with other pixels 610. For example, the FD 703 and the pixel output unit 705 may be shared among a plurality of pixels 610 arranged in a row or column direction. The pixel 610 may be composed of a plurality of photoelectric conversion units 700 and transfer units 701. The readout unit 710 may not have the discharge unit 702 and the discharge control line 712.

[0061] Fig. 8 is a timing chart showing an example of imaging operation of the image sensor 101. Fig. 8 shows an example of imaging operation in which the drive of the image sensor 101 is controlled by the transfer control signal φTX, the discharge control signal φPDRST, the reset control signal φRST, and the selection control signal φSEL. In Fig. 8, the discharge control signal φPDRST is locally controlled, and the transfer control signal φTX, the reset control signal φRST, and the selection control signal φSEL are globally controlled. Note that the suffixes at the end of each signal on the left side indicate <1> , <2> , …, <m>indicates the row number of the pixel 610 within the pixel block.

[0062] The discharge control signal φPDRST controls the timing to start exposure. The exposure start timing corresponds to the falling edge of the discharge control signal φPDRST (for example, time T1). That is, before the exposure start time T1, the discharge control signal φPDRST turns on the discharge unit 702 to discharge the charge accumulated in the photoelectric conversion unit 700, and exposure starts at the falling edge of the discharge control signal φPDRST. Since the discharge control signal φPDRST is locally controlled, the exposure time can be adjusted for each pixel block 600.

[0063] The transfer control signal φTX controls the timing to end exposure. At time T3, the transfer control signal φTX turns on the transfer unit 701, thereby transferring the charge accumulated in the photoelectric conversion unit 700 to the FD 703. The timing to end exposure corresponds to the falling edge of the transfer control signal φTX (for example, time T4). Since the transfer control signal φTX is a globally controlled signal, the timing to end exposure is the same for each pixel block 600.

[0064] The reset control signal φRST controls the timing of discharging the charge accumulated in the FD 703. At time T2, the reset control signal φRST turns on the reset unit 704, thereby discharging the charge in the FD 703. By discharging the charge in the FD 703 before the timing of the end of exposure, the influence of the charge remaining in the FD 703 when the charge is transferred from the photoelectric conversion unit 700 can be suppressed.

[0065] The selection control signal φSEL is a signal for selecting an arbitrary pixel 610. The selection control signal φSEL controls the on / off of the selection unit 752. At time T2, the selection control signal φSEL is set to high. At time T3, the pixel 610 for which the selection control signal φSEL is set to high outputs a pixel signal to the vertical signal line 720 in response to the transfer control signal φTX being on. On the other hand, the pixel 610 for which the selection control signal φSEL is not set to high does not output a pixel signal.

[0066] The image sensor 101 can change the exposure start timing for each pixel block 600 by locally controlling the discharge control signal φPDRST, thereby controlling the exposure time for each pixel block 600. The image sensor 101 may also control the exposure end timing for each pixel block 600 by locally controlling the transfer control signal φTX. The image sensor 101 may also control both the exposure start timing and end timing for each pixel block 600 by locally controlling both the transfer control signal φTX and the discharge control signal φPDRST.

[0067] <Example of operation of pixel block 600> Fig. 9 is a block diagram showing an example of the operation of the pixel block 600. In Fig. 9, the pixel block 600 is, for example, a set of pixels in 6 rows and 6 columns (m=n=6). Therefore, the ADC column 606 is also composed of six ADCs. For ease of explanation, a placement area 900 indicates the area of ​​the first semiconductor substrate where the pixel block 600 is placed and the area of ​​the second semiconductor substrate where the control block 601 is placed directly below the pixel block 600.

[0068] The local memory 604 stores the exposure value of the pixel block 600 from the peripheral circuit unit 502. In this example, the exposure value indicates, as an example, either 1 msec exposure (long exposure) or 1 / 2 msec (short exposure).

[0069] The exposure timing control circuit 603 reads the exposure value. When the exposure timing control circuit 603 receives a drive trigger of a RESET pulse and a READ pulse from the peripheral circuit unit 502, the exposure timing control circuit 603 refers to the exposure value and sets the exposure time, that is, the output timing of the RESET pulse and the READ pulse.

[0070] The RESET pulse is, for example, the discharge control signal φPDRST and the reset control signal φRST shown in FIGS. 7 and 8, and the READ pulse is, for example, the transfer control signal φTX shown in FIGS.

[0071] For example, the exposure timing control circuit 603 sets a 1 msec exposure (long exposure) or a 1 / 2 msec (short exposure) as shown in FIG. 3 and FIG.

[0072] The local memory 604 outputs a SEL pulse to the row readout selection circuit 602 for sequentially selecting pixel rows starting from the first row of the pixel block 600. The SEL pulse is, for example, the selection control signal φSEL shown in FIGS.

[0073] The row readout selection circuit 602 outputs a SEL pulse (selection control signal φSEL) from the local memory 604 to the gate terminal of the selection unit 752 in the pixel 610 in the readout row of the pixel block 600. A pixel signal is read out from the pixel 610 while the selection unit 752 is ON by the selection control signal φSEL.

[0074] Furthermore, the row readout selection circuit 602 inputs a discharge control signal φPDRST from the exposure timing control circuit 603 to the gate terminal of the discharge unit 702 to discharge the charges accumulated in the photoelectric conversion unit 700. This discharge starts the exposure time.

[0075] Furthermore, while the selection unit 752 is ON by the selection control signal φSEL, the row readout selection circuit 602 outputs the reset control signal φRST from the exposure timing control circuit 603 to the gate terminal of the reset unit 704 in the pixel 610, and outputs the transfer control signal φTX, which is a READ pulse, to the gate terminal of the transfer unit 701 in the pixel 610. This causes the pixel signal to be read out from the pixel 610.

[0076] Note that pixel signals of pixel rows in the pixel block 600 that are not selected by the row readout selection circuit 602 are not read out and are thinned out (thinned out rows). In Fig. 9, the pixel rows at m=3rd and 5th rows are the thinned out rows.

[0077] The column readout selection circuit 605 outputs an enable signal to turn on the power supply of the ADC connected to the pixel column to be read out. As a result, the ADC to which the enable signal is input among the ADC columns 606 is turned on.

[0078] An ADC that is turned ON converts the pixel signals of each of the pixels 610 in that pixel column into digital signals and outputs them to the SRAM column 607. The pixel signals of each of the pixels 610 in a pixel column of an ADC to which an enable signal is not input (an ADC shaded in black in FIG. 9) are not read out. In FIG. 9, the n=2th pixel column from the left is the thinned-out column.

[0079] The pixel signals stored in the SRAM column 607 are output as an image signal for the entire pixel block 600 from a horizontal signal line 910 to the digital signal processing unit 504 in the peripheral circuit unit 502 .

[0080] In one read control from the pixel block 600, a read including at least one of the thinned-out rows and thinned-out columns is called a "thinned-out readout", and a readout without thinning out is called a "all-pixel readout".

[0081] In addition, even in a single read control from the pixel block array 501, if it includes at least one pixel block 600 that performs thinning readout, it is called thinning readout, and if it does not include any pixel block 600 that performs thinning readout, it is called "all pixel readout."

[0082] <Operation Example of Imaging Device 500> Fig. 10 is an explanatory diagram showing an example of the operation of the imaging device 500. In Fig. 10, rows (1) to (4) show the time-series processing in the imaging device 500. (1) to (4) correspond to the numbers of the frames to be processed. Columns (A) to (C) show the processing contents in the imaging device 500 for each of the time series (1) to (4).

[0083] [(1) 1st frame] (A) Read control The image sensor 101 performs thinning readout with the entire surface of the pixel block array 501 as a thinning readout region TR. Specifically, for example, the image sensor 101 thins out the pixel blocks 600 in a specific row. That is, in the image sensor 101, the peripheral circuit unit 502 does not output a drive trigger for the pixel blocks 600 in the specific row. In addition, the peripheral circuit unit 502 may perform control so that the pixel blocks 600 in the specific row do not output an enable signal to the ADC column 606.

[0084] (B) Structured light pattern (A) The projection pattern of the structured light 200 during the exposure time during readout control is a low-resolution projection pattern, i.e., a spatial distribution 211 of light composed of a global pattern PA in which the local patterns p0 and p1 are not distinguished and a global pattern PB in which the local patterns p2 and p3 are not distinguished. In this example, the image sensor 101 can receive the structured light 200 in the spatial distribution 211 by driving with the above-mentioned long-second exposure of 1 msec.

[0085] (C) Processing of the imaging device 500 The image sensor 101 captures an image at a low resolution, and thins out and reads out the first frame Fa1, which is the image signal for the first frame, as shown in (A) read control. The first frame Fa1 becomes a low-resolution image consisting of a main subject 1001 and a background 1010, as shown in (B) structured light pattern.

[0086] The external processing device 520 uses the first frame Fa1 to execute distance image calculation 523, subject detection / movement information calculation 524, and exposure time map generation 525. It is assumed that a first main subject 1001 is detected from a background 1010 by subject detection / movement information calculation 524.

[0087] The pixel blocks 600 that receive the structured light 200 of the global pattern PA are darker than the pixel blocks 600 that receive the structured light 200 of the global pattern PB (see FIGS. 2 and 3). In the exposure time map generation 525, the image signal processing unit 522 calculates the statistics of the luminance values ​​of each pixel 610 affected by such global patterns PA, PB for each pixel block 600, and generates a first exposure time map Ma1.

[0088] In the first exposure time map Ma1, the exposure value of the pixel block 600 in which the first main subject 1001 is detected is set to a value indicating an exposure time of ½ msec, and the exposure value of the pixel block 600 in the background 1010 in which the first main subject 1001 is not detected is set to a value indicating an exposure time of 1 msec. The first exposure time map Ma1 is written to the global memory 506 of the peripheral circuit unit 502.

[0089] The first exposure time map Ma1 is divided into exposure values ​​for each pixel block 600 and written to the local memory 604 of the pixel block 600.

[0090] [(2) 2nd frame] (A) Read control The image sensor 101 sets the exposure time for capturing the second frame for each pixel block 600 based on the exposure value written to the local memory 604. That is, the pixel block 600 in which the first main subject 1001 is detected is set to a first all-pixel readout region FR1 having an exposure time of 1 / 2 msec, and the other pixel blocks 600 in the background 1010 are set to a thinned-out readout region TR having an exposure time of 1 msec. The image sensor 101 then executes thinned-out readout of the thinned-out readout region TR and all-pixel readout of the first all-pixel readout region FR1.

[0091] (B) Structured light pattern (A) The projection pattern of the structured light 200 during the exposure time during readout control is a low-resolution region with a low-resolution spatial distribution 211 in the thinned readout region TR, as in (1), but is a high-resolution projection pattern, i.e., a first high-resolution region 212a with a high-resolution spatial distribution 212, in the first all-pixel readout region FR1. The first high-resolution region 212a is a spatial distribution 212 of light composed of a global pattern PA in which local patterns p0 and p1 are distinguished, and a global pattern PB in which local patterns p2 and p3 are distinguished.

[0092] The image sensor 101 can receive the structured light 200 in the low-resolution region in a spatial distribution 211 by driving the thinned-out readout region TR with a long-second exposure of 1 msec. Also, the image sensor 101 can receive the structured light 200 in the first high-resolution region 212a in a spatial distribution 212 by driving the first all-pixel readout region FR1 with a short-second exposure of 1 / 2 msec.

[0093] (C) Processing of the imaging device 500 The image sensor 101 performs low-resolution imaging in the thinned-out readout region TR, thereby thinning-out readout of the thinned-out readout region TR that constitutes the background 1010 in the second frame Fa2, which is the image signal of the second frame.

[0094] Meanwhile, the image sensor 101 performs high-resolution imaging in the first full-pixel readout region FR1, thereby reading out all pixels of the first full-pixel readout region FR1 including the first main subject 1001 in the second frame Fa2. As a result, the second frame Fa2 is output to the external processing device 520. The second frame Fa2 becomes an image consisting of the high-resolution first main subject 1001 and the low-resolution background 1010 as shown in the (B) structured light pattern.

[0095] The external processing device 520 uses the second frame Fa2 to execute distance image calculation 523, subject detection / movement information calculation 524, and exposure time map generation 525. It is assumed that the first main subject 1001 is detected from the background 1010 by subject detection / movement information calculation 524, as in (1).

[0096] In the thinned-out readout region TR, pixel blocks 600 that receive structured light 200 of the global pattern PA are darker than pixel blocks 600 that receive structured light 200 of the global pattern PB (see FIGS. 2 and 3). In the exposure time map generation 525, the image signal processing unit 522 calculates statistics of the luminance values ​​of each pixel 610 affected by such global patterns PA, PB for each pixel block 600 that constitutes the thinned-out readout region TR.

[0097] In the first all-pixel readout region FR1, pixel blocks 600 that receive structured light 200 of local patterns p0, p2 are darker than pixel blocks 600 that receive structured light 200 of local patterns p1, p3 (see FIGS. 2 and 4). In exposure time map generation 525, the image signal processing unit 522 calculates statistics of the luminance values ​​of each pixel 610 affected by such local patterns p0 to p3 for each pixel block 600 constituting the first all-pixel readout region FR1.

[0098] The matrix structure of the statistical values ​​of the luminance values ​​for each pixel block 600 consisting of M rows and N columns becomes the second exposure time map Ma2. The global memory 506 of the peripheral circuit unit 502 is overwritten with the second exposure time map Ma2.

[0099] The second exposure time map Ma2 is divided into exposure values ​​for each pixel block 600 and written to the local memory 604 of the pixel block 600.

[0100] [(3) 3rd frame] (A) Read control The image sensor 101 sets the exposure time for capturing the third frame based on the exposure value written to the local memory 604 for each pixel block 600. That is, similar to (2), the image sensor 101 executes thinning readout of the thinning readout region TR and full pixel readout of the first full pixel readout region FR1.

[0101] (B) Structured light pattern (A) The projection pattern of the structured light 200 during the exposure time during readout control is, as in (2), a low-resolution area with a low-resolution spatial distribution 211 in the thinned readout region TR, and a first high-resolution area 212a in the first full pixel readout region FR1.

[0102] The image sensor 101 drives the thinned readout region TR with a long-second exposure of 1 msec, thereby being able to receive the structured light 200 of the low-resolution region in a spatial distribution 211. The image sensor 101 drives the first all-pixel readout region FR1 with a short-second exposure of 1 / 2 msec, thereby being able to receive the structured light 200 of the first high-resolution region 212a in a spatial distribution 212. It is assumed that the second main subject 1002 moves into the background 1010 at the timing of (3).

[0103] (C) Processing of the imaging device 500 As in (2), the image sensor 101 performs low-resolution imaging in the thinned-out readout region TR to thin-out read out the thinned-out readout region TR that constitutes the background 1010 in the third frame Fa3, which is the image signal for the third frame.

[0104] Meanwhile, the image sensor 101 performs high-resolution imaging in the first all-pixel readout region FR1, thereby reading out all pixels of the first all-pixel readout region FR1 including the first main subject 1001 in the third frame Fa3. As a result, the third frame Fa3 is output to the external processing device 520. The third frame Fa3 is an image made up of the high-resolution first main subject 1001 and the low-resolution background 1010 (including the second main subject 1002) as shown in the structured light pattern (B).

[0105] Using the third frame Fa3, the external processing device 520 executes distance image calculation 523, subject detection / movement information calculation 524, and exposure time map generation 525. By subject detection / movement information calculation 524, the first main subject 1001 is detected from the background 1010, and the second main subject 1002 is detected, as in (2).

[0106] As in (2), in the exposure time map generation 525, in the thinned readout region TR including the background 1010 and the second main subject 1002, the image signal processing unit 522 calculates statistics of the luminance values ​​of each pixel 610 affected by the global patterns PA, PB for each pixel block 600 that constitutes the thinned readout region TR.

[0107] As in (2), in the exposure time map generation 525, in the first all-pixel readout region FR1, the image signal processing unit 522 calculates statistical values ​​of the brightness values ​​of each pixel 610 affected by local patterns p0 to p3 for each pixel block 600 that constitutes the first all-pixel readout region FR1.

[0108] The matrix structure of the statistical values ​​of the luminance values ​​for each pixel block 600 composed of M rows and N columns becomes the third exposure time map Ma3. The global memory 506 of the peripheral circuit unit 502 is overwritten with the third exposure time map Ma3.

[0109] The third exposure time map Ma3 is divided into exposure values ​​for each pixel block 600 and written to the local memory 604 of the pixel block 600.

[0110] [(4) 4th frame] (A) Read control The image sensor 101 sets the exposure time for capturing the fourth frame for each pixel block 600 based on the exposure value written to the local memory 604. That is, similar to (3), the image sensor 101 executes thinning readout of the thinning readout region TR and reads out all pixels in the first all-pixel readout region FR1. In addition, the pixel block 600 in which the second main subject 1002 is detected is set to the second all-pixel readout region FR2, which has an exposure time of 1 / 2 msec. Therefore, the image sensor 101 also executes readout of all pixels in the second all-pixel readout region FR2.

[0111] (B) Structured light pattern (A) The projection pattern of the structured light 200 during the exposure time during readout control is, as in (3), a low-resolution area with a low-resolution spatial distribution 211 in the thinned readout region TR, and a first high-resolution area 212a in the first full pixel readout region FR1.

[0112] Similarly to the first all-pixel readout region FR1, the second all-pixel readout region FR2 also becomes a high-resolution projection pattern, i.e., a second high-resolution region 212b which is a high-resolution spatial distribution 212. The second high-resolution region 212b becomes a spatial distribution 212 of light which is composed of a global pattern PA in which local patterns p0 and p1 are distinguished, and a global pattern PB in which local patterns p2 and p3 are distinguished.

[0113] The image sensor 101 can receive the structured light 200 in the low resolution region in a spatial distribution 211 by driving the thinned-out readout region TR with a long-second exposure of 1 msec. Also, the image sensor 101 can receive the structured light 200 in the first high resolution region 212a and the second high resolution region 212b in a spatial distribution 212 by driving the first all-pixel readout region FR1 and the second all-pixel readout region FR2 with a short-second exposure of 1 / 2 msec.

[0114] (C) Processing of the imaging device 500 As in (3), the image sensor 101 performs low-resolution imaging in the thinned-out readout region TR to thin-out read out the thinned-out readout region TR that is the background 1010 in the fourth frame Fa4, which is the image signal for the fourth frame.

[0115] Meanwhile, the image sensor 101 performs high-resolution imaging in the first full pixel readout region FR1 and the second full pixel readout region FR2, thereby reading out all pixels of the first full pixel readout region FR1 including the first main subject 1001 and the second full pixel readout region FR2 including the second main subject 1002 in the fourth frame Fa4. As a result, the fourth frame Fa4 is output to the external processing device 520. The fourth frame Fa4 is an image made up of the high-resolution first main subject 1001 and second main subject 1002 and the low-resolution background 1010 as shown in the structured light pattern (B).

[0116] Using the fourth frame Fa4, the external processing device 520 executes, as in (3), distance image calculation 523, subject detection / movement information calculation 524, and exposure time map generation 525. In this manner, the imaging device 500 can perform imaging including low resolution and high resolution.

[0117] 10, in the (1)(A) readout control, the image sensor 101 performs thinning readout on the entire surface, but the entire surface may be read out. In this case, when the first main subject 1001 is detected in (C), in the (2)(A) readout control, the image sensor 101 sets the area other than the first all-pixel readout area FR1 as the thinning readout area TR.

[0118] The higher the speed and resolution of the image sensor 101, the more important it becomes to reduce the amount of data of pixel signals to be read and to reduce power consumption. The greater the number of pixels in the image sensor 101, the greater the number of pixels required for pattern matching, which increases the computation cost and processing time for pixel signals. On the other hand, the finer the pattern granularity of the structured light 200, the greater the number of pixels from which the image sensor 101 reads pixel signals, improving the distance measurement accuracy, whereas the coarser the pattern granularity of the structured light 200, the fewer the number of pixels from which the image sensor 101 reads pixel signals, decreasing the distance measurement accuracy.

[0119] According to the imaging device 500 of the first embodiment, the high-resolution projection pattern of the structured light 200 is received in the all-pixel readout regions FR1 and FR2 to read out pixel signals. This improves the distance measurement accuracy of the first main subject 1001 and the second main subject 1002 in the all-pixel readout regions FR1 and FR2.

[0120] Furthermore, the low-resolution projection pattern of the structured light 200 is received in the thinned readout region TR and pixel signals are read out. This makes it possible to reduce the amount of data read by the image sensor 101 for the image of the background 1010, which is the thinned readout region TR, and the power consumption, as well as reduce the computation cost and processing time during pattern matching. EXAMPLES

[0121] Next, a description will be given of Example 2. In Example 1, an image capturing device 500 in which an external processing device 520 is provided outside the image capturing element 101 is described. In Example 2, an image capturing device that generates an exposure time map within the image capturing element 101 is described. Note that the same components as those in Example 1 are given the same reference numerals, and descriptions thereof will be omitted.

[0122] <Configuration example of imaging device> 11 is a block diagram illustrating an example of the configuration of an image capturing apparatus according to Example 2. The image capturing apparatus 1100 includes a light source 510, an image sensor 1101, and an external processing apparatus 520.

[0123] The image sensor 1101 includes a pixel block array 501 and a peripheral circuit section 1102. The peripheral circuit section 1102 includes a synchronization signal processing section 503, a digital signal processing section 1104, an output IF 505, and a global memory 506.

[0124] The digital signal processing unit 1104 receives an image signal from the pixel block array 501 and executes signal processing such as CDS processing, similar to the digital signal processing unit 504 in the first embodiment. In addition, the digital signal processing unit 1104 executes region-by-region luminance statistics calculation 541, difference processing / ternarization processing 542, movement information calculation 543, and exposure time map update 544 for the image signal from the pixel block array 501.

[0125] The region-specific luminance statistics calculation 541 is a process for calculating, for each pixel block 600, a statistical value (average, median, mode, etc.) of the luminance values ​​indicated by pixel signals from the pixels 610 in the pixel block 600.

[0126] The difference processing / ternarization processing 542 includes difference processing and ternarization processing. Difference processing is processing for calculating an inter-frame difference, which is the difference between statistical values ​​of luminance values ​​for the same pixel block 600 between two temporally consecutive frames. The inter-frame difference is the difference between statistical values ​​of luminance values ​​for each pixel block 600 configured with M rows and N columns. The ternarization processing is processing for converting each element of the inter-frame difference configured with M rows and N columns into ternary values.

[0127] Movement information calculation 543 is a process for calculating the movement direction and amount of movement of the main subject based on the result of the ternarization process.

[0128] Exposure time map update 544 is a process of reading the exposure time map of the previous frame from the global memory 506, generating an exposure time map to be used for exposure control in capturing the next frame from the exposure time map of the previous frame based on the result of the ternarization process, and overwriting and saving it in the global memory 506. Note that the initial exposure time map is stored in the global memory 506 in advance or by external input.

[0129] It should be noted that the exposure time map is repeatedly updated within the image sensor 1101, but similar to the first embodiment, the image signal processing unit 522 of the external processing device 520 may generate the exposure time map by executing distance image calculation 523, subject detection / movement information calculation 524, and exposure time map generation 525, similar to the first embodiment.

[0130] The imaging device 1100 may be configured such that the frequency of generating the exposure time map in the image signal processing unit 522 in the external processing device 520 can be set by an external setting.

[0131] For example, image signal processing unit 522 may generate an exposure time map every time the exposure time map is updated a predetermined number of times in image sensor 1101, and overwrite and save the exposure time map in global memory 506. Image signal processing unit 522 generates the exposure time map based on the distance image calculated by distance image calculation 523. For this reason, the exposure time map generated by image signal processing unit 522 is based on the distance image rather than on statistics of brightness values, and therefore has a more accurate exposure value than the exposure time map generated by exposure time map update 544.

[0132] The more frequently the exposure time map is generated by the image signal processing unit 522, the more accurate the exposure time map will be, and the less frequently the exposure time map is generated by the image signal processing unit 522, the less data communication volume between the image sensor 1101 and the external processing device 520 will be, thereby enabling lower power consumption of the image sensor 1100 and faster image processing at the image sensor 1101.

[0133] <Operation Example of Imaging Device 1100> Fig. 12 is an explanatory diagram showing an example of the operation of the imaging device 1100. In Fig. 12, rows (1) to (3) indicate time-series processing in the imaging device 1100. (1) to (3) correspond to the numbers of frames to be processed.

[0134] Fig. 13 is an explanatory diagram showing an update example 1 of the exposure time map in the digital signal processing unit 1104, and Fig. 14 is an explanatory diagram showing an update example 2 of the exposure time map in the digital signal processing unit 1104. The operation of the imaging device 1100 will be described below with reference to Figs. 12 to 14. For ease of explanation, the pixel block array 501 is assumed to be a set of pixel blocks 600 with 8 rows and 8 columns.

[0135] [(1) 1st frame] 12, the image sensor 1101 performs imaging at a low resolution (step S111). Specifically, for example, as shown in FIG 13, the image sensor 1101 performs thinning readout on the entire surface of the pixel block array 501 as a thinning readout region TR, and outputs a first frame Fb1, which is an image signal for the first frame. The first frame Fb1 is a low-resolution image consisting of a main subject 1301 and a background 1302.

[0136] 12, the image sensor 1101 executes the per-region luminance statistical calculation 541 to generate a first luminance map LM1 for the first frame Fb1 as shown in FIG 13 (step S112). The first luminance map LM1 is an 8-row, 8-column matrix, similar to the pixel block array 501, and each element is a statistical value of the luminance value indicated by the pixel signal from each pixel 610 in each pixel block 600. The first luminance map LM1 becomes a luminance region 1312 corresponding to the entire surface, the thinned-out readout region TR.

[0137] 12, the external processing device 520 executes distance image calculation 523 and subject detection based on the first frame Fb1 (step S121). Then, the external processing device 520 executes exposure time map generation 525 using the distance image and the subject detection result to generate a first exposure time map Mb1 (step S122).

[0138] In the first exposure time map Mb1, the pixel block array 501 is also a matrix of 8 rows and 8 columns. In the first exposure time map Mb1, the region where the element corresponding to the main subject 1301 has a value of "1" is the high resolution region HR (1 / 2 msec exposure), and the region where the element corresponding to the background 1302 has a value of "0" is the low resolution region LR (1 msec exposure). The external processing device 520 outputs the first exposure time map Mb1 to the peripheral circuit unit 1102 of the image sensor 1101.

[0139] 12, the image sensor 1101 executes the exposure time map update 544 (step S113). Specifically, for example, the image sensor 1101 overwrites the global memory 506 with the first exposure time map Mb1 from the external processing device 520 and saves it.

[0140] [(2) 2nd frame] 12, the image sensor 1101 performs imaging using per-region exposure control (step S211). Specifically, for example, as shown in FIG 13, the image sensor 1101 refers to a first exposure time map Mb1 to set an exposure value for each pixel block 600 and performs imaging.

[0141] The image sensor 1101 then reads out all the pixels of the pixel block 600 including the main subject 1301 in the pixel block array 501 as a full pixel readout region FR, and thins out and reads out the pixel block 600 including the background 1302 as a thinned readout region TR, to output a second frame Fb2 which is an image signal for the second frame. The second frame Fb2 is an image consisting of the main subject 1301 in high resolution and the background 1302 in low resolution.

[0142] In FIG. 12, the imaging device 1101 executes luminance statistics calculation 541 for each region, and as shown in FIG. 13, generates a second luminance map LM2 for the second frame Fb2 (step S212). The second luminance map LM2 is a matrix of 8 rows and 8 columns, similar to the pixel block array 501, and each element is a statistical value of the luminance values indicated by the pixel signals from each pixel 610 of each pixel block 600. The second luminance map LM2 includes a luminance region 1321 corresponding to the entire pixel readout region FR and a luminance region 1322 corresponding to the decimated readout region TR.

[0143] In FIG. 12, the imaging device 1101 executes difference processing / ternarization processing 542 and movement information calculation 543 (step S213). Specifically, for example, as shown in FIG. 13, the imaging device 1101 outputs a difference map Δ(LM1, LM2) by subtracting the second to first luminance maps LM1 through difference processing.

[0144] Next, the imaging device 1101 executes ternarization processing on the difference map Δ(LM1, LM2) and outputs a ternarization processing result 3Δ(LM1, LM2) as shown in FIG. 13. Specifically, for example, for each element of the difference map Δ(LM1, LM2), if the value is greater than or equal to the first threshold Th1, it is converted to "1", if it is greater than or equal to the second threshold Th2 (<T1) and less than the first threshold T2, it is converted to "0", and if it is less than the second threshold Th2, it is converted to "-1".

[0145] In the ternarization processing result, the region where the element value is "1" is denoted as the difference positive region Δ+, the region where the value is "0" is denoted as the difference zero region Δ0, and the region where the value is "-1" is denoted as the difference negative region Δ-. In FIGS. 13 and 14, for example, the first threshold Th1 = 5 and the second threshold Th2 = -5.

[0146] The difference positive region Δ+ corresponds to the main subject before movement. Since the difference zero region Δ0 is a region where there is no difference between the previous frame and the current frame, it corresponds to the background. The difference negative region Δ- corresponds to the main subject after movement. When the main subject does not move between consecutive frames, the difference negative region Δ- does not exist in the ternarization processing result.

[0147] In the case of the ternarization processing result 3Δ(LM1, LM2), the positive difference area Δ+ is the area in the first frame Fb1 where the main subject 1301 exists before movement, and the zero difference area Δ0 is the area in the first frame Fb1 and the second frame Fb2 where the background 1302 exists. The negative difference area Δ- does not exist in the ternarization processing result 3Δ(LM1, LM2). Therefore, the movement information calculation 543 does not detect the movement direction and movement amount of the main subject 1301 from the ternarization processing result 3Δ(LM1, LM2).

[0148] 12, the image sensor 1101 executes exposure time map update 544 (step S213). Specifically, for example, the image sensor 1101 updates the first exposure time map Mb1 from the external processing device 520 based on the movement direction and movement amount of the main subject 1301, as shown in FIG.

[0149] In this example, since the direction and amount of movement of the main subject 1301 cannot be detected from the ternary processing result 3Δ(LM1, LM2), the first exposure time map Mb1 is directly overwritten and saved in the global memory 506. For convenience, the overwritten first exposure time map Mb1 is referred to as the second exposure time map Mb2.

[0150] In FIG. 12, the external processing device 520 executes the distance image calculation 523 and subject detection (step S221) based on the second frame Fb2 obtained by imaging using per-area exposure control (step S211).

[0151] [(3) 3rd frame] 12, the image sensor 1101 performs imaging using per-region exposure control (step S311). Specifically, for example, as shown in FIG 14, the image sensor 1101 refers to the second exposure time map Mb2 to set an exposure value for each pixel block 600 and captures an image. Because the second exposure time map Mb2 is the same map as the first exposure time map Mb1, the image sensor 1101 performs full pixel readout and thinned readout in the full pixel readout region FR and thinned readout region TR, which are located in the same positions as the first exposure time map Mb1, and outputs a third frame Fb3, which is an image signal for the third frame.

[0152] In imaging using per-area exposure control (step S311), it is assumed that the main subject 1301 moves to the left. In this case, it is assumed that the third frame Fb3 becomes an image consisting of the main subject 1301 and low-resolution background 1302 that straddles low-resolution and high-resolution areas.

[0153] 12, the image sensor 1101 executes per-region luminance statistical calculation 541 to generate a third luminance map LM3 for the third frame Fb3 as shown in Fig. 14 (step S312). The third luminance map LM3 is an 8-row, 8-column matrix, similar to the pixel block array 501, and each element is a statistical value of the luminance value indicated by the pixel signal from each pixel 610 in each pixel block 600. The third luminance map LM3 includes a luminance region 1331 corresponding to the all-pixel readout region FR and a luminance region 1332 corresponding to the thinned-out readout region TR.

[0154] 12, the image sensor 1101 executes difference processing / ternarization processing 542 and movement information calculation 543 (step S313). Specifically, for example, the image sensor 1101 performs difference processing to subtract the second luminance map LM2 from the third luminance map LM3, thereby outputting a difference map Δ(LM2, LM3), as shown in FIG.

[0155] Next, the image sensor 1101 performs ternarization processing on the difference map Δ(LM2, LM3) and outputs the ternarization processing result 3Δ(LM2, LM3) as shown in FIG.

[0156] In the case of the ternarization processing result 3Δ(LM2, LM3), the positive difference area Δ+ is the area in the second frame Fb2 where the main subject 1301 exists before movement. The zero difference area Δ0 is the area in the second frame Fb2 and the third frame Fb3 where the background 1302 exists. The negative difference area Δ- is the area in the third frame Fb3 where the main subject 1301 exists after movement.

[0157] The imaging element 1101 calculates the movement direction and movement amount from the positive difference area Δ+ to the negative difference area Δ- in the ternarization processing result 3Δ(LM2, LM3) by the movement information calculation 543. In this example, the movement direction is leftward, and the movement amount is "3".

[0158] 12, the image sensor 1101 executes exposure time map update 544 (step S313). Specifically, for example, the image sensor 1101 updates the second exposure time map Mb2 stored in the global memory 506 based on the movement direction and movement amount of the main subject 1301. In this example, since the movement direction and movement amount of the main subject 1301 have been detected from the ternarization processing result 3Δ(LM2, LM3), the image sensor 1101 moves the high resolution region HR of the second exposure time map Mb2 to the left by 3 and stores it in the global memory 506 as a third exposure time map Mb3.

[0159] Note that, when the external processing device 520 generates the third exposure time map Mb3 for the third frame, steps S312 and S313 are not executed in the image sensor 1101. In this case, the external processing device 520 executes the distance image calculation 523 and subject detection (step S321) based on the third frame Fb3 obtained by imaging using per-area exposure control (step S311).

[0160] Then, the external processing device 520 executes exposure time map generation 525 using the distance image and subject detection result of step S321 to generate a third exposure time map Mb3 (step S322). The external processing device 520 outputs the third exposure time map Mb3 to the peripheral circuit unit 1102 of the image sensor 1101. As a result, the digital signal processing unit 1104 in the peripheral circuit unit 1102 overwrites and saves the third exposure time map Mb3 from the external processing device 520 in the global memory 506 by exposure time map update 544 (step S314).

[0161] In this way, according to the second embodiment, the image sensor 1101 autonomously generates the exposure time map repeatedly, so that the exposure time map can be updated at high speed. Also, the amount of data communication between the image sensor 1101 and the external processing device 520 can be reduced.

[0162] In the above-described first and second embodiments, the spatial distribution 211 of the local patterns p0 to p3 in the 1 msec exposure and the spatial distribution 212 of the local patterns p0 to p3 in the 1 / 2 msec exposure may be similar in shape.

[0163] To generate a distance image using structured light ranging, two types of information are required: imaging data and projection pattern, but if the global pattern and local pattern are different, it is necessary to prepare two separate pieces of information: distance image calculation unit 523 inside image signal processing unit 522 and the projection pattern in the internal memory of distance image calculation unit 523. In this case, if a projection pattern that is even partially identical is used, the processing circuitry and memory area for storing projection pattern information can be reused, leading to a reduction in circuit scale.

[0164] 15 is an explanatory diagram showing an example of a spatial pattern of structured light 200. Spatial distribution 1501 is spatial distribution 211 of structured light 200 due to global patterns PA and PB (local patterns p0 to p3 in 1 msec exposure). Spatial distribution 1502 is spatial distribution 212 of structured light 200 due to local patterns p0 to p3 in 1 / 2 msec. Of spatial distribution 1502, partial spatial distribution 1520 indicated by a thick frame is similar to spatial distribution 1501.

[0165] As a result, in the case of 1 msec exposure, when thinning readout is performed on image sensor 101, 1101, structured light 200 of subspatial distribution 1510 having the same pattern as subspatial distribution 1520 is received, and thinning readout is performed. Therefore, for image signals from pixel blocks 600 of image sensor 101, 1101 corresponding to subspatial distributions 1510, 1520, distance image calculation unit 523 and an area of ​​the internal memory that holds projection pattern information can be reused. This makes it possible to reduce the circuit scale.

[0166] The present invention is not limited to the above, and may be implemented in any combination thereof. In addition, other embodiments that are conceivable within the scope of the technical concept of the present invention are also included in the scope of the present invention. [Explanation of symbols]

[0167] 101 imaging element, 200 structured light, 201 pattern area, 211, 212 spatial distribution, 500 imaging device, 501 pixel block array, 502 peripheral circuit section, 503 synchronization signal processing section, 504 digital signal processing section, 506 global memory, 510 light source, 520 external processing device, 521 synchronization signal generation section, 522 image signal processing section, 523 distance image calculation, 524 movement information calculation, 525 exposure time map generation, 541 region-by-region brightness statistics calculation, 542 difference processing / ternary processing, 543 movement information calculation, 544 exposure time map update, 600 pixel block, 601 pixel array, 602 row readout selection circuit, 603 exposure timing control circuit, 604 local memory, 605 column readout selection circuit, 606 ADC column, 607 SRAM column, 610 pixel, 700 Photoelectric conversion section, 701 transfer section, 702 discharge section, 704 reset section, 705 pixel output section, 706 load current source, 710 read section, 711 transfer control line, 712 discharge control line, 713 reset control line, 714 selection control line, 720 vertical signal line, 751 amplifier section, 752 selection section, 900 horizontal signal line, 1100 imaging device, 1101 imaging element, 1102 peripheral circuit section, 1104 digital signal processing section< / m>

Claims

1. a light source that emits structured light including a plurality of pattern lights having different light emission patterns in a first period and a second period that is shorter than the first period; a pixel unit having one or more pixels including a photoelectric conversion unit that converts incident light incident as a result of irradiation of the structured light from the light source into an electric charge; an exposure control unit that controls an accumulation time for accumulating charges in the photoelectric conversion unit to be the first period or the second period based on a pixel signal read out from the pixel; An imaging device having the above configuration.

2. 2. The imaging device according to claim 1, The photoelectric conversion unit of the pixel unit converts incident light, which is incident as a result of the structured light being irradiated from the light source, into an electric charge during an accumulation time controlled by the exposure control unit. Imaging device.

3. 2. The imaging device according to claim 1, an imaging element including the pixel unit and the exposure control unit; Imaging device.

4. 2. The imaging device according to claim 1, the structured light includes a first pattern light having a first light emission pattern, a second pattern light having a second light emission pattern, a third pattern light having a third light emission pattern, and a fourth pattern light having a fourth light emission pattern; In the first period, the first pattern light and the second pattern light are darker than the third pattern light and the fourth pattern light, In the second period, the first pattern light and the third pattern light are darker than the second pattern light and the fourth pattern light. Imaging device.

5. 2. The imaging device according to claim 1, a spatial distribution of the plurality of pattern lights in the first period and a spatial distribution of the plurality of pattern lights in the second period have similar shapes; Imaging device.

6. 2. The imaging device according to claim 1, a detection unit that detects a specific subject based on a timing of irradiation of the structured light and a timing of reception of the incident light at the pixel; the exposure control unit sets the accumulation time by the pixels within the area of ​​the specific object detected by the detection unit to the second period, and sets the accumulation time by the pixels outside the area of ​​the specific object detected by the detection unit to the first period. Imaging device.

7. 7. The imaging device according to claim 6, the pixel unit thins out a portion of the pixel signals from each of the pixels that have undergone photoelectric conversion in the first period and outputs the thinned pixel signals. Imaging device.

8. a pixel unit having one or more pixels including a photoelectric conversion unit that converts incident light incident as a result of irradiation of the structural light from a light source that irradiates the structural light including a plurality of pattern lights having different emission patterns in a first period and a second period that is shorter than the first period, into electric charges; an exposure control unit that controls an accumulation time for accumulating charges in the photoelectric conversion unit to be the first period or the second period based on a pixel signal read out from the pixel; An imaging element having

Citation Information

Patent Citations

  • Imaging apparatus

    JP2007295113A