Camera system and control method thereof

Abstract

[Problem] To achieve excellent responsiveness and reduce power consumption. [Solution] In a camera system, a first imaging element includes a plurality of first pixel circuits that simultaneously sample and non-destructively read out charges corresponding to photoelectrically converted incident light in each pixel; the second imaging element includes a plurality of second pixel circuits that simultaneously sample and non-destructively read out charges corresponding to the photoelectrically converted incident light in each pixel. The first imaging element switches whether or not to output pixel signals from at least some of the plurality of first pixel circuits within the same frame period based on pixel signals output from some of the first pixel circuits or the second pixel circuits for each frame period. The second imaging element switches whether or not to output a pixel signal from at least some of the plurality of second pixel circuits within the same frame period based on a pixel signal output from the first pixel circuit or some of the second pixel circuits per frame period.

Description

Technical Field

[0001] The present disclosure relates to a camera system and a control method thereof. Background Art

[0002] Since a stereo camera used for goggles, head-mounted displays, and the like is driven by a battery, it is desirable to suppress average power consumption and peak power. For example, in the camera system disclosed in Patent Document 1, motion detection is performed using a low-resolution camera, and pixels where motion is detected are imaged by a high-resolution camera, thereby reducing power consumption. Citation list Patent Literature

[0003] Patent Document 1: Japanese Patent Application Laid-Open No. 2012-198075 Summary of the invention Technical problem to be solved by the invention

[0004] However, since the imaging of the high-resolution camera starts after the motion is detected, the imaging performed by the high-resolution camera is performed later than the imaging performed by the low-resolution camera. Therefore, there is a risk that fast-moving objects cannot be imaged. In addition, in Patent Document 1, when the high-resolution camera is driven, the low-resolution camera is also driven, thereby increasing the peak power and deteriorating the battery.

[0005] Therefore, the present disclosure provides a camera system having excellent responsiveness and capable of reducing power consumption. Solutions to technical problems

[0006] In order to solve the above problems, according to the present disclosure, a camera system is provided, which includes: a first imaging element; a second imaging element; and a control unit for controlling the first imaging element and the second imaging element, wherein the first imaging element comprises a plurality of first pixel circuits, which simultaneously sample charges corresponding to incident light photoelectrically converted in each pixel and read out the sampled charges in a non-destructive manner, The second imaging element includes a plurality of second pixel circuits that simultaneously samples charges corresponding to incident light photoelectrically converted in each pixel and reads out the sampled charges in a non-destructive manner, The first imaging element switches whether to output pixel signals from at least some of the plurality of first pixel circuits in the same frame period based on pixel signals output from some of the first pixel circuits or some of the second pixel circuits in each frame period, and The second imaging element switches whether to output pixel signals from at least some of the plurality of second pixel circuits within the same frame period based on pixel signals output from some of the first pixel circuits or some of the second pixel circuits in each frame period.

[0007] The some of the first pixel circuits or the some of the second pixel circuits may be arranged at one or more pixel positions obtained by thinning out the plurality of first pixel circuits or the plurality of second pixel circuits in at least one of the first direction and the second direction.

[0008] The control section may include a feature detection section that detects a feature including at least one of a motion of a target object and a presence of the target object based on pixel signals output from the some of the first pixel circuits or the some of the second pixel circuits in each of the frame periods, The first imaging element may output pixel signals from at least some of the plurality of first pixel circuits during the frame period in which the feature detection section detects the feature, and The second imaging element may output pixel signals from at least some of the plurality of first pixel circuits during the frame period in which the feature detection section detects a feature.

[0009] The first imaging element may output pixel signals from the plurality of first pixel circuits within the frame period in which the feature detection section detects a feature, and The second imaging element may output pixel signals from the plurality of second pixel circuits during the frame period in which the feature detection section detects a feature.

[0010] The feature detection section may detect a feature based on pixel signals output from the some of the first pixel circuits, and The first imaging element may output pixel signals twice from some of the first pixel circuits within the frame period in which the feature detection section detects a feature, and output pixel signals once from each of the first pixel circuits except for the some of the first pixel circuits.

[0011] The first imaging element may stop outputting pixel signals from the plurality of first pixel circuits during a frame period in which the feature detection unit does not detect a feature, and The second imaging element may stop outputting pixel signals from the plurality of second pixel circuits during the frame period in which the feature detection section detects no feature.

[0012] The control unit may include a ROI setting unit, which sets a first region of interest (ROI) pixel area including one or more first pixel circuits among the plurality of first pixel circuits and sets a second ROI pixel area including one or more second pixel circuits among the plurality of second pixel circuits in each of the frame periods according to a pixel position including the feature detected by the feature detection unit, The first imaging element can output the pixel signal of the first ROI pixel area within the same frame period, and The second imaging element may output pixel signals of the second ROI pixel region within the same frame period.

[0013] The feature detection unit may detect the motion based on pixel signals output from some of the first pixel circuits or some of the second pixel circuits in each of the frame periods. The first imaging element may output pixel signals from at least some of the plurality of first pixel circuits during a frame period in which the motion is detected, and The second imaging element may output pixel signals from at least some of the plurality of second pixel circuits during the frame period in which the motion is detected.

[0014] The feature detection unit may detect the motion based on pixel signals output from some of the first pixel circuits or some of the second pixel circuits in each of the frame periods. The control section may include a ROI setting section that sets a first ROI pixel region including pixel positions of one or more of the plurality of first pixel circuits and sets a second ROI pixel region including pixel positions of one or more of the plurality of second pixel circuits, The first imaging element can output a pixel signal of the first ROI pixel area within a frame period in which the motion is detected, and The second imaging element may output a pixel signal of the second ROI pixel region within the frame period in which the motion is detected.

[0015] The control unit may include: a target object determination section that determines whether a target object is imaged based on pixel signals output from the some of the first pixel circuits or the some of the second pixel circuits in each frame period; and The ROI setting unit sets a first ROI pixel area including pixel positions of one or more of the plurality of first pixel circuits, and sets a second ROI pixel area including pixel positions of one or more of the plurality of second pixel circuits, when determining that the target object is imaged. The first imaging element may output a pixel signal from the first pixel circuit in the first ROI pixel region within a frame period in which the target object is imaged, and The second imaging element may output a pixel signal from the second pixel circuit in the second ROI pixel region during the frame period in which the target object is imaged.

[0016] The control unit may control the pixel positions of the first ROI pixel area and the second ROI pixel area in each frame period according to the position of the target object to include the target object.

[0017] Each of the plurality of first pixel circuits may include: a first floating diffusion region that accumulates charges corresponding to incident light photoelectrically converted by the first photoelectric conversion element; a first capacitor that accumulates charges corresponding to a potential of the first floating diffusion region in a state in which charges of the first floating diffusion region are reset; and a second capacitor that accumulates electric charge corresponding to incident light photoelectrically converted by the first photoelectric conversion element, Each of the plurality of second pixel circuits may include: a second floating diffusion region that accumulates charges corresponding to incident light photoelectrically converted by the second photoelectric conversion element; a third capacitor that accumulates charge corresponding to the potential of the second floating diffusion region in a state in which the charge of the second floating diffusion region is reset; and a fourth capacitor that accumulates charge corresponding to incident light photoelectrically converted by the second photoelectric conversion element, and The charges accumulated in the first capacitor, the second capacitor, the third capacitor, and the fourth capacitor may be maintained within the same frame period even after being read out.

[0018] The first imaging element may switch whether to output pixel signals from at least some of the plurality of first pixel circuits in the same frame period based on pixel signals output from some of the first pixel circuits in each frame period. The second imaging element may switch whether to output pixel signals from at least some of the plurality of second pixel circuits in the same frame period based on the pixel signals output from some of the first pixel circuits in each frame period. Each of the plurality of first pixel circuits may include: a first floating diffusion region that accumulates charges corresponding to incident light photoelectrically converted by the first photoelectric conversion element; a first capacitor that accumulates charge corresponding to a potential of the first floating diffusion region in a state in which the charge of the first floating diffusion region is reset; and a second capacitor that accumulates electric charge corresponding to incident light photoelectrically converted by the first photoelectric conversion element, Each of the plurality of second pixel circuits may include: a second floating diffusion region that accumulates charges corresponding to incident light photoelectrically converted by the second photoelectric conversion element; and a third capacitor that accumulates charges in a state where the second floating diffusion region is reset or charges corresponding to incident light photoelectrically converted by the second photoelectric conversion element, The charges accumulated in the first capacitor and the second capacitor can be maintained in the same frame even after being read out, and The third capacitor may accumulate charges in a state where the second floating diffusion region is reset within one frame period, and then accumulate charges corresponding to incident light photoelectrically converted by the second photoelectric conversion element.

[0019] The control unit can offset the timing of sampling by the first imaging element of the charge corresponding to the incident light photoelectrically converted by the first photoelectric conversion elements of at least some of the multiple first pixel circuits within the same frame period and the timing of sampling by the second imaging element of the charge corresponding to the incident light photoelectrically converted by the second photoelectric conversion elements of at least some of the multiple second pixel circuits within the same frame period.

[0020] The control unit can offset the timing at which the first imaging element samples the reset level charge of at least some of the multiple first pixel circuits within the same frame period, the timing at which the first imaging element samples the charge corresponding to the incident light photoelectrically converted by the first photoelectric conversion element, the timing at which the second imaging element samples the reset level charge of at least some of the multiple second pixel circuits within the same frame period, and the timing at which the second imaging element samples the charge corresponding to the incident light photoelectrically converted by the second photoelectric conversion element.

[0021] The control unit can offset the timing of the first imaging element outputting pixel signals from at least some of the multiple first pixel circuits within the same frame period and the timing of the second imaging element outputting pixel signals from at least some of the multiple second pixel circuits within the same frame period.

[0022] The control unit may cause at least some of the plurality of first pixel circuits to output pixel signals within the same frame period, and then cause at least some of the plurality of second pixel circuits to output pixel signals within the same frame period.

[0023] The plurality of first pixel circuits may be arranged in each of the first direction and the second direction, The plurality of second pixel circuits may be arranged in each of the first direction and the second direction, and When outputting pixel signals in the same frame period, the first imaging element and the second imaging element may alternately output pixel signals of respective pixel groups arranged in the second direction.

[0024] The first imaging element and the second imaging element may have the same number of pixels, and The first imaging element and the second imaging element may perform exposure at the same exposure timing.

[0025] According to the present disclosure, there is provided a control method for a camera system, the camera system comprising: a first imaging element including a plurality of first pixel circuits, the plurality of first pixel circuits simultaneously sampling charges corresponding to incident light photoelectrically converted in each pixel and reading out the sampled charges in a non-destructive manner; a second imaging element including a plurality of second pixel circuits that simultaneously samples charges corresponding to photoelectrically converted incident light in each pixel and reads out the sampled charges in a non-destructive manner; and a control unit for controlling the first imaging element and the second imaging element, wherein the first imaging element switches whether to output pixel signals from at least some of the plurality of first pixel circuits in the same frame period based on pixel signals output from some of the first pixel circuits or some of the second pixel circuits in each frame period, and The second imaging element switches whether to output pixel signals from at least some of the plurality of second pixel circuits within the same frame period based on pixel signals output from some of the first pixel circuits or some of the second pixel circuits in each frame period. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a block diagram showing a schematic configuration of a camera system according to the first embodiment. Figure 2 : is a block diagram showing an example of the internal configuration of the motion detection unit. Figure 3 is a block diagram showing a schematic configuration of a first imaging element. Figure 4 This is a circuit diagram of each pixel in the pixel array unit. Figure 5 It is shown Figure 3 The block diagram of the internal structure of the load MOS circuit and the column signal processing circuit is shown. Figure 6 is a timing diagram of the global shutter operation of the first imaging element and the second imaging element. Figure 7 is a timing chart of the pixel signal readout operation of the first imaging element and the second imaging element. Figure 8 is a flowchart illustrating a processing operation of the camera system according to the first embodiment. Fig. 9 is a schematic operation timing chart of the camera system according to the first embodiment. Fig.10 is a block diagram showing a schematic configuration of a camera system according to the second embodiment. Fig.11 is a schematic operation timing chart of the camera system according to the second embodiment. Fig.12 is a block diagram showing a schematic configuration of a camera system according to a third embodiment. Fig.13 is a schematic operation timing chart of the camera system according to the third embodiment. Fig.14is a block diagram showing a schematic configuration of a camera system according to a modification example of the third embodiment. Fig.15 This is an alternative circuit diagram of the pixel circuit of the second imaging element that does not perform preview readout. Fig.16 is a block diagram showing a schematic configuration of a camera system according to a fifth embodiment. Fig.17 is an operation timing chart of the camera system according to the fifteenth embodiment. Fig.18 is an operation timing chart of a camera system according to a comparative example. Fig.19 It shows that according to Fig.16 A block diagram showing a schematic configuration of a camera system according to a modified example. Fig. 20 is an operation timing chart of the camera system according to the sixth embodiment. Fig.21 is an operation timing chart according to a modification of the sixth embodiment. Fig. 22 is a block diagram showing an example of a schematic configuration of a vehicle control system. Fig.23 It is a diagram for assisting in explaining an example of the installation positions of the vehicle exterior information detection unit and the imaging unit. DETAILED DESCRIPTION

[0027] Hereinafter, an embodiment of the camera system will be described with reference to the accompanying drawings. Although the main components of the camera system will be mainly described below, the camera system may have components and functions that are not shown or described. The following description does not exclude components and functions that are not shown or described.

[0028] (First embodiment) Figure 1 : is a block diagram showing a schematic configuration of a camera system 1 according to the first embodiment. Figure 1 The camera system 1 includes at least two imaging elements and can constitute a stereo camera. Note that Figure 1 The minimum configuration of the camera system 1 according to the first embodiment is shown, and may include Figure 1 For example, the camera system 1 according to the first embodiment may include a signal processing circuit, a distance measuring circuit, an image processing circuit, etc. Figure 1 The configuration and operation of the camera system 1 according to the first embodiment will be described.

[0029] Figure 1The camera system 1 in the embodiment includes a first imaging element 2, a second imaging element 3 and a control unit 4. The first imaging element 2 and the second imaging element 3 have the same number of pixels and perform imaging at the same exposure timing. For example, the first imaging element and the second imaging element 3 can image the same subject and generate parallax images. For example, the parallax images can be used to measure the distance of the subject.

[0030] Hereinafter, as an example, an example in which the first imaging element 2 is arranged on the left and the second imaging element 3 is arranged on the right with respect to the incident direction of incident light will be described, but the arrangement positions of the first imaging element 2 and the second imaging element 3 are arbitrary.

[0031] The first imaging element 2 and the second imaging element 3 can perform photoelectric conversion in the visible light band, or can perform photoelectric conversion in the infrared light band. When the first imaging element 2 and the second imaging element 3 perform photoelectric conversion in the visible light band, a color image can be captured, or a monochrome image can be captured.

[0032] The first imaging element 2 includes a plurality of first pixel circuits, which simultaneously sample the charges corresponding to the incident light photoelectrically converted by each pixel in all pixels, and read out the sampled charges in a non-destructive manner. The second imaging element 3 includes a plurality of second pixel circuits, which simultaneously sample the charges corresponding to the incident light photoelectrically converted by each pixel in all pixels, and read out the sampled charges in a non-destructive manner. Since the first pixel circuit included in the first imaging element 2 and the pixel circuit included in the second imaging element 3 have the same circuit configuration, in this specification, the first pixel circuit and the second pixel circuit may be simply collectively referred to as a pixel circuit.

[0033] The first imaging element 2 and the second imaging element 3 are both global shutter imaging elements that perform imaging in all pixels simultaneously. By adopting the global shutter method, distortion of the captured image can be suppressed, and, for example, the distance accuracy when measuring the distance using the first imaging element 2 and the second imaging element 3 can be improved.

[0034] In addition, each pixel circuit of the first imaging element 2 and the second imaging element 3 has a circuit configuration of a so-called voltage domain method as described later. By setting the circuit configuration of the voltage domain method, miniaturization is facilitated and high resolution can be achieved.

[0035] The control section 4 controls the first imaging element 2 and the second imaging element 3. More specifically, the control section 4 controls the exposure timing, the charge transfer timing, and the pixel signal readout timing of the first imaging element 2 and the second imaging element 3.

[0036] The control section 4 includes a feature detection section 5 and an imaging control section 6. The feature detection section 5 detects a feature including at least one of motion and presence of a target object based on pixel signals output from some first pixel circuits or some second pixel circuits in each frame period.

[0037] The first imaging element 2 outputs pixel signals from at least some of the plurality of first pixel circuits during the frame period in which the feature detection unit 5 detects the feature. The second imaging element 3 outputs pixel signals from at least some of the plurality of first pixel circuits during the frame period in which the feature detection unit 5 detects the feature.

[0038] For example, the first imaging element 2 can output pixel signals from the first pixel circuits of all pixels during the frame period in which the feature detection unit 5 detects a feature. Similarly, the second imaging element 3 can output pixel signals from the second pixel circuits of all pixels during the frame period in which the feature detection unit 5 detects a feature.

[0039] In the case where the feature detection unit 5 detects a feature based on a pixel signal output from some first pixel circuits, the first imaging element 2 outputs a pixel signal twice from some first pixel circuits within a frame period in which the feature detection unit 5 detects the feature, and outputs a pixel signal once from first pixel circuits other than the some first pixel circuits. As described above, some first pixel circuits in the first imaging element 2 perform a double readout of outputting a pixel signal twice within one frame period. The reason why such a double readout can be performed is that the first pixel circuit has a circuit configuration of a voltage domain method.

[0040] The first imaging element 2 stops outputting pixel signals from the plurality of first pixel circuits during a frame period in which the feature detection unit 5 does not detect a feature. Similarly, the second imaging element 3 stops outputting pixel signals from the plurality of second pixel circuits during a frame period in which the feature detection unit 5 does not detect a feature. As described above, since pixel signals are not output from the first imaging element 2 and the second imaging element 3 during a frame period in which no feature is detected, power consumption can be reduced.

[0041] A specific example of the feature detection unit 5 is the motion detection unit 5a. Hereinafter, an example in which the feature detection unit 5 is the motion detection unit 5a will be mainly described. The motion detection unit 5a detects the motion of the target object based on the pixel signals output from some first pixel circuits or some second pixel circuits. The target object is arbitrary, and when some motion is detected in at least some pixels in the image data generated based on the pixel signals output from some first pixel circuits or some second pixel circuits, the motion detection unit 5a outputs a signal indicating that the motion is detected (hereinafter, may be referred to as a motion detection signal).

[0042] The motion detection signal output from the motion detection section 5a is input to the imaging control section 6 of the control section 4. The imaging control section 6 controls the exposure timing, sampling timing, output timing of pixel signals, etc. of the first imaging element 2 and the second imaging element 3 based on the motion detection signal.

[0043] Figure 2 2 is a block diagram showing an example of the internal configuration of the motion detection section 5 a. The motion detection section 5 a includes a storage section 7 and a comparison section 8 . Figure 2 An example is shown in which motion detection is performed based on the image data output from the first imaging element 2. The image data in units of frames output from the first imaging element 2 is input to the storage section 7 and the comparison section 8. The storage section 7 stores the image data of the previous frame. The comparison section 8 compares the image data output from the first imaging element 2 with the image data of the previous frame read out from the storage section 7, and detects the difference between the two image data as motion.

[0044] Figure 3 2 is a block diagram showing a schematic configuration of the first imaging element 2. Since the second imaging element 3 has the same block configuration as the first imaging element 2, the configuration of the first imaging element 2 will be mainly described below.

[0045] like Figure 3 As shown, the first imaging element 2 includes a pixel array section 11, a vertical scanning circuit 12, a load metal oxide semiconductor (MOS) circuit 13, a column signal processing circuit 14, a timing control circuit 15 and a digital-to-analog converter (DAC) 16.

[0046] The pixel array section 11 includes a plurality of pixels arranged along a first direction X and a second direction Y. In this specification, the first direction X may be referred to as a row direction, and the second direction Y may be referred to as a column direction. In addition, in this specification, a pixel group of a row is referred to as a pixel row, and a pixel group of a column is referred to as a pixel column.

[0047] The timing control circuit 15 controls the operation timing of the vertical scanning circuit 12, the DAC 16, and the column signal processing circuit 14 in synchronization with the vertical synchronization signal XVS supplied from the control section 4. The DAC 16 generates a ramp signal Ramp used when analog-to-digital conversion (hereinafter, AD conversion) is performed in the column signal processing circuit 14. The ramp signal Ramp is, for example, a sawtooth signal.

[0048] The vertical scanning circuit 12 sequentially drives the plurality of row scanning lines L1. The row scanning lines L1 are provided for each pixel row arranged along the second direction Y, and when the vertical scanning circuit 12 drives any one of the row scanning lines L1, the pixel signals from each pixel in the corresponding pixel row are sent to the plurality of vertical signal lines L2. The plurality of pixel signals on the plurality of vertical signal lines L2 are input to the column signal processing circuit 14 via the load MOS circuit 13.

[0049] As described later, the load MOS circuit 13 includes a constant current source that supplies a constant current to each vertical signal line L2 .

[0050] The column signal processing circuit 14 performs AD conversion on the pixel signal on each vertical signal line L2 for each vertical signal line L2, and performs correlated double sampling (CDS) processing on the digital pixel signal after AD conversion. In this specification, the output of the column signal processing circuit 14 is referred to as image data. Image data is image data in units of frames captured in a part or the entire area of ​​the pixel array section 11.

[0051] The first imaging element 2 and the second imaging element 3 both have a plurality of pixels 20. Each pixel 20 in the first imaging element 2 includes a first photoelectric conversion element and a first pixel circuit. Each pixel 20 in the second imaging element 3 includes a second photoelectric conversion element and a second pixel circuit.

[0052] Figure 4 2 is a circuit diagram of each pixel 20 in the pixel array section 11. The pixel 20 includes a photoelectric conversion element 21 and a pixel circuit 22. Figure 4 The pixel 20 in is applicable to the pixel 20 in the first imaging element 2 and the pixel 20 in the second imaging element 3, and is a pixel 20 of a global shutter method and a voltage domain method.

[0053] Figure 4 The pixel circuit 22 of the pixel 20 in FIG. 1 includes a front-stage circuit 23 , a selection circuit 24 , and a rear-stage circuit 25 .

[0054] The previous stage circuit 23 is connected to the photoelectric conversion element 21 , and includes a transfer transistor 31 , a floating diffusion region (hereinafter, referred to as a floating diffusion or FD) 32 , a first reset transistor 33 , a first amplification transistor 34 , and a current source 35 .

[0055] The photoelectric conversion element 21 accumulates the charge corresponding to the incident light by photoelectric conversion. The transfer transistor 31 transfers the charge from the photoelectric conversion element 21 to the FD 32 according to the transfer signal trg from the vertical scanning circuit 12. According to the reset signal from the vertical scanning circuit 12, the first reset transistor 33 extracts the charge from the FD 32 to initialize the voltage of the FD 32. The FD 32 accumulates the reset level charge or the charge corresponding to the incident light photoelectrically converted by the photoelectric conversion element 21, and generates a voltage corresponding to the accumulated charge. The power supply voltage VDD is applied to the drain of the first amplifier transistor 34, and the current source 35 and the output node of the previous stage circuit 23 are connected to the source of the first amplifier transistor 34.

[0056] One end of the first capacitor C1 and one end of the second capacitor C2 are connected to the output node of the preceding circuit 23. The first capacitor C1 is used to hold (sample) the reset level charge of the FD 32, and the second capacitor C2 is used to hold (sample) the charge corresponding to the incident light photoelectrically converted by the photoelectric conversion element 21. Hereinafter, the charge corresponding to the incident light photoelectrically converted by the photoelectric conversion element 21 may be referred to as a signal charge.

[0057] The selection circuit 24 includes a first selection transistor 41 and a second selection transistor 42. The subsequent stage circuit 25 includes a second amplification transistor 43 and a third selection transistor 44.

[0058] The first selection transistor 41 switches whether to connect the other end of the first capacitor C1 to the gate of the second amplifier transistor 43 in the subsequent stage circuit 25 according to the first selection signal φr from the vertical scanning circuit 12. The second selection transistor 42 switches whether to connect the other end of the second capacitor C2 to the gate of the second amplifier transistor 43 in the subsequent stage circuit 25 according to the second selection signal φs from the vertical scanning circuit 12.

[0059] The second reset transistor 45 is connected to a signal path connecting the other end of the first capacitor C1, the other end of the second capacitor C2 and the gate of the second amplifying transistor 43. The second reset transistor 45 switches whether to initialize the above signal path to the power supply voltage level according to the second reset signal rstb from the vertical scanning circuit 12.

[0060] At the start of exposure, the vertical scanning circuit 12 supplies a high-level first reset signal rst and a high-level transfer signal to all pixels 20 in the pixel array section 11. As a result, the photoelectric conversion elements 21 of all pixels 20 are initialized. Then, immediately before the end of exposure, the vertical scanning circuit 12 turns on the first reset transistor 33 to extract the charge of the FD 32, and sets the potential of the FD 32 to the reset level. In addition, by turning on the second reset transistor 45 and the first selection transistor 41, the charge corresponding to the reset level of the FD 32 is held in the first capacitor C1.

[0061] At the end of exposure, the vertical scanning circuit 12 turns on both the second reset transistor 45 and the second selection transistor 42 for all pixels 20 in the pixel array section 11, and turns on the transfer transistor 31 for a predetermined time. As a result, the signal charge corresponding to the exposure amount is transferred to the FD 32 through the transfer transistor 31 and accumulated therein. The charge corresponding to the incident light amount is accumulated in the FD 32, and the FD 32 has a potential corresponding to the accumulated capacitance. In addition, the second capacitor C2 holds the signal charge corresponding to the incident light amount.

[0062] In this manner, the first imaging element 2 and the second imaging element 3 simultaneously start exposure of all pixels 20 in each pixel array section 11 by a global shutter method, and simultaneously end exposure to perform imaging. In addition, each of the first imaging element 2 and the second imaging element 3 includes a pixel circuit (a first pixel circuit and a second pixel circuit) 22 having a circuit configuration of a voltage domain method in which a charge corresponding to the reset level of the FD 32 is held in the first capacitor C1 of each pixel 20, and a signal charge corresponding to the incident light is held in the second capacitor C2.

[0063] After the exposure is completed, the vertical scanning circuit 12 sequentially selects each pixel row in the pixel array section 11. As a result, a pixel signal of a reset level corresponding to the charge held by the first capacitor C1 of each pixel 20 included in the selected pixel row and a pixel signal of a signal charge level corresponding to the charge held by the second capacitor C2 are sequentially output to the corresponding vertical signal line L2.

[0064] Figure 5 It shows Figure 3 1 is a block diagram of the internal structure of the load MOS circuit 13 and the column signal processing circuit 14 shown in FIG. The load MOS circuit 13 includes a plurality of current sources 51 respectively connected to a plurality of vertical signal lines L2 arranged along the first direction X and extending along the second direction Y. Each vertical signal line L2 is provided for each pixel column extending along the second direction Y. Each current source 51 supplies a constant current to the corresponding vertical signal line L2.

[0065] The column signal processing circuit 14 includes a plurality of analog-to-digital converters (hereinafter, ADC) 52 and a digital signal processing section 53. The plurality of ADCs 52 are provided in association with the plurality of vertical signal lines L2. The ADCs 52 are connected to the plurality of vertical signal lines L2. Figure 3 The control signal L3 of the timing control circuit 15 shown is synchronized with the control signal L3 from Figure 3 The ramp signal Ramp of the DAC 16 shown converts the pixel signal on the corresponding vertical signal line L2 into a digital pixel signal. The digital pixel signal output from the ADC 52 is supplied to the digital signal processing section 53. The ADC 52 is, for example, a single slope ADC 52 having a comparator and a counter.

[0066] The digital signal processing section 53 performs signal processing such as CDS processing between the reset level pixel signal and the signal charge pixel signal for each vertical signal line L2 and outputs image data. The digital signal processing section 53 outputs image data in units of frames.

[0067] Figure 6 : is a timing chart of the global shutter operation of the first imaging element 2 and the second imaging element 3. The vertical scanning circuit 12 sets both the first reset signal rst and the transfer signal trg to a high level in the period from the time t0 immediately before the start of exposure to the time t1 when the exposure starts. As a result, the first reset transistor 33 and the transfer transistor 31 are turned on, all the pixels 20 in the pixel array section 11 are reset, the charge of the FD 32 is extracted, and the potential of the FD 32 becomes a reset level. Figure 6 An example is shown in which there are N pixel rows in the pixel array section 11. Figure 6 In the embodiment, [1:N] is written at the end of the signal input to each pixel circuit of N pixel rows, but this description is omitted in this specification.

[0068] At time t1, both the first reset transistor 33 and the transfer transistor 31 are turned off, and exposure is started. Then, at time t2 immediately before the end of the exposure period, the vertical scanning circuit 12 sets both the second reset signal rstb and the first selection signal φr to a high level for all pixels 20 in the pixel array section 11, and sets the first reset signal rst to a high level. As a result, the potential of the FD 32 of all pixels 20 is initialized to a reset level, and a charge corresponding to the reset level is held (sampled) in the first capacitor C1.

[0069] Then, at time t3, the first selection signal φr changes from the high level to the low level, and the first selection transistor 41 is turned off. As a result, the sampling period of the first capacitor C1 ends.

[0070] Then, at time t4, the transfer signal trg temporarily changes to a high level, and the transfer transistor 31 temporarily turns on. As a result, the charge corresponding to the incident light photoelectrically converted by the photoelectric conversion element 21 is transferred to the FD 32 through the transfer transistor 31. As the amount of incident light increases, the signal charge transferred to the FD 32 increases, and the voltage of the FD 32 decreases.

[0071] Furthermore, at time t4, the second selection signal φs becomes a high level, and the second selection transistor 42 is turned on. As a result, the second capacitor C2 holds (samples) a charge corresponding to the signal charge of the FD 32. Then, at time t5, the second selection signal φs becomes a low level, and the sampling period of the second capacitor C2 ends.

[0072] Figure 7 2 is a timing chart of the pixel signal readout operation of the first imaging element 2 and the second imaging element 3 . Figure 7 1 shows the timing in the case of performing a readout operation of the n-th pixel row among N pixel rows. Figure 7 In the description, [n] is written at the end of the signal input to each pixel circuit in the n-th pixel row, but this description is omitted in this specification.

[0073] At time t10, the vertical scanning circuit 12 sets the first reset signal rst of each pixel 20 in the n-th pixel row to a high level, sets the third selection signal sel input to the gate of the third selection transistor 44 to a high level, sets the second reset signal rstb to a low level, and sets the first selection signal φr to a high level. As a result, the potential of the FD 32 is initialized to a reset level, and a voltage corresponding to the retained charge of the first capacitor C1 is supplied to the gate of the second amplifying transistor 43. At this time, since the third selection transistor 44 is turned on, a pixel signal having a voltage level corresponding to the gate voltage of the second amplifying transistor 43 in each pixel circuit of the n-th pixel row is supplied to the vertical signal line L2.

[0074] At time t13, the first selection signal φr becomes low level and turns off the first selection transistor 41. Then, at time t14, the second reset signal rstb temporarily becomes high level and turns on the second reset transistor 45. Therefore, the other end sides of the first capacitor C1 and the second capacitor C2 increase to the power supply voltage level.

[0075] Then, at time t15, the second selection signal φs becomes a high level, and turns on the second selection transistor 42. As a result, a voltage corresponding to the charge held by the second capacitor C2 is supplied to the gate of the second amplification transistor 43. At this time, since the third selection transistor 44 is turned on, a pixel signal of a voltage level corresponding to the gate voltage of the second amplification transistor 43 of each pixel circuit in the n-th pixel row is supplied to the vertical signal line L2.

[0076] like Figure 7 As shown by the dotted line in FIG. 1 , from time t16 to time t17, the voltage level of the ramp signal Ramp input to the DAC 16 gradually increases. The ADC 52 compares the ramp signal Ramp with the pixel signal on the vertical signal line L2, and counts the period until the comparison result is inverted by a counter (not shown). A digital pixel signal is generated based on the count value of the counter.

[0077] Figure 8 is a flowchart showing the processing operation of the camera system 1 according to the first embodiment. When a shutter button (not shown) is pressed, the camera system 1 starts Figure 8 A flowchart of the processing operation.

[0078] First, the first imaging element 2 and the second imaging element 3 start exposure at the same time, and for each pixel 20, Figure 4 The first capacitor C1 shown holds a reset level charge, and the second capacitor C2 holds a signal charge (step S1). As described above, in step S1, for all pixels 20, the first imaging element 2 and the second imaging element 3 are Figure 6 The operation is performed simultaneously on each pixel 20 during the period from time t0 to time t5.

[0079] Next, pixel signals of at least some of the pixels 20 are read out from one of the first imaging element 2 and the second imaging element 3 (step S2). In this specification, the readout of the pixel signals in step S2 is referred to as preview readout. Hereinafter, an example of performing preview readout from the first imaging element 2 will be described, but preview readout can also be performed from the second imaging element 3.

[0080] In the preview readout, pixel signals of all pixels 20 of the pixel array section 11 of the first imaging element 2 are not read out, but image signals of some pixels 20 obtained by thinning out from all pixels 20 are read out. The thinning out method of the pixels 20 is arbitrary. For example, pixel signals of some pixels 20 selected for every plurality of pixels in the first direction X and the second direction Y may be read out from all pixels 20.

[0081] The reason why the pixel signals of some pixels 20 obtained by thinning out from all the pixels 20 are read out in step S2 is to reduce power consumption and speed up the motion detection process. The pixel signals read out from the pixel array section 11 are converted into digital pixel signals by the column signal processing circuit 14. The column signal processing circuit 14 performs CDS processing using the digital pixel signals at the reset level and the digital pixel signals corresponding to the signal charge to generate image data. The image data output from the column signal processing circuit 14 is input to Figure 1 The motion detection unit 5a of the control unit 4 in.

[0082] The motion detection unit 5a detects the moving pixels 20 based on the image data based on the pixel signals of some pixels 20 read out in step S2 (step S3). Next, it is determined whether there are moving pixels 20 based on the detection result of step S3 (step S4). In the case where there are no moving pixels 20, step S1 and the subsequent processing are repeated. In the case where there are moving pixels 20, the pixel signals of all pixels 20 of the pixel array unit 11 in both the first imaging element 2 and the second imaging element 3 are output (step S5). In this specification, the processing of step S5 is referred to as main readout. In the main readout according to the present embodiment, the first imaging element 2 and the second imaging element 3 both output the pixel signals of all pixels 20.

[0083] Since the charge is held in the first capacitor C1 and the second capacitor C2 when the process of step S1 is first performed, when the determination of step S4 is yes and the main readout is performed, only the pixel signal corresponding to the held charge of the first capacitor C1 and the second capacitor C2 needs to be output, and the pixel signal can be output quickly. When the process of step S5 is finished, the process of step S1 and subsequent steps is repeated.

[0084] Note that the main readout in step S5 is performed only when there is motion in at least a portion of the image data. In other words, when no motion is detected in the image data generated by the preview readout, the main readout is not performed, so that the frequency of the main readout can be reduced and power consumption can be reduced.

[0085] Since the pixel circuit 22 according to the present embodiment has a circuit configuration of a voltage domain method, even if the retained charges in the first capacitor C1 and the second capacitor C2 are read out for preview readout, the charges are still retained in the first capacitor C1 and the second capacitor C2. Therefore, in the case where it is determined to be YES in step S4, the retained charges can be read out from the first capacitor C1 and the second capacitor C2 a plurality of times within the same frame period.

[0086] Fig. 9 is a schematic operation timing chart of the camera system 1 according to the first embodiment. Fig. 9The timing at which the signal charge is held in the second capacitor C2 and read out is shown, and the timing at which the reset level of the FD 32 is held in the first capacitor C1 and read out is omitted. Fig. 9 The upper part shows the operation timing of the first imaging element 2 which performs preview readout and main readout, and the lower part shows the operation timing of the second imaging element 3 which performs main readout without performing preview readout.

[0087] At time t21, the first imaging element 2 and the second imaging element 3 start exposure simultaneously. At time t22, the exposure ends, and the first imaging element 2 and the second imaging element 3 simultaneously hold (sample) signal charges in the second capacitor C2. During the period from time t22 to time t23, the first imaging element 2 performs preview readout.

[0088] As described above, the image data based on the preview-read pixel signal is sent to the motion detection section 5a of the control section 4. The motion detection section 5a performs motion detection in the period from time t23 to time t24.

[0089] In the case where no motion is detected, neither the first imaging element 2 nor the second imaging element 3 performs main readout within the frame. Fig. 9 In frame f1, since no motion is detected in the preview readout, the main readout is not performed.

[0090] At time t24, the exposure of the next frame f2 is started. The exposure ends at time t25, and the first imaging element 2 and the second imaging element 3 simultaneously hold (sample) the signal charge in the second capacitor C2. During the period from time t25 to time t26, the first imaging element 2 performs preview readout. During the period from time t26 to time t27, the motion detection section 5a detects the motion of the image data based on the pixel signal read out in preview. As a result, when motion is detected, the first imaging element 2 and the second imaging element 3 both perform the main readout of the pixel signal for all pixels.

[0091] As described above, according to the first embodiment, in a camera system including the first imaging element 2 and the second imaging element 3 having the pixel circuit (the first pixel circuit and the second pixel circuit) 22, the pixel circuit 22 having a circuit configuration of the global shutter method and the voltage domain method, motion detection is performed on image data generated by performing preview readout in one of the first imaging element 2 and the second imaging element 3, and main readout is performed to read out pixel signals of all pixels 20 in both the first imaging element 2 and the second imaging element 3 only when motion is detected. Since the main readout is not performed unless motion is detected in the preview readout, power consumption can be reduced.

[0092] In addition, since the pixel circuit 22 according to the present embodiment has a circuit configuration of a voltage domain method, even if the charge held in the first capacitor C1 and the second capacitor C2 is read out for preview readout, the charge remains as it is. Therefore, the charge can be read out from the first capacitor C1 and the second capacitor C2 again for main readout in the same frame, and the main readout can be performed quickly after the preview readout.

[0093] (Second Embodiment) In the first embodiment, pixel signals of all pixels 20 are read out at the time of main readout, but image signals of some pixels 20 may be read out.

[0094] Fig.10 is a block diagram showing a schematic configuration of a camera system 1 according to the second embodiment. Fig.10 Camera system 1 with Figure 1 The difference lies in the internal structure of the control unit 4. Fig.10 The control section 4 of the camera system 1 in FIG. 5 includes an imaging control section 6 and a region of interest (ROI) setting section 5 b . The ROI setting section 5 b is a form of the feature detection section 5 .

[0095] The ROI setting section 5b sets a portion of a pixel region of interest in the pixel array section 11. It is arbitrary at which pixel position the ROI pixel region is set and how large the ROI pixel region is. Fig.10 The ROI setting section 5b in the image processing apparatus sets the ROI pixel region based on the image data at the time of preview reading. The ROI pixel region may be, for example, a pixel region including eyes, face, outline, etc. of a person.

[0096] The ROI setting section 5b transmits setting information of the ROI pixel region to the imaging control section 6. The imaging control section 6 transmits a control signal to the vertical scanning circuit 12 and the column signal processing circuit 14 based on the setting information of the ROI pixel region.

[0097] Fig.11 1 is a schematic operation timing chart of the camera system 1 according to the second embodiment. The first imaging element 2 and the second imaging element 3 start exposure at time t31 and end exposure at time t32. The first imaging element 2 performs preview readout in the period from time t32 to time t33.

[0098] The ROI setting section 5b sets a ROI pixel region including pixels of interest based on image data generated by preview readout in the period from time t33 to time t34. The pixels of interest are pixels 20 including, for example, eyes, faces, and the like of a person.

[0099] The ROI setting section 5b sends the setting information of the ROI pixel area to the imaging control section 6. The setting information of the ROI pixel area is, for example, information including the pixel position of the pixel of interest, the size and pixel position of the ROI pixel area set around the pixel of interest, etc. The imaging control section 6 sends a control signal to the vertical scanning circuit 12 and the column signal processing circuit 14 based on the setting information of the ROI pixel area. Based on the control signal, the vertical scanning circuit 12 drives a row scanning signal for driving a pixel row for which main readout is performed. Based on the control signal, the column signal processing circuit 14 acquires a pixel signal on the vertical signal line L2 corresponding to the pixel column for which main readout is performed, and performs AD conversion, CDS processing, etc.

[0100] The first imaging element 2 and the second imaging element 3 perform main readout in the period from time t34 to time t35 to output the pixel signal in the ROI pixel area. Since the first capacitor C1 and the second capacitor C2 of each pixel circuit 22 have already retained the charge, the pixel signal based on the retained charge of the first capacitor C1 and the second capacitor C2 can be quickly output.

[0101] During the period from time t32 to time t36, preview readout and main readout of frame f1 are performed. Similarly, during the period from time t37 to time t40, preview readout and main readout of the next frame f2 are performed. Fig.11 The example of shows an example where the position of the ROI pixel region is different between frame f1 and frame f2. Since the ROI setting unit 5b sets the ROI pixel region for each frame, when the target object moves, the ROI pixel region is set at a different position for each frame.

[0102] Fig.11 The operation timing diagram of shows an example of reading out the pixel signal of the ROI pixel area for each frame, but in some cases, the pixel of interest may not appear in the image data read out in the preview. For example, this is a case where the pixel of interest is a person's face, but there is no person in the image data, etc. In this case, the ROI setting section 5b cannot set the ROI pixel area. Therefore, depending on the frame, the first imaging element 2 and the second imaging element 3 may not perform the main readout.

[0103] As described above, in the second embodiment, the ROI pixel area is set based on the image data read out in preview, and the pixel signal of the ROI pixel area is output from the first imaging element 2 and the second imaging element 3 in the frame period of the preview read out. As a result, only the ROI pixel area can be output quickly and with low power consumption.

[0104] (Third Embodiment) In the third embodiment, a ROI pixel area is set in a pixel area where motion is detected.

[0105] Fig.12 is a block diagram showing a schematic configuration of a camera system 1 according to the third embodiment. Fig.12 Camera system 1 with Figure 1 and Fig.10 The camera system of FIG. 1 is different from that of FIG. 1 in that the internal structure of the control unit 4 is different. Fig.10 The control section 4 of the camera system 1 includes an imaging control section 6 , a motion detection section 5 a and a ROI setting section 5 b . The motion detection section 5 a and the ROI setting section 5 b are one form of the feature detection section 5 .

[0106] The motion detection section 5a detects motion based on the image data generated by the preview readout of the first imaging element 2. The ROI setting section 5b sets the ROI pixel area to include the pixels 20 whose motion is detected by the motion detection section 5a. The ROI setting section 5b sends the setting information of the ROI pixel area to the imaging control section 6. The imaging control section 6 sends a control signal to the vertical scanning circuit 12 and the column signal processing circuit 14 based on the setting information of the ROI pixel area.

[0107] Fig.13 is a schematic operation timing diagram of the camera system 1 according to the third embodiment. The first imaging element 2 and the second imaging element 3 start exposure at time t51, and end exposure at time t52, a reset level charge is held in the first capacitor C1, and a signal charge is held in the second capacitor C2. The first imaging element 2 performs preview readout in the period from time t52 to time t53.

[0108] After time t53, the motion detection section 5a detects motion based on the image data generated by the preview readout. The motion detection section 5a transmits the information of the pixel 20 in which the motion is detected to the ROI setting section 5b. The ROI setting section 5b sets the ROI pixel area including the pixel 20 in which the motion is detected. The ROI setting section 5b sends the setting information of the ROI pixel area to the imaging control section 6. The imaging control section 6 sends a control signal to the vertical scanning circuit 12 and the column signal processing circuit 14 based on the setting information of the ROI pixel area.

[0109] Fig.13 An example is shown in which motion is not detected in the period of the frame f1 (time t52 to t55 ) and motion is detected in the period of the next frame f2 (time t55 to t60 ).

[0110] When motion is detected in the image data generated by preview readout in the period from time t55 to time t56 in the period of frame f2, the motion detection section 5a sends information about the motion pixels 20 to the ROI setting section 5b. The ROI setting section 5b sets a ROI pixel area including the motion pixels 20.

[0111] The first imaging element 2 and the second imaging element 3 output pixel signals of the ROI pixel region during the period from time t57 to time t58 .

[0112] As described above, in the third embodiment, motion is detected based on the image data generated in the preview readout, and the ROI pixel area including the motion pixel 20 is set. The first imaging element 2 and the second imaging element 3 output the pixel signal of the ROI pixel area within the frame period in which the preview readout has been performed. As a result, the pixel signal of the ROI pixel area including the motion pixel 20 can be quickly output while suppressing power consumption.

[0113] Fig.12 2 shows an example of setting a ROI pixel region including the pixels 20 whose motion is detected by the motion detection unit 5a. Fig.14 As shown, the ROI setting section 5b can set the ROI pixel area independently of the motion detection section 5a.

[0114] Fig.14 is a block diagram showing a schematic configuration of a camera system 1 according to a modification of the third embodiment. Fig.12 The control unit 4 is similar to that of Fig.14 The control section 4 in the camera system 1 includes an imaging control section 6, a motion detection section 5a, and a ROI setting section 5b.

[0115] The motion detection section 5a detects motion based on the image data generated by the preview readout. Information indicating whether the motion detection section 5a detects motion is sent to the imaging control section 6. The ROI setting section 5b sets the ROI pixel area independently of the pixel 20 where the motion detection section 5a detects motion. For example, the ROI setting section 5b sets the ROI pixel area around the pixel 20 including the face of a person or the like included in the image data generated by the preview readout. The setting information of the ROI pixel area of ​​the ROI setting section 5b is transmitted to the imaging control section 6.

[0116] When motion is detected based on the image data generated by preview readout, the first imaging element 2 and the second imaging element 3 output pixel signals in the ROI pixel area in the same frame period. As a result, the pixel signals in the ROI pixel area can be output only when there is motion.

[0117] (Fourth Embodiment) For one of the first imaging element 2 and the second imaging element 3 that performs preview readout (for example, the first imaging element 2), the charges of the first capacitor C1 and the second capacitor C2 are read out twice in the same frame period. Therefore, it is necessary to adopt a pixel circuit 22 having a circuit configuration of a voltage domain method capable of reading out charges in a non-destructive manner. At the same time, the other of the first imaging element 2 and the second imaging element 3 that does not perform preview readout (for example, the second imaging element 3) does not necessarily need to adopt a pixel circuit 22 having a circuit configuration of a voltage domain method.

[0118] Fig.15 2 is an alternative circuit diagram of the pixel circuit (second pixel circuit) 22 of the second imaging element 3 that does not perform preview readout. Fig.15 The pixel circuit 22 in FIG. 1 has a charge domain type circuit configuration. Fig.15 The pixel circuit 22 in FIG. 2 includes an overflow transistor 61 , a first transfer transistor 62 , a second transfer transistor 63 , a reset transistor 64 , an amplification transistor 65 , a selection transistor 66 , a capacitor C3 , and a floating diffusion region 67 . The overflow transistor 61 is connected between the cathode of the photoelectric conversion element 21 and the power supply voltage node VDD. When the overflow signal is at a high level, the overflow transistor 61 is turned on, and the accumulated charge in the photoelectric conversion element 21 is reset.

[0119] The first transfer transistor 62 and the second transfer transistor 63 are cascode connected between the cathode of the photoelectric conversion element 21 and the floating diffusion region 67. When the first transfer signal is at a high level, the first transfer transistor 62 is turned on. When the second transfer signal is at a high level, the second transfer transistor 63 is turned on. The capacitor C3 is connected between the connection node of the first transfer transistor 62 and the second transfer transistor 63 and the ground node.

[0120] The reset transistor 64 is connected between the power supply voltage node VDD and the floating diffusion region 67. When the reset signal RST is at a high level, the reset transistor 64 is turned on and the accumulated charge in the floating diffusion region 67 is reset. The amplifier transistor 65 and the selection transistor 66 are connected in common source and common gate between the power supply voltage node VDD and the vertical signal line L2. A voltage corresponding to the accumulated charge in the floating diffusion region 67 is applied to the gate of the amplifier transistor 65. When the selection signal is at a high level, the selection transistor 66 is turned on.

[0121] The capacitor C3 holds a reset level charge obtained by resetting the accumulated charge of the photoelectric conversion element 21 within a frame period, and then holds a charge (signal charge) corresponding to incident light photoelectrically converted by the photoelectric conversion element 21 .

[0122] As mentioned above, due to Fig.15 The pixel circuit 22 in FIG. 1 holds the reset level charge and the signal charge in the same capacitor C3 in a time-shifted manner, so when CDS processing is performed in the column signal processing circuit 14, kTC noise can be completely eliminated. However, after the reset level charge held in the capacitor C3 is read out, the accumulated charge of the capacitor C3 is reset once, and then the signal charge is held again. Therefore, this is destructive readout, and double readout cannot be performed in the same frame period.

[0123] therefore, Fig.15 The pixel circuit 22 in the first imaging element 2 can be used only for the second imaging element 3 where preview readout is not performed. By configuring the pixel circuit 22 in the first imaging element 2 as a circuit configuration having a voltage domain method, and configuring the pixel circuit 22 in the second imaging element 3 as a circuit configuration having a charge domain method, preview readout and main readout can be performed in the first imaging element 2, and main readout with kTC noise further suppressed can be performed in the second imaging element 3.

[0124] (Fifth embodiment) In the first to fourth embodiments, when features including motion or a target object are detected based on image data generated by preview readout, main readout of pixel signals output from the first imaging element 2 and the second imaging element 3 is performed.

[0125] In the first to fourth embodiments, the first imaging element 2 and the second imaging element 3 start exposure at the same time, end exposure at the same time, and simultaneously hold the reset level charge in the first capacitor C1 of all pixels, and then simultaneously hold the signal charge in the second capacitor C2 of all pixels. In particular, when the charge is held in the first capacitor C1 and the second capacitor C2, a large current temporarily flows in each pixel circuit 22, thereby increasing the peak power. In addition, when the pixel signal corresponding to the held charge in the first capacitor C1 and the second capacitor C2 is output to the vertical signal line L2, the peak power also increases. Therefore, in the fifth embodiment, the peak power is suppressed.

[0126] Fig.16 is a block diagram showing a schematic configuration of a camera system 1 according to the fifth embodiment. Fig.16 The camera system 1 in the embodiment includes a first imaging element 2, a second imaging element 3, and a control unit 4. The control unit 4 performs control to suppress the peak power of the first imaging element 2 and the second imaging element 3. Each pixel 20 in the first imaging element 2 and the second imaging element 3 includes a pixel having a Figure 4 The pixel circuit 22 is constructed with a similar circuit.

[0127] Fig.171 is an operation timing chart of the camera system 1 according to the fifteenth embodiment. The time t61 is immediately before the exposure starts, and the first reset transistor 33 and the transfer transistor 31 are both turned on. As a result, the accumulated charge in the photoelectric conversion element 21 is reset. When the first reset transistor 33 and the transfer transistor 31 are turned off at the time t62, the exposure starts.

[0128] When the first selection transistor 41 in each pixel circuit 22 of the first imaging element 2 and the second imaging element 3 is turned on at time t63, during the period from time t64 to time t65, the reset level charge is held in the first capacitor C1 in each pixel circuit 22 of the first imaging element 2. Then, during the period from time t65 to time t66, the reset level charge is held in the first capacitor C1 in each pixel circuit 22 of the second imaging element 3.

[0129] When the second selection transistor 42 in each pixel circuit 22 of the first imaging element 2 and the second imaging element 3 is turned on at time t66, the signal charge is held in the second capacitor C2 in each pixel circuit 22 of the first imaging element 2 during the period from time t67 to time t68. Then, during the period from time t68 to time t69, the signal charge is held in the second capacitor C2 in each pixel circuit 22 of the second imaging element 3.

[0130] After time t69 , the first imaging element 2 and the second imaging element 3 sequentially transfer pixel signals corresponding to the charges held in the first capacitor C1 and the second capacitor C2 in each pixel circuit 22 to the vertical signal line L2 .

[0131] As mentioned above, Fig.17 As shown in the operation timing diagram of FIG. 1 , the timings at which the first imaging element 2 and the second imaging element 3 hold the reset level charge in the first capacitor C1 are offset from each other. In addition, the timings at which the signal charges of the first imaging element 2 and the second imaging element 3 are held in the second capacitor C2 are offset from each other. As a result, when the charges are held in the first capacitor C1 and the second capacitor C2, the peak power of the first imaging element 2 and the second imaging element 3 can be suppressed.

[0132] Fig.18 is an operation timing chart of the camera system 1 according to the comparative example. Fig.18 In the camera system 1 , the first imaging element 2 and the second imaging element 3 simultaneously hold the reset level charge in the first capacitor C1 (time t71 to t72 ), and then hold the signal charge in the second capacitor C2 (time t72 to t73 ).

[0133] exist Fig.18In this case, since the reset level charge is simultaneously maintained in the first capacitor C1 of all pixels of the first imaging element 2 and the second imaging element 3, and then the signal charge is simultaneously maintained in the second capacitor C2 of all pixels, the peak power of the period from time t71 to time t72 and the period from time t72 to time t73 increases.

[0134] On the other hand, in the camera system 1 according to the fifth embodiment, as Fig.17 As shown, since the first imaging element 2 and the second imaging element 3 hold the charges of the first capacitor C1 and the second capacitor C2 of all pixels in a time-shifted manner, the peak power can be suppressed to about half of that of the comparative example.

[0135] Fig.16 The camera system 1 can be combined with the camera systems according to the above-described first to fourth embodiments. Fig.19 It shows that according to Fig.16 A block diagram showing a schematic configuration of a camera system 1 according to a modified example. Fig.19 The camera system 1 is provided by Fig.16 The block structure is obtained by adding the motion detection unit 5a. Fig.19 In the embodiment, the motion detection unit 5a is provided separately from the control unit 4, but the motion detection unit 5a may be provided separately from the control unit 4. Figure 1 The control unit 4 is similarly arranged inside.

[0136] The first imaging element 2 and the second imaging element 3 start exposure at the same time and end exposure at the same time. The first imaging element 2 and the second imaging element 3 hold reset level charges in the first capacitor C1 of each pixel circuit 22 in a manner shifted in time from each other, and then hold signal charges in the second capacitor C2 of each pixel circuit 22 in a manner shifted in time from each other.

[0137] One of the first imaging element 2 and the second imaging element 3 (for example, the first imaging element 2) performs preview readout. The motion detection section 5a detects motion based on the image data generated by the preview readout, and sends the detection result to the control section 4. The control section 4 instructs the first imaging element 2 and the second imaging element 3 to perform main readout only when the motion detection section 5a detects motion. The first imaging element 2 and the second imaging element 3 output pixel signals corresponding to the charges held in the first capacitor C1 and the second capacitor C2 to the vertical signal line L2 only when the motion detection section 5a detects motion.

[0138] As described above, in the fifth embodiment, the timing at which the first capacitor C1 in each pixel circuit 22 holds the charge corresponding to the reset level is shifted between the first imaging element 2 and the second imaging element 3, and the timing at which the second capacitor C2 in each pixel circuit 22 holds the signal charge is shifted between the first imaging element 2 and the second imaging element 3. As a result, the peak power can be suppressed.

[0139] (Sixth embodiment) In the fifth embodiment, the measure for suppressing the peak power when the charge is held (sampled) in the first capacitor C1 and the second capacitor C2 has been described. However, when the pixel signal corresponding to the held charge in the first capacitor C1 and the second capacitor C2 is output to the vertical signal line L2, the peak power is also generated. In the sixth embodiment, the peak power when the pixel signal is output to the vertical signal line L2 is suppressed.

[0140] The camera system 1 according to the sixth embodiment has Fig.16 or Fig.19 Similar block composition.

[0141] Fig. 20 is an operation timing chart of the camera system 1 according to the sixth embodiment. Fig. 20 The operation timing diagram shows the timing of the signal charge sampling and readout operation, and omits the timing of the reset level charge sampling and readout operation.

[0142] The first imaging element 2 and the second imaging element 3 start exposure at time t81 and end exposure at time t82 , and hold (sample) signal charges in the second capacitor C2 for all pixels.

[0143] In the period from time t82 to time t83, the first imaging element 2 outputs, for each pixel row, a pixel signal corresponding to the charge held by the second capacitor C2 of each pixel 20. Then, in the period from time t84 to time t85, the second imaging element 3 outputs, for each pixel row, a pixel signal corresponding to the charge held by the second capacitor C2 of each pixel 20. The above operation is repeated for each frame period.

[0144] like Fig. 20 As shown, since the first imaging element 2 and the second imaging element 3 output pixel signals corresponding to the retained charges of the second capacitors C2 of all pixels in a time-shifted manner, peak power can be suppressed when the pixel signals are output from the second capacitors C2 to the vertical signal lines L2.

[0145] exist Fig. 20In the embodiment, since the second imaging element 3 outputs a frame of pixel signals after the first imaging element 2 completes the output of a frame of pixel signals, when the signal processing circuit (not shown) performs pattern matching between the image data captured by the first imaging element 2 and the image data captured by the second imaging element 3, the image data captured by the first imaging element 2 needs to be temporarily stored in a frame memory or the like. If the image data captured by the first inspection imaging element and the image data captured by the second imaging element 3 are input to the signal processing circuit at the same timing, it is not necessary to store the image data in the frame memory. Therefore, as described below, it is also possible to perform timing control in which the first imaging element 2 and the second imaging element 3 output pixel signals at approximately the same time.

[0146] Fig.21 is an operation timing chart according to a modification of the sixth embodiment. Fig.21 The timing in which the signal charge is held in the second capacitor C2 and the pixel signal corresponding to the held charge of the second capacitor C2 is output to the vertical signal line L2 is shown, and the timing in which the reset level charge is held in the first capacitor C1 and the pixel signal corresponding to the held charge of the first capacitor C1 is output to the vertical signal line L2 is omitted.

[0147] The first imaging element 2 and the second imaging element 3 start exposure at time t91, and end exposure at time t92, and the second capacitor C2 holds (samples) the signal charge. Then, during the period from time t92 to time t93, the first imaging element 2 and the second imaging element 3 alternately output pixel signals corresponding to the held charges of the corresponding second capacitors C2 to the vertical signal line L2 for each pixel row.

[0148] Fig.21 The output timing of the n-th pixel row (time T1 to T3 ), the output timing of the n+1-th pixel row (time t3 to T5 ), and the output timing of the n+2-th pixel row (time T5 to T7 ) are shown.

[0149] exist Fig.21 In this case, since the first imaging element 2 and the second imaging element 3 alternately output pixel signals of each pixel row, the timing of the column signal processing circuit 14 generating pixel 20 data of one frame is approximately the same between the first imaging element 2 and the second imaging element 3, and two types of image data can be generated approximately at the same time, and pattern matching, etc. can be performed quickly without storing the image data in the frame memory.

[0150] exist Fig.21 In the embodiment, since the first imaging element 2 and the second imaging element 3 alternately output pixel signals of each pixel row, the peak power can be suppressed to about half compared with the case where the first imaging element 2 and the second imaging element 3 output pixel signals simultaneously.

[0151] As described above, in the sixth embodiment, when the pixel signal corresponding to the signal charge held in the second capacitor C2 is output to the vertical signal line L2, the output timing is shifted between the first imaging element 2 and the second imaging element 3, so that the peak power can be suppressed.

[0152] <Application Examples> The technology according to the present disclosure can be applied to various products. For example, the technology according to the present disclosure can also be implemented as a device installed on any type of mobile body such as an automobile, an electric car, a hybrid electric car, a motorcycle, a bicycle, a personal mobile device, an airplane, an unmanned aerial vehicle, a ship, a robot, a construction machine, or an agricultural machine (tractor).

[0153] Fig. 22 7000 is a block diagram showing an example of a schematic configuration of a vehicle control system 7000 as an example of a mobile body control system to which the technology according to the embodiment of the present disclosure can be applied. The vehicle control system 7000 includes a plurality of electronic control units connected to each other via a communication network 7010. Fig. 22 In the example shown, the vehicle control system 7000 includes a drive system control unit 7100, a body system control unit 7200, a battery control unit 7300, an external information detection unit 7400, an internal information detection unit 7500, and an integrated control unit 7600. For example, the communication network 7010 connecting the plurality of control units to each other may be an in-vehicle communication network conforming to any standard such as a controller area network (CAN), a local interconnect network (LIN), a local area network (LAN), FlexRay (registered trademark), or the like.

[0154] Each control unit includes: a microcomputer that executes algorithm processing according to various programs; a storage unit that stores programs executed by the microcomputer, parameters for various operations, etc.; and a drive circuit that drives various control object devices. Each control unit also includes: a network interface (I / F) for communicating with other control units via the communication network 7010; and a communication I / F for communicating with devices, sensors, etc. inside and outside the vehicle through wired communication or wireless communication. Fig. 22 The functional configuration of the integrated control unit 7600 shown includes a microcomputer 7610, a general communication I / F 7620, a dedicated communication I / F 7630, a positioning unit 7640, a beacon receiving unit 7650, an in-vehicle device I / F 7660, a sound / image output unit 7670, an in-vehicle network I / F 7680, and a storage unit 7690. Other control units also include a microcomputer, a communication I / F, a storage unit, and the like.

[0155] The drive system control unit 7100 controls the operation of devices related to the drive system of the vehicle according to various programs. For example, the drive system control unit 7100 is used as a control device for the following devices: a drive force generating device such as an internal combustion engine, a drive motor, etc. for generating the drive force of the vehicle, a drive force transmitting mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, a braking device for generating the braking force of the vehicle, etc. The drive system control unit 7100 may have a function as a control device for an anti-lock braking system (ABS), an electronic stability control (ESC), etc.

[0156] The drive system control unit 7100 is connected to the vehicle state detection unit 7110. For example, the vehicle state detection unit 7110 includes at least one of a gyro sensor for detecting the angular velocity of the axial rotational motion of the vehicle body, an acceleration sensor for detecting the acceleration of the vehicle, and a sensor for detecting the amount of operation of the accelerator pedal, the amount of operation of the brake pedal, the steering angle of the steering wheel, the engine speed, or the wheel speed, etc. The drive system control unit 7100 performs algorithm processing using the signal input from the vehicle state detection unit 7110, and controls the internal combustion engine, the drive motor, the electric steering device, the brake device, etc.

[0157] The body system control unit 7200 controls the operation of various devices provided on the body according to various programs. For example, the body system control unit 7200 is used as a control device for a keyless entry system, a smart key system, a power window device, or various lights such as headlights, reverse lights, brake lights, turn signals, fog lights, etc. In this case, radio waves or signals of various switches transmitted from a mobile device as a substitute for a key may be input to the body system control unit 7200. The body system control unit 7200 receives these input radio waves or signals and controls the door lock device, power window device, lights, etc. of the vehicle.

[0158] The battery control unit 7300 controls the secondary battery 7310 as a power source for driving the motor according to various programs. For example, information about the battery temperature, the battery output voltage, the remaining amount of power in the battery, etc. is provided to the battery control unit 7300 from the battery device including the secondary battery 7310. The battery control unit 7300 performs algorithm processing using these signals, and performs control for adjusting the temperature of the secondary battery 7310 or control of a cooling device provided in the battery device, etc.

[0159] The vehicle exterior information detection unit 7400 detects information about the exterior of the vehicle including the vehicle control system 7000. For example, the vehicle exterior information detection unit 7400 is connected to at least one of the imaging unit 7410 and the vehicle exterior information detection unit 7420. The imaging unit 7410 includes at least one of a time-of-flight (ToF) camera, a stereo camera, a monocular camera, an infrared camera, and other cameras. For example, the vehicle exterior information detection unit 7420 includes at least one of an environmental sensor for detecting current atmospheric conditions or weather conditions and a peripheral information detection sensor for detecting other vehicles, obstacles, pedestrians, and the like around the vehicle including the vehicle control system 7000.

[0160] For example, the environmental sensor may be at least one of a raindrop sensor for detecting rain, a fog sensor for detecting fog, a sunshine sensor for detecting the degree of sunshine, and a snow sensor for detecting snowfall. The peripheral information detection sensor may be at least one of an ultrasonic sensor, a radar device, and a LIDAR (LIDAR: light detection and ranging or laser imaging detection and ranging) device. Each of the imaging unit 7410 and the vehicle exterior information detection unit 7420 may be provided as an independent sensor or device, or may be provided as a device in which a plurality of sensors or devices are integrated.

[0161] Fig.23 A diagram showing an example of the installation position of the imaging unit 7410 and the vehicle exterior information detection unit 7420. For example, the imaging units 7910, 7912, 7914, 7916 and 7918 are arranged at least one of the positions on the front nose, side mirrors, rear bumper and rear door of the vehicle 7900 and the position on the upper part of the windshield inside the vehicle. The imaging unit 7910 arranged on the front nose and the imaging unit 7918 arranged on the upper part of the windshield inside the vehicle mainly obtain images in front of the vehicle 7900. The imaging units 7912 and 7914 arranged on the side mirrors mainly obtain images on the side of the vehicle 7900. The imaging unit 7916 arranged on the rear bumper or rear door mainly obtains images behind the vehicle 7900. The imaging unit 7918 arranged on the upper part of the windshield inside the vehicle is mainly used to detect the front vehicle, pedestrians, obstacles, signals, traffic signs, lanes, etc.

[0162] By the way, Fig.23Examples of the shooting ranges of the respective imaging units 7910, 7912, 7914, and 7916 are shown. Imaging range a represents the imaging range of the imaging unit 7910 provided on the front nose. Imaging ranges b and c represent the imaging ranges of the imaging units 79124 and 79142 provided on the side mirrors, respectively. Imaging range d represents the imaging range of the imaging unit 7916 provided on the rear bumper or the rear door. For example, a bird's-eye view image of the vehicle 7900 viewed from above can be obtained by superimposing image data imaged by the imaging units 7910, 7912, 7914, and 7916.

[0163] For example, the vehicle exterior information detection units 7920, 7922, 7924, 7926, 7928, and 7930 disposed at the front, rear, sides, and corners of the vehicle 7900 and the upper portion of the windshield inside the vehicle may be ultrasonic sensors or radar devices. For example, the vehicle exterior information detection units 7920, 7926, and 7930 disposed at the front nose of the vehicle 7900, the rear bumper, the rear door of the vehicle 7900, and the upper portion of the windshield inside the vehicle may be LIDAR devices. These vehicle exterior information detection units 7920-7930 are mainly used to detect vehicles ahead, pedestrians, obstacles, and the like.

[0164] return Fig. 22 , continue to explain. The vehicle exterior information detection unit 7400 enables the imaging unit 7410 to image the image outside the vehicle and receive the image data of the imaging. In addition, the vehicle exterior information detection unit 7400 receives detection information from the vehicle exterior information detection unit 7420 connected to the vehicle exterior information detection unit 7400. In the case where the vehicle exterior information detection unit 7420 is an ultrasonic sensor, a radar device or a LIDAR device, the vehicle exterior information detection unit 7400 transmits ultrasonic waves, electromagnetic waves, etc., and receives information of the received reflected waves. The vehicle exterior information detection unit 7400 can perform detection processing of objects such as people, vehicles, obstacles, signs, characters on the road surface, etc. based on the received information, or perform detection processing of the distance therebetween. The vehicle exterior information detection unit 7400 can perform environmental recognition processing of identifying rainfall, fog, road conditions, etc. based on the received information. The vehicle exterior information detection unit 7400 can calculate the distance to the object outside the vehicle based on the received information.

[0165] In addition, the vehicle exterior information detection unit 7400 can perform image recognition processing for identifying people, vehicles, obstacles, signs, characters on the road surface, etc. based on the received image data, or perform detection processing of the distance therebetween. The vehicle exterior information detection unit 7400 can perform processing such as distortion correction, alignment, etc. on the received image data, and synthesize the image data imaged by multiple different imaging units 7410 to generate a bird's-eye view image or a panoramic image. The vehicle exterior information detection unit 7400 can perform viewpoint conversion processing using the image data imaged by the imaging unit 7410 including different imaging parts.

[0166] The in-vehicle information detection unit 7500 detects information about the interior of the vehicle. For example, the in-vehicle information detection unit 7500 is connected to a driver state detection unit 7510 that detects the driver's state. The driver state detection unit 7510 may include a camera that images the driver, a biosensor that detects the driver's biometric information, a microphone that collects sounds inside the vehicle, and the like. For example, the biosensor is arranged in a seat surface, a steering wheel, and the like, and detects the biometric information of an occupant sitting in a seat or a driver holding the steering wheel. The in-vehicle information detection unit 7500 can calculate the driver's fatigue level or the driver's concentration level based on the detection information input from the driver state detection unit 7510, or can determine whether the driver is taking a nap. The in-vehicle information detection unit 7500 can perform processing such as noise elimination processing on the audio signal obtained by sound collection.

[0167] The integrated control unit 7600 controls the overall operation within the vehicle control system 7000 according to various programs. The integrated control unit 7600 is connected to the input unit 7800. For example, the input unit 7800 is implemented by a device such as a touch panel, a button, a microphone, a switch, a joystick, etc., which can be input by the occupant. Data obtained by voice recognition of a voice input via a microphone can be supplied to the integrated control unit 7600. For example, the input unit 7800 can be a remote control device using infrared or other radio waves, or an external connection device such as a mobile phone, a personal digital assistant (PDA), etc. that supports the operation of the vehicle control system 7000. For example, the input unit 7800 can be a camera. In this case, the occupant can input information through gestures. Alternatively, data obtained by detecting the movement of a wearable device worn by the occupant can be input. In addition, for example, the input unit 7800 may include an input control circuit that generates an input signal based on information input by the occupant or the like using the above-mentioned input unit 7800 and outputs the generated input signal to the integrated control unit 7600. The occupant or the like inputs various data or gives instructions for processing operations to the vehicle control system 7000 through the operation input unit 7800 .

[0168] The storage unit 7690 may include a read-only memory (ROM) storing various programs executed by the microcomputer and a random access memory (RAM) storing various parameters, operation results, sensor values, etc. In addition, the storage unit 7690 may be implemented by a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, a magneto-optical storage device, or the like.

[0169] The general communication I / F 7620 is a widely used communication I / F that mediates communication with various devices present in the external environment 7750. The general communication I / F 7620 may implement cellular communication protocols such as Global System for Mobile Communications (GSM (registered trademark)), Worldwide Interoperability for Microwave Access (WiMAX (registered trademark)), Long Term Evolution (LTE (registered trademark)), Advanced LTE (LTE-A), or other wireless communication protocols such as wireless LAN (also known as Wireless Fidelity (Wi-Fi (registered trademark)), Bluetooth (registered trademark), etc.). For example, the general communication I / F 7620 may be connected to a device (e.g., an application server or a control server) present on an external network (e.g., the Internet, a cloud network, or a company private network) via a base station or an access point. In addition, for example, the general communication I / F 7620 may be connected to a terminal present near the vehicle (e.g., the terminal is a terminal of a driver, a pedestrian, or a store, or a machine type communication (MTC) terminal) using a peer-to-peer (P2P) technology.

[0170] The dedicated communication I / F 7630 is a communication I / F that supports a communication protocol developed for vehicle use. For example, the dedicated communication I / F 7630 may implement a standard protocol such as Wireless Access in Vehicular Environments (WAVE), a dedicated short range communication (DSRC), or a cellular communication protocol that is a combination of the Institute of Electrical and Electronics Engineers (IEEE) 802.11p as a lower layer and IEEE 1609 as an upper layer. The dedicated communication I / F 7630 generally performs V2X communication as a concept including one or more of communication between vehicles (vehicle to vehicle), communication between roads and vehicles (vehicle to infrastructure), communication between vehicles and residences (vehicle to residence), and communication between pedestrians and vehicles (vehicle to pedestrian).

[0171] For example, the positioning unit 7640 performs positioning by receiving a GNSS satellite signal (e.g., a GPS signal from a Global Positioning System (GPS) satellite) from a satellite of a Global Navigation Satellite System (GNSS), and generates position information including the latitude, longitude, and altitude of the vehicle. Incidentally, the positioning unit 7640 may identify the current position by exchanging signals with a wireless access point, or may obtain position information from a terminal such as a mobile phone, a personal handyphone system (PHS), or a smart phone having a positioning function.

[0172] For example, the beacon receiving unit 7650 receives radio waves or electromagnetic waves transmitted from a radio station installed on the road, etc., thereby obtaining information on the current position, congestion, closed roads, necessary time, etc. Incidentally, the function of the beacon receiving unit 7650 may be included in the above-mentioned dedicated communication I / F 7630.

[0173] The in-vehicle device I / F 7660 is a communication interface that mediates the connection between the microcomputer 7610 and various in-vehicle devices 7760 present in the vehicle. The in-vehicle device I / F 7660 can establish a wireless connection using a wireless communication protocol such as wireless LAN, Bluetooth (registered trademark), near field communication (NFC), or wireless universal serial bus (WUSB). In addition, the in-vehicle device I / F 7660 can establish a wired connection through a universal serial bus (USB), a high-definition multimedia interface (HDMI (registered trademark)), a mobile high-definition link (MHL), etc. via a connection terminal not shown in the figure (if necessary, it can also be via a cable). For example, the in-vehicle device 7760 may include at least one of a mobile device and a wearable device owned by an occupant and an information device carried into or attached to the vehicle. The in-vehicle device 7760 may also include a navigation device that searches for a path to an arbitrary destination. The in-vehicle device I / F 7660 exchanges control signals or data signals with these in-vehicle devices 7760.

[0174] The in-vehicle network I / F 7680 is an interface which mediates communication between the microcomputer 7610 and the communication network 7010. The in-vehicle network I / F 7680 transmits and receives signals and the like according to a predetermined protocol supported by the communication network 7010.

[0175] The microcomputer 7610 of the integrated control unit 7600 controls the vehicle control system 7000 according to various programs based on information obtained via at least one of the general communication I / F 7620, the dedicated communication I / F 7630, the positioning unit 7640, the beacon receiving unit 7650, the in-vehicle device I / F 7660, and the in-vehicle network I / F 7680. For example, the microcomputer 7610 can calculate the control target value of the driving force generation device, the steering mechanism, or the braking device based on the information obtained about the inside and outside of the vehicle, and output a control command to the drive system control unit 7100. For example, the microcomputer 7610 can perform cooperative control aimed at realizing the functions of the advanced driver assistance system (ADAS), which include vehicle collision avoidance or collision mitigation, follow-up driving based on the following distance, vehicle speed maintenance driving, vehicle collision warning, lane departure warning of the vehicle, etc. In addition, the microcomputer 7610 can perform cooperative control aimed at automatic driving, which makes the vehicle automatically travel without relying on the operation of the driver, etc. by controlling the driving force generation device, the steering mechanism, the braking device, etc. based on the information obtained about the surroundings of the vehicle.

[0176] The microcomputer 7610 can generate three-dimensional distance information between the vehicle and objects such as surrounding structures and people based on information obtained via at least one of the general communication I / F 7620, the dedicated communication I / F 7630, the positioning unit 7640, the beacon receiving unit 7650, the in-vehicle device I / F 7660, and the vehicle network I / F 7680, and generate local map information including information about the surroundings of the current position of the vehicle. In addition, the microcomputer 7610 can predict dangers such as vehicle collision, approach of pedestrians, etc., entering a closed road, etc. based on the obtained information, and generate a warning signal. For example, the warning signal can be a signal for generating a warning sound or lighting a warning light.

[0177] The sound / image output unit 7670 transmits an output signal of at least one of sound and image to an output device, which can visually or auditorily notify the vehicle occupants or the outside of the vehicle of information. Fig. 22In the example of , an audio speaker 7710, a display unit 7720, and an instrument panel 7730 are shown as output devices. For example, the display unit 7720 may include at least one of an in-vehicle display and a head-up display. The display unit 7720 may have an augmented reality (AR) display function. The output device may be a device other than these devices, and may be other devices such as headphones, wearable devices such as glasses-type displays worn by passengers, projectors, lights, etc. In the case where the output device is a display device, the display device visually displays the results obtained by various processes performed by the microcomputer 7610 or information received from other control units in various forms such as text, images, tables, charts, etc. In addition, in the case where the output device is an audio output device, the audio output device converts an audio signal composed of reproduced audio data or sound data, etc. into an analog signal, and outputs the analog signal audibly.

[0178] By the way, in Fig. 22 In the example shown, at least two control units connected to each other via the communication network 7010 can be integrated into one control unit. Alternatively, each control unit may include multiple control units. In addition, the vehicle control system 7000 may include other control units not shown in the figure. In addition, part or all of the functions performed by one of the control units in the above description can be assigned to other control units. In other words, as long as information is sent and received via the communication network 7010, the predetermined algorithm processing can be performed by any control unit. Similarly, a sensor or device connected to a control unit in the control unit can be connected to other control units, and multiple control units can send and receive detection information to each other via the communication network 7010.

[0179] Note that for reference Figures 1 to 21 The computer program for each function of the camera system 1 according to the present embodiment described above can be installed on any control unit or the like. In addition, a computer-readable recording medium storing such a computer program can be provided. The recording medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, or the like. In addition, without using a recording medium, the above-mentioned computer program can be distributed via, for example, a network.

[0180] In the vehicle control system 7000, referring to Figures 1 to 21 The camera system according to the present embodiment described above can be applied to Fig. 22 An integrated control unit 7600 of the application example shown.

[0181] In addition, refer to Figures 1 to 21 At least some components of the camera system 1 described may be located in Fig. 22The integrated control unit 7600 shown in FIG. 7 is implemented in a module (eg, an integrated circuit module including a die). Alternatively, referring to Figures 1 to 21 The camera system 1 described can be Fig. 22 The vehicle control system 7000 shown is implemented by multiple control units.

[0182] Note that the present technology can have the following configurations. (1) A camera system comprising: a first imaging element; a second imaging element; and a control unit for controlling the first imaging element and the second imaging element, wherein the first imaging element comprises a plurality of first pixel circuits, which simultaneously sample charges corresponding to incident light photoelectrically converted in each pixel and read out the sampled charges in a non-destructive manner, The second imaging element includes a plurality of second pixel circuits that simultaneously samples charges corresponding to incident light photoelectrically converted in each pixel and reads out the sampled charges in a non-destructive manner, The first imaging element switches whether to output pixel signals from at least some of the plurality of first pixel circuits in the same frame period based on pixel signals output from some of the first pixel circuits or some of the second pixel circuits in each frame period, and The second imaging element switches whether to output pixel signals from at least some of the plurality of second pixel circuits within the same frame period based on pixel signals output from some of the first pixel circuits or some of the second pixel circuits in each frame period. (2) The camera system according to (1), wherein the some of the first pixel circuits or the some of the second pixel circuits are arranged at one or more pixel positions obtained by thinning out the plurality of first pixel circuits or the plurality of second pixel circuits in at least one of the first direction and the second direction. (3) The camera system according to (1) or (2), wherein the control unit includes a feature detection unit that detects a feature including at least one of a motion of a target object and a presence of the target object based on pixel signals output from the some of the first pixel circuits or the some of the second pixel circuits in each of the frame periods, The first imaging element outputs pixel signals from at least some of the plurality of first pixel circuits during the frame period in which the feature detection section detects a feature, and The second imaging element outputs pixel signals from at least some of the plurality of first pixel circuits during the frame period in which the feature detection section detects a feature. (4) The camera system according to (3), wherein the first imaging element outputs pixel signals from the plurality of first pixel circuits within the frame period in which the feature detection section detects a feature, and The second imaging element outputs pixel signals from the plurality of second pixel circuits during the frame period in which the feature detection section detects a feature. (5) The camera system according to (4), wherein the feature detection section detects the feature based on pixel signals output from the some of the first pixel circuits, and The first imaging element outputs pixel signals twice from the some first pixel circuits within the frame period in which the feature detection section detects a feature, and outputs pixel signals once from each of the first pixel circuits except for the some first pixel circuits. (6) The camera system according to any one of (4) to (5), wherein the first imaging element stops outputting pixel signals from the plurality of first pixel circuits during a frame period in which the feature detection unit does not detect a feature, and The second imaging element stops outputting pixel signals from the plurality of second pixel circuits during the frame period in which the feature detection section does not detect a feature. (7) The camera system according to any one of (3) to (6), wherein the control unit includes an ROI setting unit that sets a first region of interest (ROI) pixel area including one or more first pixel circuits among the plurality of first pixel circuits and sets a second ROI pixel area including one or more second pixel circuits among the plurality of second pixel circuits in each of the frame periods, based on a pixel position including the feature detected by the feature detection unit. The first imaging element outputs the pixel signal of the first ROI pixel area within the same frame period, and The second imaging element outputs pixel signals of the second ROI pixel area within the same frame period. (8) The camera system according to any one of (3) to (7), wherein the feature detection section detects the motion based on pixel signals output from the some of the first pixel circuits or the some of the second pixel circuits for each of the frame periods, The first imaging element outputs pixel signals from at least some of the plurality of first pixel circuits during a frame period in which the motion is detected, and The second imaging element outputs pixel signals from at least some of the plurality of second pixel circuits during the frame period in which the motion is detected. (9) The camera system according to any one of (3) to (7), wherein the feature detection section detects the motion based on pixel signals output from the some of the first pixel circuits or the some of the second pixel circuits for each of the frame periods, The control unit includes a ROI setting unit, which sets a first ROI pixel area including pixel positions of one or more of the plurality of first pixel circuits and sets a second ROI pixel area including pixel positions of one or more of the plurality of second pixel circuits, The first imaging element outputs a pixel signal of the first ROI pixel area within a frame period in which the motion is detected, and The second imaging element outputs a pixel signal of the second ROI pixel area within the frame period in which the motion is detected. (10) The camera system according to (1) or (2), wherein the control unit includes: a target object determination section that determines whether a target object is imaged based on pixel signals output from the some of the first pixel circuits or the some of the second pixel circuits in each frame period; and The ROI setting unit sets a first ROI pixel area including pixel positions of one or more of the plurality of first pixel circuits, and sets a second ROI pixel area including pixel positions of one or more of the plurality of second pixel circuits, when determining that the target object is imaged. The first imaging element outputs a pixel signal from the first pixel circuit in the first ROI pixel region during a frame period in which the target object is imaged, and The second imaging element outputs a pixel signal from the second pixel circuit in the second ROI pixel region during the frame period in which the target object is imaged. (11) The camera system according to (10), wherein the control section controls pixel positions of the first ROI pixel area and the second ROI pixel area in each frame period according to a position of the target object so as to include the target object. (12) The camera system according to any one of (1) to (11), wherein each of the plurality of first pixel circuits includes: a first floating diffusion region that accumulates charges corresponding to incident light photoelectrically converted by the first photoelectric conversion element; a first capacitor that accumulates charges corresponding to a potential of the first floating diffusion region in a state in which charges of the first floating diffusion region are reset; and a second capacitor that accumulates electric charge corresponding to incident light photoelectrically converted by the first photoelectric conversion element, Each of the plurality of second pixel circuits comprises: a second floating diffusion region that accumulates charges corresponding to incident light photoelectrically converted by the second photoelectric conversion element; a third capacitor that accumulates charge corresponding to the potential of the second floating diffusion region in a state in which the charge of the second floating diffusion region is reset; and a fourth capacitor that accumulates charge corresponding to incident light photoelectrically converted by the second photoelectric conversion element, and The charges accumulated in the first capacitor, the second capacitor, the third capacitor, and the fourth capacitor are maintained within the same frame period even after being read out. (13) A camera system according to any one of (1) to (11), wherein the first imaging element switches whether to output pixel signals from at least some of the plurality of first pixel circuits in the same frame period based on pixel signals output from some of the first pixel circuits in each frame period, The second imaging element switches whether to output pixel signals from at least some of the plurality of second pixel circuits in the same frame period based on the pixel signals output from some of the first pixel circuits in each frame period, Each of the plurality of first pixel circuits comprises: a first floating diffusion region that accumulates charges corresponding to incident light photoelectrically converted by the first photoelectric conversion element; a first capacitor that accumulates charge corresponding to a potential of the first floating diffusion region in a state in which the charge of the first floating diffusion region is reset; and a second capacitor that accumulates electric charge corresponding to incident light photoelectrically converted by the first photoelectric conversion element, Each of the plurality of second pixel circuits comprises: a second floating diffusion region that accumulates charges corresponding to incident light photoelectrically converted by the second photoelectric conversion element; and a third capacitor that accumulates charges in a state where the second floating diffusion region is reset or charges corresponding to incident light photoelectrically converted by the second photoelectric conversion element, The charges accumulated in the first capacitor and the second capacitor are maintained in the same frame even after being read out, and The third capacitor accumulates charges in a state where the second floating diffusion region is reset within one frame period, and then accumulates charges corresponding to incident light photoelectrically converted by the second photoelectric conversion element. (14) A camera system according to any one of (1) to (13), wherein the control unit offsets the timing at which the first imaging element samples the charge corresponding to the incident light photoelectrically converted by the first photoelectric conversion elements of at least some of the multiple first pixel circuits within the same frame period and the timing at which the second imaging element samples the charge corresponding to the incident light photoelectrically converted by the second photoelectric conversion elements of at least some of the multiple second pixel circuits within the same frame period. (15) A camera system according to (14), wherein the control unit offsets the timing at which the first imaging element samples the reset level charge of at least some of the multiple first pixel circuits within the same frame period, the timing at which the first imaging element samples the charge corresponding to the incident light photoelectrically converted by the first photoelectric conversion element, the timing at which the second imaging element samples the reset level charge of at least some of the multiple second pixel circuits within the same frame period, and the timing at which the second imaging element samples the charge corresponding to the incident light photoelectrically converted by the second photoelectric conversion element. (16) A camera system according to any one of (1) to (15), wherein the control unit offsets the timing at which the first imaging element outputs pixel signals from at least some of the plurality of first pixel circuits within the same frame period and the timing at which the second imaging element outputs pixel signals from at least some of the plurality of second pixel circuits within the same frame period. (17) A camera system according to (16), wherein the control unit causes at least some of the plurality of first pixel circuits to output pixel signals within the same frame period, and then causes at least some of the plurality of second pixel circuits to output pixel signals within the same frame period. (18) The camera system according to any one of (1) to (15), wherein the plurality of first pixel circuits are arranged in each of a first direction and a second direction, The plurality of second pixel circuits are arranged in each of the first direction and the second direction, and When outputting pixel signals in the same frame period, the first imaging element and the second imaging element alternately output pixel signals of respective pixel groups arranged in the second direction. (19) The camera system according to any one of (1) to (18), wherein the first imaging element and the second imaging element have the same number of pixels, and The first imaging element and the second imaging element perform exposure at the same exposure timing. (20) A control method for a camera system, the camera system comprising: a first imaging element including a plurality of first pixel circuits that simultaneously samples charges corresponding to incident light photoelectrically converted in each pixel and reads out the sampled charges in a non-destructive manner; a second imaging element including a plurality of second pixel circuits that simultaneously samples charges corresponding to photoelectrically converted incident light in each pixel and reads out the sampled charges in a non-destructive manner; and a control unit for controlling the first imaging element and the second imaging element, wherein the first imaging element switches whether to output pixel signals from at least some of the plurality of first pixel circuits in the same frame period based on pixel signals output from some of the first pixel circuits or some of the second pixel circuits in each frame period, and The second imaging element switches whether to output pixel signals from at least some of the plurality of second pixel circuits within the same frame period based on pixel signals output from some of the first pixel circuits or some of the second pixel circuits in each frame period.

[0183] The aspects of the present disclosure are not limited to the above-mentioned embodiments, but include various modifications that can be thought of by those skilled in the art, and the effects of the present disclosure are not limited to the above-mentioned contents. That is, various additions, modifications and partial deletions can be made without departing from the conceptual ideas and spirit of the present disclosure derived from the matters defined in the claims and their equivalents. Reference numerals list 1 Camera System 2First imaging element 3. Second imaging element 4. Control Unit 5Feature Detection Unit 5a Detection Department 5bROI Setting Department 6Imaging control unit 7 Storage 8Comparison 11 Pixel array unit 12Vertical scanning circuit 13 Load MOS circuit 14 columns of signal processing circuit 15 Timing control circuit 16 Digital-to-Analog Converters (DACs) 20 pixels 21 Photoelectric conversion element 22 pixel circuit 23 Pre-amplifier circuit 24 Selection circuit 25 post-stage circuit 31 pass transistor 32 floating diffusion area 33 First reset transistor 34 first amplifier transistor 35 Current Source 41 first selection transistor 42 Second selection transistor 43 Second amplifier transistor 44 third selection transistor 45 Second reset transistor 51 Current Source 52 analog-to-digital converter 53Digital Signal Processing Unit 61 Overflow transistor 62 first transfer transistor 63 Second transfer transistor 64 Reset transistor 65 Amplifier Transistor 66 Select transistor 67 Floating Diffusion Area

Claims

1. A camera system comprising: a first imaging element; a second imaging element; and a control unit for controlling the first imaging element and the second imaging element, wherein the first imaging element comprises a plurality of first pixel circuits, which simultaneously sample charges corresponding to incident light photoelectrically converted in each pixel and read out the sampled charges in a non-destructive manner, The second imaging element includes a plurality of second pixel circuits that simultaneously samples charges corresponding to incident light photoelectrically converted in each pixel and reads out the sampled charges in a non-destructive manner, The first imaging element switches whether to output pixel signals from at least some of the plurality of first pixel circuits in the same frame period based on pixel signals output from some of the first pixel circuits or some of the second pixel circuits in each frame period, and The second imaging element switches whether to output pixel signals from at least some of the plurality of second pixel circuits within the same frame period based on pixel signals output from some of the first pixel circuits or some of the second pixel circuits in each frame period.

2. The camera system according to claim 1, wherein: The some of the first pixel circuits or the some of the second pixel circuits are arranged at one or more pixel positions obtained by thinning out the plurality of first pixel circuits or the plurality of second pixel circuits in at least one of the first direction and the second direction.

3. The camera system according to claim 1 , wherein the control section includes a feature detection section that detects a feature including at least one of a motion of a target object and a presence of the target object based on pixel signals output from the some of the first pixel circuits or the some of the second pixel circuits in each of the frame periods, The first imaging element outputs pixel signals from at least some of the plurality of first pixel circuits during the frame period in which the feature detection section detects a feature, and The second imaging element outputs pixel signals from at least some of the plurality of first pixel circuits during the frame period in which the feature detection section detects a feature.

4. The camera system according to claim 3, wherein the first imaging element outputs pixel signals from the plurality of first pixel circuits within the frame period in which the feature detection section detects a feature, and The second imaging element outputs pixel signals from the plurality of second pixel circuits during the frame period in which the feature detection section detects a feature.

5. The camera system according to claim 4, wherein the feature detection section detects the feature based on the pixel signals output from the some of the first pixel circuits, and The first imaging element outputs pixel signals twice from the some first pixel circuits within the frame period in which the feature detection section detects a feature, and outputs pixel signals once from each of the first pixel circuits except for the some first pixel circuits.

6. The camera system according to claim 4, wherein the first imaging element stops outputting pixel signals from the plurality of first pixel circuits during a frame period in which the feature detection section does not detect a feature, and The second imaging element stops outputting pixel signals from the plurality of second pixel circuits during the frame period in which the feature detection section does not detect a feature.

7. The camera system according to claim 3, wherein the control section includes a ROI setting section that sets a first region of interest (ROI) pixel area including one or more first pixel circuits among the plurality of first pixel circuits and sets a second ROI pixel area including one or more second pixel circuits among the plurality of second pixel circuits in each of the frame periods according to a pixel position including the feature detected by the feature detection section, The first imaging element outputs the pixel signal of the first ROI pixel area within the same frame period, and The second imaging element outputs pixel signals of the second ROI pixel area within the same frame period.

8. The camera system according to claim 3, wherein the feature detection section detects the motion based on pixel signals output from the some of the first pixel circuits or the some of the second pixel circuits in each of the frame periods, The first imaging element outputs pixel signals from at least some of the plurality of first pixel circuits during a frame period in which the motion is detected, and The second imaging element outputs pixel signals from at least some of the plurality of second pixel circuits during the frame period in which the motion is detected.

9. The camera system according to claim 3, wherein the feature detection section detects the motion based on pixel signals output from the some of the first pixel circuits or the some of the second pixel circuits in each of the frame periods, The control unit includes a ROI setting unit, which sets a first ROI pixel area including pixel positions of one or more of the plurality of first pixel circuits and sets a second ROI pixel area including pixel positions of one or more of the plurality of second pixel circuits, The first imaging element outputs a pixel signal of the first ROI pixel area within a frame period in which the motion is detected, and The second imaging element outputs a pixel signal of the second ROI pixel area within the frame period in which the motion is detected.

10. The camera system according to claim 1, wherein the control section comprises: a target object determination unit that determines whether a target object is imaged based on pixel signals output from the some of the first pixel circuits or the some of the second pixel circuits in each frame period; and The ROI setting unit sets a first ROI pixel area including pixel positions of one or more of the plurality of first pixel circuits, and sets a second ROI pixel area including pixel positions of one or more of the plurality of second pixel circuits, when determining that the target object is imaged. The first imaging element outputs a pixel signal from the first pixel circuit in the first ROI pixel region during a frame period in which the target object is imaged, and The second imaging element outputs a pixel signal from the second pixel circuit in the second ROI pixel region during the frame period in which the target object is imaged. 11 . The camera system according to claim 10 , wherein the control section controls pixel positions of the first ROI pixel area and the second ROI pixel area in each frame period according to a position of the target object so as to include the target object.

12. The camera system of claim 1, wherein Each of the plurality of first pixel circuits comprises: a first floating diffusion region that accumulates charges corresponding to incident light photoelectrically converted by the first photoelectric conversion element; a first capacitor that accumulates charge corresponding to a potential of the first floating diffusion region in a state in which the charge of the first floating diffusion region is reset; and a second capacitor that accumulates electric charge corresponding to incident light photoelectrically converted by the first photoelectric conversion element, Each of the plurality of second pixel circuits comprises: a second floating diffusion region that accumulates charges corresponding to incident light photoelectrically converted by the second photoelectric conversion element; a third capacitor that accumulates charge corresponding to the potential of the second floating diffusion region in a state in which the charge of the second floating diffusion region is reset; and a fourth capacitor that accumulates charge corresponding to incident light photoelectrically converted by the second photoelectric conversion element, and The charges accumulated in the first capacitor, the second capacitor, the third capacitor, and the fourth capacitor are maintained within the same frame period even after being read out.

13. The camera system according to claim 1 , wherein the first imaging element switches whether to output pixel signals from at least some of the plurality of first pixel circuits in the same frame period based on pixel signals output from some of the first pixel circuits in each frame period, The second imaging element switches whether to output pixel signals from at least some of the plurality of second pixel circuits in the same frame period based on the pixel signals output from some of the first pixel circuits in each frame period, Each of the plurality of first pixel circuits comprises: a first floating diffusion region that accumulates charges corresponding to incident light photoelectrically converted by the first photoelectric conversion element; a first capacitor that accumulates charge corresponding to a potential of the first floating diffusion region in a state in which the charge of the first floating diffusion region is reset; and a second capacitor that accumulates electric charge corresponding to incident light photoelectrically converted by the first photoelectric conversion element, Each of the plurality of second pixel circuits comprises: a second floating diffusion region that accumulates charges corresponding to incident light photoelectrically converted by the second photoelectric conversion element; and a third capacitor that accumulates charges in a state where the second floating diffusion region is reset or charges corresponding to incident light photoelectrically converted by the second photoelectric conversion element, The charges accumulated in the first capacitor and the second capacitor are maintained in the same frame even after being read out, and The third capacitor accumulates charges in a state where the second floating diffusion region is reset within one frame period, and then accumulates charges corresponding to incident light photoelectrically converted by the second photoelectric conversion element.

14. A camera system according to claim 1, wherein the control unit offsets the timing at which the first imaging element samples the charge corresponding to the incident light photoelectrically converted by the first photoelectric conversion elements of at least some of the multiple first pixel circuits within the same frame period and the timing at which the second imaging element samples the charge corresponding to the incident light photoelectrically converted by the second photoelectric conversion elements of at least some of the multiple second pixel circuits within the same frame period.

15. A camera system according to claim 14, wherein the control unit offsets the timing at which the first imaging element samples the reset level charge of at least some of the multiple first pixel circuits within the same frame period, the timing at which the first imaging element samples the charge corresponding to the incident light photoelectrically converted by the first photoelectric conversion element, the timing at which the second imaging element samples the reset level charge of at least some of the multiple second pixel circuits within the same frame period, and the timing at which the second imaging element samples the charge corresponding to the incident light photoelectrically converted by the second photoelectric conversion element.

16. A camera system according to claim 1, wherein the control unit offsets the timing at which the first imaging element outputs pixel signals from at least some of the multiple first pixel circuits within the same frame period and the timing at which the second imaging element outputs pixel signals from at least some of the multiple second pixel circuits within the same frame period.

17. A camera system according to claim 16, wherein the control unit causes at least some of the plurality of first pixel circuits to output pixel signals within the same frame period, and then causes at least some of the plurality of second pixel circuits to output pixel signals within the same frame period.

18. The camera system according to claim 1, wherein the plurality of first pixel circuits are arranged in each of a first direction and a second direction, The plurality of second pixel circuits are arranged in each of the first direction and the second direction, and When outputting pixel signals in the same frame period, the first imaging element and the second imaging element alternately output pixel signals of respective pixel groups arranged in the second direction.

19. The camera system of claim 1, wherein the first imaging element and the second imaging element have the same number of pixels, and The first imaging element and the second imaging element perform exposure at the same exposure timing.

20. A control method for a camera system, the camera system comprising: a first imaging element including a plurality of first pixel circuits that simultaneously samples charges corresponding to photoelectrically converted incident light in each pixel and reads out the sampled charges in a non-destructive manner; a second imaging element including a plurality of second pixel circuits that simultaneously samples charges corresponding to the photoelectrically converted incident light in each pixel and reads out the sampled charges in a non-destructive manner; and a control unit for controlling the first imaging element and the second imaging element, wherein the first imaging element switches whether to output pixel signals from at least some of the plurality of first pixel circuits in the same frame period based on pixel signals output from some of the first pixel circuits or some of the second pixel circuits in each frame period, and The second imaging element switches whether to output pixel signals from at least some of the plurality of second pixel circuits within the same frame period based on pixel signals output from some of the first pixel circuits or some of the second pixel circuits in each frame period.

Images