Solid-state imaging device, image processing device, and imaging system

Abstract

This enables faster signal readout control tailored to different scenarios. [Solution] The solid-state imaging apparatus according to the embodiment includes a pixel unit that outputs long-storage images or short-storage images with different exposure times for each frame, a processing unit that processes the long-storage images or short-storage images for each frame, a detection unit that detects the brightness of the long-storage images or short-storage images for each frame, and a control unit that controls the processing unit for each frame based on brightness information relating to the brightness.

Description

Technical Field

[0001] The present disclosure relates to a solid-state imaging device, an image processing device, and an imaging system.

Background Art

[0002] In a solid-state imaging device (solid-state imaging element), there is a limitation in the dynamic range, and in a scene with a wide dynamic range, white blooming or black crushing occurs in the image. For this reason, a technique for capturing an HDR (High Dynamic Range) image with an expanded dynamic range has been developed. This technique synthesizes a plurality of images captured with different exposure times to generate an HDR image (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in order to obtain the plurality of images described above, signal reading is performed multiple times, but the drive change of the solid-state imaging device cannot follow the speed of change of the scene. For this reason, control of faster signal reading according to the scene is required.

[0005] Therefore, the present disclosure proposes a technique capable of realizing control of faster signal reading according to the scene.

Means for Solving the Problems

[0006] The solid-state imaging apparatus according to this embodiment includes a pixel unit that outputs long-exposure images or short-exposure images with different exposure times for each frame, a processing unit that processes the long-exposure images or short-exposure images for each frame, a detection unit that detects the brightness of the long-exposure images or short-exposure images for each frame, and a control unit that controls the processing unit for each frame based on brightness information relating to the brightness.

[0007] The image processing apparatus according to the embodiment includes a confirmation unit that confirms that image data of both or one of a long-exposure image and a short-exposure image with different exposure times have been input; a multiple image processing unit that synthesizes the long-exposure image and the short-exposure image when both of the long-exposure image and the short-exposure image have been input; and a single image processing unit that processes either the long-exposure image or the short-exposure image when one of the image data of the long-exposure image and the short-exposure image has been input.

[0008] The imaging system according to this embodiment includes a solid-state imaging device and an image processing device, wherein the solid-state imaging device has a pixel unit that outputs long-storage images or short-storage images with different exposure times for each frame, a processing unit that processes the long-storage image or short-storage image for each frame, a detection unit that detects the brightness of the long-storage image or short-storage image for each frame, a control unit that controls the processing unit based on brightness information for each frame, and an output unit that outputs image data of both or one of the long-storage image and the short-storage image, wherein the image processing device has a confirmation unit that confirms that the image data of both or one of the long-storage image and the short-storage image has been input, a multiple image processing unit that synthesizes the long-storage image and the short-storage image when both of the image data of the long-storage image and the short-storage image have been input, and a single image processing unit that processes the long-storage image or the short-storage image when one of the image data of the long-storage image or the short-storage image has been input. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example configuration of an imaging system according to the first embodiment. [Figure 2] This diagram illustrates the individual processes of the image brightness detection unit and the image brightness reflection control unit according to the first embodiment. [Figure 3] This figure illustrates the control of image brightness reflection when the brightness of a long-stored image is saturated, according to the first embodiment. [Figure 4] This figure illustrates the image brightness reflection control when the brightness of a long-stored image is not saturated, according to the first embodiment. [Figure 5] This figure shows the timing chart for image brightness reflection control according to the first embodiment. [Figure 6] This diagram shows the flow of image brightness reflection control according to the first embodiment. [Figure 7] This figure illustrates the analog signal from the sensor to the AP according to the first embodiment and comparative example. [Figure 8] This figure illustrates the output data from the sensor to the AP according to the first embodiment and comparative example. [Figure 9] This figure shows an example configuration of an imaging system according to the second embodiment. [Figure 10] This figure shows an example configuration of an imaging system according to the third embodiment. [Figure 11] This figure illustrates the image brightness reflection control when the amount of change in image brightness exceeds a threshold according to the third embodiment. [Figure 12] This figure illustrates the image brightness reflection control when the amount of change in image brightness according to the third embodiment is below a threshold. [Figure 13] This figure shows an example of the application of the aforementioned imaging system. [Figure 14] This figure shows an example configuration of an imaging device related to an application example. [Figure 15] This figure shows an example configuration of a distance measuring device related to an application example. [Modes for carrying out the invention]

[0010] The embodiments of the present disclosure will be described in detail below with reference to the drawings. The embodiments include examples and modifications. Note that the technology according to the present disclosure is not limited by the embodiments. Also, in the following embodiments, the same reference numerals are basically assigned to the same parts to omit redundant explanations.

[0011] The present disclosure will be described in accordance with the item order shown below. 1. First Embodiment 1-1. Configuration Example of Imaging System 1-2. Individual Processing Examples of Image Brightness Detection Unit and Image Brightness Reflection Control Unit 1-3. Example of Image Brightness Reflection Control 1-4. Example of Timing Chart of Image Brightness Reflection Control 1-5. Example of Flow of Image Brightness Reflection Control 1-6. Example of Output Signal or Output Data 2. Second Embodiment 3. Third Embodiment 4. Fourth Embodiment 5. Overview of Each Embodiment 6. Other Embodiments 7. Application Examples 7-1. Various Devices 7-2. Imaging Device 7-3. Distance Measuring Device 8. Supplementary Note

[0012] <1. First Embodiment> <1-1. Configuration Example of Imaging System> A configuration example of an imaging system 10 according to the first embodiment will be described with reference to FIG. 1. FIG. 1 is a diagram showing a configuration example of the imaging system 10 according to the first embodiment. In the example of FIG. 1, solid arrows are image processing lines, and dotted arrows are control lines.

[0013] As shown in FIG. 1, the imaging system 10 according to the first embodiment includes a sensor 20 and an AP (application processor) 30. The sensor 20 and the AP 30 are formed to be able to communicate with each other. The sensor 20 is an example of a solid-state imaging device, and the AP 30 is an example of an image processing device.

[0014] The sensor 20 includes a pixel unit 21, a processing unit 22, an output unit 23, a control unit 24, an image brightness detection unit 25, and an image brightness reflection control unit 26. The image brightness detection unit 25 is an example of a detection unit, and the image brightness reflection control unit 26 is an example of a control unit. For example, the image brightness reflection control unit 26 may be included in the control unit 24.

[0015] The pixel section 21 is composed of multiple pixels. Each pixel is arranged, for example, in a two-dimensional matrix. Each pixel has a photoelectric conversion unit that performs photoelectric conversion of incident light and generates a pixel signal based on the irradiated incident light. For example, a photodiode is used as the photoelectric conversion unit.

[0016] The pixel unit 21 supplies an image (pixel signal) of the subject or other object to the processing unit 22. For example, the pixel unit 21 alternately outputs long-exposure images and short-exposure images (individual pixel signals) with different exposure times for each frame. The long-exposure image is a long-exposure image (long-exposure signal) obtained with an exposure time predetermined. The short-exposure image is a short-exposure image (short-exposure signal) obtained with an exposure time shorter than the predetermined exposure time of the long-exposure image.

[0017] The processing unit 22 generates image data based on the image (pixel signal) transmitted from the pixel unit 21. For example, the processing unit 22 processes the image (pixel signal) frame by frame and generates image data for each frame. As an example, the processing unit 22 processes long-storage images or short-storage images (individual pixel signals) and generates individual image data for the long-storage images or short-storage images.

[0018] This processing unit 22 includes, for example, an AD conversion unit 221, a signal reordering unit 222, and an image signal processing unit 223. The AD conversion unit 221 converts analog data of pixel signals into digital data and outputs an image signal. The signal reordering unit 222 reorders the image signal output from the AD conversion unit 221 and outputs it. For example, the signal reordering unit 222 reorders the image signal according to the rolling shutter. This signal reordering unit 222 is not mandatory and is provided as needed. The image signal processing unit 223 processes the image signal output from the signal reordering unit 222. For example, the image signal processing unit 223 performs various processes on the image signal, such as correction processing specific to the sensor 20 (for example, including correction processing of the optical system) or adjustment processing.

[0019] The output unit 23 outputs image data (image signals) generated by the processing unit 22 to the external AP 30. For example, the output unit 23 outputs image data such as long-cycle images and short-cycle images transmitted from the processing unit 22 to the AP 30. The AP 30 receives the image data output from the output unit 23 and processes the received image data.

[0020] The control unit 24 controls each part. For example, the control unit 24 supplies control signals such as predetermined clock signals and timing signals to the pixel unit 21 in response to control signals from AP30, thereby controlling the driving of the pixel unit 21. The control unit 24 also supplies control signals such as predetermined clock signals and timing signals to the processing unit 22 (for example, the AD conversion unit 221, the signal reordering unit 222, and the image signal processing unit 223) and the output unit 23 in response to control signals from AP30, thereby controlling the driving of the processing unit 22 and the output unit 23, respectively. However, since the control unit 24 controls each part based on control signals from AP30 via communication such as I2C (Inter-Integrated Circuit) communication, high-speed processing is difficult.

[0021] The image brightness detection unit 25 detects the brightness of the image, which is the pixel signal output from the signal rearrangement unit 222, and generates brightness information related to the brightness of the image. For example, the image brightness detection unit 25 detects the brightness of each individual image, either a long-series or short-series image, for each frame and generates brightness information for each. The brightness information includes, for example, the brightness intensity of the image. The image brightness detection unit 25 receives the image signal output from the signal rearrangement unit 222, but it may also receive the image signal processed by, for example, the AD conversion unit 221 or the image signal processing unit 223.

[0022] The image brightness reflection control unit 26 controls the processing unit 22 and the output unit 23. For example, the image brightness reflection control unit 26 controls the driving of the processing unit 22 (e.g., the AD conversion unit 221, the signal reordering unit 222, and the image signal processing unit 223) and the output unit 23 based on the brightness information for each frame generated by the image brightness detection unit 25. Although the image brightness reflection control unit 26 may be included in the control unit 24 as described above, it is desirable that it be provided separately from the control unit 24 in order to achieve high-speed processing.

[0023] AP30 includes a data verification unit 31, a multiple image processing unit 32, a single image processing unit 33, and a signal generation unit 34. The data verification unit 31 is an example of a verification unit.

[0024] The data verification unit 31 verifies the image data transmitted from the output unit 23 of the sensor 20. The image data may include, for example, long-storage image data or short-storage image data. For example, the data verification unit 31 verifies that both or either of the long-storage and short-storage image data have been input based on the image data transmitted from the output unit 23.

[0025] The multiple image processing unit 32 processes the multiple images according to the verification result of the data verification unit 31. For example, if both long-range and short-range image data are input to the multiple image processing unit 32, it combines the long-range and short-range images and generates an HDR image with an expanded dynamic range from the long-range and short-range images. The multiple image processing unit 32 also performs processing on the HDR image according to the application (AP) use. In addition, the multiple image processing unit 32 may perform various processing on the multiple images, such as correction processing or adjustment processing specific to the sensor 20, instead of the image signal processing unit 223 of the sensor 20 described above.

[0026] The single image processing unit 33 processes the single image according to the confirmation result of the data confirmation unit 31. For example, if either a long-storage image or a short-storage image is input to the single image processing unit 33, it performs processing on either the long-storage image or the short-storage image in accordance with the AP's application. Alternatively, the single image processing unit 33 may perform, for example, correction or adjustment processing specific to the sensor 20 on either the long-storage image or the short-storage image, instead of the image signal processing unit 223 of the sensor 20 described above.

[0027] The signal generation unit 34 generates various signals. For example, the signal generation unit 34 generates signals such as control signals and transmits them to the control unit 24 of the sensor 20. The control unit 24 performs various processes (for example, imaging processing or changing imaging conditions) according to the signals transmitted from the signal generation unit 34. The signal generation unit 34 communicates with the sensor 20, for example, by I2C communication. I2C communication is an example of synchronous serial communication that transmits data in synchronization with a clock.

[0028] In an imaging system 10 with this configuration, the image brightness detection unit 25 detects brightness information from the image for each frame. The image brightness reflection control unit 26 transmits control signals (e.g., interrupt signals) to the processing unit 22 and output unit 23 in the sensor 20 based on the brightness information of a given frame. For example, the control signal is a signal that turns on or off the individual drives of the processing unit 22 and output unit 23 for the next frame. By changing the drives of both the processing unit 22 and output unit 23 using this on / off control signal, for example, the power consumption of the imaging system 10 (e.g., the entire camera set) can be reduced (details will be described later). Alternatively, the on / off control signal may be used to change the drive of only one of the processing unit 22 or output unit 23.

[0029] For example, the image brightness reflection control unit 26 may send a clock signal as a control signal to the processing unit 22 and the output unit 23 when it turns on the drive of both the processing unit 22 and the output unit 23 for the next frame, but may send a gating clock signal as a control signal to the processing unit 22 and the output unit 23 when it turns off the drive of both the processing unit 22 and the output unit 23 for the next frame (clock gating).

[0030] Furthermore, for example, if the image brightness reflection control unit 26 turns on the operation of both the processing unit 22 and the output unit 23 for the next frame, it connects the processing unit 22 and the output unit 23 to the power supply, but if it turns off the operation of both the processing unit 22 and the output unit 23 for the next frame, it may disconnect the processing unit 22 and the output unit 23 from the power supply (power gating).

[0031] Individual parts or all of the aforementioned sensors 20 and AP30 may be composed of, for example, either hardware and software, or both, and their configuration is not particularly limited. For example, individual parts of sensor 20 or AP30 may be implemented by integrated circuits such as ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), and CPLDs (Complex Programmable Logic Devices). Alternatively, individual parts of sensor 20 or AP30 may be implemented by a computer such as a CPU (Central Processing Unit), MCU (Micro Controller Unit), or MPU (Micro Processing Unit), where a program pre-stored in ROM (Read Only Memory) is executed using RAM (Random Access Memory) as a working area.

[0032] <1-2. Examples of individual processes in the image brightness detection unit and the image brightness reflection control unit> Examples of individual processes of the image brightness detection unit 25 and the image brightness reflection control unit 26 according to the first embodiment will be described with reference to Figure 2. Figure 2 is a diagram illustrating the individual processes of the image brightness detection unit 25 and the image brightness reflection control unit 26 according to the first embodiment.

[0033] As shown in Figure 2, the image brightness detection unit 25 divides an image (RAW image), such as a long-storage image or a short-storage image, into multiple regions A1 (step S1). Each region A1 is divided, for example, into a two-dimensional matrix. The image brightness detection unit 25 calculates the brightness (y_i) for each region A1 (step S2). The brightness (y_i) of region A1 is, for example, the average value of the brightness (pixel value) of each pixel in region A1.

[0034] The image brightness reflection control unit 26 determines whether the brightness (y_i) of each region A1 is greater than or equal to a predetermined first threshold Y (step S3). Furthermore, the image brightness reflection control unit 26 determines whether the total number of regions A1 where the brightness (y_i) is greater than or equal to the predetermined first threshold Y is greater than or equal to a predetermined second threshold Z (step S4). Regions A1 where the brightness (y_i) is greater than or equal to the predetermined first threshold Y are brightness saturation regions. Brightness saturation regions are, for example, regions where highlights are blown out.

[0035] Subsequently, if the total number of regions A1 where the luminance (y_i) is greater than or equal to a predetermined first threshold Y is greater than or equal to a predetermined second threshold Z, the image brightness reflection control unit 26 treats the current scene as a scene with many saturated regions in the image, permits the operation of the processing unit 22 and the output unit 23, and reads the image of the next frame. On the other hand, if the total number of regions A1 where the luminance (y_i) is greater than or equal to a predetermined first threshold Y is less than a predetermined second threshold Z, the image brightness reflection control unit 26 treats the current scene as a scene with few saturated regions in the image, restricts the operation of the processing unit 22 and the output unit 23, and does not read the image of the next frame.

[0036] As described above, the image brightness reflection control unit 26 determines the brightness saturation region of the image for each frame based on the brightness of each region A1 for each frame, and limits the operation of the processing unit 22 and the output unit 23 according to the size (area) of the brightness saturation region for each frame determined. Specifically, the image brightness reflection control unit 26 determines whether the brightness (y_i) of each region A1 is greater than or equal to a predetermined first threshold Y, determines the total number of regions A1 whose brightness is greater than or equal to the first threshold Y as the size of the brightness saturation region, and limits the operation of the processing unit 22 and the output unit 23 according to whether the total number determined is greater than or equal to a predetermined second threshold Z.

[0037] <1-3. An example of image brightness reflection control> An example of image brightness reflection control according to the first embodiment will be described with reference to Figures 3 and 4. Figure 3 is a diagram illustrating image brightness reflection control when the brightness of a long-stored image is saturated (long-stored image saturation) according to the first embodiment. Figure 4 is a diagram illustrating image brightness reflection control when the brightness of a long-stored image is not saturated (long-stored image non-saturation) according to the first embodiment. In the examples of Figures 3 and 4, as in the example of Figure 1, solid arrows are image processing lines and dotted arrows are control lines.

[0038] As shown in Figures 3 and 4, light enters the pixel section 21 whether the long-storage image is saturated or unsaturated, and the AD conversion section 221 and the signal rearrangement section 222 read out the signal of the long-storage image of the target frame (long-storage frame). This signal of the long-storage image is input to the image signal processing section 223 and the image brightness detection section 25. The signal input to the image signal processing section 223 is output to the subsequent AP30 through the image signal processing section 223 and the output section 23 (output of long-storage image data).

[0039] Meanwhile, the signal input to the image brightness detection unit 25 is converted into brightness information by the image brightness detection unit 25, and this brightness information is input to the image brightness reflection control unit 26. Based on the brightness information, the image brightness reflection control unit 26 transmits control signals (for example, on / off control signals) to the processing unit 22 and the output unit 23, respectively, which determine the drive within the sensor 20 for the short image of the next frame (short image frame) following the aforementioned target frame. Note that the drive within the sensor 20 refers to, for example, the drive of the processing unit 22 and the output unit 23, respectively.

[0040] When the long-storage image is saturated, as shown in Figure 3, the image brightness reflection control unit 26 in the sensor 20 transmits a control signal (an ON control signal) to the AD conversion unit 221, the signal reordering unit 222, the image signal processing unit 223, and the output unit 23, thereby allowing them to operate. The image brightness reflection control unit 26 drives the AD conversion unit 221, the signal reordering unit 222, the image signal processing unit 223, and the output unit 23 to perform image processing on the short-storage image of the next frame of the long-storage image and output data to the subsequent stage (output of short-storage image data). In AP30, the data verification unit 31 verifies the image data, and the multiple image processing units 32 are driven according to the verification result.

[0041] On the other hand, when the long-saturated image is not saturated, as shown in Figure 4, the image brightness reflection control unit 26 in the sensor 20 transmits a control signal (off control signal) to the AD conversion unit 221, signal reordering unit 222, image signal processing unit 223, and output unit 23, restricting the operation of these units and preventing image processing of the short-saturated image of the next frame of the long-saturated image and data output to subsequent stages (non-output of short-saturated image data). In AP30, the data verification unit 31 verifies the image data, and the single image processing unit 33 is driven according to the verification result.

[0042] With this image brightness reflection control, the image brightness detection unit 25 detects the brightness of the target frame and transmits the detection result to the image brightness reflection control unit 26. The image brightness reflection control unit 26 adaptively changes the drive within the sensor 20 in the next frame after the target frame according to the aforementioned detection result. This makes it possible to change the drive within the sensor 20 according to the brightness of the scene. For example, it becomes possible to reduce power consumption due to unnecessary drive within the sensor 20 that does not correspond to the brightness of the scene, thereby reducing the power consumption of the imaging system 10 (for example, the entire camera set).

[0043] In the above explanation, an example was given in which the operation of each part within the sensor 20 (for example, the processing unit 22 and the output unit 23) for the short-storage image is turned on / off based on the brightness information of the long-storage image. However, the operation of each part within the sensor 20 for the long-storage image may also be turned on / off based on the brightness information of the short-storage image.

[0044] Furthermore, although the above explanation described an example where no data is output from sensor 20 to AP30, for example, dummy data may be output from sensor 20 to AP30 for consistency with AP30 (details will be provided later). Even in this case, the power consumption of the imaging system 10 (for example, the entire camera set) can be reduced.

[0045] <1-4. An example of a timing chart for image brightness reflection control> An example of a timing chart for image brightness reflection control according to the first embodiment will be described with reference to Figure 5. Figure 5 is a diagram showing the timing chart for image brightness reflection control according to the first embodiment. In the example in Figure 5, short-storage B1 shows the pixel signal (short-storage pixel signal) from shutter operation to read operation, and long-storage B2 shows the pixel signal (long-storage pixel signal) from shutter operation to read operation.

[0046] As shown in Figure 5, the processing unit 22 reads out the long-saturated pixel signal (long-saturated image) of the Nth frame, and the brightness information of the long-saturated image is generated by the image brightness detection unit 25. If the long-saturated image is an image with many saturated regions (for example, if the total number of the long-saturated images is greater than or equal to a predetermined second threshold Z), the image brightness reflection control unit 26 outputs a control signal (an ON control signal). In response to this control signal, processing is performed on the short-saturated image of the N+1th frame, and the image data of that short-saturated image is output to AP30 (data + common blank).

[0047] Furthermore, the processing unit 22 reads out the long-saturated image signal (long-saturated image) of the N+2 frame, and the brightness information of that long-saturated image is generated by the image brightness detection unit 25. If the long-saturated image is an image with many unsaturated regions (for example, if the total number of the long-saturated image is smaller than a predetermined second threshold Z), the image brightness reflection control unit 26 outputs a control signal (off control signal). In response to this control signal, processing of the short-saturated image of the N+3 frame is not performed, and the image data of that short-saturated image is not output to AP30 (blank + common blank).

[0048] <1-5. An example of the flow of image brightness reflection control> An example of the image brightness reflection control flow according to the first embodiment will be described with reference to Figure 6. Figure 6 is a diagram showing the image brightness reflection control flow according to the first embodiment.

[0049] As shown in Figure 6, in step S11, the sensor 20 reads a long-storage image (pixel signal) from the pixel unit 21 and outputs the long-storage image to the subsequent AP30 through the processing unit 22 (AD conversion unit 221, signal reordering unit 222, image signal processing unit 223) and the output unit 23. The signal reordering unit 222 also transmits the long-storage image to the image brightness detection unit 25.

[0050] In step S12, the image brightness detection unit 25 detects the brightness of the long-stored image transmitted from the signal rearrangement unit 222. For example, the image brightness detection unit 25 performs a process to convert the signal value of the long-stored image into brightness intensity (for example, brightness for each pixel) and transmits the brightness information related to that brightness intensity to the image brightness reflection control unit 26.

[0051] In step S13, the image brightness reflection control unit 26 performs brightness detection result processing based on the brightness information (saturated or unsaturated). For example, the image brightness reflection control unit 26 determines, based on the brightness information, whether the long-storage image is an image with many saturated regions or an image with many unsaturated regions. Based on the above determination result, the image brightness reflection control unit 26 creates control signals (on / off control signals) to turn on / off the driving of the processing unit 22 and the output unit 23, respectively, in the next frame.

[0052] In step S14, the image brightness reflection control unit 26 performs brightness reflection control. For example, the image brightness reflection control unit 26 transmits the control signal created in step S13 to the processing unit 22 and the output unit 23. If the long-storage image is an image with many saturated regions, the image brightness reflection control unit 26 transmits an ON control signal to the processing unit 22 and the output unit 23, allowing the processing unit 22 and the output unit 23 to operate. On the other hand, if the long-storage image is an image with many unsaturated regions, the image brightness reflection control unit 26 transmits an OFF control signal to the processing unit 22 and the output unit 23, restricting the operation of the processing unit 22 and the output unit 23.

[0053] If the long-storage image is an image with many saturated regions, in step S15, the sensor 20 reads the short-storage image of the next frame from the pixel unit 21 and outputs the image data of the short-storage image to the subsequent AP30 via the processing unit 22 and output unit 23. Subsequently, the multiple image processing unit 32 of the AP30 is activated and combines the long-storage image and the short-storage image to generate an HDR image.

[0054] On the other hand, if the long-storage image is an image with many unsaturated regions, in step S16, the sensor 20 does not read the short-storage image of the next frame, does not perform short-storage image processing, and does not output the image data of the short-storage image to the subsequent AP30. After that, the single-image processing unit 33 of AP30 is driven to generate an image based on the long-storage image.

[0055] <1-6. Example of output signal or output data> An example of an output signal or output data according to the first embodiment will be described with reference to Figures 7 and 8. The output signal is, for example, an analog signal. Figure 7 is a diagram illustrating the analog signal from sensor 20 to AP30 according to the first embodiment and comparative example. Figure 8 is a diagram illustrating the output data from sensor 20 to AP30 according to the first embodiment and comparative example.

[0056] (Output signal) As shown in Figure 7, the analog signal C1, which is the output signal in the comparative example, repeatedly includes the waveform of one frame of image data. On the other hand, the analog signal C2, which is the output signal in the first embodiment, repeatedly includes the waveform of one frame of image data, but there is a period in which it does not include the waveform of one frame of image data. This period is, for example, a period in which it does not include the waveform of the image data of one frame of short image. According to the analog signal C2 in the first embodiment, there is a period in which no short image is output to AP30, so for example, the data toggle period may be reduced or shortened. Note that each of the analog signals C1 and C2 can be observed, for example, with an oscilloscope.

[0057] When such an analog signal C2 is detected between sensor 20 and AP30, it can be seen that the image data transmitted from sensor 20 to AP30 is not transmitted continuously. Therefore, the discontinuity of the image data can be used to confirm whether or not the aforementioned image brightness reflection control is being performed.

[0058] (Output data) As shown in Figure 8, the frame data D10 in the comparative example includes embedded data D11, data D12 such as OPB (black level data), and image data D13. On the other hand, the frame data D20 in the first embodiment includes embedded data D21 with a skip flag and blank D22. The embedded data D21 with a skip flag is data in which a flag data indicating that it is skipped data has been inserted into the embedded data. The embedded data is packet data output from sensor 20 to AP30.

[0059] Furthermore, another frame data D30 according to the first embodiment includes embedded data D31 and dummy data D32. Another frame data D40 includes dummy data D41. In either frame data D30 or frame data D40, the skipped data is filled with dummy data D32 or dummy data D41.

[0060] By examining the output data of each frame data D20, D30, and D40 (for example, the image packet data), it can be determined that there is no image data (for example, image data of a short image) being transmitted from sensor 20 to AP30. Therefore, it is possible to confirm whether or not the aforementioned image brightness reflection control is being executed.

[0061] <2. Second Embodiment> An example configuration of the imaging system 10A according to the second embodiment will be described with reference to Figure 9. Figure 9 is a diagram showing an example configuration of the imaging system 10A according to the second embodiment.

[0062] As shown in Figure 9, the imaging system 10A according to the second embodiment comprises a plurality of sensors 20 and one AP 30. This imaging system 10A is a compound eye imaging system. Each of the sensors 20 and AP 30 is configured to communicate with each other.

[0063] In the individual driving of each sensor 20 (compound eye driving), the imaging system 10A uses the sensor 20 with higher sensitivity in the next frame when there are many non-saturated regions in the image (when the total number is less than a predetermined second threshold Z). For example, the main sensor 20 in the imaging system 10A selects and drives the sensor 20 with the highest sensitivity from among the multiple sensors 20 (including the main sensor 20) in the next frame.

[0064] Furthermore, in scenes with many saturated areas in the image (when the total number mentioned above is greater than or equal to a predetermined second threshold Z), the imaging system 10A uses a sensor 20 with lower sensitivity in the next frame. For example, the main sensor 20 in the imaging system 10A selects and drives the sensor 20 with the lowest sensitivity from among multiple sensors 20 (including the main sensor 20) in the next frame.

[0065] According to this second embodiment, it is possible to select and use an appropriate sensor 20 from a plurality of sensors 20 with different sensitivities depending on the scene, and to acquire an image for HDR synthesis with appropriate brightness.

[0066] In the first embodiment described above, the image brightness reflection control unit 26 controls the driving of the processing unit 22 based on brightness information. However, for example, the imaging conditions of the pixel unit 21 or the image processing of the processing unit 22 may be changed based on brightness information. Imaging conditions include, for example, shutter timing and exposure time. Image processing includes, for example, multiplex AD conversion processing and analog gain adjustment processing.

[0067] For example, the image brightness reflection control unit 26 updates the shutter operation for the next frame and takes another shot. This changes the exposure time. The scenes that can be updated are, for example, scenes where a shorter exposure time than the original setting is optimal. When updating the shutter operation, for example, the image brightness reflection control unit 26 outputs a control signal to the pixel unit 21 instructing it to update the shutter operation. In this way, by updating the shutter operation according to the scene (for example, changing the exposure time), an image for HDR synthesis with the appropriate brightness can be obtained.

[0068] Furthermore, in scenes with many saturated areas in the image, multiple AD conversion is performed to increase the signal resolution. Multiple AD conversion refers to, for example, performing AD conversion a predetermined number of times. On the other hand, in scenes with many unsaturated areas in the image, AD conversion is performed only once without multiple AD conversion. The image brightness reflection control unit 26, for example, outputs a control signal to the AD conversion unit 221 instructing it to perform multiple AD conversion when multiple AD conversion is to be performed. In this way, by performing multiple AD conversion according to the scene, an image for HDR synthesis with appropriate brightness can be obtained.

[0069] Furthermore, in scenes with a large amount of saturated image regions, the analog gain is reduced in the next frame. On the other hand, in scenes with a large amount of unsaturated image regions, the analog gain is not changed in the next frame. For example, when the image brightness reflection control unit 26 reduces the analog gain in the next frame, it outputs a control signal to the image signal processing unit 223 instructing it to reduce the analog gain. In this way, by adjusting the analog gain according to the scene, an image with appropriate brightness for HDR synthesis can be obtained.

[0070] <3. Third Embodiment> An example configuration of the imaging system 10 according to the third embodiment will be described with reference to Figures 10 to 12. Figure 10 is a diagram showing an example configuration of the imaging system 10 according to the third embodiment. Figure 11 is a diagram illustrating the image brightness reflection control when the amount of change in image brightness according to the third embodiment exceeds a threshold. Figure 12 is a diagram illustrating the image brightness reflection control when the amount of change in image brightness according to the third embodiment is less than or equal to a threshold. In the examples of Figures 10 to 12, as in the example of Figure 1, solid arrows are image processing lines and dotted arrows are control lines.

[0071] In the third embodiment, unlike the first embodiment described above, the amount of change in image brightness between frames is used. In other words, the detection process of the image brightness detection unit 25 is different from that of the first embodiment described above.

[0072] As shown in Figure 10, the image brightness detection unit 25 detects the amount of change in image brightness for each frame and generates brightness information, i.e., brightness change information, related to the amount of change in brightness. For example, if a short image has N frames, the image brightness detection unit 25 detects the amount of change in brightness between the Nth frame and the N+1th frame (the long image frame). The image brightness detection unit 25 transmits the brightness change information, which indicates the amount of change in brightness, to the image brightness reflection control unit 26. Based on the brightness change information, the image brightness reflection control unit 26 changes the individual drives of the processing unit 22 and the output unit 23 in the N+2th frame (the short image frame). This controls the on / off of image data output according to the scene.

[0073] The image brightness detection unit 25 can read the images of the Nth frame and the N+1th frame consecutively and detect the change in brightness between the Nth frame and the N+1th frame. However, to detect the change in brightness between the Nth frame and the N+1th frame, it may also have a storage unit that temporarily stores the image of the Nth frame as needed.

[0074] As shown in Figures 11 and 12, light enters the pixel unit 21, and the AD conversion unit 221 and the signal rearrangement unit 222 read out the image signals of the Nth frame and the (N+1)th frame. The readout results are input to the image signal processing unit 223 and the image brightness detection unit 25. The signals input to the image signal processing unit 223 are output to the subsequent AP30 through the image signal processing unit 223 and the output unit 23.

[0075] Meanwhile, the signal input to the image brightness detection unit 25 is converted into brightness change information by the image brightness detection unit 25, and this brightness change information is input to the image brightness reflection control unit 26. Based on the brightness change information, the image brightness reflection control unit 26 transmits a control signal (for example, an on / off control signal) to the processing unit 22 and the output unit 23, respectively, which determines the drive within the sensor 20 for the N+2 frame short image.

[0076] When the amount of change in brightness exceeds a predetermined threshold (when the change in scene brightness is large), as shown in Figure 11, the image brightness reflection control unit 26 in the sensor 20 transmits a control signal (an ON control signal) to the AD conversion unit 221, the signal reordering unit 222, the image signal processing unit 223, and the output unit 23, thereby authorizing the operation of the AD conversion unit 221, the signal reordering unit 222, the image signal processing unit 223, and the output unit 23. The image brightness reflection control unit 26 drives the AD conversion unit 221, the signal reordering unit 222, the image signal processing unit 223, and the output unit 23 to perform image processing on the N+2 frame short image and output data to the next stage (output of the N+2 frame short image data). In AP30, the data verification unit 31 verifies the image data, and the multiple image processing units 32 are driven according to the verification result.

[0077] On the other hand, if the amount of change in brightness is below a predetermined threshold (when the change in scene brightness is small), as shown in Figure 12, the sensor 20's image brightness reflection control unit 26 transmits a control signal (off control signal) to the AD conversion unit 221, signal reordering unit 222, image signal processing unit 223, and output unit 23, restricting the operation of these units and preventing image processing and data output to subsequent stages for the N+2 frame short image (non-output of the N+2 frame short image data). In AP30, the data verification unit 31 verifies the image data, and the single image processing unit 33 is driven according to the verification result.

[0078] According to this third embodiment, the image brightness detection unit 25 detects the change in brightness between the Nth frame and the N+1th frame and transmits the detection result to the image brightness reflection control unit 26. The image brightness reflection control unit 26 adaptively changes the drive within the sensor 20 in the N+2nd frame according to the aforementioned detection result. This makes it possible to change the drive within the sensor 20 in accordance with the rapid changes in the scene. For example, it becomes possible to reduce power consumption due to unnecessary drive within the sensor 20 that does not correspond to the brightness of the scene, thereby reducing the power consumption of the imaging system 10 (for example, the entire camera set).

[0079] In the third embodiment, the image brightness reflection control unit 26 controls the operation of the processing unit 22 based on brightness information indicating the amount of brightness change, i.e., brightness change information. However, for example, the image processing of the processing unit 22 may be changed based on the brightness change information.

[0080] For example, the processing unit 22 may crop and read only the areas in the image where the scene brightness changes significantly in the next frame, and perform processing only on those areas. Cropping means cutting out a part of the image. Alternatively, the processing unit 22 may, for example, decimate areas in the next frame where the scene brightness changes little (for example, scene movement), or it may not read out the signal in those areas at all. Or, the processing unit 22 may read out the short image only when the amount of brightness change exceeds a predetermined threshold.

[0081] <4. Fourth Embodiment> A configuration example of the imaging system 10A according to the fourth embodiment will be described with reference to Figure 9. The configuration example of the imaging system 10A according to the fourth embodiment is basically the same as the configuration example of the imaging system 10A according to the second embodiment (see Figure 9).

[0082] However, in the fourth embodiment, unlike the second embodiment described above, the amount of change in image brightness between frames is used, similar to the third embodiment described above. In other words, the detection process of the image brightness detection unit 25 is the same as in the third embodiment described above, unlike the second embodiment described above.

[0083] In each sensor 20, the image brightness detection unit 25 detects the amount of change in image brightness for each frame, similar to the third embodiment described above, and generates brightness information related to the amount of change in brightness, i.e., brightness change information. The image brightness reflection control unit 26 changes the drive within the sensor 20 for the next frame based on the brightness change information.

[0084] For example, in scenes with a lot of subject movement (when the amount of change in brightness exceeds a predetermined threshold), the main sensor 20 of the imaging system 10A may activate another sensor 20 with a wider field of view than its own in the next frame, and perform imaging with multiple sensors 20 (multi-eye drive). This makes it possible to obtain an image for HDR synthesis with fewer artifacts.

[0085] Furthermore, for example, in scenes where the subject movement is minimal (when the amount of change in brightness is below a predetermined threshold), the main sensor 20 of the imaging system 10A may stop all other sensors 20 in the next frame and perform imaging with a single sensor 20 (monocular drive). This reduces the power consumption of the imaging system 10A (for example, the entire camera set).

[0086] According to this fourth embodiment, the main sensor 20 appropriately changes the driving of other sensors 20 depending on whether the scene has a lot of movement or little movement of the subject, thereby enabling the acquisition of HDR composite images with fewer artifacts and reducing the power consumption of the imaging system 10A.

[0087] In the fourth embodiment, the image brightness reflection control unit 26 of the main sensor 20 controls the driving of another sensor 20 based on brightness change information. However, for example, the image processing of the other sensor 20 (for example, various processing of the processing unit 22) may be changed based on brightness change information.

[0088] For example, in scenes with a lot of movement of the subject, the processing unit 22 may perform high-speed readout, such as additive readout or decimal readout, as part of image processing in the next frame. Additive readout is the process of reading out each pixel value by treating multiple pixels as a single pixel. Decimal readout is the process of reading out each pixel value from a predetermined number of pixels, rather than all pixels, based on predetermined decimation conditions, for example.

[0089] <5. Overview of Each Embodiment> As described above, the solid-state imaging device (e.g., sensor 20) according to each embodiment includes a pixel unit 21 that outputs long-storage images or short-storage images with different exposure times for each frame, a processing unit 22 that processes the long-storage images or short-storage images for each frame, a detection unit (e.g., image brightness detection unit 25) that detects the brightness of the long-storage images or short-storage images for each frame, and a control unit (e.g., image brightness reflection control unit 26) that controls the processing unit 22 based on brightness information for each frame (see Figure 1, etc.). This makes it possible to control the processing unit 22 according to the brightness of the long-storage images or short-storage images, i.e., the brightness of the scene (e.g., scene changes), thereby enabling faster signal readout control according to the scene.

[0090] Furthermore, the control unit (for example, the image brightness reflection control unit 26) may control the processing unit 22 in the frame following the target frame based on the brightness information of the target frame (see Figures 3 to 6). This ensures that faster signal readout control according to the scene can be reliably achieved.

[0091] Furthermore, the control unit (for example, the image brightness reflection control unit 26) may restrict the operation of the processing unit 22 in the frame following the target frame based on the brightness information of the target frame (see Figures 3 to 6). This ensures reliable control according to the scene. For example, since it becomes possible to restrict the operation of the processing unit 22 according to the scene, power saving can be achieved.

[0092] Furthermore, the pixel unit 21 may alternately output long-storage images and short-storage images, and the control unit (for example, the image brightness reflection control unit 26) may restrict the operation of the processing unit 22 in the frame following the target frame based on the brightness information of the long-storage or short-storage image of the target frame (see Figures 3 to 6). This makes it possible to reliably achieve, for example, power saving.

[0093] Furthermore, the control unit (for example, the image brightness reflection control unit 26) may limit the operation of the processing unit 22 by restricting the reading of long-storage or short-storage images by the processing unit 22 (see Figures 3 to 6). This ensures, for example, power saving.

[0094] Furthermore, the control unit (for example, the image brightness reflection control unit 26) may limit the operation of the processing unit 22 by clock gating or power gating (see Figures 3 to 6). This ensures, for example, power saving.

[0095] Furthermore, the solid-state imaging device (e.g., sensor 20) may further include an output unit 23 that outputs image data of both or either long-storage and short-storage images, and the control unit (e.g., image brightness reflection control unit 26) may limit the operation of the output unit 23 in the frame following the target frame based on the brightness information of the target frame (see Figures 3 to 7). This makes it possible to reliably achieve, for example, power saving.

[0096] Furthermore, the solid-state imaging device (e.g., sensor 20) may also include an output unit 23 that outputs dummy data (e.g., dummy data D32, dummy data D41) as image data for the long-storage or short-storage image of the next frame when the control unit (e.g., image brightness reflection control unit 26) limits the driving of the processing unit 22 (see Figure 8). This eliminates the need for the subsequent AP30 to perform processing corresponding to the absence of image data input, thus enabling high-speed processing.

[0097] Furthermore, brightness information includes brightness intensity or the amount of change in brightness, and the control unit (for example, the image brightness reflection control unit 26) may control the processing unit 22 according to the brightness intensity or the amount of change in brightness (see Figures 3 to 6 and 10 to 12). This ensures that faster signal readout control according to the scene can be reliably achieved.

[0098] Furthermore, the control unit (for example, the image brightness reflection control unit 26) may detect moving objects in the long-storage image or short-storage image according to the amount of change in brightness (see Figures 10 to 12). This ensures that faster signal readout control according to the scene can be reliably achieved.

[0099] Furthermore, the control unit (for example, the image brightness reflection control unit 26) may change the imaging conditions of the pixel unit 21 in the frame following the target frame based on the brightness information of the target frame. This ensures reliable control according to the scene. For example, by changing the imaging conditions according to the scene, it becomes possible to obtain an appropriate image for HDR synthesis, thereby achieving higher image quality.

[0100] Furthermore, the control unit (for example, the image brightness reflection control unit 26) may change the image processing of the processing unit 22 in the frame following the target frame based on the brightness information of the target frame. This ensures reliable control according to the scene. For example, by changing the image processing according to the scene, it becomes possible to obtain an appropriate image for HDR synthesis, thereby achieving higher image quality.

[0101] Furthermore, the control unit (for example, the image brightness reflection control unit 26) may determine the brightness saturation region of the image based on brightness information for each frame and control the processing unit 22 according to the size of the brightness saturation region for each frame (see Figure 2). This ensures that faster signal readout control according to the scene can be reliably achieved.

[0102] Alternatively, the detection unit (for example, the image brightness detection unit 25) may divide the image into multiple regions A1 for each frame, determine the brightness of each of the multiple regions A1, and the control unit (for example, the image brightness reflection control unit 26) may determine the brightness saturation region of the image based on the brightness of each of the multiple regions A1 for each frame, and control the processing unit 22 according to the size of the brightness saturation region (see Figure 2). This ensures that faster signal readout control according to the scene can be reliably achieved.

[0103] Furthermore, the control unit (for example, the image brightness reflection control unit 26) may determine whether the brightness of each of the multiple regions A1 is equal to or greater than a predetermined first threshold Y, calculate the total number of regions A1 whose brightness is equal to or greater than the first threshold Y as the size of the brightness saturation region, and control the processing unit 22 according to whether the total number is equal to or greater than a predetermined second threshold Z (see Figure 2). This ensures that faster signal readout control according to the scene can be reliably achieved.

[0104] Furthermore, the image processing device (e.g., AP30) includes a confirmation unit (e.g., data confirmation unit 31) that confirms that image data of either or both long-exposure and short-exposure images with different exposure times have been input, a multiple image processing unit 32 that combines the long-exposure and short-exposure images when both are input, and a single image processing unit 33 that processes either the long-exposure or short-exposure image when only one of the long-exposure or short-exposure images is input (see Figure 1, etc.). This enables faster signal readout control according to the scene.

[0105] Furthermore, multiple solid-state imaging devices (e.g., sensor 20) are provided, and the control unit (e.g., image brightness reflection control unit 26) may change the individual drive of the multiple solid-state imaging devices according to the brightness information of each solid-state imaging device (see Figure 9). This ensures that faster signal readout control according to the scene can be reliably achieved.

[0106] Furthermore, the control unit (for example, the image brightness reflection control unit 26) may drive the solid-state imaging device with the highest or lowest sensitivity among the multiple solid-state imaging devices (for example, the sensor 20) according to the brightness information of each solid-state imaging device (see Figure 9). This ensures that faster signal readout control according to the scene can be reliably achieved.

[0107] Furthermore, the control unit (for example, the image brightness reflection control unit 26) may drive the solid-state imaging device with the widest or narrowest field of view among the multiple solid-state imaging devices (for example, the sensor 20) according to the brightness information of each solid-state imaging device (see Figure 9). This ensures that faster signal readout control according to the scene can be reliably achieved.

[0108] <6. Other Embodiments> Each configuration and process relating to the above-described embodiments (including examples and modifications) may be implemented in various different forms other than those described above. For example, each configuration and process may be in various forms, not limited to the examples described above. Furthermore, information such as the configurations, processing procedures, specific names, and various data and parameters shown in the above document and drawings may be arbitrarily changed unless otherwise specified.

[0109] Furthermore, the configurations and processes described in the above-mentioned embodiments (including examples and modifications) do not necessarily have to be physically configured as shown in the illustrations. In other words, the specific forms of distribution and integration of each configuration and process are not limited to those shown in the illustrations, and all or part of them may be functionally or physically distributed and integrated in any unit depending on various loads and usage conditions.

[0110] Furthermore, the various configurations and processes described in the above-mentioned embodiments (including examples and modifications) may be combined as appropriate. For example, at least a part of one embodiment may be combined with at least a part of another embodiment as appropriate. Also, the effects described in the embodiments are merely illustrative and not limiting, and other effects may also occur.

[0111] <7. Examples of Application> <7-1. Various devices> Examples of applications of the aforementioned imaging system 10 (including imaging system 10A) will be explained with reference to Figure 13. Figure 13 is a diagram showing examples of applications of the aforementioned imaging system 10. The imaging system 10 may be applied to various cases, such as various devices (an example of electronic equipment).

[0112] As shown in Figure 13, the imaging system 10 can be used in, for example, "devices that capture images for viewing purposes, such as digital cameras and portable devices with camera functions," "devices used for traffic purposes, such as in-vehicle sensors that capture images of the front, rear, surroundings, and interior of a vehicle for safe driving such as automatic stopping and recognition of the driver's condition, surveillance cameras that monitor moving vehicles and roads, and distance measuring sensors that measure distances between vehicles," "devices used in home appliances such as TVs, refrigerators, and air conditioners to capture user gestures and operate the device according to those gestures," "devices used for medical and healthcare purposes, such as endoscopes and devices that perform angiography by receiving infrared light," "devices used for security purposes, such as surveillance cameras for crime prevention and cameras for person recognition," "devices used for beauty purposes, such as skin measuring devices that capture images of the skin and microscopes that capture images of the scalp," "devices used for sports purposes, such as action cameras and wearable cameras for sports use," and "devices used for agriculture, such as cameras for monitoring the condition of fields and crops."

[0113] Furthermore, the technology disclosed herein can be applied to a variety of products. For example, the technology disclosed herein may be implemented as electronic equipment mounted on any type of mobile device, such as automobiles, electric vehicles, hybrid electric vehicles, motorcycles, bicycles, personal mobility devices, airplanes, drones, ships, robots, construction machinery, or agricultural machinery (tractors). Alternatively, for example, the technology disclosed herein may be implemented as electronic equipment mounted on endoscopic surgical systems or microsurgical systems.

[0114] <7-2. Imaging Equipment> The imaging device 1000 relating to the application example will be described with reference to Figure 14. Figure 14 is a diagram showing an example configuration of the imaging device 1000 relating to the application example. This imaging device 1000 is an example of an electronic device to which the aforementioned imaging system 10 is applied. Examples of electronic devices such as digital still cameras, video cameras, smartphones, and mobile phones can be used as the imaging device 1000.

[0115] As shown in Figure 14, the imaging device 1000 comprises an optical system 1001, a shutter device 1002, an image sensor (solid-state imager) 1003, a control circuit (drive circuit) 1004, a signal processing circuit 1005, a monitor 1006, and a memory 1007. This imaging device 1000 is capable of capturing both still and moving images.

[0116] The optical system 1001 has one or more lenses. This optical system 1001 guides light from the subject (incident light) to the image sensor 1003 and forms an image on the light-receiving surface of the image sensor 1003.

[0117] The shutter device 1002 is positioned between the optical system 1001 and the image sensor 1003. This shutter device 1002 controls the light illumination period and the light shielding period for the image sensor 1003 according to the control of the control circuit 1004.

[0118] The image sensor 1003 accumulates signal charge for a certain period of time in response to light that is imaged onto the light-receiving surface via the optical system 1001 and shutter device 1002. The signal charge accumulated in the image sensor 1003 is transferred according to a drive signal (timing signal) supplied from the control circuit 1004.

[0119] The control circuit 1004 drives the image sensor 1003 and the shutter device 1002 by outputting drive signals that control the transfer operation of the image sensor 1003 and the shutter operation of the shutter device 1002.

[0120] The signal processing circuit 1005 performs various signal processing operations on the signal charge output from the image sensor 1003. The image (image data) obtained by the signal processing circuit 1005 is supplied to the monitor 1006 and also to the memory 1007.

[0121] The monitor 1006 displays a video or still image captured by the image sensor 1003 based on image data supplied from the signal processing circuit 1005. For example, the monitor 1006 may be a panel-type display device such as a liquid crystal panel or an organic EL (Electro-Luminescence) panel.

[0122] The memory 1007 stores image data supplied from the signal processing circuit 1005, that is, image data of moving or still images captured by the image sensor 1003. Various types of storage devices can be used as the memory 1007.

[0123] In the imaging device 1000 configured in this way, the same effects as in the above embodiment can be obtained by applying the aforementioned imaging system 10.

[0124] <7-3. Distance measuring device> The distance measuring device 2000 in the application example will be described with reference to Figure 15. Figure 15 is a diagram showing an example configuration of the distance measuring device 2000 in the application example. This distance measuring device 2000 is an example of an electronic device to which the aforementioned imaging system 10 is applied.

[0125] As shown in Figure 15, the distance measuring device (distance image sensor) 2000 comprises a light source unit 2001, an optical system 2002, an image sensor (solid-state imager) 2003, a control circuit (drive circuit) 2004, a signal processing circuit 2005, a monitor 2006, and a memory 2007. This distance measuring device 2000 projects light from the light source unit 2001 toward the subject and receives the light (modulated light or pulsed light) reflected from the surface of the subject, thereby acquiring a distance image corresponding to the distance to the subject.

[0126] The light source unit 2001 projects light toward the subject. Examples of light sources for the light source unit 2001 include a vertical cavity surface-emitting laser (VCSEL) array that emits laser light as a surface light source, and a laser diode array in which laser diodes are arranged in a line. The laser diode array is supported by a predetermined drive unit (not shown) and scanned in a direction perpendicular to the arrangement of the laser diodes.

[0127] The optical system 2002 has one or more lenses. This optical system 2002 guides light from the subject (incident light) to the image sensor 2003 and forms an image on the light-receiving surface (sensor part) of the image sensor 2003.

[0128] The image sensor 2003 accumulates signal charge in response to light formed on the light-receiving surface via the optical system 2002. A distance signal indicating the distance, determined from the light-receiving signal (APD OUT) output from the image sensor 2003, is supplied to the signal processing circuit 2005. As the image sensor 2003, a solid-state imaging device such as an image sensor is used, for example.

[0129] The control circuit 2004 outputs drive signals (control signals) that control the operation of the light source unit 2001 and the image sensor 2003, and drives the light source unit 2001 and the image sensor 2003.

[0130] The signal processing circuit 2005 performs various signal processing operations on the distance signal supplied from the image sensor 2003. For example, the signal processing circuit 2005 performs image processing (e.g., histogram processing and peak detection processing) to construct a distance image based on the distance signal. The image (image data) obtained by the signal processing circuit 2005 is supplied to the monitor 2006 and also to the memory 2007.

[0131] The monitor 2006 displays the depth image captured by the image sensor 2003 based on the image data supplied from the signal processing circuit 2005. For example, a panel-type display device such as a liquid crystal panel or an organic EL panel can be used as the monitor 2006.

[0132] Memory 2007 stores image data supplied from the signal processing circuit 2005, that is, image data of the distance image captured by the image sensor 2003. Various types of storage devices can be used as memory 2007.

[0133] In the distance measuring device 2000 configured in this way, the same effects as in the above embodiment can be obtained by applying the aforementioned imaging system 10.

[0134] As described above, the imaging system 10 can be implemented in various electronic devices. For example, in addition to the imaging device 1000 and the distance measuring device 2000, the imaging system 10 may be installed in various electronic devices such as HDDs (hard disk drives), notebook PCs (personal computers), mobile devices (e.g., smartphones and tablet PCs), PDAs (personal digital assistants), wearable devices, game consoles, and music players.

[0135] <8. Addendum> Furthermore, this technology can also be configured as follows. (1) A pixel unit that outputs long-exposure images or short-exposure images with different exposure times for each frame, A processing unit that processes the long-storage image or the short-storage image for each frame, A detection unit that detects the brightness of the long-storage image or the short-storage image for each frame, A control unit controls the processing unit for each frame based on brightness information relating to the brightness, A solid-state imaging device equipped with the following features. (2) The control unit controls the processing unit in the next frame after the target frame based on the brightness information of the target frame. (1) The solid-state imaging device described above. (3) The control unit restricts the operation of the processing unit in the next frame based on the brightness information of the target frame. (2) The solid-state imaging device described above. (4) The pixel unit alternately outputs the long-storage image and the short-storage image. The control unit restricts the operation of the processing unit in the next frame based on the brightness information of the long-storage image or short-storage image of the target frame. (3) The solid-state imaging device described above. (5) The control unit restricts the operation of the processing unit by restricting the reading of the long-storage image or the short-storage image by the processing unit. (3) or (4) The solid-state imaging device described above. (6) The control unit limits the operation of the processing unit by clock gating or power gating. (3) or (4) The solid-state imaging device described above. (7) The system further includes an output unit that outputs image data of both or either of the long-storage image and the short-storage image. The control unit, based on the brightness information of the target frame, limits the driving of the output unit in the next frame. A solid-state imaging device as described in any one of (3) to (6). (8) The system further includes an output unit that outputs dummy data as image data for the long-storage image or short-storage image of the next frame when the control unit restricts the operation of the processing unit. A solid-state imaging device as described in any one of (3) to (6). (9) The brightness information includes the intensity of brightness or the amount of change in brightness. The control unit controls the processing unit according to the brightness intensity or the amount of change in brightness. A solid-state imaging device as described in any one of (1) to (8). (10) The control unit detects moving objects in the long-storage image or the short-storage image according to the amount of change in brightness. (9) The solid-state imaging device described above. (11) The control unit changes the imaging conditions of the pixel portion in the next frame after the target frame, based on the brightness information of the target frame. A solid-state imaging device as described in any one of (1) to (10). (12) The control unit modifies the image processing of the processing unit in the next frame after the target frame, based on the brightness information of the target frame. A solid-state imaging device as described in any one of (1) to (11). (13) The control unit, For each frame, the brightness saturation region of the long-storage image or the short-storage image is determined based on the brightness information. For each frame, the processing unit is controlled according to the size of the brightness saturation region. A solid-state imaging device as described in any one of (1) to (12). (14) The detection unit divides the long-storage image or the short-storage image into multiple regions for each frame, and determines the brightness of each of the multiple regions. The control unit determines the brightness saturation region of the long-storage image or the short-storage image based on the brightness of each of the plurality of regions for each frame, and controls the processing unit according to the size of the brightness saturation region. (13) The solid-state imaging device described above. (15) The control unit determines whether the brightness of each of the plurality of regions is equal to or greater than a predetermined first threshold, calculates the total number of regions whose brightness is equal to or greater than the first threshold as the size of the brightness saturation region, and controls the processing unit according to whether the total number is equal to or greater than a predetermined second threshold. (14) The solid-state imaging device described above. (16) A confirmation unit that confirms that image data of both or either of the long-exposure and short-exposure images with different exposure times have been input, When both the long-storage image and the short-storage image data are input, a multiple image processing unit is provided to synthesize the long-storage image and the short-storage image. When the image data of either the long-storage image or the short-storage image is input, a single image processing unit processes the long-storage image or the short-storage image. An image processing device equipped with the following features. (17) Solid-state imaging device, Image processing device and Equipped with, The solid-state imaging device is A pixel unit that outputs long-exposure images or short-exposure images with different exposure times for each frame, A processing unit that processes the long-storage image or the short-storage image for each frame, A detection unit that detects the brightness of the long-storage image or the short-storage image for each frame, A control unit controls the processing unit for each frame based on brightness information relating to the brightness, An output unit that outputs image data of both or one of the long-storage image and the short-storage image, It has, The aforementioned image processing device is A confirmation unit that confirms that both or either of the long-storage image and the short-storage image data have been input, When both the long-storage image and the short-storage image data are input, a multiple image processing unit is provided to synthesize the long-storage image and the short-storage image. When the image data of either the long-storage image or the short-storage image is input, a single image processing unit processes the long-storage image or the short-storage image. An imaging system having the following features. (18) Multiple solid-state imaging devices are provided, The control unit changes the individual drive of the plurality of solid-state imaging devices according to the brightness information for each solid-state imaging device. The imaging system described in (17). (19) The control unit drives the solid-state imaging device with the highest or lowest sensitivity among the plurality of solid-state imaging devices according to the brightness information for each solid-state imaging device. The imaging system described in (18). (20) The control unit drives the solid-state imaging device with the widest or narrowest field of view among the plurality of solid-state imaging devices, according to the brightness information for each solid-state imaging device. The imaging system described in (18) or (19). (twenty one) An imaging system comprising a solid-state imaging device as described in any one of (1) to (15). [Explanation of Symbols]

[0136] 10 Imaging System 10A Imaging System 20 sensors 21 pixel section 22 Processing Units 23 Output section 24 Control Unit 25 Image brightness detection unit 26 Image Brightness Reflection Control Unit 30 AP (Application Processor) 31 Data Verification Section 32 Multiple Image Processing Unit 33 Single Image Processing Unit 34 Signal Generation Unit 221 AD Conversion Unit 222 Signal rearrangement section 223 Image signal processing unit

Claims

1. A pixel unit that outputs long-exposure images or short-exposure images with different exposure times for each frame, A processing unit that processes the long-storage image or the short-storage image for each frame, A detection unit that detects the brightness of the long-storage image or the short-storage image for each frame, A control unit controls the processing unit for each frame based on brightness information relating to the brightness, A solid-state imaging device equipped with the following features.

2. The control unit controls the processing unit in the next frame after the target frame based on the brightness information of the target frame. The solid-state imaging apparatus according to claim 1.

3. The control unit restricts the operation of the processing unit in the next frame based on the brightness information of the target frame. The solid-state imaging apparatus according to claim 2.

4. The pixel unit alternately outputs the long-storage image and the short-storage image. The control unit restricts the operation of the processing unit in the next frame based on the brightness information of the long-storage image or short-storage image of the target frame. The solid-state imaging apparatus according to claim 3.

5. The control unit restricts the operation of the processing unit by restricting the reading of the long-storage image or the short-storage image by the processing unit. The solid-state imaging apparatus according to claim 3.

6. The control unit limits the operation of the processing unit by clock gating or power gating. The solid-state imaging apparatus according to claim 3.

7. The system further includes an output unit that outputs image data of both or either of the long-storage image and the short-storage image. The control unit, based on the brightness information of the target frame, limits the driving of the output unit in the next frame. The solid-state imaging apparatus according to claim 3.

8. The system further includes an output unit that outputs dummy data as image data for the long-storage image or short-storage image of the next frame when the control unit restricts the operation of the processing unit. The solid-state imaging apparatus according to claim 3.

9. The brightness information includes the intensity of brightness or the amount of change in brightness. The control unit controls the processing unit according to the brightness intensity or the amount of change in brightness. The solid-state imaging apparatus according to claim 1.

10. The control unit detects moving objects in the long-storage image or the short-storage image according to the amount of change in brightness. The solid-state imaging apparatus according to claim 9.

11. The control unit changes the imaging conditions of the pixel portion in the next frame after the target frame, based on the brightness information of the target frame. The solid-state imaging apparatus according to claim 1.

12. The control unit modifies the image processing of the processing unit in the next frame after the target frame, based on the brightness information of the target frame. The solid-state imaging apparatus according to claim 1.

13. The control unit, For each frame, the brightness saturation region of the long-storage image or the short-storage image is determined based on the brightness information. For each frame, the processing unit is controlled according to the size of the brightness saturation region. The solid-state imaging apparatus according to claim 1.

14. The detection unit divides the long-storage image or the short-storage image into multiple regions for each frame, and determines the brightness of each of the multiple regions. The control unit determines the brightness saturation region of the long-storage image or the short-storage image based on the brightness of each of the plurality of regions for each frame, and controls the processing unit according to the size of the brightness saturation region. The solid-state imaging apparatus according to claim 13.

15. The control unit determines whether the brightness of each of the plurality of regions is equal to or greater than a predetermined first threshold, calculates the total number of regions whose brightness is equal to or greater than the first threshold as the size of the brightness saturation region, and controls the processing unit according to whether the total number is equal to or greater than a predetermined second threshold. The solid-state imaging apparatus according to claim 14.

16. A confirmation unit that confirms that image data of both or either of the long-exposure and short-exposure images with different exposure times have been input, When both the long-storage image and the short-storage image data are input, a multiple image processing unit is provided to synthesize the long-storage image and the short-storage image. When the image data of either the long-storage image or the short-storage image is input, a single image processing unit processes the long-storage image or the short-storage image. An image processing device equipped with the following features.

17. Solid-state imaging device, Image processing device and Equipped with, The solid-state imaging device is A pixel unit that outputs long-exposure images or short-exposure images with different exposure times for each frame, A processing unit that processes the long-storage image or the short-storage image for each frame, A detection unit that detects the brightness of the long-storage image or the short-storage image for each frame, A control unit controls the processing unit for each frame based on brightness information relating to the brightness, An output unit that outputs image data of both or one of the long-storage image and the short-storage image, It has, The aforementioned image processing device is A confirmation unit that confirms that both or either of the long-storage image and the short-storage image data have been input, When both the long-storage image and the short-storage image data are input, a multiple image processing unit is provided to synthesize the long-storage image and the short-storage image. When the image data of either the long-storage image or the short-storage image is input, a single image processing unit processes the long-storage image or the short-storage image. An imaging system having the following features.

18. Multiple solid-state imaging devices are provided, The control unit changes the individual drive of the plurality of solid-state imaging devices according to the brightness information for each solid-state imaging device. The imaging system according to claim 17.

19. The control unit drives the solid-state imaging device with the highest or lowest sensitivity among the plurality of solid-state imaging devices according to the brightness information for each solid-state imaging device. The imaging system according to claim 18.

20. The control unit drives the solid-state imaging device with the widest or narrowest field of view among the plurality of solid-state imaging devices, according to the brightness information for each solid-state imaging device. The imaging system according to claim 18.

Images