Imaging device

Abstract

Provided is an imaging device which can adaptively acquire a captured image in accordance with imaging conditions. An imaging device (1) comprises: an imaging unit (10) that includes a pixel array (110) having a plurality of pixel groups each containing N×N pixels (100) (N being an integer of 2 or more), and that outputs pixel signals read out from each pixel; and a switching unit (14) that switches reading modes for reading out the pixel signals from each of the pixels of the imaging unit. The switching unit switches the reading modes between an adding mode in which each of the pixel signals read out from each of the N×N pixels contained in the pixel groups are added together to produce one pixel signal, and an individual mode in which each of the pixel signals read out from each of the N×N pixels contained in the pixel groups are output individually.

Description

[Technical field]

[0001] The present disclosure relates to an imaging device. [Background technology]

[0002] 2. Description of the Related Art Conventionally, there has been known a technique for imaging an image by controlling an exposure time, a gain for a pixel signal, and the like, based on the brightness of a subject or an imaging environment detected by an imaging element. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2013-066146 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the past, even when an object does not require high resolution, it was imaged at the same resolution as normal, making it difficult to optimize power consumption and image processing resources.

[0005] An object of the present disclosure is to provide an imaging device capable of adaptively acquiring a captured image in accordance with imaging conditions. [Means for solving the problem]

[0006] The imaging device according to the present disclosure comprises: An imaging device mounted on a vehicle, the imaging device comprising: The image pickup unit includes a pixel array having a plurality of pixel groups each including N×N pixels (N is an integer of 2 or more), and outputs a pixel signal read from each pixel. The image pickup unit includes a switching unit that switches a read mode in which the image pickup unit reads out the pixel signals from each of the pixels, and the switching unit According to the brightness of the surroundings of the vehicle acquired as vehicle information from the vehicle,The readout mode is switched between an addition mode in which the pixel signals read out from each of the N x N pixels included in the pixel group are added to form a single pixel signal, and an individual mode in which the pixel signals read out from each of the N x N pixels included in the pixel group are output individually. [Brief description of the drawings]

[0007] [Figure 1] FIG. 2 is a functional block diagram illustrating an example of functions of an imaging apparatus applicable to each embodiment. [Diagram 2] FIG. 1 is a diagram showing an example of a Bayer array. [Diagram 3] 1A to 1C are diagrams illustrating examples of color filter arrays that can be applied to each embodiment. [Figure 4] FIG. 2 is a block diagram showing an example of a configuration of an imaging unit applicable to each embodiment. [Diagram 5] FIG. 2 is a block diagram showing an example of a hardware configuration of an imaging apparatus applicable to each embodiment. [Figure 6] 1A to 1C are schematic diagrams for explaining detection of a gradient between the same colors that can be applied to an embodiment. [Figure 7] 1A to 1C are schematic diagrams for explaining detection of a gradient between different colors that can be applied to an embodiment. [Figure 8] FIG. 13 is a diagram illustrating an example of a direction of gradient calculation applicable to the embodiment. [Figure 9] FIG. 13 is a diagram illustrating an example of a direction of gradient calculation applicable to the embodiment. [Figure 10] FIG. 11 is a schematic diagram for explaining an example of a method for reading out pixel signals in a divided Bayer type RGB array. [Figure 11] FIG. 1 is a schematic diagram for explaining a first example of HDR imaging. [Figure 12] FIG. 11 is a schematic diagram for explaining a second example of HDR imaging. [Figure 13] FIG. 13 is a schematic diagram for explaining an example in which the additive mode is applied to the second example of HDR imaging. [Figure 14]FIG. 13 is a schematic diagram for explaining an example of a case where an individual mode is applied to a second example of HDR imaging and re-mosaic processing is performed. [Figure 15] FIG. 2 is a functional block diagram illustrating an example of functions of the imaging device according to the first embodiment. [Figure 16] FIG. 2 is a functional block diagram illustrating an example of a function of a pixel processing unit according to the first embodiment. [Figure 17] 4 is a schematic diagram for explaining an example of generating a switching control signal CNG in accordance with two imaging conditions according to the first embodiment. FIG. [Figure 18A] FIG. 11 is a schematic diagram for explaining a processing region according to a first modified example of the first embodiment. [Figure 18B] FIG. 11 is a schematic diagram for explaining a processing region according to a first modified example of the first embodiment. [Figure 19] FIG. 11 is a functional block diagram illustrating an example of functions of an image processing unit applicable to a first modified example of the first embodiment. [Figure 20] FIG. 11 is a schematic diagram for explaining a processing region according to a second modified example of the first embodiment. [Figure 21] FIG. 11 is a functional block diagram illustrating an example of functions of an image processing unit applicable to a second modified example of the first embodiment. [Figure 22A] FIG. 1 is a schematic diagram showing an example of a pixel array applicable to the present disclosure. [Figure 22B] FIG. 1 is a schematic diagram showing an example of a pixel array applicable to the present disclosure. [Figure 22C] FIG. 1 is a schematic diagram showing an example of a pixel array applicable to the present disclosure. [Figure 22D] FIG. 1 is a schematic diagram showing an example of a pixel array applicable to the present disclosure. [Figure 22E] FIG. 1 is a schematic diagram showing an example of a pixel array applicable to the present disclosure. [Figure 23A] 13A and 13B are schematic diagrams for explaining an example of a method of switching between an individual mode and an addition mode in a pixel group according to a second embodiment. [Figure 23B]13A and 13B are schematic diagrams for explaining an example of a method of switching between an individual mode and an addition mode in a pixel group according to a second embodiment. [Figure 23C] 13A and 13B are schematic diagrams for explaining an example of a method of switching between an individual mode and an addition mode in a pixel group according to a second embodiment. [Figure 24] FIG. 11 is a functional block diagram illustrating an example of functions of an image processing unit that can be applied to a second embodiment. [Diagram 25] FIG. 13 is a block diagram showing a configuration of an example of an imaging unit according to a modified example of the second embodiment. [Figure 26] 1A and 1B are diagrams illustrating examples of use of an imaging device according to the present disclosure. [Figure 27] 1 is a block diagram showing an example of a system configuration of a vehicle in which an imaging device according to the present disclosure can be mounted. [Figure 28] FIG. 2 is a block diagram showing an example of a configuration of a front sensing camera of a vehicle system. [Figure 29] 1 is a block diagram showing a schematic configuration example of a vehicle control system that is an example of a mobile object control system to which the technology according to the present disclosure can be applied. [Diagram 30] FIG. 4 is a diagram showing an example of an installation position of an imaging unit. [Diagram 31] 1 is a diagram illustrating an example of a schematic configuration of an endoscopic surgery system. [Diagram 32] 2 is a block diagram showing an example of a functional configuration of a camera head and a CCU. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are denoted by the same reference numerals, and duplicated descriptions will be omitted.

[0009] Hereinafter, embodiments of the present disclosure will be described in the following order. 1. Technology Applicable to Each Embodiment of the Present Disclosure 1-1. Configurations applicable to each embodiment 1-2. Overview of re-mosaic processing 1-3. Example of pixel readout method 2. First embodiment 2-1. First modified example of the first embodiment 2-2. Second Modification of the First Embodiment 2-3. Third Modification of the First Embodiment 3. Second embodiment 3-1. Modification of the second embodiment 4. Third embodiment 4-0. Application examples for mobile devices 4-1. Application example to endoscopic surgery system

[0010] [1. Technology applicable to each embodiment of the present disclosure] (1-1. Configurations Applicable to Each Embodiment) First, a technique applicable to each embodiment of the present disclosure will be described. Fig. 1 is a functional block diagram of an example for explaining functions of an imaging device applicable to each embodiment.

[0011] 1, the imaging device 1 includes an imaging unit 10, an optical unit 11, an image processing unit 12, an output processing unit 13, and a control unit 14. When the imaging device 1 is used for mounting on a vehicle, the imaging device 1 further includes a vehicle information acquisition unit 15.

[0012] The imaging unit 10 has a pixel array in which a plurality of pixels, each of which includes one or more light receiving elements, are arranged in a matrix. The optical unit 11 includes a lens, an aperture mechanism, a focus mechanism, etc., and guides light from a subject to the light receiving surface of the pixel array.

[0013] The imaging unit 10 reads out pixel signals from each pixel exposed for a specified exposure time, performs signal processing such as noise removal and gain adjustment on the read out pixel signals, and converts them into digital pixel data. The imaging unit 10 outputs pixel data based on these pixel signals. This series of operations by the imaging unit 10, from exposure and reading out pixel signals from the exposed pixels to outputting them as pixel data, is called imaging.

[0014] The image processing unit 12 performs predetermined image processing on the pixel data output from the imaging unit 10, and outputs the data as image data for each frame, for example. The image processing performed by the image processing unit 12 on the pixel data includes color adjustment processing such as pixel data synthesis, conversion processing, and white balance processing.

[0015] The output processing unit 13 converts the image data output from the image processing unit 12 into a format suitable for output from the imaging device 1. The output image data output from the output processing unit 13 is supplied to, for example, a display (not shown) and displayed as an image. Without being limited to this, the output image data may be supplied to another device, for example, a device that performs recognition processing on the output image data, or a control device that performs control based on the output image data.

[0016] The control unit 14 controls the overall operation of the imaging device 1. The control unit 14 includes, for example, a CPU (Central Processing Unit) and an interface circuit for communicating with each part of the imaging device 1, and the CPU operates according to a predetermined program to generate various control signals and controls each part of the imaging device 1 by the generated control signals.

[0017] The image processing unit 12 and the output processing unit 13 described above can be configured, for example, by a DSP (Digital Signal Processor) or an ISP (Image Signal Processor) that operate according to a predetermined program. Without being limited to this, one or both of the image processing unit 12 and the output processing unit 13 may be realized by a program that operates on a CPU together with the control unit 14. These programs may be stored in advance in a non-volatile memory included in the imaging device 1, or may be supplied to the imaging device 1 from the outside and written into the memory.

[0018] When the imaging device 1 is mounted on a vehicle and used, the vehicle information acquisition unit 15 acquires vehicle information from the vehicle. The vehicle information acquisition unit 15 acquires vehicle information such as the speed of the vehicle and the brightness in front of the vehicle. The vehicle information acquired by the vehicle information acquisition unit 15 is passed to the control unit 14. The control unit 14 can instruct the imaging unit 10 and the image processing unit 12 to perform processing according to the passed vehicle information.

[0019] Each pixel included in the pixel array of the imaging unit 10 is provided with an optical filter that transmits light in a predetermined wavelength range. Unless otherwise specified, the optical filter that transmits light in the predetermined wavelength range will be described as a color filter. When obtaining full-color image data, three types of color filters are generally used: a color filter that transmits light in the R (red) wavelength range (hereinafter, R color filter), a color filter that transmits light in the G (green) wavelength range (hereinafter, G color filter), and a color filter that transmits light in the B (blue) wavelength range (hereinafter, B color filter). The arrangement of these R, G, and B color filters can be variously considered, but generally, an arrangement called a Bayer arrangement is used.

[0020] Fig. 2 is a diagram showing an example of a Bayer array. In Fig. 2, the Bayer array 120 includes two pixels 100G in which a G color filter is arranged, one pixel 100R in which an R color filter is arranged, and a pixel 100B in which a B color filter is arranged. In the Bayer array, these four pixels are arranged in a 2 pixel x 2 pixel grid so that two pixels 100G are not adjacent to each other. In other words, the Bayer array is an array in which pixels 100 in which color filters that transmit light of the same wavelength band are arranged are not adjacent to each other.

[0021] In the following, unless otherwise specified, the "pixel 100R on which an R color filter is arranged" will be referred to as the "R pixel 100R" or simply as the "pixel 100R." The same applies to the pixel 100G on which a G color filter is arranged and the pixel 100B on which a B color filter is arranged. In addition, when the color filters are not of particular concern, the pixels 100R, 100G, and 100B will be described as being representative of the pixel 100.

[0022] Fig. 3 is a diagram showing an example of a color filter array applicable to each embodiment. In the array in Fig. 3, a pixel block is a unit in which 2 pixels x 2 pixels of the same color are arranged in a grid pattern, and each pixel block is composed of four R pixels 100R, four G pixels 100G, and four B pixels 100B, and is configured to be arranged in a pixel array conforming to the Bayer array. Hereinafter, unless otherwise specified, such a pixel array is called a 4-division Bayer type RGB array.

[0023] More specifically, in the four-division Bayer RGB array, pixel blocks each made up of R pixels 100R, G pixels 100G, and B pixels 100B are arranged in a 2×2 grid such that the ratio of the numbers of pixels 100R, pixels 100G, and pixels 100B is 1:2:1, and pixel blocks made up of pixels of the same color are not adjacent to each other. In the example of Fig. 3, pixel blocks made up of G pixels 100G are arranged to the left and below the pixel block made up of R pixels 100R, and pixel blocks made up of B pixels 100B are arranged diagonally from the pixel block made up of R pixels 100R.

[0024] 4 is a block diagram showing an example of the configuration of the imaging unit 10 applicable to each embodiment. In FIG. 4, the imaging unit 10 includes a pixel array section 110, a vertical scanning section 20, a horizontal scanning section 21, and a control section 22.

[0025] The pixel array section 110 includes a plurality of pixels 100 each having a light receiving element that generates a voltage according to received light. A photodiode can be used as the light receiving element. In the pixel array section 110, the plurality of pixels 100 are arranged in a matrix in the horizontal direction (row direction) and the vertical direction (column direction). In the pixel array section 110, the row direction arrangement of the pixels 100 is called a line. In the pixel array section 110, one frame of image (image data) is formed based on pixel signals read out from a predetermined number of lines. For example, when one frame of image is formed with 3000 pixels x 2000 lines, the pixel array section 110 includes at least 2000 lines each including at least 3000 pixels 100.

[0026] In addition, in the pixel array section 110, pixel signal lines HCTL are connected to the rows and columns of the pixels 100 for each row, and vertical signal lines VSL are connected to the columns.

[0027] The end of the pixel signal line HCTL that is not connected to the pixel array unit 110 is connected to the vertical scanning unit 20. The vertical scanning unit 20 transmits a plurality of control signals, such as drive pulses used for reading pixel signals from the pixels 100, to the pixel array unit 110 via the pixel signal line HCTL in accordance with a control signal supplied from, for example, the control unit 14. The end of the vertical signal line VSL that is not connected to the pixel array unit 110 is connected to the horizontal scanning unit 21.

[0028] The horizontal scanning unit 21 includes an AD (Analog to Digital) conversion unit, an output unit, and a signal processing unit. The pixel signals read out from the pixels 100 are transmitted to the AD conversion unit of the horizontal scanning unit 21 via the vertical signal lines VSL.

[0029] The following provides an overview of the control of reading out pixel signals from the pixels 100. Reading out pixel signals from the pixels 100 is performed by transferring charges accumulated in the light receiving elements upon exposure to a floating diffusion layer (FD) and converting the transferred charges into a voltage in the floating diffusion layer. The voltage into which the charges are converted in the floating diffusion layer is output to the vertical signal line VSL via an amplifier.

[0030] More specifically, in the pixel 100, during exposure, the light receiving element and the floating diffusion layer are turned off (open) to accumulate electric charges generated in response to the incident light by photoelectric conversion in the light receiving element. After the exposure is completed, the floating diffusion layer and the vertical signal line VSL are connected in response to a selection signal supplied via the pixel signal line HCTL. Furthermore, the floating diffusion layer is connected to a supply line of a power supply voltage VDD or a black level voltage for a short period of time in response to a reset pulse supplied via the pixel signal line HCTL to reset the floating diffusion layer. A voltage (referred to as voltage P) of a reset level of the floating diffusion layer is output to the vertical signal line VSL. After that, a transfer pulse supplied via the pixel signal line HCTL turns the light receiving element and the floating diffusion layer to an on (closed) state, and the electric charges accumulated in the light receiving element are transferred to the floating diffusion layer. A voltage (referred to as voltage Q) corresponding to the amount of electric charge in the floating diffusion layer is output to the vertical signal line VSL.

[0031] In the horizontal scanning unit 21, the AD conversion unit includes an AD converter provided for each vertical signal line VSL, and the pixel signals supplied from the pixels 100 via the vertical signal lines VSL are subjected to AD conversion processing by the AD converter, and two digital values ​​(values ​​corresponding to voltage P and voltage Q, respectively) are generated for correlated double sampling (CDS) processing, which performs noise reduction.

[0032] The two digital values ​​generated by the AD converter are subjected to CDS processing by the signal processor, and a pixel signal (pixel data) is generated as a digital signal. The generated pixel data is output from the imaging unit 10.

[0033] The horizontal scanning unit 21, under the control of the control unit 22, performs selective scanning to select the AD converters for each vertical signal line VSL in a predetermined order, thereby causing each digital value temporarily held by each AD converter to be output sequentially to the signal processing unit. The horizontal scanning unit 21 realizes this operation by a configuration including, for example, a shift register and an address decoder.

[0034] The control unit 22 performs drive control of the vertical scanning unit 20, the horizontal scanning unit 21, etc. The control unit 22 generates various drive signals that serve as references for the operations of the vertical scanning unit 20 and the horizontal scanning unit 21. The control unit 22 generates control signals that the vertical scanning unit 20 supplies to each pixel 100 via the pixel signal line HCTL, based on a vertical synchronization signal or an external trigger signal supplied from the outside (e.g., the control unit 14) and a horizontal synchronization signal. The control unit 22 supplies the generated control signals to the vertical scanning unit 20.

[0035] The vertical scanning unit 20 supplies various signals including drive pulses to the pixel signal lines HCTL of a selected pixel row of the pixel array unit 110 for each pixel 100 on a line basis based on a control signal supplied from the control unit 22, and causes each pixel 100 to output a pixel signal to a vertical signal line VSL. The vertical scanning unit 20 is configured using, for example, a shift register, an address decoder, etc.

[0036] The imaging unit 10 configured in this manner is a column AD type CMOS (Complementary Metal Oxide Semiconductor) image sensor in which AD converters are arranged for each column.

[0037] Fig. 5 is a block diagram showing an example of a hardware configuration of an imaging device 1 applicable to each embodiment. In Fig. 5, the imaging device 1 includes a CPU 2000, a ROM (Read Only Memory) 2001, a RAM (Random Access Memory) 2002, an imaging unit 2003, a storage 2004, a data I / F 2005, an operation unit 2006, and a display control unit 2007, each of which is connected by a bus 2020. The imaging device 1 also includes an image processing unit 2010 and an output I / F 2012, each of which is connected by the bus 2020.

[0038] A CPU 2000 controls the overall operation of the imaging device 1 in accordance with a program prestored in a ROM 2001 and using a RAM 2002 as a work memory.

[0039] 1, the imaging unit 2003 captures an image and outputs pixel data. The pixel data output from the imaging unit 2003 is supplied to an image processing unit 2010. The image processing unit 2010 uses a frame memory 2011 to perform predetermined image processing on the pixel data supplied from the imaging unit 10, and generates image data in units of frames.

[0040] The output I / F 2012 is an interface for externally outputting image data generated by the image processing unit 2010. The output I / F 2012 can convert image data supplied from the image processing unit 2010 into image data in a predetermined format and output the converted image data.

[0041] The storage 2004 is, for example, a flash memory, and can store and accumulate image data output from the image processing unit 2010. The storage 2004 can also store a program for operating the CPU 2000. Furthermore, the storage 2004 is not limited to being built into the imaging device 1, and may be detachable from the imaging device 1.

[0042] The data I / F 2005 is an interface for the imaging device 1 to transmit and receive data to and from an external device. For example, a Universal Serial Bus (USB) can be used as the data I / F 2005. Also, an interface for short-distance wireless communication such as Bluetooth (registered trademark) can be used as the data I / F 2005.

[0043] The operation unit 2006 accepts user operations on the imaging device 1. The operation unit 2006 includes operators such as dials and buttons as input devices that accept user inputs. The operation unit 2006 may include a touch panel as an input device that outputs a signal according to a touched position.

[0044] The display control unit 2007 generates a display signal that can be displayed by the display 2008, based on the display control signal passed by the CPU 2000. The display 2008 uses, for example, an LCD (Liquid Crystal Display) as a display device, and displays a screen according to the display signal generated by the display control unit 2007. Depending on the application of the imaging device 1, the display control unit 2007 and the display 2008 may be omitted.

[0045] (1-2. Overview of re-mosaic processing) Here, the above-mentioned re-mosaic processing will be briefly described. As processing modes of pixel signals of each pixel in a four-division Bayer RGB array, there are a mode (called an addition mode) in which pixel signals of four pixels included in a pixel group are added and processed as one pixel signal, and a mode (called an individual mode) in which the pixel signals of the four pixels are processed individually. Among these, in the individual mode, the pixel value of each pixel of the four-division Bayer RGB array can be converted into the pixel value of each pixel in, for example, a Bayer array by the re-mosaic processing.

[0046] The image processing unit 12 detects the gradient of pixel values ​​at the position of the pixel of interest for each direction, and determines the direction in which the detected gradient is the smallest. Based on the pixel values ​​of the pixels in the detected direction, the image processing unit 12 predicts the pixel value of a pixel in the Bayer array corresponding to the position of the pixel of interest, and replaces the pixel value of the pixel of interest with the predicted pixel value. In this way, the pixel values ​​of each pixel in the four-division Bayer RGB array are converted to the pixel values ​​of each pixel in the Bayer array.

[0047] More specifically, the image processing unit 12 detects gradients between pixels of the same color (same-color gradients) as shown in FIG. 6, and gradients between pixels of different colors (different-color gradients) as shown in FIG. 7.

[0048] In Fig. 6 and Fig. 7, as well as in the following similar drawings, each square indicates a pixel, and the letters (R, G, and B) in each square indicate the color (R, G, and B) of the color filter corresponding to that pixel. Here, the pixels on which the R, G, and B color filters are provided are called pixel R, pixel G, and pixel B, respectively. Also, the upper left square of the grid is the origin, the horizontal direction of the drawing is the x direction, and the vertical direction is the y direction, and the coordinates (x, y) are expressed in pixel units. For example, in Fig. 6, the pixel at the origin is pixel R with coordinates (0, 0), and is described as pixel (0, 0).

[0049] 6, for pixel G, gradients are detected along the x direction for each pair of same-color pixels G(0,2) and (1,2), G(4,2) and (5,2), G(0,3) and (1,3), and G(4,3) and (5,3).For pixel B, gradients are detected for each pair of same-color pixels B(2,2) and (3,2), and B(2,3) and (3,3).

[0050] On the other hand, in the example of Figure 7, gradients are detected along the x direction for each pair of different colors: pixel G(1,2) and pixel B(2,2), pixel G(1,3) and pixel B(2,3), pixel B(3,2) and pixel G(4,2), and pixel B(3,3) and pixel G(4,3).

[0051] Here, detection of the same-color gradient is a gradient calculation between pixels with the same sensitivity. In contrast, detection of the different-color gradient is a gradient calculation between pixels with different sensitivities, and a gradient that depends on the sensitivity difference between the colors of the pixels is calculated. Therefore, the image processing unit 12 detects the gradient by multiplying each pixel value of the pixel R and pixel B included in the input image signal by a predetermined coefficient.

[0052] The direction determination process of the image processing unit 12 applicable to each embodiment will be described in more detail. The image processing unit 12 calculates gradients in a plurality of directions in a local region. Figures 8 and 9 are diagrams showing examples of gradient calculation directions applicable to the embodiments. In Figures 8 and 9 and the following similar figures, the size of the local region is 6 pixels x 6 pixels.

[0053] The image processing unit 12 performs gradient calculation along four directions, namely, the horizontal direction (H), the vertical direction (V), the 45° direction to the upper right (A), and the 45° direction to the lower right (D), as shown in Fig. 8. Furthermore, the image processing unit 12 performs gradient calculation along four directions, namely, the 22.5° direction to the upper right (A2), the 67.5° direction to the upper right (A3), the 22.5° direction to the lower right (D2), and the 67.5° direction to the lower right (D3), as shown in Fig. 9. That is, in the embodiment, the image processing unit 12 performs gradient calculation along eight directions that differ in angle by 22.5° each.

[0054] The four-division Bayer RGB array described using Fig. 3 has a sparser sampling interval for same-color components than the Bayer array shown in Fig. 2, so aliasing occurs at 1 / 2 Nyquist frequency. As shown in Fig. 8 and Fig. 9, by performing gradient calculations along each of the eight directions, it becomes possible to detect such a frequency repetition pattern with high accuracy.

[0055] The image processing unit 12 predicts the pixel value of a pixel corresponding to the position of the pixel of interest in the Bayer array based on the gradient determination result, and replaces the pixel value of the pixel of interest with the predicted pixel value.

[0056] (1-3. Example of pixel readout method) Next, an example of reading pixel signals in a four-division Bayer RGB array will be described. Fig. 10 is a schematic diagram for explaining an example of a method for reading pixel signals in a divided Bayer RGB array. In Fig. 10, array 121 shows a pixel array in a four-division Bayer RGB array. Array 130 shows an example of a pixel data array in which the pixel values ​​of the array 121 are converted into pixel values ​​in a Bayer array by reading the pixel values ​​in the individual mode described above and performing a re-mosaic process on the array 121.

[0057] Array 131 shows an example of an array obtained by performing readout in the above-mentioned addition mode on array 121. In this way, in the addition mode, four pixels included in a pixel group are collectively treated as one pixel. In the example of FIG. 10, as shown in array 131, four pixels R included in a pixel group by pixel R in a four-division Bayer type RGB array (array 121) are treated as one pixel R(+) by adding up pixel values. Similarly, four pixels G included in a pixel group by pixel G in array 121 are treated as one pixel G(+) by adding up pixel values, and four pixels B included in a pixel group by pixel B are treated as one pixel B(+) by adding up pixel values.

[0058] Array 132 shows an example of an array in which the exposure time of at least one pixel among the four pixels included in the pixel group shown in array 121 is made different from the exposure time of the other pixels included in the pixel group. In the example of Fig. 10, the exposure time of one pixel among the four pixels included in the pixel group is set to a first exposure time, the exposure time of two pixels is set to a second exposure time longer than the first exposure time, and the exposure time of the remaining pixel is set to a third exposure time longer than the second exposure time. Hereinafter, readout at the first exposure time is called short accumulation, readout at the second exposure time is called medium accumulation, and readout at the third exposure time is called long accumulation.

[0059] 10, for example, taking the pixel group of pixel R as an example, the upper left is the long accumulation pixel R(L), the upper right and lower left are the medium accumulation pixel R(M), and the lower right is the short accumulation pixel R(S). Similarly, the pixel group of pixel G is also set as the upper left is the long accumulation pixel G(L), the upper right and lower left are the medium accumulation pixel G(M), and the lower right is the short accumulation pixel G(S), and the pixel group of pixel B is also set as the upper left is the long accumulation pixel B(L), the upper right and lower left are the medium accumulation pixel B(M), and the lower right is the short accumulation pixel B(S).

[0060] In the case of array 132, pixel data (image data) read out by short accumulation, medium accumulation, and long accumulation are synthesized to generate one image data. By synthesizing image data based on pixel data exposed with different exposure times, image data with a wider dynamic range can be generated. Image data of an image with a wide dynamic range generated in this way is called an HDR (High Dynamic Range) image. In addition, such an imaging method is called HDR imaging.

[0061] 10, as shown in the array 131HDR, for example, in the case of a pixel group of pixel R, pixel R(L), two pixels R(M), and pixel R(S) are synthesized by HDR imaging to generate pixel R(HDR) having the size of the pixel group. Similarly, in the case of each pixel group of pixels G and B, pixels G(HDR) and B(HDR) having the size of the pixel group are generated.

[0062] Fig. 11 is a schematic diagram for explaining a first example of HDR imaging. In Fig. 11 and Fig. 12 described later, the passage of time is shown to the right. Lines are also shown in the vertical direction, with the line at the top of the frame being line Ln#1, and readout is shown to be performed line-sequentially from line Ln#1 toward the bottom of the frame (rolling shutter method).

[0063] In a first example of HDR imaging, exposures of long accumulation, medium accumulation, and short accumulation are executed in parallel, and readout from pixels of each of the long accumulation, medium accumulation, and short accumulation is executed simultaneously.

[0064] For example, in line Ln#1, exposure by long accumulation is started at time t0 in one frame time, and exposure by medium accumulation is started at time t1 when a predetermined time has elapsed from time t0. Furthermore, exposure by short accumulation is started at time t2 when a predetermined time has elapsed from time t1. At time t3 when a predetermined time has elapsed from time t2, the exposures of the long accumulation, medium accumulation, and short accumulation are each ended, and reading is performed from the pixels of the long accumulation, medium accumulation, and short accumulation.

[0065] 12 is a schematic diagram for explaining a second example of HDR imaging. In the second example of HDR imaging, long accumulation, medium accumulation, and short accumulation exposures are performed sequentially, and reading from pixels is performed for each exposure.

[0066] For example, in line Ln#1, exposure by long accumulation is started at time t0 in one frame time, and exposure by long accumulation is ended at time t1 when a predetermined time has elapsed since time t0, and reading is performed from the pixels of long accumulation. Next, exposure by medium accumulation is started at time t1, and exposure by medium accumulation is ended at time t2 when a predetermined time has elapsed since time t1, and reading is performed from the pixels of medium accumulation. Furthermore, exposure by short accumulation is started at time t2, and exposure by short accumulation is ended at time t3 when a predetermined time has elapsed since time t2, and reading is performed from the pixels of short accumulation. That is, in this second example of HDR imaging, three exposures of long accumulation, medium accumulation, and short accumulation are performed within one frame time in each line.

[0067] In this second example of HDR imaging, either the additive mode or the individual mode described above can be applied. Fig. 13 is a schematic diagram for explaining an example in which the additive mode is applied to the second example of HDR imaging.

[0068] In FIG. 13, the arrays 131L, 131M, and 131S shown in the upper part show examples of arrays in which exposure is performed by long accumulation, medium accumulation, and short accumulation, and reading is performed by addition mode, respectively, for the array 121. The arrays 131L, 131M, and 131S show examples of arrays in which exposure is performed by long accumulation, medium accumulation, and short accumulation, and reading is performed by addition mode, respectively. As an example, regarding a pixel group by pixel R, in the array 131L, four pixels R(L) exposed by long accumulation are treated as one pixel R(L+) by adding pixel values. In the array 131M, four pixels R(M) exposed by medium accumulation are treated as one pixel R(M+) by adding pixel values. Also, in the array 131S, four pixels R(M) exposed by medium accumulation are treated as one pixel R(M+) by adding pixel values. The same is true for the pixel groups by the other pixels G and B.

[0069] The array 131HDR+ shows an example of combining pixels from the arrays 131L, 131M, and 131S. In the array 131HDR+, the pixels R(L+), R(M+), and R(S+) are combined to obtain a pixel R(HDR+) with a wide dynamic range. Similarly, the pixels G(L+), G(M+), and G(S+), and the pixels B(L+), B(M+), and B(S+), are combined to obtain pixels R(HDR+), G(HDR+), and B(HDR+) with a high dynamic range. That is, the array 131HDR+ is an array in which the pixels from the arrays 131L, 131M, and 131S are combined in units of pixel groups.

[0070] Fig. 14 is a schematic diagram for explaining an example of the case where the individual mode is applied to the second example of HDR imaging and re-mosaic processing is performed. Arrays 133L, 133M, and 133S correspond to the arrays 131L, 131M, and 131S of Fig. 13, respectively, and show four pixels included in each pixel group separately. Arrays 134L, 134M, and 134S show an example in which re-mosaic processing is performed on each pixel of these arrays 133L, 133M, and 133S, and the pixel values ​​of the four pixels included in each pixel group are converted into pixel values ​​of each pixel in the Bayer array. In this case, the array 134L includes only each pixel value based on pixel data of long accumulation, and similarly, the arrays 134M and 134S include only each pixel value based on pixel data of medium accumulation and short accumulation, respectively.

[0071] Array 135HDR shown in the lower part of Fig. 14 is a combination of arrays 134L, 134M, and 134S at corresponding pixels. In this way, in the second example of HDR imaging, a high dynamic range can be obtained in each pixel corresponding to the Bayer array.

[0072] Here, the difference between the first and second examples of the above-mentioned HDR imaging will be described with reference to Fig. 10, Fig. 13, and Fig. 14. In the first example of HDR imaging, for example, pixels assigned to long accumulation (or medium accumulation, short accumulation) are 4 pixels among 16 pixels included in a 4-division Bayer type RGB array (see array 132 in Fig. 10). In contrast, in the second example of HDR imaging, all 16 pixels are assigned to long accumulation (or medium accumulation, short accumulation) (see arrays 131L, 131M, and 131S in Fig. 13). Therefore, when the resolution is the same, the second example is advantageous over the first example in terms of SNR (Signal-Noise Ratio).

[0073] Also, in the first example of HDR imaging, the exposure times of the four pixels included in the pixel group are different (see array 132 in FIG. 10), making it difficult to apply re-mosaic processing. In contrast, in the second example of HDR imaging, the exposure times of the 16 pixels included in the four-division Bayer RGB array are the same, making it easy to apply re-mosaic processing (see FIG. 14). Therefore, in terms of resolution, the second example is advantageous over the first example.

[0074] On the other hand, when comparing the additive mode and the individual mode, the additive mode allows for faster pixel readout and is advantageous in terms of SNR compared to the individual mode. Also, the individual mode allows for re-mosaic processing, so is advantageous in terms of resolution compared to the additive mode. In each embodiment, the pixel readout in the imaging unit 10 is executed by adaptively switching between the additive mode and the individual mode. This makes it possible to execute imaging suited to imaging conditions such as the imaging purpose and imaging environment.

[0075] [2. First embodiment] Next, a first embodiment of the present disclosure will be described. The first embodiment is an example in which switching between an addition mode and an individual mode is realized by image processing. Fig. 15 is a functional block diagram of an example for explaining the function of the imaging device 1 according to the first embodiment.

[0076] The imaging device 1 shown in Fig. 15 performs imaging according to the second example of HDR imaging described above. That is, in the imaging section 10, R, G, and B color filters are provided for each pixel 100 of the pixel array section 110 according to a four-division Bayer type RGB arrangement. The control section 14 controls the imaging section 10 so as to perform imaging by long accumulation, medium accumulation, and short accumulation described with reference to Fig. 12. The imaging section 10 supplies pixel data imaged by the exposure of each of the long accumulation, medium accumulation, and short accumulation and read out from each pixel 100 to the image processing section 12.

[0077] In the imaging device 1 of the first embodiment, the image processing unit 12 includes a pixel processing unit 1200, an HDR synthesis processing unit 1201, a WB (white balance) processing unit 1202, a gradation compression processing unit 1203, a demosaic processing unit 1204, a gamma correction processing unit 1205, and an output processing unit 1206.

[0078] The pixel processing unit 1200, HDR synthesis processing unit 1201, WB processing unit 1202, gradation compression processing unit 1203, demosaic processing unit 1204, gamma correction processing unit 1205, and output processing unit 1206 are realized by executing a predetermined program on a processor such as a DSP, ISP, CPU, etc. Without being limited to this, some or all of the pixel processing unit 1200, HDR synthesis processing unit 1201, WB processing unit 1202, gradation compression processing unit 1203, demosaic processing unit 1204, gamma correction processing unit 1205, and output processing unit 1206 may be realized by hardware circuits operating in cooperation with each other.

[0079] The pixel processing unit 1200 executes, on the pixel data supplied from the imaging unit 10, an addition process of adding pixel values ​​of four pixels included in a pixel group in an addition mode, and a re-mosaic process based on each pixel data in an individual mode. The pixel processing unit 1200 switches between and executes this addition process and the re-mosaic process in accordance with a switching control signal CNG supplied from the control unit 14. The pixel processing unit 1200 executes the addition process or the re-mosaic process on each of the pixel data of long accumulation, medium accumulation, and short accumulation supplied from the imaging unit 10.

[0080] The pixel data of the long accumulation, medium accumulation, and short accumulation that have been subjected to the addition process or the re-mosaic process in the pixel processing unit 1200 are supplied to the HDR synthesis processing unit 1201. Note that each pixel data at this point is RAW data that corresponds to the color of the color filter corresponding to the position or the color of the position subjected to the re-mosaic process.

[0081] The HDR synthesis processing unit 1201 executes HDR synthesis processing for generating an HDR image by synthesizing the pixel data of the long accumulation, the medium accumulation, and the short accumulation supplied from the pixel processing unit 1200. The pixel data subjected to the HDR synthesis processing has a bit depth of about 20 to 24 bits, and has a wider dynamic range than normal pixel data having a bit depth of about 8 to 10 bits, for example.

[0082] When the HDR synthesis processing unit 1201 executes HDR synthesis processing on pixel data for one frame, for example, the pixel data for one frame generated by the HDR synthesis processing is output as image data for each frame. The image data (HDR image data) output from the HDR synthesis processing unit 1201 is supplied to the WB processing unit 1202, where known white balance processing is performed, and the image data is supplied to the gradation compression processing unit 1203. The gradation compression processing unit 1203 compresses the gradation of the supplied image data, and generates image data having gradation suitable for processing in the subsequent stage. For example, the gradation compression processing unit 1203 converts the bit depth of each pixel data included in the image data from a bit depth of 20 to 24 bits by the HDR synthesis processing to a bit depth of, for example, about 8 to 10 bits, and performs gradation compression.

[0083] The gradation compression processing unit 1203 supplies the gradation compressed image data to the demosaic processing unit 1204. The demosaic processing unit 1204 performs a known demosaic processing on the supplied image data, and gives each pixel a pixel value of R, G, and B. The demosaic processing performed by the demosaic processing unit 1204 is, for example, a process of giving each pixel value of R, G, and B to one pixel of interest using the pixel value of the pixel of interest and the pixel values ​​of each pixel in the vicinity of the pixel of interest.

[0084] The demosaic processing unit 1204 supplies data of each of the R, G, and B components of the image data that has been subjected to the demosaic processing to the gamma correction processing unit 1205. The gamma correction processing unit 1205 executes gamma correction processing based on each of the supplied data, and supplies the gamma-corrected data of each of the R, G, and B components to the output processing unit 13. The output processing unit 13 converts the data of each of the R, G, and B components supplied from the gamma correction processing unit 1205 into data in a format suitable for subsequent processing. In the example of FIG. 15, the output processing unit 13 converts the data of each of the R, G, and B components into image data having a luminance component Y and color difference components RY and BY, and outputs the converted data.

[0085] Fig. 16 is a functional block diagram of an example for explaining the function of the pixel processing unit 1200 according to the first embodiment. In the example of Fig. 16, the pixel processing unit 1200 includes a pixel changeover switch 210, frame memories 200L, 200M, and 200S, processing changeover switches 201L, 201M, and 201S, pixel addition units 202L, 202M, and 202S, re-mosaic processing units 203L, 203M, and 203S, and selectors 204L, 204M, and 204 selectors.

[0086] Among these, the frame memory 200L, the process changeover switch 201L, the pixel addition unit 202L, the re-mosaic processing unit 203L, and the selector 204 are components related to processing of pixel data by long accumulation. Similarly, the frame memories 200M and 200S, the process changeover switches 201M and 201S, the pixel addition units 202M and 202S, the re-mosaic processing units 203M and 203S, and the selectors 204M and 204M are components related to processing of pixel data by medium accumulation and short accumulation, respectively.

[0087] The pixel data output from the imaging unit 10 is input to the pixel changeover switch 210. The pixel changeover switch 210 switches between the first, second and third output terminals in response to a control signal (not shown) supplied from the control unit 14, for example. Here, the first output terminal is for selecting the long accumulation configuration and is connected to the frame memory 200L. The second output terminal is for selecting the medium accumulation configuration and is connected to the frame memory 200M. Moreover, the third output terminal is for selecting the short accumulation configuration and is connected to the frame memory 200S.

[0088] More specifically, for example, the control unit 14 switches the output destination of the pixel changeover switch 210 to the output terminal corresponding to the readout among the first, second, and third output terminals in synchronization with the readout timing of each of the long accumulation, medium accumulation, and short accumulation in the imaging unit 10. The pixel changeover switch 210 switches the first, second, and third output terminals for each line, for example, at timings corresponding to times t1, t2, and t3 shown in FIG.

[0089] The operation of the pixel processing unit 1200 will be described by taking the processing related to pixel data by long-term accumulation as an example. When the pixel changeover switch 210 is switched to the first output terminal, one line of pixel data by long-term accumulation is written to the frame memory 200L. The processing changeover switch 201L changes the supply destination of the pixel data read out from the frame memory 200L to either the pixel addition unit 202L or the re-mosaic processing unit 203L according to the changeover control signal CNG supplied from the control unit 14.

[0090] In addition, the processing changeover switches 201M and 201M in each configuration related to medium accumulation and short accumulation are switched in synchronization with the processing changeover switch 201L according to the switching control signal CNG, and either the pixel addition unit 202M or the re-mosaic processing unit 203M, and either the pixel addition unit 202S or the re-mosaic processing unit 203S are switched.

[0091] First, a case will be described where pixel data for one frame is written to the frame memory 200L and the processing changeover switch 201L is switched to the first output terminal. In this case, the pixel addition unit 202L reads out pixel data in units of pixel groups from the frame memory 200L via the pixel changeover switch 210, and adds up the pixel values ​​of each of the read pixel data for each pixel group. The pixel addition unit 202L outputs one frame of pixel data based on the pixel values ​​added up for each pixel group as image data in the addition mode. The image data in the addition mode output from the pixel addition unit 202L is supplied to a first input terminal of the selector 204L.

[0092] Next, a case will be described where pixel data for one frame is written to the frame memory 200L and the process changeover switch 201L is switched to the first output terminal. In this case, the re-mosaic processor 203L reads out pixel data in processing units (e.g., 6 pixels x 6 pixels) of the re-mosaic process from the frame memory 200L, and performs the re-mosaic process as described above based on each of the read pixel data. The re-mosaic processor 203L outputs the pixel data for one frame that have been subjected to the re-mosaic process as image data in the individual mode. This image data in the individual mode output from the re-mosaic processor 203L is supplied to the second input terminal of the selector 204L.

[0093] The selector 204L and the selectors 204M and 204S select one of the first and second input terminals in synchronization with the above-mentioned processing changeover switches 201L, 201M, and 201S according to the changeover control signal CNG. The outputs of the selector 204L, the selectors 204M, and 204S are output from the pixel processing unit 1200 as image data of long accumulation, medium accumulation, and short accumulation, respectively.

[0094] In addition, the processing of the pixel data of medium accumulation and short accumulation by the frame memory 200M and 200S, the processing changeover switch 201M and 201S, the pixel summing unit 202M and 202S, and the re-mosaic processing unit 203M and 203S is equivalent to the processing by the above-mentioned frame memory 200L, the processing changeover switch 201L, the pixel summing unit 202L, and the re-mosaic processing unit 203L, so the description here is omitted.

[0095] The control unit 14 generates a switching control signal CNG for instructing whether to execute processing in the additive mode or processing in the individual mode in accordance with, for example, a predetermined imaging condition, and supplies the signal to the pixel processing unit 1200. The control unit 14 can apply, for example, the speed of the imaging device 1 relative to the subject as an imaging condition for generating the switching control signal CNG. As another example, the control unit 14 can apply the brightness of the subject or the brightness of the imaging environment as an imaging condition for generating the switching control signal CNG.

[0096] A more specific description will be given by taking as an example a case where the imaging device 1 is mounted on a vehicle. The imaging device 1 acquires vehicle information including information related to the traveling of the vehicle from the vehicle by the vehicle information acquisition unit 15 (see FIG. 1). In the imaging device 1, the control unit 14 extracts information indicating the traveling speed from the acquired vehicle information, and compares the extracted traveling speed with a threshold value v th The control unit 14 compares the traveling speed with the threshold value v th If so, a switching control signal CNG for instructing the individual mode is generated and supplied to the pixel processing unit 1200.

[0097] On the other hand, when the driving speed is the threshold v th If it is less than this, control unit 14 generates a switching control signal CNG for instructing the addition mode and supplies it to pixel processing unit 1200. In the addition mode, the amount of processing in pixel processing unit 1200 and each unit downstream of pixel processing unit 1200 is less than in the individual mode, making it possible to realize power saving.

[0098] The imaging device 1 also acquires information indicating the brightness of the imaging environment (e.g., a luminance value) based on the output of, for example, another sensor mounted on the imaging device 1 or the vehicle. In the imaging device 1, the control unit 14 calculates the brightness by comparing the acquired brightness with a threshold value Y th The control unit 14 compares the brightness with the threshold value Y th If the brightness is equal to or greater than the threshold Y th If it is less than this, the control unit 14 generates a switching control signal CNG for instructing the individual mode, and supplies it to the pixel processing unit 1200.

[0099] In the above description, the control unit 14 generates the switching control signal CNG according to one imaging condition, but this is not limited to this example. The control unit 14 may generate the switching control signal CNG according to a plurality of imaging conditions.

[0100] FIG. 17 is a schematic diagram for explaining an example of generating a switching control signal CNG according to two imaging conditions according to the first embodiment. In the example of FIG. 17, the addition mode and the individual mode are switched based on the vehicle traveling speed and the brightness of the imaging environment described above. That is, the control unit 14 switches between the addition mode and the individual mode when the traveling speed is greater than or equal to the threshold value v th and the brightness is greater than or equal to the threshold Y th If the driving speed is equal to or greater than the threshold value v th Less than Y th If it is less than this, the control circuit 1200 generates a switching control signal CNG for instructing the addition mode and supplies it to the pixel processing unit 1200.

[0101] In this way, by generating the switching control signal CNG for instructing the individual mode and the switching control signal CNG for instructing the additive mode based on the imaging conditions, it is possible to optimize the sensing performance based on the captured image. In addition, by generating the switching control signal CNG based on multiple imaging conditions, more detailed control is possible.

[0102] In addition, in the individual mode, four times as much processing is required for pixel data as in the additive mode. In this case, it is possible to reduce the readout from the imaging unit 10 in the individual mode and the area to be subjected to image processing, for example, to half both horizontally and vertically. This makes it possible to save power and suppress consumption of calculation resources in the image processing unit 12.

[0103] In the above description, the imaging method applied to the first embodiment is described as the rolling shutter method, but this is not limited to this example. That is, the first embodiment can also be applied with the global shutter method, which is an imaging method that performs exposure on all lines simultaneously.

[0104] In the above description, each pixel data of the long accumulation, medium accumulation, and short accumulation that has been subjected to the re-mosaic processing in the pixel processing unit 1200 is supplied to the HDR synthesis processing unit 1201, but this is not limited to this example. For example, after the HDR synthesis processing in the HDR synthesis processing unit 1201, it is also possible to execute the re-mosaic processing by the pixel processing unit 1200. More specifically, the HDR synthesis processing is executed on each pixel data of the long accumulation, medium accumulation, and short accumulation output from the imaging unit 10, and the re-mosaic processing is executed on, for example, one frame's worth of pixel data that has been subjected to the HDR synthesis processing.

[0105] [2-1. First Modification of the First Embodiment] Next, a first modified example of the first embodiment will be described. In the above-mentioned first embodiment, readout processing is performed on the entire surface of a frame in a single mode of the additive mode and the individual mode. In contrast, in the first modified example of the first embodiment, a processing region in which processing is performed in the individual mode and a processing region in which processing is performed in the additive mode are set in the frame.

[0106] 18A and 18B are schematic diagrams for explaining a processing region according to a first modified example of the first embodiment. As shown in FIG. 18A, in the first modified example of the first embodiment, a first processing region 301 for performing processing in an individual mode is set in the center of a frame 300, for example, and a region in the frame 300 other than the first processing region 301 (the periphery of the frame 300) is set as a second processing region 302 for performing processing in an additive mode. The first processing region 301 may have horizontal and vertical sizes that are, for example, half the horizontal and vertical sizes of the frame 300, respectively.

[0107] 18B, when viewed from a vehicle 304 equipped with the imaging device 1 according to the first modification of the first embodiment, for example, a first processing area 301 in the center of a frame 300 includes an image of a gaze area 303 that is farther away in the traveling direction of the vehicle 304 and that the driver of the vehicle should gaze at. Therefore, it is preferable that the first processing area 301 executes image processing in an individual mode with a higher resolution. On the other hand, a second processing area 302 outside the first processing area 301 executes image processing in an additive mode with a lower resolution than the first processing area 301.

[0108] In this way, the first modification of the first embodiment generates the switching control signal CNG with the area in the frame as the imaging condition. By separately setting the individual mode and the additive mode for the central part and the peripheral part of the frame 300, it is possible to save power while including necessary information in the output image.

[0109] Fig. 19 is a functional block diagram of an example for explaining the function of the pixel processing unit 1200a applicable to the first modified example of the first embodiment. Note that Fig. 19 is a diagram corresponding to Fig. 16 described above, and shows the configuration related to long accumulation, and omits the configuration related to medium accumulation and short accumulation.

[0110] The pixel processing unit 1200a shown in FIG. 19 is configured by adding a frame memory 2020L connected to the pixel addition unit 202L and a frame memory 2030L connected to the re-mosaic processing unit 203L to the pixel processing unit 1200 shown in FIG. 16 described above.

[0111] The control unit 14 controls the switching control signal CNG to be switched at the boundary between the first processing area 301 and the second processing area 302, so as to perform processing in the individual mode in the first processing area 301 and processing in the additive mode in the second processing area 302.

[0112] As a more specific example, the control unit 14 supplies the switching control signal CNG to the pixel processing unit 1200 in line sequence, and supplies the switching control signal CNG to the pixel processing unit 1200 from the left end to the right end of the frame 300 for each line.

[0113] The control unit 14 generates a switching control signal CNG for instructing the addition mode from the top end of the frame 300 to the line immediately before the top end of the first processing region 301, and supplies it to the pixel processing unit 1200. For the lines from the top end to the bottom end of the first processing region 301, the control unit 14 generates a switching control signal CNG for instructing the addition mode from the left end of the frame 300 to immediately before the left end of the first processing region 301 for each line, and supplies it to the pixel processing unit 1200. The control unit 14 generates a switching control signal CNG for instructing the individual mode from the left end to the right end of the first processing region 301, and supplies it to the pixel processing unit 1200. The control unit 14 generates a switching control signal CNG for instructing the addition mode from immediately after the right end of the first processing region 301 to the right end of the frame 300, and supplies it to the pixel processing unit 1200. Furthermore, the control unit 14 generates a switching control signal CNG for instructing the addition mode from the line immediately after the bottom edge of the first processing region 301 to the line at the bottom edge of the frame 300, and supplies this to the pixel processing unit 1200.

[0114] While a switching control signal CNG for instructing the addition mode is being supplied, the pixel addition unit 202L reads out pixel data corresponding to the second processing region 302 from the frame memory 200L, for example, line sequentially, and writes the read pixel data to the frame memory 2020L. At this time, in a line including the first processing region 301, the pixel addition unit 202L skips, for example, pixel data inside the first processing region 301. The pixel addition unit 202L performs an addition process on the pixel data written to the frame memory 2020L in pixel group units, and updates the frame memory 2020L with the pixel value resulting from the addition process.

[0115] Similarly, while a switching control signal CNG for instructing the individual mode is being supplied, the re-mosaic processing unit 203L reads out pixel data corresponding to the first processing region 301 from the frame memory 200L and writes the read pixel data to the frame memory 2030L. At this time, since several pixels surrounding the pixel of interest are used in the re-mosaic processing, the re-mosaic processing unit 203L reads out pixel data from the frame memory 200L, taking into account the surrounding pixels. The re-mosaic processing unit 203L executes the re-mosaic processing using the pixel data written to the frame memory 2030L, and updates the frame memory 2030L with pixel values ​​resulting from the re-mosaic processing.

[0116] The pixel data that has been subjected to the addition process by the pixel addition unit 202L and the re-mosaic process by the re-mosaic processing unit 203L and written into the frame memories 2020L and 2030L, respectively, is output via a selector 204L in which a first and a second input terminal are switched according to a switching control signal CNG, whereby the pixel data is synthesized and output as a long-accumulated frame image.

[0117] [2-2. Second Modification of the First Embodiment] Next, a second modified example of the first embodiment will be described. In the first modified example of the first embodiment described above, the processing area for performing the individual mode process and the processing area for performing the additive mode process are set fixedly. In contrast, in the second modified example of the first embodiment, the processing area for performing the individual mode process and the processing area for performing the additive mode process are adaptively set.

[0118] 20 is a schematic diagram for explaining a processing region according to a second modified example of the first embodiment. In the second modified example of the first embodiment, a specific object included in an image based on pixel data read from the imaging unit 10 is detected, and a processing region for performing individual mode processing and a processing region for performing additive mode processing are set based on the detected specific object.

[0119] In the example of Fig. 20, two traffic lights, a sign, and two oncoming vehicles are detected as specific objects from an image 310 based on pixel data read out from the imaging unit 10. The specific objects include, for example, a pedestrian walking on the road. An area 320 including the detected sign, areas 321 and 322 including the two detected traffic lights, and areas 323 and 324 including the two detected oncoming vehicles are set as processing areas for performing individual mode processing, and areas other than these areas 320 to 324 in the image 310 are set as processing areas for performing additive mode processing.

[0120] In this way, the second modified example of the first embodiment generates a switching control signal CNG with an object included in an image of a frame and an area including the object as imaging conditions. For example, processing is performed in the individual mode for an area including an object important for vehicle driving to generate a high-resolution image. On the other hand, for example, an image with a lower resolution than that of the individual mode is generated by the addition mode for an area less important for vehicle driving. This makes it easier to recognize objects important for vehicle driving, and improves driving safety. Furthermore, processing by the addition mode is inferior to the individual mode in terms of the resolution of the generated image, but the processing load is smaller than that of the individual mode. This makes it possible to reduce computing resources and power consumption overall.

[0121] Fig. 21 is a functional block diagram of an example for explaining the function of pixel processing unit 1200b applicable to the second modified example of the first embodiment. Note that pixel processing unit 1200b shown in Fig. 19 is configured such that detection unit 220 is added to the configuration of Fig. 16 described above, and processing changeover switches 201L, 201M, and 201S, and selectors 204L, 204M, and 204S are changed over in accordance with a changeover control signal CNG output from detection unit 220.

[0122] More specifically, the detection unit 220 detects a specific object included in an image based on one frame of image data based on pixel data written in the frame memory 200L, for example. The detection unit 220 can detect the object by pattern recognition based on a pattern registered in advance, for example. However, the detection unit 220 may recognize the object by machine learning processing using a learning model that has been learned in advance using predetermined teacher data.

[0123] The switching control of the process changeover switches 201L, 201M, and 201S, and the selectors 204L, 204M, and 204S by the changeover control signal CNG is similar to the control described using FIG. 19 in the first variant of the first embodiment described above, and therefore will not be described here.

[0124] In addition, the detection unit 220 has been described here as detecting a specific object based on image data in the frame memory 200L, but this is not limited to this example. For example, the detection unit 220 may detect a specific object based on image data written in at least one of the frame memories 200L, 200M, and 200S. In addition, in FIG. 21, the pixel processing unit 1200b is shown as having the function of the detection unit 220, but this is not limited to this example. For example, the control unit 14 may have the function of the detection unit 220.

[0125] [2-3. Third Modification of the First Embodiment] Next, a third modified example of the first embodiment will be described. In the above-mentioned first embodiment and the first and second modified examples of the first embodiment, the pixel array in the imaging unit 10 has been described as a four-division Bayer type RGB array, but the pixel array applicable to the present disclosure is not limited to the four-division Bayer type RGB array. In other words, the present disclosure can be applied to pixel arrays other than the four-division Bayer type RGB array as long as the pixel array has optical filters that transmit light of the same wavelength band arranged in 2 pixels x 2 pixels.

[0126] 22A to 22E are schematic diagrams showing examples of pixel arrays applicable to the present disclosure. Fig. 22A is an example of a pixel array in which a pixel W provided with a filter that transmits light in the entire visible light range is arranged instead of pixel G in a four-division Bayer type RGB array. Fig. 22B is an example of a pixel array in which a pixel Ye provided with a yellow color filter is arranged instead of pixel G in a four-division Bayer type RGB array. Fig. 22C is an example of a pixel array in which a pixel Cy provided with a cyan color filter is arranged instead of pixel B in the pixel array of Fig. 22B.

[0127] The pixel arrays shown in Figs. 22A, 22B, and 22C can obtain higher sensitivity than a 4-division Bayer type RGB array. In addition, the pixel arrays shown in Figs. 22B and 22C, which include pixel Ye, have characteristics that are less susceptible to the effects of lens aberration. On the other hand, none of the pixel arrays shown in Figs. 22A to 22C include G (green) among the three primary colors in additive mixing, and therefore cannot reproduce full color. Therefore, each pixel array shown in Figs. 22A to 22C is suitable for use in sensing applications.

[0128] FIG. 22D shows an example of a pixel array in which pixel groups of 2 pixels x 2 pixels are arranged in 4 columns x 4 rows. In the pixel array shown in FIG. 22D, two pixel groups of pixels R, two pixel groups of pixels B, four pixel groups of pixels G, and eight pixel groups of pixels W are arranged so that pixel groups of pixels of the same color are not adjacent to each other. In the pixel array of FIG. 22D, the eight pixel groups of pixels W can increase sensitivity, and full color reproduction is possible with the pixel groups of pixels R, G, and B arranged in a ratio of 1:2:1. On the other hand, the pixel groups of each color are spaced apart from each other widely compared to, for example, a 4-division Bayer type RGB array, which is disadvantageous in terms of resolution.

[0129] FIG. 22E is an example of a pixel array including a pixel IR provided with an IR (Infrared) filter that transmits light in the infrared region. By using the pixel IR, distance measurement using reflection of infrared light becomes possible. In the example of FIG. 22E, a pixel IR is arranged instead of a pixel B in a four-division Bayer type RGB array. Note that in FIG. 22E, all four pixels B included in the pixel group of pixel B in the four-division Bayer type RGB array are replaced with pixels IR, but this is not limited to this example. For example, a pixel array in which 1 to 3 of the four pixels included in the pixel group of pixel B (or pixel R, or pixel G) are replaced with pixels IR may be used.

[0130] The pixel array shown in Fig. 22E can be used in combination with, for example, a four-division Bayer type RGB array. For example, it is possible to insert the pixel array of Fig. 22E at a predetermined interval into the four-division Bayer type RGB array repeatedly arranged corresponding to the frame. It is also possible to arrange a predetermined number of pixel arrays of Fig. 22E into the four-division Bayer type RGB array repeatedly arranged corresponding to the frame.

[0131] [3. Second embodiment] Next, a second embodiment of the present disclosure will be described. In the above-mentioned first embodiment and each of its modified examples, the addition process of the pixel values ​​of each pixel included in the pixel group in the addition mode is executed in the image processing unit 12. This is not limited to this example, and the addition process can also be executed inside each pixel array unit 110 in the imaging unit 10.

[0132] 23A, 23B, and 23C are schematic diagrams for explaining an example of a method of switching between the individual mode and the addition mode in a pixel group according to the second embodiment.

[0133] Fig. 23A is a schematic diagram for explaining a method for reading out signals from each pixel 100 included in a pixel block according to the second embodiment. Here, the pixel block of pixel G will be explained as an example. As shown in Fig. 23A, pixels 100G1, 100G2, 100G3, and 100G4 included in the pixel block share one floating diffusion layer.

[0134] First, an example of a readout method when the addition mode is instructed by the switching control signal CNG will be described with reference to the time chart in Fig. 23B. Note that in Fig. 23B and Fig. 23C described later, the passage of time is indicated to the right, and pixels 100G1 to 100G4 are indicated as pixels G1, G2, G3, and G4, respectively.

[0135] In the addition mode, the charges accumulated in the light receiving elements of the pixels 100G1, 100G2, 100G3, and 100G4 included in the pixel group are added together within the pixel block and then read out.

[0136] For example, under the control of the control unit 22, the vertical scanning unit 20 00 In the pixel block, the floating diffusion layer is reset, and then charges are read from each light receiving element in the pixels 100G1 to 100G4, and the read charges are transferred to the floating diffusion layer. In the floating diffusion layer, the charges transferred from each light receiving element are added in an adder 140. In this case, the adder 140 corresponds to a floating diffusion layer common to the pixels 100G1 to 100G4. In the floating diffusion layer, the charges transferred from each light receiving element and added are converted into a voltage according to the amount of charge, and output to the vertical signal line VSL as a total pixel signal of each of the pixels 100G1 to 100G4.

[0137] The total pixel signals of the pixels 100G1 to 100G4 are converted into pixel data by an AD converter included in the horizontal scanning unit 21 and supplied to the pixel processing unit 1200c.

[0138] Next, an example of a read method when the individual mode is instructed by the switch control signal CNG will be described with reference to the time chart of FIG. 23C.

[0139] For example, under the control of the control unit 22, the vertical scanning unit 20 10 In the pixel block, the floating diffusion layer is reset, and then, in the pixel 100G1, charges are read from the light receiving element and the read charges are transferred to the floating diffusion layer. In the floating diffusion layer, the transferred charges are converted into a voltage according to the amount of charge, and output to the vertical signal line VSL as a pixel signal read from the pixel 100G1. Note that the addition process by the adder 140 is not performed.

[0140] Next, the vertical scanning unit 20 detects the time t 11In the pixel block, the floating diffusion layer is reset, and then, in the pixel 100G2, charges are read from the light receiving element and the read charges are transferred to the floating diffusion layer. In the floating diffusion layer, the transferred charges are converted into a voltage according to the amount of charge, and output to the vertical signal line VSL as a pixel signal read from the pixel 100G2. Note that the addition process by the adder 140 is not performed.

[0141] The vertical scanning unit 20 similarly reads out pixel signals from the pixels 100G3 and 100G4, resets the floating diffusion layers, and then reads out charges from the light receiving elements and transfers the read out charges to the floating diffusion layers (time t 12 and t 13 ) In each case, the addition process by the adder 140 is not performed.

[0142] Each pixel 100G1~100G 4 The pixel signals read out from the horizontal scanning unit 21 are converted into pixel data by an AD converter included in the horizontal scanning unit 21 and supplied to the pixel processing unit 1200c.

[0143] Fig. 24 is a functional block diagram of an example for explaining the function of the pixel processing unit 1200c applicable to the second embodiment. Note that Fig. 24 is a diagram corresponding to Fig. 16 described above, and shows the configuration related to long accumulation, and omits the configuration related to medium accumulation and short accumulation.

[0144] 24, pixel processing unit 1200c is provided with a frame memory 240L instead of pixel addition unit 202L in pixel processing unit 1200 shown in FIG 16 described above. When pixel data is output from imaging unit 10 in the addition mode, processing changeover switch 230L changes the output destination of the pixel data to frame memory 240L in accordance with changeover control signal CNG. The pixel data output from imaging unit 10 is written to frame memory 240L.

[0145] On the other hand, when pixel data is output from the imaging unit 10 in the individual mode, the output destination of the pixel data is switched to the re-mosaic processing unit 203L. The pixel data output from the imaging unit 10 is supplied to the re-mosaic processing unit 203L and written to the frame memory 2030L. The re-mosaic processing unit 203L performs re-mosaic processing based on the pixel data written to the frame memory 2030L, and updates, for example, the frame memory 2030L with the pixel data subjected to the re-mosaic processing.

[0146] In this way, by executing the addition process of pixel values ​​in the addition mode inside each pixel array section 110 in the imaging section 10, it is possible to reduce the configuration of the image processing section 12 (pixel processing section 1200c).

[0147] [3-1. Modification of the second embodiment] Next, a modified example of the second embodiment will be described. In the modified example of the second embodiment, in a configuration in which pixel value addition processing in the addition mode can be performed inside each pixel array unit 110 in the imaging unit 10, a processing region in which processing in the individual mode is performed and a processing region in which processing in the addition mode is performed can be set within a frame.

[0148] Fig. 25 is a block diagram showing an example of the configuration of the imaging section 10 according to a modified example of the second embodiment. In Fig. 25, the pixel array section 110a is obtained by adding a switching control signal generating section (V) 151V and a switching control signal generating section (H) 151H to the pixel array section 110 described in Fig. 4. Also, in Fig. 25, the pixels are shown grouped into pixel groups each including four pixels, 1001, 1002, 1003, and 1004. Furthermore, an AND circuit 150 is provided for each pixel group.

[0149] In FIG. 25, each pixel group is shown connected to the vertical signal line VSL, but in reality, each of the pixels 1001, 1002, 1003, and 1004 in each pixel group is connected to the vertical signal line VSL.

[0150] The switching control signal generation unit (V) 151V generates a switching control signal CNGV for instructing the additive mode and the individual mode for each column of each pixel group included in the pixel array unit 110a, based on a control signal supplied from, for example, the control unit 22. Furthermore, the switching control signal generation unit (H) 151H generates a switching control signal CNGH for instructing the additive mode and the individual mode for each row of each pixel group included in the pixel array unit 110a, based on a control signal supplied from, for example, the control unit 22.

[0151] In each pixel group, the switching control signal CNGV is input to one input terminal of an AND circuit, and the switching control signal CNGH is input to the other input terminal. The AND circuit 150 takes a logical product of the switching control signals CNGV and CNGH. The AND circuit 150 outputs the logical product of the switching control signals CNGV and CNGH as a switching control signal CNG, which is supplied to the pixel group to which the AND circuit 150 belongs.

[0152] By configuring the pixel array section 110a in this manner, as in the first and second modified examples of the first embodiment described above, it is possible to switch between reading in the additive mode and reading in the individual mode for each pixel group included in the pixel array section 110a in a rectangular area including one or more pixel groups.

[0153] [4. Third embodiment] Next, a usage example of the imaging device to which the technology according to the present disclosure is applied will be described. Fig. 26 is a diagram showing a usage example of the imaging device 1 according to the present disclosure described above.

[0154] The imaging device 1 described above can be used in various cases for sensing light such as visible light, infrared light, ultraviolet light, and X-rays, for example, as described below.

[0155] -Devices that take images for viewing, such as digital cameras and mobile devices with a shooting function. Devices used for traffic purposes, such as on-board sensors that photograph the front, rear, surroundings, and interior of a vehicle for safe driving such as automatic stopping, and for recognition of the driver's condition, surveillance cameras that monitor moving vehicles and roads, and distance measuring sensors that measure distances between vehicles. A device used in home appliances such as TVs, refrigerators, and air conditioners to capture user gestures and operate the appliance in accordance with those gestures. - Equipment used for medical or healthcare purposes, such as endoscopes and devices that take blood vessel images by receiving infrared light. - Equipment used for security purposes, such as surveillance cameras for crime prevention and cameras for person authentication. - Equipment used for beauty purposes, such as skin measuring devices that take pictures of the skin and microscopes that take pictures of the scalp. Equipment used for sports, such as action cameras and wearable cameras for sports purposes. Agricultural equipment, such as cameras for monitoring fields and crop conditions.

[0156] (4-0. Application examples for mobile objects) The technology according to the present disclosure (the present technology) can be applied to various products as described above. For example, the technology according to the present disclosure may be realized as a device mounted on any type of moving body such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility device, an airplane, a drone, a ship, a robot, etc.

[0157] (More specific example of mounting the imaging device of the present disclosure on a vehicle) As an application example of the imaging device 1 according to the present disclosure, a more specific example in which the imaging device 1 is mounted on a vehicle and used will be described.

[0158] (First installation example) First, a first mounting example of the imaging device 1 according to the present disclosure will be described. Fig. 27 is a block diagram showing a system configuration example of a vehicle in which the imaging device 1 according to the present disclosure can be mounted. In Fig. 27, a vehicle system 13200 includes each unit connected to a CAN (Controller Area Network) provided in a vehicle 13000.

[0159] The front sensing camera 13001 is a camera that captures an image of the area ahead in the vehicle travel direction. In general, the camera is not used for image display, but is specialized for sensing. The front sensing camera 13001 is disposed, for example, near the rearview mirror on the inside of the windshield.

[0160] A front camera ECU 13002 receives image data captured by the front sensing camera 13001 and performs image signal processing including image recognition processing such as image quality improvement and object detection. The image recognition result by the front camera ECU is sent to CAN communication.

[0161] Note that ECU is an abbreviation for "Electronic Control Unit."

[0162] The self-driving ECU 13003 is an ECU that controls autonomous driving, and is composed of, for example, a CPU, an ISP, a GPU (Graphics Processing Unit), etc. The image recognition results by the GPU are sent to a server, and the server executes deep learning such as a deep neural network and returns the learned results to the self-driving ECU 13003.

[0163] A GPS (Global Positioning System) 13004 is a location information acquisition unit that receives radio waves from a GPS to obtain the current location. The location information acquired by the GPS 13004 is sent to CAN communication.

[0164] The display 13005 is a display device disposed inside the vehicle body 13000. The display 13005 is disposed in the center of an instrument panel in the vehicle body 13000, inside a rear-view mirror, or the like. The display 13005 may be configured integrally with a car navigation device mounted on the vehicle 13000.

[0165] A communication unit 13006 is responsible for transmitting and receiving data in vehicle-to-vehicle communication, pedestrian-to-vehicle communication, and road-to-vehicle communication. The communication unit 13006 also transmits and receives data to and from a server. The communication unit 13006 can use various types of wireless communication.

[0166] The integrated ECU 13007 is an integrated ECU in which various ECUs are collected. In this example, the integrated ECU 13007 includes an ADAS ECU 13008, a self-driving ECU 13003, and a battery ECU 13010. The battery ECU 13010 controls batteries (such as a 200V battery 13023 and a 12V battery 13024). The integrated ECU 13007 is disposed in the center of the vehicle body 13000, for example.

[0167] The turn signal 13009 is a direction indicator, and its illumination is controlled by the integrated ECU 13007.

[0168] An ADAS ECU (Advanced Driver Assistance System ECU) 13008 generates control signals for controlling components of the vehicle system 13200 in response to driver operations, image recognition results, etc. The ADAS ECU 13008 transmits and receives signals to and from each unit through CAN communication.

[0169] In the vehicle system 13200, the drive source (engine, motor) is controlled by a power train ECU (not shown). During cruise control, the power train ECU controls the drive source according to the image recognition result.

[0170] When image recognition indicates that the vehicle is about to deviate from the white line, the Steering 13011 drives the electronic power steering motor in response to a control signal generated by the ADAS ECU 13008.

[0171] A speed sensor 13012 detects the traveling speed of the vehicle 13000. The speed sensor 13012 calculates acceleration and the derivative of acceleration (jerk) from the traveling speed. The acceleration information is used to calculate an estimated time until a collision with an object. The jerk is an index that affects the riding comfort of the passengers.

[0172] The Radar 13013 is a sensor that measures distances using electromagnetic waves with longer wavelengths such as millimeter waves. The Lidar 13014 is a sensor that measures distances using light.

[0173] The head lamp 13015 includes a lamp and a driving circuit for the lamp, and switches between high beam and low beam depending on the presence or absence of headlights of an oncoming vehicle detected by image recognition. Alternatively, the head lamp 13015 emits a high beam to avoid an oncoming vehicle.

[0174] The side view camera 13016 is a camera disposed inside the housing of the side mirror or near the side mirror. Image data output from the side view camera 13016 is used for image display. The side view camera 13016 captures, for example, an image of the blind spot of the driver. The side view camera 13016 also captures images used for the left and right areas of the around view monitor.

[0175] A side view camera ECU 13017 performs signal processing on the image captured by the side view camera 13016. The side view camera ECU 13017 improves image quality, such as white balance. The image data that has been signal-processed by the side view camera ECU 13017 is sent via a cable different from the CAN.

[0176] The front view camera 13018 is a camera disposed near the front grill. Image data captured by the front view camera 13018 is used for image display. The front view camera 13018 captures an image of a blind spot area in front of the vehicle. The front view camera 13018 also captures an image used in the upper area of ​​the around view monitor. The front view camera 13018 has a different frame layout from the front sensing camera 13001 described above.

[0177] A front view camera ECU 13019 performs signal processing on the image captured by the front view camera 13018. The front view camera ECU 13019 improves image quality, such as white balance. The image data that has been signal-processed by the front view camera ECU 13019 is sent via a cable different from the CAN.

[0178] The vehicle body 13000 includes an engine (ENG) 13020, a generator (GEN) 13021, and a drive motor (MOT) 13022. The engine 13020, the generator 13021, and the drive motor 13022 are controlled by a powertrain ECU (not shown).

[0179] The 200V battery 13023 is a power source for driving and the air conditioner. The 12V battery 13024 is a power source for the driving and the air conditioner. The 12V battery 13024 supplies power to each camera and each ECU mounted on the vehicle body 13000.

[0180] The rear view camera 13025 is a camera disposed, for example, near the license plate on the tailgate. Image data captured by the rear view camera 13025 is used for image display. The rear view camera 13025 captures images of blind spots behind the vehicle. The rear view camera 13025 also captures images to be used in the lower area of ​​an around view monitor. The rear view camera 13025 is activated, for example, by shifting the shift lever to "R (reverse)."

[0181] A rear view camera ECU 13026 performs signal processing on the image captured by the rear view camera 13025. The rear view camera ECU 13026 improves image quality such as white balance. The image data that has been signal processed by the rear view camera ECU 13026 is sent via a cable different from the CAN.

[0182] FIG. 28 is a block diagram showing an example of the configuration of the front sensing camera 13001 of the vehicle system 13200.

[0183] The front camera module 13100 includes a lens 13101, an imager 13102, a front camera ECU 13002, and an MCU (Micro Controller Unit) 13103. The lens 13101 and the imager 13102 configure the above-mentioned front sensing camera 13001. The front camera module 13100 is disposed, for example, near the rearview mirror inside the windshield.

[0184] The imager 13102 can be applied to the imaging unit 10 according to the present disclosure, and captures a forward image using a light receiving element included in a pixel, and outputs pixel data. For example, a Bayer array is used as a color filter array of the pixel. The imager 13102 may be formed of a single-layer chip, or may be a stacked imager in which two or more chips are stacked. The imager 13102 outputs the pixel data as, for example, RAW data. The front camera ECU 13002 includes, for example, an image processing unit 12, an output processing unit 13, and a control unit 14 according to the present disclosure. That is, the imaging device 1 according to the present disclosure is configured by including the imager 13102 and the front camera ECU 13002.

[0185] Either serial transmission or parallel transmission may be applied for data transmission between the imager 13102 and the front camera ECU 13002. Also, it is preferable that the imager 13102 has a function of detecting a failure of the imager 13102 itself.

[0186] The MCU 13103 has a function of interfacing with a CAN bus 13104. The various parts (self-driving ECU 13003, communication unit 13006, ADAS ECU 13008, steering 13011, headlamp 13015, engine 13020, drive motor 13022, ...) shown in Fig. 27 are connected to the CAN bus 13104. In addition, a brake system (Brake) 13030 is also connected to the CAN bus 13040.

[0187] The front camera module 13100 can acquire vehicle information (traveling speed, surrounding brightness, etc.) of the vehicle 13000 from the CAN bus 13040. For example, in the front camera module 13100, the front camera ECU 13002 can instruct whether to execute the pixel readout in the imager 13102 in the addition mode or the individual mode based on the acquired vehicle information. This makes it possible to output image data according to the traveling speed, brightness, etc., and also enables power saving.

[0188] In the above description, the imaging device 1 according to the present disclosure is applied to the front sensing camera 13001, but this is not limited to this example. For example, the imaging device 1 according to the present disclosure may be applied to a front view camera 13018, a side view camera 13016, or a rear view camera 13025.

[0189] (Second installation example) Next, a second installation example of the imaging device 1 according to the present disclosure will be described. Fig. 29 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a moving body control system to which the technology according to the present disclosure can be applied.

[0190] The vehicle control system 12000 includes a plurality of electronic control units connected via a communication network 12001. In the example shown in Fig. 29, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside-vehicle information detection unit 12030, an inside-vehicle information detection unit 12040, and an integrated control unit 12050. Also, as functional configurations of the integrated control unit 12050, a microcomputer 12051, an audio / video output unit 12052, and an in-vehicle network I / F (interface) 12053 are illustrated.

[0191] The drivetrain control unit 12010 controls the operation of devices related to the drivetrain of the vehicle in accordance with various programs. For example, the drivetrain control unit 12010 functions as a control device for a drive force generating device for generating a drive force of the vehicle, such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating a braking force of the vehicle.

[0192] The body system control unit 12020 controls the operation of various devices mounted on the vehicle body according to various programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as head lamps, back lamps, brake lamps, turn signals, and fog lamps. In this case, radio waves or signals of various switches transmitted from a portable device that replaces a key may be input to the body system control unit 12020. The body system control unit 12020 receives the input of these radio waves or signals and controls the door lock device, power window device, lamps, and the like of the vehicle.

[0193] The outside-vehicle information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000. For example, an imaging unit 12031 is connected to the outside-vehicle information detection unit 12030. The outside-vehicle information detection unit 12030 causes the imaging unit 12031 to capture an image outside the vehicle and receives the captured image. The outside-vehicle information detection unit 12030 may perform object detection processing or distance detection processing for a person, a car, an obstacle, a sign, or characters on a road surface, based on the received image.

[0194] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal according to the amount of light received. The imaging unit 12031 can output the electrical signal as an image, or as distance measurement information. The light received by the imaging unit 12031 may be visible light or invisible light such as infrared light.

[0195] The in-vehicle information detection unit 12040 detects information inside the vehicle. For example, a driver state detection unit 12041 that detects the state of the driver is connected to the in-vehicle information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera that captures an image of the driver, and the in-vehicle information detection unit 12040 may calculate the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 12041, or may determine whether the driver is dozing.

[0196] The microcomputer 12051 can calculate control target values ​​for the driving force generating device, the steering mechanism, or the braking device based on the information inside and outside the vehicle acquired by the outside-of-vehicle information detection unit 12030 or the inside-vehicle information detection unit 12040, and output a control command to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control aimed at realizing the functions of an ADAS (Advanced Driver Assistance System), including vehicle collision avoidance or impact mitigation, following driving based on the inter-vehicle distance, vehicle speed maintenance driving, vehicle collision warning, vehicle lane departure warning, etc.

[0197] In addition, the microcomputer 12051 can perform cooperative control for the purpose of autonomous driving, which allows the vehicle to travel autonomously without relying on the driver's operation, by controlling the driving force generating device, steering mechanism, braking device, etc. based on information about the surroundings of the vehicle acquired by the outside vehicle information detection unit 12030 or the inside vehicle information detection unit 12040.

[0198] Furthermore, the microcomputer 12051 can output a control command to the body system control unit 12020 based on the information outside the vehicle acquired by the outside-vehicle information detection unit 12030. For example, the microcomputer 12051 can control the headlamps according to the position of a preceding vehicle or an oncoming vehicle detected by the outside-vehicle information detection unit 12030, and perform cooperative control for the purpose of preventing glare, such as switching from high beams to low beams.

[0199] The audio / video output unit 12052 transmits at least one output signal of audio and video to an output device capable of visually or audibly notifying information to passengers in the vehicle or the outside of the vehicle. In the example of Fig. 29, an audio speaker 12061, a display unit 12062, and an instrument panel 12063 are exemplified as the output device. The display unit 12062 may include at least one of an on-board display and a head-up display, for example.

[0200] FIG. 30 is a diagram showing an example of the installation position of the imaging unit 12031.

[0201] In FIG. 30, a vehicle 12100 has imaging units 12101, 12102, 12103, 12104, and 12105 as an imaging unit 12031.

[0202] The imaging units 12101, 12102, 12103, 12104, and 12105 are provided at positions such as the front nose, side mirrors, rear bumper, back door, and the upper part of the windshield inside the vehicle cabin of the vehicle 12100. The imaging unit 12101 provided at the front nose and the imaging unit 12105 provided at the upper part of the windshield inside the vehicle cabin mainly acquire images of the front of the vehicle 12100. The imaging units 12102 and 12103 provided at the side mirrors mainly acquire images of the sides of the vehicle 12100. The imaging unit 12104 provided at the rear bumper or back door mainly acquires images of the rear of the vehicle 12100. The images of the front acquired by the imaging units 12101 and 12105 are mainly used to detect a preceding vehicle, a pedestrian, an obstacle, a traffic light, a traffic sign, a lane, or the like.

[0203] 30 shows an example of the imaging ranges of the imaging units 12101 to 12104. Imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose, imaging ranges 12112 and 12113 indicate the imaging ranges of the imaging units 12102 and 12103 provided on the side mirrors, respectively, and imaging range 12114 indicates the imaging range of the imaging unit 12104 provided on the rear bumper or back door. For example, an overhead image of the vehicle 12100 viewed from above is obtained by superimposing the image data captured by the imaging units 12101 to 12104.

[0204] At least one of the imaging units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera made up of a plurality of imaging elements, or may be an imaging element having pixels for detecting a phase difference.

[0205] For example, the microcomputer 12051 can extract, as a preceding vehicle, a three-dimensional object that is the closest three-dimensional object on the travel path of the vehicle 12100 and travels at a predetermined speed (for example, 0 km / h or more) in approximately the same direction as the vehicle 12100, by calculating the distance to each three-dimensional object in the imaging ranges 12111 to 12114 and the change over time of this distance (relative speed with respect to the vehicle 12100) based on the distance information obtained from the imaging units 12101 to 12104. Furthermore, the microcomputer 12051 can set a vehicle distance to be secured in advance in front of the preceding vehicle, and perform automatic brake control (including follow-up stop control) and automatic acceleration control (including follow-up start control). In this way, cooperative control can be performed for the purpose of automatic driving that travels autonomously without relying on the driver's operation.

[0206] For example, the microcomputer 12051 classifies and extracts three-dimensional object data on three-dimensional objects into two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects based on distance information obtained from the imaging units 12101 to 12104, and can use the data for automatic obstacle avoidance. For example, the microcomputer 12051 distinguishes obstacles around the vehicle 12100 into obstacles that are visible to the driver of the vehicle 12100 and obstacles that are difficult to see. Then, the microcomputer 12051 determines a collision risk indicating the risk of collision with each obstacle, and when the collision risk is equal to or greater than a set value and a collision is possible, the microcomputer 12051 can provide driving assistance for collision avoidance by outputting an alarm to the driver via the audio speaker 12061 or the display unit 12062, or by performing forced deceleration or avoidance steering via the drive system control unit 12010.

[0207] At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether or not a pedestrian is present in the captured images of the imaging units 12101 to 12104. The recognition of such a pedestrian is performed, for example, by a procedure of extracting feature points in the captured images of the imaging units 12101 to 12104 as infrared cameras, and a procedure of performing pattern matching processing on a series of feature points that indicate the contour of an object to determine whether or not the object is a pedestrian. When the microcomputer 12051 determines that a pedestrian is present in the captured images of the imaging units 12101 to 12104 and recognizes the pedestrian, the audio / image output unit 12052 controls the display unit 12062 to superimpose a rectangular contour line for emphasis on the recognized pedestrian. The audio / image output unit 12052 may also control the display unit 12062 to display an icon or the like indicating a pedestrian at a desired position.

[0208] An example of a vehicle control system to which the technology according to the present disclosure can be applied has been described above. The technology according to the present disclosure can be applied to, for example, the imaging unit 12031 among the above-described configurations. Specifically, the imaging device 1 according to any of the first and second embodiments of the present disclosure and their respective modified examples can be applied to the imaging unit 12031.

[0209] In this case, the imaging unit 12031 can acquire vehicle information (traveling speed, surrounding brightness, etc.) of the vehicle from the communication network 12001. For example, the imaging unit 12031 can instruct whether to read out pixels in the pixel array of the imaging unit 12031 in an additive mode or an individual mode based on the acquired vehicle information. This makes it possible to output image data according to the traveling speed, brightness, etc., and also enables power saving.

[0210] (4-1. Application example to endoscopic surgery system) As a further application example of the imaging device 1 according to the present disclosure, a more specific example in which the imaging device 1 is applied to an endoscopic surgery system will be described.

[0211] FIG. 31 is a diagram showing an example of a schematic configuration of an endoscopic surgery system to which the technology according to the present disclosure (the present technology) can be applied.

[0212] 31 shows a state in which an operator (doctor) 11131 is performing surgery on a patient 11132 on a patient bed 11133 using an endoscopic surgery system 11000. As shown in the figure, the endoscopic surgery system 11000 is composed of an endoscope 11100, other surgical tools 11110 such as an insufflation tube 11111 and an energy treatment tool 11112, a support arm device 11120 that supports the endoscope 11100, and a cart 11200 on which various devices for endoscopic surgery are mounted.

[0213] The endoscope 11100 is composed of a lens barrel 11101, a region of a predetermined length from the tip of which is inserted into a body cavity of a patient 11132, and a camera head 11102 connected to the base end of the lens barrel 11101. In the illustrated example, the endoscope 11100 is configured as a so-called rigid scope having a rigid lens barrel 11101, but the endoscope 11100 may be configured as a so-called flexible scope having a flexible lens barrel.

[0214] An opening into which an objective lens is fitted is provided at the tip of the lens barrel 11101. A light source device 11203 is connected to the endoscope 11100, and light generated by the light source device 11203 is guided to the tip of the lens barrel by a light guide extending inside the lens barrel 11101, and is irradiated via the objective lens toward an observation target in the body cavity of the patient 11132. The endoscope 11100 may be a direct-viewing endoscope, an oblique-viewing endoscope, or a side-viewing endoscope.

[0215] An optical system and an image sensor are provided inside the camera head 11102, and reflected light (observation light) from an observation target is collected on the image sensor by the optical system. The observation light is photoelectrically converted by the image sensor to generate an electrical signal corresponding to the observation light, i.e., an image signal corresponding to an observation image. The image signal is transmitted to a camera control unit (CCU) 11201 as RAW data.

[0216] The CCU 11201 is configured with a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc., and performs overall control of the operations of the endoscope 11100 and the display device 11202. Furthermore, the CCU 11201 receives an image signal from the camera head 11102, and performs various types of image processing on the image signal, such as development processing (demosaic processing), for displaying an image based on the image signal.

[0217] Under the control of the CCU 11201, the display device 11202 displays an image based on an image signal that has been subjected to image processing by the CCU 11201.

[0218] The light source device 11203 is composed of a light source such as an LED (Light Emitting Diode), and supplies the endoscope 11100 with irradiation light when photographing an operation site or the like.

[0219] The input device 11204 is an input interface for the endoscopic surgery system 11000. A user can input various information and instructions to the endoscopic surgery system 11000 via the input device 11204. For example, the user inputs an instruction to change the imaging conditions (type of irradiated light, magnification, focal length, etc.) of the endoscope 11100.

[0220] The treatment tool control device 11205 controls the driving of the energy treatment tool 11112 for cauterizing tissue, incising, sealing blood vessels, etc. The insufflation device 11206 sends gas into the body cavity of the patient 11132 via the insufflation tube 11111 to inflate the body cavity for the purpose of securing the field of view of the endoscope 11100 and securing the working space of the surgeon. The recorder 11207 is a device capable of recording various information related to the surgery. The printer 11208 is a device capable of printing various information related to the surgery in various formats such as text, image, or graph.

[0221] The light source device 11203 that supplies irradiation light to the endoscope 11100 when photographing the surgical site can be composed of a white light source composed of, for example, an LED, a laser light source, or a combination of these. When the white light source is composed of a combination of RGB laser light sources, the output intensity and output timing of each color (each wavelength) can be controlled with high precision, so that the white balance of the captured image can be adjusted in the light source device 11203. In this case, it is also possible to capture images corresponding to each of the RGB colors in a time-division manner by irradiating the observation object with laser light from each of the RGB laser light sources in a time-division manner and controlling the driving of the image sensor of the camera head 11102 in synchronization with the irradiation timing. According to this method, a color image can be obtained without providing a color filter to the image sensor.

[0222] The light source device 11203 may be controlled to change the intensity of the light it outputs at predetermined time intervals. The driving of the image sensor of the camera head 11102 may be controlled in synchronization with the timing of the change in the light intensity to obtain images in a time-division manner, and the images may be synthesized to generate an image with a high dynamic range that is free of so-called blackout and whiteout.

[0223] The light source device 11203 may be configured to supply light of a predetermined wavelength band corresponding to the special light observation. In the special light observation, for example, by utilizing the wavelength dependency of light absorption in body tissue, a narrow band light is irradiated compared to the irradiated light (i.e., white light) during normal observation, and a predetermined tissue such as blood vessels on the mucous membrane surface is photographed with high contrast, so-called narrow band imaging is performed. Alternatively, in the special light observation, a fluorescent observation may be performed in which an image is obtained by fluorescence generated by irradiating an excitation light. In the fluorescent observation, it is possible to irradiate an excitation light to a body tissue and observe the fluorescence from the body tissue (autofluorescence observation), or to locally inject a reagent such as indocyanine green (ICG) into the body tissue and irradiate the body tissue with excitation light corresponding to the fluorescent wavelength of the reagent to obtain a fluorescent image. The light source device 11203 may be configured to supply narrow band light and / or excitation light corresponding to such special light observation.

[0224] FIG. 32 is a block diagram showing an example of the functional configuration of the camera head 11102 and the CCU 11201 shown in FIG.

[0225] The camera head 11102 has a lens unit 11401, an imaging unit 11402, a drive unit 11403, a communication unit 11404, and a camera head control unit 11405. The CCU 11201 has a communication unit 11411, an image processing unit 11412, and a control unit 11413. The camera head 11102 and the CCU 11201 are connected to each other by a transmission cable 11400 so as to be able to communicate with each other.

[0226] The lens unit 11401 is an optical system provided at the connection portion with the lens barrel 11101. Observation light taken in from the tip of the lens barrel 11101 is guided to the camera head 11102 and enters the lens unit 11401. The lens unit 11401 is configured by combining a plurality of lenses including a zoom lens and a focus lens.

[0227] The imaging unit 11402 is composed of an imaging element. The imaging element constituting the imaging unit 11402 may be one (so-called single-plate type) or multiple (so-called multi-plate type). When the imaging unit 11402 is composed of a multi-plate type, for example, each imaging element may generate an image signal corresponding to each of RGB, and a color image may be obtained by combining the image signals. Alternatively, the imaging unit 11402 may be configured to have a pair of imaging elements for acquiring image signals for the right eye and the left eye corresponding to 3D (Dimensional) display. By performing 3D display, the surgeon 11131 can more accurately grasp the depth of the biological tissue in the surgical site. Note that when the imaging unit 11402 is composed of a multi-plate type, multiple lens units 11401 may be provided corresponding to each imaging element.

[0228] Furthermore, the imaging unit 11402 does not necessarily have to be provided in the camera head 11102. For example, the imaging unit 11402 may be provided inside the lens barrel 11101, immediately behind the objective lens.

[0229] The driving unit 11403 is configured by an actuator, and moves the zoom lens and focus lens of the lens unit 11401 by a predetermined distance along the optical axis under the control of the camera head control unit 11405. This allows the magnification and focus of the image captured by the imaging unit 11402 to be appropriately adjusted.

[0230] The communication unit 11404 is configured by a communication device for transmitting and receiving various information to and from the CCU 11201. The communication unit 11404 transmits the image signal obtained from the imaging unit 11402 to the CCU 11201 via the transmission cable 11400 as RAW data.

[0231] Furthermore, the communication unit 11404 receives a control signal for controlling the driving of the camera head 11102 from the CCU 11201, and supplies the control signal to the camera head control unit 11405. The control signal includes information on the imaging conditions, such as information specifying the frame rate of the captured image, information specifying the exposure value at the time of capturing the image, and / or information specifying the magnification and focus of the captured image.

[0232] The image capturing conditions such as the frame rate, exposure value, magnification, and focus may be appropriately specified by a user, or may be automatically set by the control unit 11413 of the CCU 11201 based on an acquired image signal. In the latter case, the endoscope 11100 is equipped with a so-called AE (Auto Exposure) function, AF (Auto Focus) function, and AWB (Auto White Balance) function.

[0233] The camera head control unit 11405 controls the driving of the camera head 11102 based on a control signal received from the CCU 11201 via the communication unit 11404.

[0234] The communication unit 11411 is configured with a communication device for transmitting and receiving various information to and from the camera head 11102. The communication unit 11411 receives an image signal transmitted from the camera head 11102 via the transmission cable 11400.

[0235] Furthermore, the communication unit 11411 transmits, to the camera head 11102, a control signal for controlling the driving of the camera head 11102. The image signal and the control signal can be transmitted by electrical communication, optical communication, or the like.

[0236] The image processing unit 11412 performs various types of image processing on the image signal, which is RAW data sent from the camera head 11102 .

[0237] The control unit 11413 performs various controls related to imaging of the surgical site etc. by the endoscope 11100 and display of the captured image obtained by imaging the surgical site etc. For example, the control unit 11413 generates a control signal for controlling the driving of the camera head 11102.

[0238] Further, the control unit 11413 causes the display device 11202 to display the captured image showing the surgical site, etc., based on the image signal that has been image-processed by the image processing unit 11412. At this time, the control unit 11413 may recognize various objects in the captured image using various image recognition techniques. For example, the control unit 11413 can recognize surgical tools such as forceps, specific living body parts, bleeding, mist when the energy treatment tool 11112 is used, etc., by detecting the shape and color of the edge of an object included in the captured image. When the control unit 11413 causes the display device 11202 to display the captured image, it may use the recognition result to superimpose various types of surgery support information on the image of the surgical site. By superimposing and presenting the surgery support information to the surgeon 11131, the burden on the surgeon 11131 can be reduced and the surgeon 11131 can proceed with the surgery reliably.

[0239] The transmission cable 11400 connecting the camera head 11102 and the CCU 11201 is an electrical signal cable for communication of electrical signals, an optical fiber for optical communication, or a composite cable of these.

[0240] Here, in the illustrated example, communication is performed wired using a transmission cable 11400, but communication between the camera head 11102 and the CCU 11201 may also be performed wirelessly.

[0241] An example of an endoscopic surgery system to which the technology according to the present disclosure can be applied has been described above. The technology according to the present disclosure can be applied to, for example, the endoscope 11100 and the imaging unit 11402 of the camera head 11102 among the configurations described above. Specifically, the imaging element described above can be applied to the imaging unit 10112.

[0242] The imaging unit 10112 to which the imaging element according to the present disclosure is applied can instruct whether to read out pixels in the pixel array of the imaging unit 12031 in an additive mode that allows high-speed readout and has a superior SNR, or in an individual mode that has superior resolution. This allows, for example, the surgeon 11131 to obtain an image corresponding to the surgical site and the situation of the surgical site. This allows, for example, the surgeon 11131 to proceed with the surgery more reliably and also enables power saving.

[0243] Although an endoscopic surgery system has been described here as an example, the technology disclosed herein may also be applied to other systems, such as a microsurgery system.

[0244] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0245] The present technology can also be configured as follows. (1) an imaging unit including a pixel array having a plurality of pixel groups each including N×N pixels (N is an integer equal to or greater than 2), and outputting pixel signals read out from each pixel; a switching unit that switches a readout mode in which the image capturing unit reads out the pixel signals from each of the pixels; Equipped with The switching unit is The read mode is an addition mode in which the pixel signals read from each of the N×N pixels included in the pixel group are added together to form one pixel signal; an individual mode in which the pixel signals read from each of the N×N pixels included in the pixel group are individually output; Switch with Imaging device. (2) The imaging device is mounted on a vehicle, The switching unit is switching the read mode based on vehicle information acquired from the vehicle; The imaging device according to (1) above. (3) The switching unit is switching the read mode depending on the speed of the vehicle acquired as the vehicle information; The imaging device according to (2) above. (4) The switching unit is switching the read mode depending on the brightness around the vehicle acquired as the vehicle information; The imaging device according to (2) or (3). (5) The switching unit is switching the read mode to the individual mode when the speed of the vehicle acquired as the vehicle information is equal to or higher than a predetermined value and the brightness of the surroundings of the vehicle acquired as the vehicle information is equal to or higher than a predetermined value; The imaging device according to any one of (2) to (4). (6) The switching unit is switching the readout mode depending on an area of ​​a frame image based on the pixel signals; The imaging device according to any one of (1) to (5). (7) The switching unit is the readout mode is set to the individual mode in a predetermined region in the center of the frame image, and the readout mode is set to the additive mode in a region other than the predetermined region of the frame image; The imaging device according to (6) above. (8) The switching unit is switching the readout mode based on an object included in a frame image based on the pixel signal; The imaging device according to any one of (1) to (7). (9) The switching unit is when the image includes a specific object, the readout mode is switched between a specific area of ​​the image including the specific object and an area other than the specific area; The imaging device according to (8) above. (10) The imaging device is mounted on a vehicle and used, The switching unit is at least one of a traffic light, a traffic sign, an oncoming vehicle, and a pedestrian is set as the specific object, and the read mode of the specific area including the specific object is switched to the individual mode; The imaging device according to (9) above. (11) The switching unit is switching the readout mode by controlling readout of the pixels included in the pixel array; The imaging device according to any one of (1) to (10) above. (12) The switching unit is switching the readout mode by controlling image processing on the pixel signal output by the imaging unit; The imaging device according to any one of (1) to (10) above. (13) The image forming apparatus further includes a synthesis unit that synthesizes a first image based on a first pixel signal in which the pixel is exposed for a first exposure time and a second image based on a second pixel signal in which the pixel is exposed for a second exposure time following the first exposure. The imaging device according to any one of (1) to (12) above. (14) The pixel array includes: The plurality of pixel groups a first pixel group that receives light transmitted through a first optical filter; a second pixel group that receives light transmitted through the second optical filter; a third pixel group that receives light transmitted through a third optical filter; Including, the first pixel group, the second pixel group, and the third pixel group are arranged such that different pixel groups among the first pixel group, the second pixel group, and the third pixel group are adjacent to each other; The imaging device according to any one of (1) to (13) above. (15) a re-mosaic processing unit that performs a re-mosaic process to convert each pixel included in each of the first pixel group, the second pixel group, and the third pixel group into a pixel having a characteristic according to a position of each pixel when the readout mode is the individual mode, The imaging device according to (14) above. [Explanation of symbols]

[0246] 1. Imaging device 10. Imaging unit 12 Image processing section 13 Output Processing Section 14,22 Control section 15 Vehicle information acquisition unit 110, 110a Pixel array section 201L, 201M, 201S Processing changeover switch 202L, 202M, 202S Pixel addition section 203L, 203M, 203S Re-mosaic processing section 204L, 204M, 204S Selector 220 Detection unit 1200, 1200a, 1200b Pixel processing unit 1201 HDR compositing processing unit

Claims

1. An imaging device mounted on a vehicle, comprising: The imaging device includes: an imaging unit including a pixel array having a plurality of pixel groups each including N×N pixels (N is an integer equal to or greater than 2), and outputting pixel signals read out from each pixel; a switching unit that switches a readout mode in which the image capturing unit reads out the pixel signals from each of the pixels; Equipped with The switching unit is The read mode is selected according to the brightness of the surroundings of the vehicle, which is acquired as vehicle information from the vehicle. an addition mode in which the pixel signals read from each of the N×N pixels included in the pixel group are added together to form one pixel signal; an individual mode in which the pixel signals read from each of the N×N pixels included in the pixel group are individually output; Switch with Imaging device.

2. The switching unit is switching the read mode depending on the speed of the vehicle acquired as the vehicle information; The imaging device according to claim 1 .

3. The switching unit is switching the read mode to the individual mode when the speed of the vehicle acquired as the vehicle information is equal to or higher than a predetermined value and the brightness of the surroundings of the vehicle acquired as the vehicle information is equal to or higher than a predetermined value; The imaging device according to claim 1 .

4. The switching unit is switching the readout mode depending on an area of ​​a frame image based on the pixel signals; The imaging device according to claim 1 .

5. The switching unit is the readout mode is set to the individual mode in a predetermined region in the center of the frame image, and the readout mode is set to the additive mode in a region other than the predetermined region of the frame image; The imaging device according to claim 4.

6. The switching unit is switching the readout mode based on an object included in a frame image based on the pixel signal; The imaging device according to claim 1 .

7. The switching unit is when a specific object is included in the frame image, the readout mode is switched between a specific area of ​​the frame image including the specific object and an area other than the specific area; The imaging device according to claim 6.

8. The switching unit is at least one of a traffic light, a traffic sign, an oncoming vehicle, and a pedestrian is set as the specific object, and the read mode of the specific area including the specific object is switched to the individual mode; The imaging device according to claim 7.

9. The switching unit is switching the readout mode by controlling readout of the pixels included in the pixel array; The imaging device according to claim 1 .

10. The switching unit is switching the readout mode by controlling image processing on the pixel signal output by the imaging unit; The imaging device according to claim 1 .

11. a synthesis unit that synthesizes a first image based on a first pixel signal in which the pixel is exposed for a first exposure time and a second image based on a second pixel signal in which the pixel is exposed for a second exposure time following the first exposure, The imaging device according to claim 1 .

12. The pixel array includes: The plurality of pixel groups a first pixel group that receives light transmitted through a first optical filter; a second pixel group that receives light transmitted through the second optical filter; a third pixel group that receives light transmitted through a third optical filter; Including, the first pixel group, the second pixel group, and the third pixel group are arranged such that different pixel groups among the first pixel group, the second pixel group, and the third pixel group are adjacent to each other; The imaging device according to claim 1 .

13. a re-mosaic processing unit that performs a re-mosaic process to convert each pixel included in each of the first pixel group, the second pixel group, and the third pixel group into a pixel having a characteristic corresponding to a position of each pixel when the readout mode is the individual mode, The imaging device according to claim 12.

Images