imaging device

Abstract

An imaging apparatus capable of adaptively acquiring a captured image according to imaging conditions is provided. An imaging apparatus (1) includes an imaging unit (10) including a pixel array (110) having a plurality of pixel groups each containing N x N pixels (100) (N is an integer of 2 or more), and outputs a pixel signal read from each pixel, and a switching unit (14) that switches a readout mode for reading the pixel signal from each pixel of the imaging unit. The switching unit switches the readout mode between an addition mode in which each of the pixel signals read from each of the N x N pixels contained in the pixel group is added to generate one pixel signal, and an individual mode in which each of the pixel signals read from each of the N x N pixels contained in the pixel group is output respectively.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to an imaging device. BACKGROUND

[0002] Conventionally, a technique is known in which an imaging device performs imaging by controlling an exposure time, a gain of a pixel signal, or the like, based on a subject, a brightness of an imaging environment, or the like, detected by an imaging element.

[0003] LIST OF CITATIONS

[0004] PATENT LITERATURE

[0005] Patent Literature 1: JP 2013-066146 A SUMMARY

[0006] PROBLEMS TO BE SOLVED BY THE INVENTION

[0007] However, conventionally, for example, even for an imaging target for which high resolution is not required, imaging is performed at a resolution similar to that in a normal case, and it is difficult to optimize power consumption and image processing resources.

[0008] An object of the present disclosure is to provide an imaging device capable of adaptively acquiring a captured image according to imaging conditions.

[0009] SOLUTION TO PROBLEM

[0010] In order to solve the above problem, an imaging device according to one aspect of the present disclosure has an imaging unit and a switching unit. The imaging unit includes a pixel array including a plurality of pixel groups each including N x N pixels (N is an integer of 2 or more), and outputs a pixel signal read from each pixel; and the switching unit switches a readout mode in which the pixel signal is read from each pixel by the imaging unit, wherein the switching unit switches the readout mode between an addition mode in which the pixel signals read from the N x N pixels included in the pixel group are added to form one pixel signal, and an individual mode in which each of the pixel signals read from the N x N pixels included in the pixel group is outputted respectively. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 is a functional block diagram for describing an example of the function of the imaging device applicable to each embodiment.

[0012] Figure 2 is a diagram showing an example of a Bayer array.

[0013] Figure 3 is a diagram showing an example of a color filter array applicable to each embodiment.

[0014] Figure 4 is a block diagram showing an example of a configuration of an imaging unit suitable for each embodiment.

[0015] Figure 5 is a block diagram showing an example of a hardware configuration of an imaging device suitable for each embodiment.

[0016] Figure 6 is a diagram for describing detection of a gradient between the same color suitable for the embodiment.

[0017] Figure 7 is a diagram for describing detection of a gradient between different colors suitable for the embodiment.

[0018] Figure 8 is a diagram showing an example of a gradient calculation direction suitable for the embodiment.

[0019] Figure 9 is a diagram showing an example of a gradient calculation direction suitable for the embodiment.

[0020] Figure 10 is a diagram for describing an example of a readout method of a pixel signal in a split Bayer type RGB array.

[0021] Figure 11 is a diagram for describing a first example of HDR imaging.

[0022] Figure 12 is a diagram for describing a second example of HDR imaging.

[0023] Figure 13 is a diagram for describing an example of a case where an additive mode is applied to the second example of HDR imaging.

[0024] Figure 14 is a diagram for describing an example of a case where remosaicking is performed by applying an individual mode to the second example of HDR imaging.

[0025] Figure 15 is a functional block diagram for describing an example of a function of an imaging device according to the first embodiment.

[0026] Figure 16 is a functional block diagram for describing an example of a function of a pixel processing unit according to the first embodiment.

[0027] Figure 17 is a diagram for describing an example of generation of a switching control signal CNG according to two imaging conditions according to the first embodiment.

[0028] Figure 18Ais a schematic diagram for describing a processing region according to the first embodiment of the first modification example.

[0029] Figure 18B is a schematic diagram for describing a processing region according to the first embodiment of the first modification example.

[0030] Figure 19 is a functional block diagram for describing an example of the function of the image processing unit applicable to the first modification example of the first embodiment.

[0031] Figure 20 is a schematic diagram for describing a processing region according to the first embodiment of the second modification example.

[0032] Figure 21 is a functional block diagram for describing an example of the function of the image processing unit applicable to the second modification example of the first embodiment.

[0033] Figure 22A is a schematic diagram showing an example of a pixel array applicable to the present disclosure.

[0034] Figure 22B is a schematic diagram showing an example of a pixel array applicable to the present disclosure.

[0035] Figure 22C is a schematic diagram showing an example of a pixel array applicable to the present disclosure.

[0036] Figure 22D is a schematic diagram showing an example of a pixel array applicable to the present disclosure.

[0037] Figure 22E is a schematic diagram showing an example of a pixel array applicable to the present disclosure.

[0038] Figure 23A is a schematic diagram for describing an example of a switching method between an individual mode and an addition mode in a pixel group according to the second embodiment.

[0039] Figure 23B is a schematic diagram for describing an example of a switching method between an individual mode and an addition mode in a pixel group according to the second embodiment.

[0040] Figure 23C is a schematic diagram for describing an example of a switching method between an individual mode and an addition mode in a pixel group according to the second embodiment.

[0041] Figure 24 is a functional block diagram for describing an example of the function of the pixel processing unit applicable to the second embodiment.

[0042] Figure 25is a block diagram showing an example of a configuration of an imaging unit according to a modification of the second embodiment.

[0043] Figure 26 is a diagram showing a use example of the imaging apparatus according to the present disclosure.

[0044] Figure 27 is a block diagram showing an example of a system configuration of a vehicle on which the imaging apparatus according to the present disclosure can be mounted.

[0045] Figure 28 is a block diagram showing an example of a configuration of a front-facing sensing camera of a vehicle system.

[0046] Figure 29 is a block diagram showing an example of a schematic configuration of a vehicle control system as an example of a mobile body control system to which the technology according to the present disclosure can be applied.

[0047] Figure 30 is a diagram showing an example of a mounting position of an imaging unit.

[0048] Figure 31 is a diagram showing an example of a schematic configuration of an endoscope surgery system.

[0049] Figure 32 is a block diagram showing an example of a functional configuration of a camera head and a camera control unit (CCU). DETAILED DESCRIPTION

[0050] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that, in the following embodiments, the same reference signs are used for the same parts, and overlapping description will be omitted.

[0051] Hereinafter, embodiments of the present disclosure will be described in the following order.

[0052] 1. Technology applicable to each of the embodiments of the present disclosure

[0053] 1-1. Configuration applicable to each of the embodiments

[0054] 1-2. Outline of remosaicking processing

[0055] 1-3. Example of pixel readout method

[0056] 2. First embodiment

[0057] 2-1. First modification of the first embodiment

[0058] 2-2. Second modification of the first embodiment

[0059] 2-3. Third modification of the first embodiment

[0060] 3. Second embodiment

[0061] 3-1. Modification of the second embodiment

[0062] 4. Third embodiment

[0063] 4-0. Application example of the mobile body

[0064] 4-1. Application example of the endoscope surgery system

[0065] (1. Technology applicable to each of the embodiments of the present disclosure)

[0066] (1-1. Configuration applicable to each of the embodiments)

[0067] First, the technology applicable to each of the embodiments of the present disclosure will be described. Figure 1 is a functional block diagram for describing an example of the function of the imaging device applicable to each of the embodiments.

[0068] In Figure 1 , the imaging device 1 includes an imaging unit 10, an optical unit 11, an image processing unit 12, an output processing unit 13, and a control unit 14. Further, in the case where the imaging device 1 is used in a vehicle, the imaging device 1 further includes a vehicle information acquisition unit 15.

[0069] The imaging unit 10 includes a pixel array in which a plurality of pixel arrays each including one or a plurality of light-receiving elements are arranged in a matrix. Further, the optical unit 11 includes a lens, an aperture mechanism, a focusing mechanism, and the like, and guides light from a subject to a light-receiving surface of the pixel array.

[0070] The imaging unit 10 reads out a pixel signal from each of the pixels exposed at a designated exposure time, performs signal processing such as noise removal or gain adjustment or the like on the read-out pixel signal, and converts the pixel signal into digital pixel data. The imaging unit 10 outputs the pixel data based on the pixel signal. The series of operations of exposure performed by the imaging unit 10, reading out of the pixel signal from the exposed pixel, and outputting of the pixel signal as pixel data are referred to as imaging.

[0071] The image processing unit 12 performs predetermined image processing on the pixel data output from the imaging unit 10, and outputs the pixel data as image data, for example, in units of frames. The image processing performed by the image processing unit 12 includes, for example, synthesis and conversion processing of the pixel data and color adjustment processing such as white balance processing or the like.

[0072] The output processing unit 13 converts the image data output from the image processing unit 12 into a format suitable for output from the imaging device 1. For example, the output image data output from the output processing unit 13 is supplied to a display (not shown) and displayed as an image. Alternatively, the output image data can be supplied to another device, such as a device that performs recognition processing on the output image data or a control device that performs control based on the output image data, and the like.

[0073] The control unit 14 controls the overall operation of the imaging device 1. For example, the control unit 14 includes a central processing unit (CPU) and an interface circuit for performing communication with each unit of the imaging device 1, generates various control signals by the CPU operating according to a predetermined program, and controls each unit of the imaging device 1 according to the generated control signals.

[0074] Note that, for example, the image processing unit 12 and the output processing unit 13 described above can include a digital signal processor (DSP) or an image signal processor (ISP) operating according to a predetermined program. Alternatively, one or both of the image processing unit 12 and the output processing unit 13 can be implemented by a program operating on a CPU together with the control unit 14. These programs can be stored in advance in a nonvolatile memory included in the imaging device 1, or can be supplied from the outside to the imaging device 1 and written in the memory.

[0075] In a case where the imaging device 1 is installed on a vehicle and used, the vehicle information acquisition unit 15 acquires vehicle information from the vehicle. The vehicle information acquisition unit 15 acquires the speed of the vehicle, the brightness of the front of the vehicle, and the like as the vehicle information. The vehicle information acquired by the vehicle information acquisition unit 15 is transmitted to the control unit 14. The control unit 14 can instruct the imaging unit 10 or the image processing unit 12 to perform processing according to the transmitted vehicle information.

[0076] Each pixel included in the pixel array included in the imaging unit 10 is provided with a light filter that transmits light in a predetermined wavelength region. Unless otherwise specified, the light filter that transmits light in a predetermined wavelength region will be described as a color filter. In a case where full-color image data is obtained, generally, three types of color filters including a color filter that transmits light in a wavelength band of a red (R) color (hereinafter referred to as an R color filter), a color filter that transmits light in a wavelength band of a green (G) color (hereinafter referred to as a G color filter), and a color filter that transmits light in a wavelength band of a blue (B) color (hereinafter referred to as a B color filter) are used. The array of the R color, G color, and B color filters can have diversity, but generally, an array called a Bayer array is used.

[0077] Figure 2 is a diagram showing an example of a Bayer array. In Figure 2In the Bayer array 120, two pixels 100G on which a G color filter is provided, one pixel 100R on which an R color filter is provided, and one pixel 100B on which a B color filter is provided. In the Bayer array, the four pixels are arranged in a 2 x 2 pixel grid pattern such that the two pixels 100G are not adjacent to each other. In other words, the Bayer array is an array in which the pixels 100 on which the color filters that transmit light of the same wavelength band are provided are not adjacent to each other.

[0078] Hereinafter, unless otherwise specified, the pixel 100R on which the R color filter is provided is referred to as an "R color pixel 100R" or simply as a "pixel 100R". The same applies to the pixel 100G on which the G color filter is provided and the pixel 100B on which the B color filter is provided. Furthermore, when it is not necessary to particularly distinguish the color filters, each of the pixels 100R, 100G, and 100B will be collectively referred to as a pixel 100.

[0079] Figure 3 is a diagram illustrating an example of a color filter array applicable to each of the embodiments. In Figure 3 In the array of the Bayer array, each pixel block including four R color pixels 100R, four G color pixels 100G, and four B color pixels 100B is arranged as a pixel array in the Bayer array, taking a pixel block in which 2 x 2 pixels of the same color are arranged in a grid pattern as a unit. Hereinafter, unless otherwise specified, such a pixel array is referred to as a Quad Bayer type RGB array.

[0080] More specifically, in the Quad Bayer type RGB array, the respective pixel blocks of the R color pixels 100R, the G color pixels 100G, and the B color pixels 100B are arranged in a 2 x 2 grid pattern such that the ratio between the number of pixels 100R, the number of pixels 100G, and the number of pixels 100B is 1:2:1, and the pixel blocks of the pixels of the same color are not adjacent to each other. In Figure 3 In the example of the Bayer array, the pixel block of the G color pixel 100G is arranged below and to the left of the pixel block of the R color pixel 100R, and the pixel block of the B color pixel 100B is arranged diagonally to the pixel block of the R color pixel 100R.

[0081] Figure 4 is a block diagram illustrating an example of a configuration of the imaging unit 10 applicable to each of the embodiments. In Figure 4 In the example of the imaging unit 10, the imaging unit 10 includes a pixel array unit 110, a vertical scanning unit 20, a horizontal scanning unit 21, and a control unit 22.

[0082] The pixel array unit 110 includes a plurality of pixels 100 each of which includes a light-receiving element that generates a voltage corresponding to received light. A photodiode can be used as the light-receiving element. In the pixel array unit 110, the plurality of pixels 100 are arranged in a matrix in a horizontal direction (row direction) and a vertical direction (column direction). In the pixel array unit 110, the arrangement of the pixels 100 in the row direction is referred to as a row. An image (image data) of one frame is formed based on pixel signals read out from a predetermined number of rows in the pixel array unit 110. For example, in a case where an image of one frame is formed with 3000 pixels x 2000 rows, the pixel array unit 110 includes at least 2000 rows each of which includes at least 3000 pixels 100.

[0083] Further, in the pixel array unit 110, a pixel signal line HCTL is connected to each row of the pixels 100, and a vertical signal line VSL is connected to each column of the pixels 100.

[0084] An end portion of the pixel signal line HCTL that is not connected to the pixel array unit 110 is connected to the vertical scanning unit 20. For example, the vertical scanning unit 20 transmits, to the pixel array unit 110, a plurality of control signals such as a drive pulse at the time of reading out a pixel signal from the pixel 100, via the pixel signal line HCTL, in accordance with a control signal provided by the control unit 14. An end portion of the vertical signal line VSL that is not connected to the pixel array unit 110 is connected to the horizontal scanning unit 21.

[0085] The horizontal scanning unit 21 includes an analog-digital (AD) conversion unit, an output unit, and a signal processing unit. The pixel signal read out from the pixel 100 is transmitted to the AD conversion unit of the horizontal scanning unit 21 via the vertical signal line VSL.

[0086] A control to read out a pixel signal from the pixel 100 will be described in outline. The pixel signal is read out from the pixel 100 by transferring charge accumulated in the light-receiving element by exposure to a floating diffusion (FD) layer, and the charge transferred to the floating diffusion layer is converted into a voltage. The voltage obtained by converting the charge in the floating diffusion layer is output to the vertical signal line VSL via an amplifier.

[0087] More specifically, in the pixel 100, during exposure, the photoelectric conversion element and the floating diffusion layer are disconnected from each other (open), and the electric charge corresponding to the incident light generated by photoelectric conversion is accumulated in the photoelectric conversion element. After the exposure is completed, in accordance with a selection signal supplied via the pixel signal line HCTL, the floating diffusion layer is connected to the vertical signal line VSL. Further, in accordance with a reset pulse supplied via the pixel signal line HCTL, the floating diffusion layer is connected to a supply line of the power supply voltage VDD or the black level voltage for a short time, and the floating diffusion layer is reset. The reset level voltage of the floating diffusion layer (referred to as voltage P) is output to the vertical signal line VSL. Thereafter, the photoelectric conversion element and the floating diffusion layer are connected to each other (closed) by a transfer pulse supplied via the pixel signal line HCTL, and the electric charge accumulated in the photoelectric conversion element is transferred to the floating diffusion layer. A voltage corresponding to the amount of electric charge of the floating diffusion layer (referred to as voltage Q) is output to the vertical signal line VSL.

[0088] In the horizontal scanning unit 21, the AD conversion unit includes an AD converter provided for each of the vertical signal lines VSL, and performs AD conversion processing on the pixel signal supplied from the pixel 100 via the vertical signal line VSL by the AD converter, and generates two digital values (values corresponding to the voltage P and the voltage Q, respectively) for performing a correlated double sampling (CDS) process for noise reduction.

[0089] The two digital values generated by the AD converter are subjected to the CDS process by the signal processing unit, and a pixel signal (pixel data) corresponding to the digital signal is generated. The generated pixel data is output from the imaging unit 10.

[0090] Under the control of the control unit 22, the horizontal scanning unit 21 performs selective scanning to select the AD converter for each of the vertical signal lines VSL in a predetermined order, so as to sequentially output each of the digital values temporarily held by the AD converter to the signal processing unit. The horizontal scanning unit 21 realizes this operation by a configuration including, for example, a shift register, an address decoder, and the like.

[0091] The control unit 22 performs drive control of the vertical scanning unit 20, the horizontal scanning unit 21, and the like. The control unit 22 generates various drive signals used as a basis for the operation of the vertical scanning unit 20 and the horizontal scanning unit 21. The control unit 22 generates a control signal supplied to each of the pixels 100 by the vertical scanning unit 20 on the basis of a vertical synchronization signal or an external trigger signal supplied from the outside (for example, the control unit 14) and a horizontal synchronization signal. The control unit 22 supplies the generated control signal to the vertical scanning unit 20.

[0092] Based on a control signal supplied from the control unit 22, the vertical scanning unit 20 supplies various signals including a drive pulse to each of the pixels 100 through the pixel signal line HCTL of the selected pixel row of the pixel array unit 110, and causes each of the pixels 100 to output a pixel signal to the vertical signal line VSL. The vertical scanning unit 20 is implemented by using, for example, a shift register, an address decoder, or the like.

[0093] The imaging unit 10 configured as described above is a column AD system complementary metal-oxide semiconductor (CMOS) image sensor in which an AD converter is provided for each column.

[0094] Figure 5 is a block diagram showing an example of a hardware configuration of the imaging apparatus 1 applicable to each embodiment. In Figure 5 , the imaging apparatus 1 includes a CPU 2000, a read only memory (ROM) 2001, a random access memory (RAM) 2002, an imaging unit 2003, a storage 2004, a data interface (I / F) 2005, an operation unit 2006, and a display control unit 2007, each of which is connected through a bus 2020. Further, the imaging apparatus 1 includes an image processing unit 2010 and an output I / F 2012, each of which is connected through the bus 2020.

[0095] The CPU 2000 controls the overall operation of the imaging apparatus 1 by using the RAM 2002 as a work memory in accordance with a program stored in advance in the ROM 2001.

[0096] The imaging unit 2003 corresponds to the imaging unit 10 in Figure 1 , performs imaging, and outputs pixel data. The pixel data output from the imaging unit 2003 is supplied to the image processing unit 2010. The image processing unit 2010 performs predetermined image processing on the pixel data supplied from the imaging unit 10 by using a frame memory 2011 to generate image data in units of frames.

[0097] The output I / F 2012 is an interface for outputting the image data generated by the image processing unit 2010 to the outside. The output I / F 2012 can convert the image data supplied from the image processing unit 2010 into image data in a predetermined format and output the image data.

[0098] The storage 2004 is, for example, a flash memory, and can store and accumulate the image data output from the image processing unit 2010. The storage 2004 can also store a program for operating the CPU 2000. Further, the storage 2004 is not limited to a configuration built in the imaging apparatus 1, and can be detached from the imaging apparatus 1.

[0099] The data I / F 2005 is an interface for the image forming apparatus 1 to transmit and receive data to and from an external device. For example, as the data I / F 2005, a Universal Serial Bus (USB) can be applied. Further, as the data I / F 2005, an interface that performs short distance wireless communication such as Bluetooth (registered trademark) can be applied.

[0100] The operation unit 2006 receives a user operation with respect to the image forming apparatus 1. The operation unit 2006 includes an operable element such as a dial or a button as an input device that receives a user input. The operation unit 2006 can include a touch panel that outputs a signal corresponding to a contact position as an input device.

[0101] The display control unit 2007 generates a display signal that can be displayed by the display 2008 based on a display control signal transmitted by the CPU 2000. The display 2008 uses, for example, a liquid crystal display (LCD) as a display device, and displays a screen in accordance with the display signal generated by the display control unit 2007. Note that, depending on the use of the image forming apparatus 1, the display control unit 2007 and the display 2008 can be omitted.

[0102] (1-2. Outline of Remosaic Processing)

[0103] Here, the above-described remosaic processing will be described in outline. The processing mode of the pixel signal of each pixel in the 4-division Bayer type RGB array includes a mode in which the pixel signals of the four pixels included in the pixel group are added and processed as one pixel signal (referred to as an addition mode) and a mode in which each of the pixel signals of the four pixels is processed (referred to as an individual mode). In the individual mode, for example, the pixel value of each pixel of the 4-division Bayer type RGB array can be converted into the pixel value of each pixel in the Bayer array by the remosaic processing.

[0104] The image processing unit 12 detects the gradient of the pixel value at the position of the target pixel in each direction, and determines the direction in which the detected gradient is the smallest. The image processing unit 12 predicts the pixel value of the pixel on the Bayer array corresponding to the position of the target pixel based on the pixel value of the pixel in the detected direction, and replaces the pixel value of the target pixel with the predicted pixel value. Thus, the pixel value of each pixel in the 4-division Bayer type RGB array is converted into the pixel value of each pixel in the Bayer array.

[0105] More specifically, the image processing unit 12 detects the gradient between pixels of the same color (gradient between the same colors) as shown in Figure 6 and the gradient between pixels of different colors (gradient between different colors) as shown inFigure 7 The gradient between pixels of different colors is shown in the figure.

[0106] It is important to note that, in Figure 6 and Figure 7 In the accompanying diagrams and similar figures below, each square represents a pixel, and the character (R, G, or B) in each square represents the color of the corresponding color filter (R, G, or B). Here, pixels with R, G, and B color filters respectively are referred to as pixel R, pixel G, and pixel B. Furthermore, the top-left square of the grid is the origin, the horizontal direction is the x-axis, the vertical direction is the y-axis, and the coordinates (x, y) are expressed in pixels. For example, in... Figure 6 In the original, the pixel at the origin is the pixel R with coordinates (0, 0) and is described as pixel (0, 0).

[0107] exist Figure 6 In the example, for pixel G, the gradient of each pair of the same color in the x-direction is detected, including pixels G(0,2) and G(1,2), pixels G(4,2) and G(5,2), pixels G(0,3) and G(1,3), and pixels G(4,3) and G(5,3). Similarly, for pixel B, the gradient of each pair of the same color in the x-direction is detected, including pixels B(2,2) and B(3,2), and pixels B(2,3) and B(3,3).

[0108] On the other hand, Figure 7 In the example, the gradient of each different color in the x-direction is detected, including pixels G(1,2) and B(2,2), G(1,3) and B(2,3), B(3,2) and G(4,2), and B(3,3) and G(4,3).

[0109] Here, gradient detection between pixels of the same color involves calculating the gradient of pixels with the same sensitivity. On the other hand, gradient detection between pixels of different colors involves calculating the gradient of pixels with different sensitivities, and calculating the gradient based on the difference in sensitivity for each color of the pixel. Therefore, the image processing unit 12 detects gradients by multiplying the pixel value of each of pixels R and B included in the input image signal by a predetermined coefficient.

[0110] The direction determination process performed by the image processing unit 12, applicable to each embodiment, will be described in more detail. The image processing unit 12 performs gradient calculations for local regions in multiple directions. Figure 8 and Figure 9 This is a diagram illustrating an example of gradient calculation directions suitable for the implementation scheme. Figure 8 and Figure 9And in similar attached figures below, the size of the local area is 6×6 pixels.

[0111] like Figure 8 As shown, the image processing unit 12 performs gradient calculations in four directions: horizontal (H), vertical (V), upper right 45° (A), and lower right 45° (D). Further, as... Figure 9 As shown, the image processing unit 12 performs gradient calculations in four directions: the upper right 22.5° direction (A2), the upper right 67.5° direction (A3), the lower right 22.5° direction (D2), and the lower right 67.5° direction (D3). That is, in this embodiment, the image processing unit 12 performs gradient calculations in eight directions with angles differing from each other by 22.5°.

[0112] and Figure 2 Compared to the Bayer array shown, in the reference array... Figure 3 In the described 4-segment Bayer RGB array, the sampling interval for the same color component is larger, therefore, foldback occurs at 1 / 2 Nyquist frequency. This is achieved by performing actions such as... Figure 8 and Figure 9 The gradient calculation in each of the eight directions shown can detect such frequency repetition patterns with high precision.

[0113] The image processing unit 12 predicts the pixel value of the pixel corresponding to the position of the target pixel in the Bayer array based on the gradient determination result, and replaces the pixel value of the target pixel with the predicted pixel value.

[0114] (1-3. Examples of pixel readout methods)

[0115] Next, an example of reading pixel signals from a 4-segment Bayer RGB array will be described. Figure 10 This is a schematic diagram illustrating an example of a readout method for segmented Bayer RGB array pixel signals. Figure 10 In the diagram, array 121 represents a pixel array based on a 4-segment Bayer RGB array. Array 130 represents an example of a pixel data array obtained by performing readout relative to array 121 in the aforementioned individual mode and performing re-mosaic processing to convert the pixel value of each pixel into a pixel value in the Bayer array.

[0116] Array 131 represents an example of an array obtained by performing a readout relative to array 121 in the addition mode described above. As mentioned above, in addition mode, four pixels included in a pixel group are collectively treated as one pixel. Figure 10In the example of FIG. 12, as illustrated in the array 131, four pixels R included in the pixel group of the pixel R in the 4-split Bayer-type RGB array (array 121) are regarded as one pixel R (+) by addition of the pixel values. Similarly, four pixels G included in the pixel group of the pixel G in the array 121 are regarded as one pixel G (+) by addition of the pixel values, and four pixels B included in the pixel group of the pixel B are regarded as one pixel B (+) by addition of the pixel values.

[0117] The array 132 represents an example of an array obtained by making the exposure time of at least one of the four pixels included in the pixel group in the array 121 different from the exposure time of the other pixels included in the pixel group. In the example of FIG. 13, the exposure time of the pixel R(L) in the upper left of the pixel group of the pixel R is made different from the exposure time of the other pixels included in the pixel group. In the example of FIG. 13, the exposure time of the pixel R(L) is made longer than the exposure time of the other pixels included in the pixel group. In the example of FIG. 13, the exposure time of the pixel G(L) in the upper left of the pixel group of the pixel G is made different from the exposure time of the other pixels included in the pixel group. In the example of FIG. 13, the exposure time of the pixel G(L) is made longer than the exposure time of the other pixels included in the pixel group. In the example of FIG. 13, the exposure time of the pixel B(L) in the upper left of the pixel group of the pixel B is made different from the exposure time of the other pixels included in the pixel group. In the example of FIG. 13, the exposure time of the pixel B(L) is made longer than the exposure time of the other pixels included in the pixel group. Figure 10 In the example of FIG. 13, the exposure time of one of the four pixels included in the pixel group is a first exposure time, the exposure time of two pixels is a second exposure time longer than the first exposure time, and the exposure time of the remaining one pixel is a third exposure time longer than the second exposure time. Hereinafter, the readout of the first exposure time is referred to as short accumulation, the readout of the second exposure time is referred to as medium accumulation, and the readout of the third exposure time is referred to as long accumulation.

[0118] In the example of FIG. 13, the pixel data (image data) read out by the short accumulation, the medium accumulation, and the long accumulation is synthesized to generate one piece of image data. By synthesizing the image data based on the pixel data exposed with different exposure times, image data of a higher dynamic range can be generated. The high dynamic range image generated in this way is referred to as a high dynamic range (HDR) image. Further, the imaging method of this kind is referred to as HDR imaging. Figure 10 In the example of FIG. 13, for example, in the pixel group of the pixel R, the pixel in the upper left is a long-accumulation pixel R(L), the pixels in the upper right and the lower left are medium-accumulation pixels R(M), and the pixel in the lower right is a short-accumulation pixel R(S). Similarly, in the pixel group of the pixel G, the pixel in the upper left is a long-accumulation pixel G(L), the pixels in the upper right and the lower left are medium-accumulation pixels G(M), and the pixel in the lower right is a short-accumulation pixel G(S). Further, in the pixel group of the pixel B, the pixel in the upper left is a long-accumulation pixel B(L), the pixels in the upper right and the lower left are medium-accumulation pixels B(M), and the pixel in the lower right is a short-accumulation pixel B(S).

[0119] In the array 132, the pixel data (image data) read out by the short accumulation, the medium accumulation, and the long accumulation is synthesized to generate one piece of image data. By synthesizing the image data based on the pixel data exposed with different exposure times, image data of a higher dynamic range can be generated. The high dynamic range image generated in this way is referred to as a high dynamic range (HDR) image. Further, the imaging method of this kind is referred to as HDR imaging.

[0120] In the example of FIG. 13, the pixel data (image data) read out by the short accumulation, the medium accumulation, and the long accumulation is synthesized to generate one piece of image data. By synthesizing the image data based on the pixel data exposed with different exposure times, image data of a higher dynamic range can be generated. The high dynamic range image generated in this way is referred to as a high dynamic range (HDR) image. Further, the imaging method of this kind is referred to as HDR imaging. Figure 10In the example of FIG. 13, as shown in the array 131HDR, for example, in the pixel group of the pixel R, the pixels R(L), two pixels R(M), and the pixel R(S) are synthesized by the HDR imaging, and the pixel R(HDR) having a size corresponding to the pixel group is generated. Similarly, in each of the pixel groups of the pixel G and the pixel B, the pixels G(HDR) and B(HDR) having a size corresponding to the pixel group, respectively, are generated.

[0121] Figure 11 is a schematic diagram for describing a first example of the HDR imaging. In the first example of the HDR imaging, which will be described later, Figure 11 and Figure 12 In the first example of the HDR imaging, the long accumulation exposure, the middle accumulation exposure, and the short accumulation exposure are executed in parallel, and the readout from the long accumulation pixels, the readout from the middle accumulation pixels, and the readout from the short accumulation pixels are executed simultaneously.

[0122] In the first example of the HDR imaging, the long accumulation exposure, the middle accumulation exposure, and the short accumulation exposure are executed in parallel, and the readout from the long accumulation pixels, the readout from the middle accumulation pixels, and the readout from the short accumulation pixels are executed simultaneously.

[0123] For example, in the row Ln#1, the long accumulation exposure is started at the time t0 in a frame time, and the middle accumulation exposure is started at the time t1 which is a predetermined time from the time t0. Further, the short accumulation exposure is started at the time t2 which is a predetermined time from the time t1. At the time t3 which is a predetermined time from the time t2, each of the long accumulation exposure, the middle accumulation exposure, and the short accumulation exposure is terminated, and the readout from the pixels of each of the long accumulation, the middle accumulation, and the short accumulation is executed.

[0124] Figure 12 is a schematic diagram for describing a second example of the HDR imaging. In the second example of the HDR imaging, the long accumulation exposure, the middle accumulation exposure, and the short accumulation exposure are executed sequentially, and the readout from the pixels is executed at each exposure.

[0125] For example, in the row Ln#1, the long accumulation exposure is started at the time t0 in a frame time, and the middle accumulation exposure is started at the time t1 which is a predetermined time from the time t0. Further, the short accumulation exposure is started at the time t2 which is a predetermined time from the time t1. At the time t3 which is a predetermined time from the time t2, each of the long accumulation exposure, the middle accumulation exposure, and the short accumulation exposure is terminated, and the readout from the pixels of each of the long accumulation, the middle accumulation, and the short accumulation is executed. In the example of FIG. 13, as shown in the array 131HDR, for example, in the pixel group of the pixel R, the pixels R(L), two pixels R(M), and the pixel R(S) are synthesized by the HDR imaging, and the pixel R(HDR) having a size corresponding to the pixel group is generated. Similarly, in each of the pixel groups of the pixel G and the pixel B, the pixels G(HDR) and B(HDR) having a size corresponding to the pixel group, respectively, are generated.

[0121] Figure 11 is a schematic diagram for describing a first example of the HDR imaging. In the first example of the HDR imaging, which will be described later, Figure 11 and Figure 12 In the first example of the HDR imaging, the long accumulation exposure, the middle accumulation exposure, and the short accumulation exposure are executed in parallel, and the readout from the long accumulation pixels, the readout from the middle accumulation pixels, and the readout from the short accumulation pixels are executed simultaneously.

[0122] In the first example of the HDR imaging, the long accumulation exposure, the middle accumulation exposure, and the short accumulation exposure are executed in parallel, and the readout from the long accumulation pixels, the readout from the middle accumulation pixels, and the readout from the short accumulation pixels are executed simultaneously.

[0123] For example, in the row Ln#1, the long accumulation exposure is started at the time t0 in a frame time, and the middle accumulation exposure is started at the time t1 which is a predetermined time from the time t0. Further, the short accumulation exposure is started at the time t2 which is a predetermined time from the time t1. At the time t3 which is a predetermined time from the time t2, each of the long accumulation exposure, the middle accumulation exposure, and the short accumulation exposure is terminated, and the readout from the pixels of each of the long accumulation, the middle accumulation, and the short accumulation is executed.

[0124] Figure 12 is a schematic diagram for describing a second example of the HDR imaging. In the second example of the HDR imaging, the long accumulation exposure, the middle accumulation exposure, and the short accumulation exposure are executed sequentially, and the readout from the pixels is executed at each exposure.

[0125] For example, in the row Ln#1, the long accumulation exposure is started at the time t0 in a frame time, and the middle accumulation exposure is started at the time t1 which is a predetermined time from the time t0. Further, the short accumulation exposure is started at the time t2 which is a predetermined time from the time t1. At the time t3 which is a predetermined time from the time t2, each of the long accumulation exposure, the middle accumulation exposure, and the short accumulation exposure is terminated, and the readout from the pixels of each of the long accumulation, the middle accumulation, and the short accumulation is executed. In the example of FIG. 13, as shown in the array 131HDR, for example, in the pixel group of the pixel R, the pixels R(L), two pixels R(M), and the pixel R(S) are synthesized by the HDR imaging, and the pixel R(HDR) having a size corresponding to the pixel group is generated. Similarly, in each of the pixel groups of the pixel G and the pixel B, the pixels G(HDR) and B(HDR) having a size corresponding to the pixel group, respectively, are generated.

[0121] Figure 11 is a schematic diagram for describing a first example of the HDR imaging. In the first example of the HDR imaging, which will be described later, Figure 11 and Figure 12 In the first example of the HDR imaging, the long accumulation exposure, the middle accumulation exposure, and the short accumulation exposure are executed in parallel, and the readout from the long accumulation pixels, the readout from the middle accumulation pixels, and the readout from the short accumulation pixels are executed simultaneously.

[0122] In the first example of the HDR imaging, the long accumulation exposure, the middle accumulation exposure, and the short accumulation exposure are executed in parallel, and the readout from the long accumulation pixels, the readout from the middle accumulation pixels, and the readout from the short accumulation pixels are executed simultaneously.

[0123] For example, in the row Ln#1, the long accumulation exposure is started at the time t0 in a frame time, and the middle accumulation exposure is started at the time t1 which is a predetermined time from the time t0. Further, the short accumulation exposure is started at the time t2 which is a predetermined time from the time t1. At the time t3 which is a predetermined time from the time t2, each of the long accumulation exposure, the middle accumulation exposure, and the short accumulation exposure is terminated, and the readout from the pixels of each of the long accumulation, the middle accumulation, and the short accumulation is executed.

[0124] Figure 12 is a schematic diagram for describing a second example of the HDR imaging. In the second example of the HDR imaging, the long accumulation exposure, the middle accumulation exposure, and the short accumulation exposure are executed sequentially, and the readout from the pixels is executed at each exposure.

[0125] For example, in the row Ln#1, the long accumulation exposure is started at the time t0 in a frame time, and the middle accumulation exposure is started at the time t1 which is a predetermined time from the time t0. Further, the short accumulation exposure is started at the time t2 which is a predetermined time from the time t1. At the time t3 which is a predetermined time from the time t2, each of the long accumulation exposure, the middle accumulation exposure, and the short accumulation exposure is terminated, and the readout from the pixels of each of the long accumulation, the middle accumulation, and the short accumulation is executed.

[0126] In the second example of the HDR imaging, both the addition mode and the individual mode described above can be applied. Figure 13 is a schematic diagram for describing an example of a case where the addition mode is applied to the second example of the HDR imaging.

[0127] In Figure 13 In the upper portion of FIG. 13, the array 131L represents an example of an array obtained by performing a long accumulation exposure on the array 121 and performing readout in the addition mode, the array 131M represents an example of an array obtained by performing a middle accumulation exposure on the array 121 and performing readout in the addition mode, and the array 131S represents an example of an array obtained by performing a short accumulation exposure on the array 121 and performing readout in the addition mode. The array 131L represents an example of an array obtained by performing a long accumulation exposure and performing readout in the addition mode, the array 131M represents an example of an array obtained by performing a middle accumulation exposure and performing readout in the addition mode, and the array 131S represents an example of an array obtained by performing a short accumulation exposure and performing readout in the addition mode. For example, in the pixel group of the pixel R in the array 131L, four pixels R(L) that have undergone a long accumulation exposure are regarded as one pixel R(L+) by addition of pixel values. In the array 131M, four pixels R(M) that have undergone a middle accumulation exposure are regarded as one pixel R(M+) by addition of pixel values. Further, in the array 131S, four pixels R(S) that have undergone a short accumulation exposure are regarded as one pixel R(S+) by addition of pixel values. The same applies to the pixel groups of the other pixels G and B.

[0128] The array 131HDR+ represents an example in which the pixels in the arrays 131L, 131M, and 131S are synthesized. In the array 131HDR+, the pixels R(L+), R(M+), and R(S+) are synthesized to obtain a pixel R(HDR+) having a high dynamic range. Similarly, the pixels G(L+), G(M+), and G(S+) are synthesized and the pixels B(L+), B(M+), and B(S+) are synthesized to obtain pixels G(HDR+) and B(HDR+) each having a high dynamic range, respectively. That is, the array 131HDR+ is an array obtained by synthesizing the pixels in the arrays 131L, 131M, and 131S in units of pixel groups.

[0129] Figure 14 is a schematic diagram for describing an example of a case where the individual mode is applied to the second example of the HDR imaging. The arrays 133L, 133M, and 133S respectively correspond to the arrays 131L, 131M, and 131S of FIG. 13. Figure 13The arrays 131L, 131M, and 131S in FIG. 13 correspond to each other, and four pixels included in each pixel group are shown individually. The arrays 134L, 134M, and 134S represent an example in which the re-mosaicking process is performed on each pixel of the arrays 133L, 133M, and 133S, and the pixel value of four pixels included in each pixel group is converted into the pixel value of each pixel in the Bayer array. In this case, the array 134L includes only each pixel value based on the long-accumulation pixel data, and similarly, the array 134M includes only each pixel value based on the middle-accumulation pixel data, and the array 134S includes only each pixel value based on the short-accumulation pixel data.

[0130] In the lower part of FIG. 13, Figure 14 The array 135HDR shown in the lower part of FIG. 13 is obtained by synthesizing the pixels in the arrays 134L, 134M, and 134S, which correspond to each other in position. As described above, in the second example of the HDR imaging, a high dynamic range can be obtained in each pixel corresponding to the Bayer array.

[0131] Here, the difference between the first example and the second example of the HDR imaging described above will be described with reference to Figure 10 , Figure 13 and Figure 14 In the first example of the HDR imaging, for example, the pixels assigned to the long-accumulation (alternatively, the middle-accumulation or the short-accumulation) are four pixels out of 16 pixels included in the 4-division Bayer-type RGB array (see the array 132 in FIG. 12). On the other hand, in the second example of the HDR imaging, all of the 16 pixels are assigned to the long-accumulation (alternatively, the middle-accumulation or the short-accumulation) (see the arrays 131L, 131M, and 131S in FIG. 13). Therefore, in the case of the same resolution, the second example is advantageous in signal-to-noise ratio (SNR) over the first example. Figure 10 Figure 13 In addition, in the first example of the HDR imaging, since the exposure times of the four pixels included in the pixel group are different (see the array 132 in FIG. 12), it is difficult to apply the re-mosaicking process. On the other hand, in the second example of the HDR imaging, since the exposure times of the 16 pixels included in the 4-division Bayer-type RGB array are the same, it is easy to apply the re-mosaicking process (see FIG. 13). Therefore, in the resolution, the second example is advantageous over the first example.

[0132] In addition, in the first example of the HDR imaging, since the exposure times of the four pixels included in the pixel group are different (see the array 132 in FIG. 12), it is difficult to apply the re-mosaicking process. On the other hand, in the second example of the HDR imaging, since the exposure times of the 16 pixels included in the 4-division Bayer-type RGB array are the same, it is easy to apply the re-mosaicking process (see FIG. 13). Therefore, in the resolution, the second example is advantageous over the first example. Figure 10 Figure 14

[0133] ​​​On the other hand, the additive mode is more advantageous than the individual mode in terms of the fact that pixel readout can be performed at high speed and in terms of SNR. Furthermore, since remosaicking processing can be performed in the individual mode, the individual mode is more advantageous than the additive mode in terms of resolution. In each of the embodiments, pixel readout performed by the imaging unit 10 is performed by adaptively switching between the additive mode and the individual mode. This makes it possible to perform imaging that is appropriate for the imaging purpose or the imaging conditions such as the imaging environment.

[0134] (2. First Embodiment)

[0135] Next, the first embodiment of the present disclosure will be described. The first embodiment is an example of a case where switching between the additive mode and the individual mode is implemented by image processing. Figure 15 is a functional block diagram for describing an example of the function of the imaging apparatus 1 according to the first embodiment.

[0136] Figure 15 The imaging apparatus 1 shown in FIG. 1 performs imaging according to the second example of HDR imaging described above. That is, in the imaging unit 10, a color filter of R color, G color, and B color is provided for the pixels 100 of the pixel array unit 110 according to the 4-division Bayer type RGB array. The control unit 14 controls the imaging unit 10 so as to refer to the color filter of each pixel 100 and perform imaging by the long exposure, the middle exposure, and the short exposure. Figure 12 Imaging is performed by each of the long exposure, the middle exposure, and the short exposure described above. The imaging unit 10 supplies pixel data that is captured by each of the long exposure, the middle exposure, and the short exposure and read out from each of the pixels 100 to the image processing unit 12.

[0137] In the imaging apparatus 1 according to the first embodiment, the image processing unit 12 includes a pixel processing unit 1200, an HDR synthesis processing unit 1201, a white balance (WB) processing unit 1202, a gradation compression processing unit 1203, a demosaicing processing unit 1204, a gamma correction processing unit 1205, and an output processing unit 1206.

[0138] For example, the pixel processing unit 1200, the HDR synthesis processing unit 1201, the WB processing unit 1202, the gradation compression processing unit 1203, the demosaicing processing unit 1204, the gamma correction processing unit 1205, and the output processing unit 1206 are implemented by executing a predetermined program on a processor such as a DSP, an ISP, or a CPU. Alternatively, some or all of the pixel processing unit 1200, the HDR synthesis processing unit 1201, the WB processing unit 1202, the gradation compression processing unit 1203, the demosaicing processing unit 1204, the gamma correction processing unit 1205, and the output processing unit 1206 can be implemented by hardware circuits that operate in cooperation with each other.

[0139] The pixel processing unit 1200 performs an addition process of adding pixel values of four pixels included in a pixel group to the pixel data supplied from the imaging unit 10 in an addition mode and a remosaicing process based on each of the pixel data in an individual mode. The pixel processing unit 1200 performs switching between the addition process and the remosaicing process to perform the corresponding process in accordance with the switching control signal CNG supplied from the control unit 14. The pixel processing unit 1200 performs the addition process or the remosaicing process on each of the long-accumulation pixel data, the middle-accumulation pixel data, and the short-accumulation pixel data supplied from the imaging unit 10.

[0140] Each of the long-accumulation pixel data, the middle-accumulation pixel data, and the short-accumulation pixel data that has undergone the addition process or the remosaicing process in the pixel processing unit 1200 is supplied to the HDR synthesis processing unit 1201. Note that each of the pixel data at this point is original data corresponding to the color of the color filter located at the corresponding position or the color of the position after the remosaicing process.

[0141] The HDR synthesis processing unit 1201 performs an HDR synthesis process of synthesizing the long-accumulation pixel data, the middle-accumulation pixel data, and the short-accumulation pixel data supplied from the pixel processing unit 1200 to generate an HDR image. The pixel data that has undergone the HDR synthesis process has a bit depth of about 20 bits to 24 bits, and has, for example, a higher dynamic range than normal pixel data having a bit depth of about 8 bits to 10 bits.

[0142] For example, when the HDR synthesis process is performed on the pixel data corresponding to one frame, the HDR synthesis processing unit 1201 outputs the pixel data corresponding to one frame generated by the HDR synthesis process as image data in units of frames. The image data (HDR image data) output from the HDR synthesis processing unit 1201 is supplied to the WB processing unit 1202, undergoes a known white balance process by the WB processing unit 1202, and is supplied to the gradation compression processing unit 1203. The gradation compression processing unit 1203 compresses the gradation of the supplied image data and generates image data having a gradation suitable for a later-stage process. For example, the gradation compression processing unit 1203 converts the bit depth of each of the pixel data included in the image data from the bit depth of 20 bits to 24 bits resulting from the HDR synthesis process to a bit depth of, for example, about 8 bits to 10 bits, and performs gradation compression.

[0143] The gray scale compression processing unit 1203 supplies the image data subjected to the gray scale compression processing to the demosaicing processing unit 1204. The demosaicing processing unit 1204 executes a known demosaicing process on the supplied image data, and assigns a pixel value of each of R color, G color, and B color to each pixel. The demosaicing process executed by the demosaicing processing unit 1204 is, for example, a process of assigning a pixel value of each of R color, G color, and B color to one target pixel by using a pixel value of the target pixel and a pixel value of each pixel in the vicinity of the target pixel.

[0144] The demosaicing processing unit 1204 supplies data of components of R color, G color, and B color of the image data subjected to the demosaicing process to the gamma correction processing unit 1205. The gamma correction processing unit 1205 executes a gamma correction process on the basis of the supplied data, and supplies data of each component of R color, G color, and B color subjected to the gamma correction to the output processing unit 13. The output processing unit 13 converts the data of each component of R color, G color, and B color supplied from the gamma correction processing unit 1205 into data in a format suitable for subsequent processes. In Figure 15 In the example, the output processing unit 13 converts the data of each component of R color, G color, and B color into image data including a luminance component Y and each of chroma components R-Y and B-Y, and outputs the image data.

[0145] Figure 16 is a functional block diagram for describing an example of the function of the pixel processing unit 1200 according to the first embodiment. In Figure 16 In the example, the pixel processing unit 1200 includes the pixel switching switch 210, the frame memories 200L, 200M, and 200S, the processing switching switches 201L, 201M, and 201S, the pixel addition units 202L, 202M, and 202S, the remosaicing processing units 203L, 203M, and 203S, and the selectors 204L, 204M, and 204S.

[0146] Among them, the frame memory 200L, the processing switching switch 201L, the pixel addition unit 202L, the remosaicing processing unit 203L, and the selector 204L are components related to the processing for long-accumulation pixel data. Similarly, the frame memory 200M, the processing switching switch 201M, the pixel addition unit 202M, the remosaicing processing unit 203M, and the selector 204M are components related to the processing for middle-accumulation pixel data, and the frame memory 200S, the processing switching switch 201S, the pixel addition unit 202S, the remosaicing processing unit 203S, and the selector 204S are components related to the processing for short-accumulation pixel data.

[0147] The pixel data output from the imaging unit 10 is input to the pixel switching switch 210. The pixel switching switch 210 performs switching among the first, second, and third output terminals in accordance with a control signal (not shown) supplied from the control unit 14, for example. Here, for selection of components related to long accumulation, the first output terminal is connected to the frame memory 200L. For selection of components related to middle accumulation, the second output terminal is connected to the frame memory 200M. Further, for selection of components related to short accumulation, the third output terminal is connected to the frame memory 200S.

[0148] More specifically, for example, the control unit 14 switches the output destination of the pixel switching switch 210 to the output terminal corresponding to the readout among the first, second, and third output terminals in synchronization with the readout timing of each of the long accumulation, middle accumulation, and short accumulation in the imaging unit 10. For example, the pixel switching switch 210 performs switching among the first, second, and third output terminals for each row at a timing corresponding to each of the times tl, t2, and t3 shown in FIG. 6. Figure 12

[0149] The operation of the pixel processing unit 1200 will be described by taking the processing related to long accumulation pixel data as an example. When the pixel switching switch 210 is switched to the first output terminal, the long accumulation pixel data of one row is written into the frame memory 200L. The processing switching switch 201L switches the supply destination of the pixel data read out from the frame memory 200L to the pixel addition unit 202L or the remosaicking processing unit 203L in accordance with a switching control signal CNG supplied from the control unit 14.

[0150] It is to be noted that, in each of the components related to middle accumulation and short accumulation, the processing switching switch 201M is switched to either one of the pixel addition unit 202M and the remosaicking processing unit 203M, and the processing switching switch 201S is switched to either one of the pixel addition unit 202S and the remosaicking processing unit 203S in synchronization with the processing switching switch 201L in accordance with the switching control signal CNG.

[0151] First, a case where the pixel data corresponding to one frame is written into the frame memory 200L and the processing switching switch 201L is switched to the pixel addition unit 202L will be described. In this case, the pixel addition unit 202L reads out the pixel data from the frame memory 200L in units of pixel groups via the processing switching switch 201L, and performs an addition operation on the pixel values of each of the read-out pixel data per pixel group. The pixel addition unit 202L outputs the pixel data corresponding to one frame based on the pixel values added for each pixel group as the addition mode image data. The addition mode image data output from the pixel addition unit 202L is supplied to the first input terminal of the selector 204L. ​

[0152] Next, a case where pixel data corresponding to one frame is written in the frame memory 200L and the processing switching switch 201L is switched to the remosaicking processing unit 203L will be described. In this case, the remosaicking processing unit 203L reads out pixel data from the frame memory 200L in units of processing of remosaicking processing (for example, 6 x 6 pixels), and performs remosaicking processing as described above based on each read-out pixel data. The remosaicking processing unit 203L outputs each of the pixel data corresponding to one frame subjected to remosaicking processing as individual mode image data. The individual mode image data output from the remosaicking processing unit 203L is supplied to the second input terminal of the selector 204L.

[0153] For the selectors 204L, 204M, and 204S, one of the first and second input terminals is selected in synchronization with the processing switching switch 201L, 201M, or 201S described above in accordance with the switching control signal CNG. The outputs of the selectors 204L, 204M, and 204S are output from the pixel processing unit 1200 as long-accumulation image data, middle-accumulation image data, or short-accumulation image data, respectively.

[0154] Note that the processing related to middle-accumulation pixel data performed by the frame memory 200M, the processing switching switch 201M, the pixel adding unit 202M, and the remosaicking processing unit 203M, and the processing related to short-accumulation pixel data performed by the frame memory 200S, the processing switching switch 201S, the pixel adding unit 202S, and the remosaicking processing unit 203S are identical to the processing performed by the frame memory 200L, the processing switching switch 201L, the pixel adding unit 202L, and the remosaicking processing unit 203L described above, and thus the description thereof is omitted here.

[0155] For example, in accordance with a predetermined imaging condition, the control unit 14 generates the switching control signal CNG for instructing which one of the processing in the addition mode and the processing in the individual mode is to be performed, and supplies the switching control signal CNG to the pixel processing unit 1200. For example, the control unit 14 can use the speed of the imaging device 1 with respect to a subject as the imaging condition for generating the switching control signal CNG. As another example, the control unit 14 can use the brightness of the subject or the brightness of the imaging environment as the imaging condition for generating the switching control signal CNG.

[0156] A case where the imaging device 1 is installed on a vehicle and used will be described more specifically as an example. In the imaging device 1, the vehicle information acquisition unit 15 acquires vehicle information including information related to the travel of the vehicle from the vehicle (see Figure 1In the imaging device 1, the control unit 14 extracts information indicating the driving speed from the acquired vehicle information, and compares the extracted driving speed with a threshold v. th Comparison. When the driving speed is equal to or higher than the threshold v. th In this case, the control unit 14 generates a switching control signal CNG to indicate the individual mode and provides the switching control signal CNG to the pixel processing unit 1200.

[0157] On the other hand, when the driving speed is below the threshold v th In this case, the control unit 14 generates a switching control signal CNG to indicate the addition mode and provides the switching control signal CNG to the pixel processing unit 1200. In addition mode, the processing load of the pixel processing unit 1200 or each unit of its subsequent stages is smaller compared to individual mode, and therefore, power saving can be achieved.

[0158] Furthermore, the imaging device 1 acquires information (e.g., brightness values) indicating the brightness of the imaging environment based on the output of the imaging device 1 or, for example, another sensor mounted on the vehicle. In the imaging device 1, the control unit 14 compares the acquired brightness with a threshold Y. th Compare the values. When the brightness is equal to or higher than the threshold Y... th In the case of [unspecified event], the control unit 14 generates a switching control signal CNG to indicate the individual mode and provides the switching control signal CNG to the pixel processing unit 1200. On the other hand, when the brightness is below the threshold Y [unspecified event], [unspecified event] occurs. th In this case, the control unit 14 generates a switching control signal CNG to indicate the addition mode and provides the switching control signal CNG to the pixel processing unit 1200.

[0159] In the above description, the control unit 14 generates a switching control signal CNG based on one imaging condition, but the number of imaging conditions is not limited to this. The control unit 14 can generate a switching control signal CNG based on multiple imaging conditions.

[0160] Figure 17 This is a schematic diagram illustrating an example of generating a switching control signal CNG based on two imaging conditions according to a first embodiment. Figure 17 In the example, the switching between additive mode and individual mode is based on the vehicle's speed and the brightness of the imaging environment. That is, when the vehicle speed is equal to or higher than a threshold v... th and brightness equal to or higher than threshold Y th In the case of [specific situation], the control unit 14 generates a switching control signal CNG to indicate the individual mode and provides the switching control signal CNG to the pixel processing unit 1200. On the other hand, when the driving speed is below the threshold v...th or the luminance is lower than Y th In this case, the control unit 14 generates a switching control signal CNG for instructing the addition mode, and supplies the switching control signal CNG to the pixel processing unit 1200.

[0161] In this way, the switching control signal CNG for instructing the individual mode and the switching control signal CNG for instructing the addition mode are generated based on the imaging conditions, so that the sensing performance based on the captured image can be optimized. Furthermore, the switching control signal CNG is generated based on a plurality of imaging conditions, so that finer control can be performed.

[0162] Note that, in the individual mode, 4 times as much processing of the pixel data is required as compared with the addition mode. At this time, it is conceivable that, for example, the region to be read out and image-processed in the individual mode from the imaging unit 10 is reduced to 1 / 2 in both the horizontal and vertical directions. Thus, power saving can be achieved, and consumption of the computational resources in the image processing unit 12 can be suppressed.

[0163] In the above description, it has been described that the imaging method applied to the first embodiment is the rolling shutter method, but the imaging method is not limited thereto. That is, the global shutter method, which is an imaging method in which exposure is performed at the same time in all rows, can also be applied to the first embodiment.

[0164] Furthermore, in the above description, each of the long-accumulation pixel data, the middle-accumulation pixel data, and the short-accumulation pixel data subjected to the remosaicking processing in the pixel processing unit 1200 is supplied to the HDR synthesis processing unit 1201, but the present disclosure is not limited to this example. For example, the remosaicking processing can be performed by the pixel processing unit 1200 after the HDR synthesis processing is performed by the HDR synthesis processing unit 1201. More specifically, the HDR synthesis processing is performed on each of the long-accumulation pixel data, the middle-accumulation pixel data, and the short-accumulation pixel data output from the imaging unit 10, and, for example, the remosaicking processing is performed on the pixel data corresponding to one frame subjected to the HDR synthesis processing.

[0165] (2-1. First Modification of the First Embodiment)

[0166] Next, the first modification of the first embodiment will be described. In the above-described first embodiment, the readout processing is performed in one of the addition mode and the individual mode for the entire frame. On the other hand, in the first modification of the first embodiment, a processing region to be processed in the individual mode and a processing region to be processed in the addition mode are set within a frame.

[0167] Figure 18A and Figure 18Bis a diagram for describing a processing region of a first modification of the first embodiment. As Figure 18A In the first modification of the first embodiment, as illustrated in

[0168] As illustrated in Figure 18B For example, when viewed from a vehicle 304 on which the imaging device 1 according to the first modification of the first embodiment is mounted, for example, the first processing region 301 at the central portion of the frame 300 includes an image of a gaze region 303 located at a position farther in the traveling direction of the vehicle 304 and which the driver of the vehicle needs to gaze at. Therefore, it is preferable that the first processing region 301 perform image processing in the individual mode at a high resolution. On the other hand, the second processing region 302 outside the first processing region 301 performs image processing in the additive mode at a lower resolution than the first processing region 301.

[0169] As described above, in the first modification of the first embodiment, the switching control signal CNG is generated in accordance with the region in the frame as the imaging condition. By individually setting the individual mode and the additive mode for the central portion and the peripheral portion of the frame 300, respectively, it is possible to save power while including necessary information in the output image.

[0170] Figure 19 is a functional block diagram for describing an example of the function of the pixel processing unit 1200a applicable to the first modification of the first embodiment. Note that, Figure 19 is a diagram corresponding to Figure 16 above, and only components related to long accumulation are illustrated and components related to middle accumulation and short accumulation are omitted.

[0171] In the pixel processing unit 1200a illustrated in Figure 19 , the frame memory 2020L connected to the pixel addition unit 202L and the frame memory 2030L connected to the remosaicking processing unit 203L are added to the pixel processing unit 1200 illustrated in Figure 16 above.

[0172] The control unit 14 performs control by switching the switching control signal CNG at the boundary between the first processing region 301 and the second processing region 302 so that the first processing region 301 is processed in the individual mode and the second processing region 302 is processed in the additive mode.

[0173] As a more specific example, the control unit 14 supplies the switching control signal CNG to the pixel processing unit 1200a in row order, and supplies the switching control signal CNG to the pixel processing unit 1200a for the region from the left end to the right end of the frame 300 for each row.

[0174] The control unit 14 generates the switching control signal CNG for instructing the addition mode for the row at the upper end of the frame 300 to the row immediately before the upper end of the first processing region 301, and supplies the switching control signal CNG to the pixel processing unit 1200a. For the row from the upper end to the lower end of the first processing region 301, the control unit 14 generates the switching control signal CNG for instructing the addition mode for each row for the region from the left end of the frame 300 to the region immediately before the left end of the first processing region 301, and supplies the switching control signal CNG to the pixel processing unit 1200a. The control unit 14 generates the switching control signal CNG for instructing the individual mode for the region from the left end to the right end of the first processing region 301, and supplies the switching control signal CNG to the pixel processing unit 1200a. The control unit 14 generates the switching control signal CNG for instructing the addition mode for the region from the region immediately after the right end of the first processing region 301 to the right end of the frame 300, and supplies the switching control signal CNG to the pixel processing unit 1200a. Further, the control unit 14 generates the switching control signal CNG for instructing the addition mode for the region from the row immediately after the lower end of the first processing region 301 to the row at the lower end of the frame 300, and supplies the switching control signal CNG to the pixel processing unit 1200a.

[0175] For example, when the switching control signal CNG for instructing the addition mode is supplied, the pixel addition unit 202L reads out the pixel data corresponding to the second processing region 302 from the frame memory 200L in row order, and writes the read pixel data to the frame memory 2020L. At this time, for example, the pixel addition unit 202L skips the pixel data inside the first processing region 301 in the row including the first processing region 301. The pixel addition unit 202L performs addition processing on the pixel data written to the frame memory 2020L in units of pixel groups, and updates the frame memory 2020L with the pixel value of the addition processing result.

[0176] Similarly, when the switching control signal CNG for instructing the individual mode is supplied, the remosaicking processing unit 203L reads out pixel data corresponding to the first processing region 301 from the frame memory 200L and writes the read-out pixel data to the frame memory 2030L. At this time, since a peripheral pixel corresponding to several pixels is used for a target pixel in the remosaicking processing, the remosaicking processing unit 203L reads out pixel data from the frame memory 200L taking the peripheral pixel into account. The remosaicking processing unit 203L performs the remosaicking processing by using the pixel data written to the frame memory 2030L, and updates the frame memory 2030L with the pixel values of the remosaicking processing result.

[0177] The plurality of pieces of pixel data subjected to the addition processing performed by the pixel addition unit 202L and the remosaicking processing performed by the remosaicking processing unit 203L and written in the frame memories 2020L and 2030L, respectively, are synthesized by the selector 204L output via its first and second input terminals in accordance with the switching control signal CNG, and output as a long-accumulation frame image.

[0178] (2-2. Second Modification of the First Embodiment)

[0179] Next, the second modification of the first embodiment will be described. In the first modification of the first embodiment described above, the processing region processed in the individual mode and the processing region processed in the addition mode are fixedly set. On the other hand, in the second modification of the first embodiment, the processing region processed in the individual mode and the processing region processed in the addition mode are adaptively set.

[0180] Figure 20 is a schematic view for describing the processing region according to the second modification of the first embodiment. In the second modification of the first embodiment, a specific object included in an image based on the pixel data read out from the imaging unit 10 is detected, and the processing region processed in the individual mode and the processing region processed in the addition mode are set based on the detected specific object.

[0181] In the example of Figure 20 , two traffic signal lights, a sign, and two oncoming vehicles are detected as specific objects from the image 310 based on the pixel data read out from the imaging unit 10. Examples of the specific object include a pedestrian walking on the road. A region 320 including the detected sign, regions 321 and 322 including the detected two traffic signal lights, respectively, and regions 323 and 324 including the detected two oncoming vehicles, respectively, are processing regions to be processed in the individual mode, and a region other than these regions 320 to 324 in the image 310 is a processing region to be processed in the addition mode.

[0182] As described above, in the second modification of the first embodiment, the switching control signal CNG is generated by using an object included in an image of a frame and a region including the object as an imaging condition. For example, a region including an object important for running of a vehicle is processed in an individual mode to generate a high-resolution image. On the other hand, for example, a region less important for running of a vehicle is processed in an additive mode to generate an image having a resolution lower than that in the case of the individual mode. Thus, it becomes easy to recognize an object important for running of a vehicle, and it is possible to improve running safety. Further, processing in the additive mode is inferior to the individual mode in terms of resolution of a generated image, but the processing load is smaller than that in the individual mode. Thus, overall, it is possible to suppress a computing resource and power consumption.

[0183] Figure 21 is a functional block diagram for describing an example of a function of the pixel processing unit 1200b suitable for the second modification of the first embodiment. Note that, Figure 21 The pixel processing unit 1200b illustrated in Figure 16 is configured such that the detection unit 220 is added to the above-described configuration of the pixel processing unit 1200a, and switches the processing switching switches 201L, 201M, and 201S and the selectors 204L, 204M, and 204S in accordance with the switching control signal CNG output from the detection unit 220.

[0184] More specifically, for example, the detection unit 220 detects a specific object included in an image based on image data of one frame based on the pixel data written in the frame memory 200L. For example, the detection unit 220 can detect an object by pattern recognition based on a pre-registered pattern. The present disclosure is not limited to this, and the detection unit 220 can recognize an object by machine learning processing using a learning model pre-trained by using predetermined teacher data.

[0185] Note that the switching control of the processing switching switches 201L, 201M, and 201S and the selectors 204L, 204M, and 204S in accordance with the switching control signal CNG is similar to the control described with reference to Figure 19 in the above-described first modification of the first embodiment, and thus the description thereof is omitted here

[0186] Further, here, it has been described that the detection unit 220 detects a specific object based on the image data of the frame memory 200L, but the present disclosure is not limited to this example. For example, the detection unit 220 can detect a specific object based on image data written in at least one of the frame memories 200L, 200M, and 200S. Further, in Figure 21In the present embodiment, the pixel processing unit 1200b is shown as having the function of the detection unit 220, but the present disclosure is not limited to this example. For example, the control unit 14 can have the function of the detection unit 220.

[0187] (2-3. Third Modification of the First Embodiment)

[0188] Next, a third modification of the first embodiment will be described. In the above-described first embodiment and the first and second modifications of the first embodiment, it has been described that the pixel array in the imaging unit 10 is a 4-division Bayer-type RGB array, but the pixel array applicable to the present disclosure is not limited to the 4-division Bayer-type RGB array. That is, the present disclosure is also applicable to a pixel array other than the 4-division Bayer-type RGB array, as long as the pixel array is a pixel array in which filters that transmit light of the same wavelength band are arranged in 2x2 pixels.

[0189] Figure 22A to Figure 22E is a schematic diagram showing an example of a pixel array applicable to the present disclosure. Figure 22A is an example of a pixel array in which a pixel W on which a filter that transmits light in the entire visible light region is provided is arranged instead of a pixel G in a 4-division Bayer-type RGB array. Figure 22B is an example of a pixel array in which a pixel Ye on which a yellow color filter is provided is arranged instead of a pixel G in a 4-division Bayer-type RGB array. Further, Figure 22C is an example of a pixel array in which a pixel Cy on which a cyan color filter is further arranged instead of a pixel B in the pixel array of Figure 22B .

[0190] With the pixel arrays shown in Figure 22A , Figure 22B and Figure 22C , higher sensitivity can be obtained compared to a 4-division Bayer-type RGB array. Further, the pixel arrays shown in Figure 22B and Figure 22C have a characteristic that they are hardly affected by lens aberration. On the other hand, Figure 22A to Figure 22C all of the pixel arrays of do not include G (green) color in the three primary colors in additive mixing, and thus, full-color reproduction cannot be performed. Therefore, Figure 22A to Figure 22C each of the pixel arrays shown in is suitable for sensing applications.

[0191] Figure 22D is an example of a pixel array in which pixel groups of 2x2 pixels are arranged in four columns by four rows. In Figure 22DIn the pixel array shown in FIG. 1, two pixel groups of pixel R, two pixel groups of pixel B, four pixel groups of pixel G, and eight pixel groups of pixel W are arranged such that the pixel groups of the same color are not adjacent to each other. In the pixel array shown in FIG. 1, the pixel groups of pixel R, pixel G, and pixel B arranged at a ratio of 1:2:1 can achieve full-color reproduction. On the other hand, for example, since the interval between the pixel groups of each color is larger than that of the 4-division Bayer-type RGB array, it is disadvantageous in terms of resolution. Figure 22D In the pixel array of FIG. 2, high sensitivity can be obtained by the eight pixel groups of pixel W, and full-color reproduction can be achieved by the pixel groups of pixel R, pixel G, and pixel B arranged at a ratio of 1:2:1. On the other hand, for example, since the interval between the pixel groups of each color is larger than that of the 4-division Bayer-type RGB array, it is disadvantageous in terms of resolution.

[0192] Figure 22E is an example of a pixel array including a pixel IR on which an infrared (IR) filter that transmits light in an infrared region is provided. Distance measurement using reflection of infrared light and the like can be performed by using the pixel IR. Figure 22E An example of FIG. 2 is an example in which the pixel IR is arranged instead of the pixel B in the 4-division Bayer-type RGB array. Note that, in the pixel array shown in FIG. 2, the pixel groups of pixel R, pixel G, and pixel B arranged at a ratio of 1:2:1 can achieve full-color reproduction. On the other hand, for example, since the interval between the pixel groups of each color is larger than that of the 4-division Bayer-type RGB array, it is disadvantageous in terms of resolution. Figure 22E In FIG. 2, all four pixels B included in the pixel group of pixel B in the 4-division Bayer-type RGB array are replaced with the pixel IR, but the present disclosure is not limited to this. For example, a pixel array in which one to three of the four pixels included in the pixel group of pixel B (optionally, pixel R or pixel G) are replaced with the pixel IR can be used.

[0193] For example, Figure 22E The pixel array shown in FIG. 1 can be used in combination with the 4-division Bayer-type RGB array. For example, it is conceivable to insert the pixel array of FIG. 1 at predetermined intervals in the 4-division Bayer-type RGB array arranged repeatedly corresponding to frames. Figure 22E The pixel array of FIG. 2 can be used in combination with the 4-division Bayer-type RGB array. For example, it is conceivable to insert the pixel array of FIG. 2 at predetermined intervals in the 4-division Bayer-type RGB array arranged repeatedly corresponding to frames. Further, it is also conceivable to arrange a predetermined number of the pixel array of FIG. 2 with respect to the 4-division Bayer-type RGB array arranged repeatedly corresponding to frames. Figure 22E The pixel array shown in FIG. 1 can be used in combination with the 4-division Bayer-type RGB array. For example, it is conceivable to insert the pixel array of FIG. 1 at predetermined intervals in the 4-division Bayer-type RGB array arranged repeatedly corresponding to frames.

[0194] (3. Second Embodiment)

[0195] Next, the second embodiment of the present disclosure will be described. In the first embodiment described above and its modified examples, the addition processing of adding the pixel values of the individual pixels included in the pixel group is performed in the image processing unit 12 in the addition mode. The present disclosure is not limited to this, and the addition processing can be performed inside each pixel array unit 110 of the imaging unit 10.

[0196] Figure 23A 、 Figure 23B and Figure 23C are schematic diagrams for describing examples of a switching method between the individual mode and the addition mode in the pixel group according to the second embodiment.

[0197] Figure 23A This is a schematic diagram illustrating a method for reading signals from each pixel 100 included in a pixel block according to a second embodiment. Here, a pixel block of pixel G is described as an example. Figure 23A As shown, pixels 100G1, 100G2, 100G3 and 100G4 in the pixel block share a floating diffusion layer.

[0198] First, refer to Figure 23B The timing diagram illustrates an example of the readout method when the addition mode is indicated by the switching control signal CNG. It is important to note that... Figure 23B And will be described later Figure 23C In the middle, time increases to the right, and pixels 100G1 to 100G4 are shown as pixels G1, G2, G3 and G4 respectively.

[0199] In addition mode, the accumulated charges in each photoreceiving element of pixels 100G1, 100G2, 100G3 and 100G4 included in the pixel group are added and read out within the pixel block.

[0200] For example, under the control of the control unit 22, the vertical scanning unit 20 in Figure 23B In time t 00 The floating diffusion layer of the reset pixel block is then used to read out the charge from each light-receiving element in pixels 100G1 to 100G4 and transfer the read-out charge to the floating diffusion layer. In the floating diffusion layer, the charges transferred from each light-receiving element are added in the addition unit 140. In this case, the addition unit 140 corresponds to the floating diffusion layer shared by pixels 100G1 to 100G4. In the floating diffusion layer, the charges transferred from each light-receiving element and added are converted into a voltage corresponding to the charge amount, and the voltage is output as the total pixel signal for each pixel 100G1 to 100G4 to the vertical signal line VSL.

[0201] The total pixel signal of each pixel 100G1 to 100G4 is converted into pixel data by the AD converter included in the horizontal scanning unit 21 and provided to the pixel processing unit 1200c.

[0202] Next, refer to Figure 23C The timing diagram illustrates an example of the readout method when the individual mode is indicated by the switching control signal CNG.

[0203] For example, under the control of the control unit 22, the vertical scanning unit 20 in Figure 23C Time t 10The floating diffusion layer of the reset pixel block is then reset, and the charge is read from the light-receiving element of pixel 100G1 and transferred to the floating diffusion layer. In the floating diffusion layer, the transferred charge is converted into a voltage corresponding to the charge amount, and the voltage is output as a pixel signal read from pixel 100G1 to the vertical signal line VSL. It should be noted that the addition process performed by the addition unit 140 is not executed.

[0204] Next, the vertical scanning unit 20 in Figure 23C Time t 11 The floating diffusion layer of the reset pixel block is then reset, and the charge is read from the light-receiving element of pixel 100G2 and transferred to the floating diffusion layer. In the floating diffusion layer, the transferred charge is converted into a voltage corresponding to the charge amount, and the voltage is output as a pixel signal read from pixel 100G2 to the vertical signal line VSL. It should be noted that the addition process performed by the addition unit 140 is not executed.

[0205] Similarly, during the process of reading pixel signals from pixels 100G3 and 100G4, the vertical scanning unit 20 resets the floating diffusion layer, then reads the charge from the light receiving element, and transfers the read-out charge to the floating diffusion layer. Figure 23C In time t 12 and t 13 It should be noted that for each of them, the addition process performed by the addition unit 140 is not executed.

[0206] The AD converter included in the horizontal scanning unit 21 converts the pixel signals read from each of pixels 100G1 to 100G4 into pixel data and provides them to the pixel processing unit 1200c.

[0207] Figure 24 This is a functional block diagram illustrating an example of the functionality of the pixel processing unit 1200c suitable for the second embodiment. It should be noted that... Figure 24 It is the same as the above. Figure 16 The corresponding diagram shows only the components related to long accumulations, omitting the components related to medium and short accumulations.

[0208] exist Figure 24 In the pixel processing unit 1200c shown, a frame memory 240L is provided instead of the aforementioned. Figure 16The pixel addition unit 202L of the pixel processing unit 1200 shown in FIG. 12 is described. Further, in a case where the pixel data is output from the imaging unit 10 in the addition mode, the processing switching switch 230L switches the output destination of the pixel data to the frame memory 240L in accordance with the switching control signal CNG. The pixel data output from the imaging unit 10 is written in the frame memory 240L.

[0209] On the other hand, in a case where the pixel data is output from the imaging unit 10 in the individual mode, the output destination of the pixel data is switched to the remosaicking processing unit 203L. The pixel data output from the imaging unit 10 is supplied to the remosaicking processing unit 203L and written in the frame memory 2030L. The remosaicking processing unit 203L performs remosaicking processing based on the pixel data written in the frame memory 2030L, and, for example, updates the frame memory 2030L with the remosaicked pixel data.

[0210] As described above, by performing the addition processing of the pixel values in the addition mode inside each of the pixel array units 110 of the imaging unit 10, it is possible to simplify the configuration of the image processing unit 12 (the pixel processing unit 1200c).

[0211] (3-1. Modification of the Second Embodiment)

[0212] Next, a modification of the second embodiment will be described. In the modification of the second embodiment, in a configuration in which the addition processing of adding the pixel values in the addition mode can be performed inside each of the pixel array units 110 in the imaging unit 10, it is possible to set a processing region processed in the individual mode and a processing region processed in the addition mode in a frame.

[0213] Figure 25 is a block diagram showing an example of the configuration of the imaging unit 10 according to the modification of the second embodiment. In Figure 25 In the pixel array unit 110a, a switching control signal generating unit (V) 151V and a switching control signal generating unit (H) 151H are added to the pixel array unit 110 described in Figure 4 In the pixel array unit 110 described in Figure 25 In the pixel array unit 110 described in

[0214] Note that, although Figure 25 is shown that each of the pixel groups is connected to the vertical signal line VSL, in actual implementation, each of the pixels 1001, 1002, 1003, and 1004 of each of the pixel groups is connected to the vertical signal line VSL.

[0215] For example, the switching control signal generating unit (V) 151V generates a switching control signal CNGV for indicating the addition mode and the individual mode for each column of the respective pixel groups included in the pixel array unit 110a based on the control signal supplied from the control unit 22. Further, for example, the switching control signal generating unit (H) 151H generates a switching control signal CNGH for indicating the addition mode and the individual mode for each row of the respective pixel groups included in the pixel array unit 110a based on the control signal supplied from the control unit 22.

[0216] For each of the pixel groups, the switching control signal CNGV is input to one input terminal of the AND circuit, and the switching control signal CNGH is input to the other input terminal. In the AND circuit 150, a logical product of the switching control signal CNGV and the switching control signal CNGH is obtained. The AND circuit 150 outputs the result of the logical product of the switching control signal CNGV and the switching control signal CNGH as the switching control signal CNG, and supplies the switching control signal CNG to the pixel group including the AND circuit 150.

[0217] With this configuration of the pixel array unit 110a, similarly to the first and second modified examples of the first embodiment described above, the readout in the addition mode and the readout in the individual mode for each of the pixel groups included in the pixel array unit 110a can be switched and indicated for a rectangular region including one or more pixel groups.

[0218] (4. Third Embodiment)

[0219] Next, a use example of the imaging device to which the technology according to the present disclosure is applied will be described. Figure 26 is a diagram showing a use example of the imaging device 1 according to the present disclosure described above.

[0220] For example, the imaging device 1 described above can be used for various cases of sensing light such as visible light, infrared light, ultraviolet light, and X-rays, as described below.

[0221] - a device for taking an image for viewing, such as a digital camera and a portable device having an imaging function, and the like.

[0222] - a device for transportation, a vehicle-mounted sensor for taking an image of a front, rear, surrounding, or internal region of a vehicle for the purpose of safe driving such as automatic stop and recognition of a driver's condition, a monitoring camera for monitoring a traveling vehicle or a road, or a distance measuring sensor for measuring a distance between vehicles, and the like.

[0223] - a device for a home appliance such as a television (TV), a refrigerator, and an air conditioner, to take an image of a user's gesture and perform a device operation according to the gesture.

[0224] - devices for healthcare, such as an endoscope or a device for taking an image of a blood vessel by receiving infrared light, and the like.

[0225] - devices for security, such as a monitoring camera for security or a camera for personal authentication, and the like.

[0226] - devices for beauty, such as a skin measurement device for taking an image of skin or a microscope for taking an image of a scalp, and the like.

[0227] - devices for sports, such as an action camera or a wearable camera for sports, and the like.

[0228] - devices for agriculture, such as a camera for monitoring the condition of a field and a crop, and the like.

[0229] (4-0. Application examples of mobile bodies)

[0230] The technology according to the present disclosure (the present technology) can be applied to the various products described above. For example, the technology according to the present disclosure can be implemented as a device mounted on any one of mobile bodies such as a car, an electric car, a hybrid car, a motorcycle, a bicycle, a personal mobility device, an airplane, a drone, a ship, and a robot.

[0231] (More specific examples in the case where the imaging device of the present disclosure is mounted on a vehicle)

[0232] As an application example of the imaging device 1 according to the present disclosure, a more specific example in the case where the imaging device 1 is mounted on a vehicle and used will be described.

[0233] (First mounting example)

[0234] First, a first mounting example of the imaging device 1 according to the present disclosure will be described. Figure 27 is a block diagram showing a system configuration example of a vehicle on which the imaging device 1 according to the present disclosure can be mounted. In Figure 27 In the vehicle system 13200, units connected to a controller area network (CAN) provided for the vehicle 13000 are included.

[0235] The front sensing camera 13001 is a camera that takes an image of a front region in a vehicle traveling direction. Usually, this camera is not used for image display but is a camera dedicated to sensing. For example, the front sensing camera 13001 is disposed in the vicinity of a rearview mirror located on the inner side of a windshield.

[0236] The front camera ECU 13002 receives image data taken by the front sensing camera 13001, and performs image signal processing including image recognition processing such as image quality improvement and object detection. The result of image recognition performed by the front camera ECU is transmitted through CAN communication.

[0237] Note that ECU is an abbreviation for "electronic control unit".

[0238] The self-driving ECU 13003 is an ECU that controls automatic driving, and is realized by, for example, a CPU, an ISP, a graphics processor (GPU), or the like. The result of image recognition performed by the GPU is transmitted to a server, and the server performs deep learning such as a deep neural network and returns the learning result to the self-driving ECU 13003.

[0239] The global positioning system (GPS) 13004 is a position information acquisition unit that receives a GPS radio wave and obtains a current position. The position information acquired by the GPS 13004 is transmitted through CAN communication.

[0240] The display 13005 is a display device arranged on the vehicle 13000. The display 13005 is arranged in a central portion of an instrument panel, an interior of a rearview mirror, or the like of the vehicle 13000. The display 13005 can be integrally constituted with a car navigation device mounted on the vehicle 13000.

[0241] The communication unit 13006 is used to perform data transmission and reception in vehicle-to-vehicle communication, pedestrian-to-vehicle communication, and road-to-vehicle communication. The communication unit 13006 also performs transmission and reception with a server. Various types of wireless communication can be applied to the communication unit 13006.

[0242] The integrated ECU 13007 is an integrated ECU in which various ECUs are integrated. In this example, the integrated ECU 13007 includes the ADAS ECU 13008, the self-driving ECU 13003, and a battery ECU 13010. The battery ECU 13010 controls a battery (200V battery 13023, 12V battery 13024, or the like). For example, the integrated ECU 13007 is arranged in a central portion of the vehicle 13000.

[0243] The turn signal 13009 is a direction indicator, and its illumination is controlled by the integrated ECU 13007.

[0244] The advanced driver assistance system (ADAS) ECU 13008 generates a control signal for controlling components of the vehicle system 13200 in accordance with the driver's operation, the image recognition result, and so on. The ADAS ECU 13008 sends and receives a signal to and from each unit through CAN communication.

[0245] In the vehicle system 13200, a drive source (engine or electric motor) is controlled by a powertrain ECU (not shown). During cruise control, the powertrain ECU controls the drive source in accordance with the image recognition result.

[0246] When the vehicle is about to deviate from the white line in the image recognition, the steering 13011 drives an electric power steering motor in accordance with the control signal generated by the ADAS ECU 13008.

[0247] A speed sensor 13012 detects the running speed of the vehicle 13000. The speed sensor 13012 calculates acceleration and a differential of the acceleration (jerk) from the running speed. The acceleration information is used to calculate an estimated time before a collision with an object. The jerk is an index that affects the ride comfort of the passenger.

[0248] A radar 13013 is a sensor that performs distance measurement by using an electromagnetic wave having a long wavelength such as a millimeter wave. A laser radar 13014 is a sensor that performs distance measurement by using light.

[0249] A headlamp 13015 includes a lamp and a driving circuit of the lamp, and performs switching between high beam and low beam in accordance with the presence or absence of a headlamp of an oncoming vehicle detected by image recognition. Alternatively, the headlamp 13015 emits high beam so as to avoid an oncoming vehicle.

[0250] A side view camera 13016 is a camera that is disposed in a housing of a side mirror or in the vicinity of the side mirror. Image data output from the side view camera 13016 is used for image display. For example, the side view camera 13016 captures an image of a blind spot area of the driver. Further, the side view camera 13016 captures an image of a left and right area for a surround monitor.

[0251] A side view camera ECU 13017 performs signal processing on an image captured by the side view camera 13016. The side view camera ECU 13017 improves image quality such as white balance. Image data subjected to signal processing by the side view camera ECU 13017 is transmitted through a cable different from CAN.

[0252] The forward-looking camera 13018 is a camera arranged near the front grille. Image data taken by the forward-looking camera 13018 is used for image display. The forward-looking camera 13018 takes an image of a blind spot area in front of the vehicle. Further, the forward-looking camera 13018 takes an image used in the upper area of the surround monitor. The forward-looking camera 13018 differs from the aforementioned front-sensing camera 13001 in terms of frame layout.

[0253] The forward-looking camera ECU 13019 performs signal processing on an image taken by the forward-looking camera 13018. The forward-looking camera ECU 13019 improves image quality such as white balance. Image data subjected to signal processing by the forward-looking camera ECU 13019 is transmitted through a cable different from the CAN.

[0254] The vehicle 13000 includes an engine (ENG) 13020, a generator (GEN) 13021, and a drive motor (MOT) 13022. The engine 13020, the generator 13021, and the drive motor 13022 are controlled by a powertrain ECU (not shown).

[0255] The 200V battery 13023 is a power source for driving and air conditioning. The 12V battery 13024 is a power source other than the power source for driving and air conditioning. The 12V battery 13024 supplies power to each camera and each ECU mounted on the vehicle 13000.

[0256] The rear-view camera 13025, for example, is a camera arranged near the license plate of the tailgate. Image data taken by the rear-view camera 13025 is used for image display. The rear-view camera 13025 takes an image of a blind spot area behind the vehicle. Further, the rear-view camera 13025 takes an image used in the lower area of the surround monitor. The rear-view camera 13025 is activated, for example, by moving the shift lever to "R (reverse)".

[0257] The rear-view camera ECU 13026 performs signal processing on an image taken by the rear-view camera 13025. The rear-view camera ECU 13026 improves image quality such as white balance. Image data subjected to signal processing by the rear-view camera ECU 13026 is transmitted through a cable different from the CAN.

[0258] Figure 28 is a block diagram showing an example configuration of the front-sensing camera 13001 of the vehicle system 13200.

[0259] The front camera module 13100 includes a lens 13101, an imager 13102, a front camera ECU 13002, and a micro controller unit (MCU) 13103. The lens 13101 and the imager 13102 are included in the front sensing camera 13001 described above. The front camera module 13100 is disposed, for example, in the vicinity of a rearview mirror located on the inner side of a windshield.

[0260] The imager 13102 can be implemented by using the imaging unit 10 according to the present disclosure, and captures a front region image by using light-receiving elements included in pixels and outputs pixel data. For example, a Bayer array is used as a color filter array of the pixels. The imager 13102 can be formed of a single layer chip, or can be a stacked imager in which two or more chips are stacked. The imager 13102 outputs pixel data as raw data, for example. The front camera ECU 13002 includes the image processing unit 12, the output processing unit 13, and the control unit 14 according to the present disclosure, for example. That is, the imaging device 1 according to the present disclosure includes the imager 13102 and the front camera ECU 13002.

[0261] Note that serial transmission or parallel transmission can be applied to data transmission between the imager 13102 and the front camera ECU 13002. Further, preferably, the imager 13102 has a function of detecting a failure of the imager 13102 itself.

[0262] The MCU 13103 has a function of interfacing with the CAN bus 13040. Figure 27 Each unit (the self-driving ECU 13003, the communication unit 13006, the ADAS ECU 13008, the steering 13011, the headlamp 13015, the engine 13020, the drive motor 13022, and the like) illustrated in FIG. 13 is connected to the CAN bus 13040. The brake system 13030 is also connected to the CAN bus 13040.

[0263] The front camera module 13100 can obtain vehicle information (traveling speed, environmental brightness, and the like) of the vehicle 13000 from the CAN bus 13040. In the front camera module 13100, for example, the front camera ECU 13002 can instruct whether to perform readout of the pixels in the imager 13102 in an additive mode or an individual mode on the basis of the acquired vehicle information. This makes it possible to output image data corresponding to the traveling speed, the brightness, and the like, and to save power.

[0264] Note that, in the above description, the imaging device 1 according to the present disclosure has been described as being applied to the front sensing camera 13001, but the present disclosure is not limited thereto. For example, the imaging device 1 according to the present disclosure can be applied to the front view camera 13018, the side view camera 13016, and the rear view camera 13025.

[0265] (Second mounting example)

[0266] Next, a second mounting example of the imaging device 1 according to the present disclosure will be described. Figure 29 is a block diagram illustrating an example of a schematic configuration of a vehicle control system that is an example of a mobile body control system to which the technology according to the present disclosure can be applied.

[0267] The vehicle control system 12000 includes a plurality of electronic control units connected via a communication network 12001. In the example shown in FIG. 12, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside information detecting unit 12030, an inside information detecting unit 12040, and an integrated control unit 12050. Further, a microcomputer 12051, a sound and image output unit 12052, and a vehicle-mounted network interface (I / F) 12053 are shown as functional constituents of the integrated control unit 12050. Figure 29

[0268] The drive system control unit 12010 controls the operation of devices related to the drive system of the vehicle in accordance with various programs. For example, the drive system control unit 12010 functions as a control device for a drive force generating device such as an internal combustion engine or a drive motor, which generates the drive force of the vehicle, a drive force transmission mechanism that transmits the drive force to the wheels, a steering mechanism that adjusts the steering angle of the vehicle, and a brake device that generates the braking force of the vehicle, and the like.

[0269] The body system control unit 12020 controls the operation of various devices mounted to the body of the vehicle in accordance with various programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as a headlamp, a tail lamp, a brake lamp, a turn signal lamp, or a fog lamp. In this case, an electric wave transmitted from a portable device that substitutes for a key or a signal of various switches can be input to the body system control unit 12020. The body system control unit 12020 receives the electric wave or the signal to control the door lock device, the power window device, the lamps, and the like of the vehicle.

[0270] ​The vehicle exterior information detection unit 12030 detects information about the outside of the vehicle on which the vehicle control system 12000 is mounted. For example, the vehicle exterior information detection unit 12030 is connected with an imaging unit 12031. The vehicle exterior information detection unit 12030 causes the imaging unit 12031 to image an image of an area outside the vehicle, and receives the captured image. On the basis of the received image, the vehicle exterior information detection unit 12030 can perform an object detection process or a distance detection process on, for example, a person, a vehicle, an obstacle, a sign, or a character on a road surface.

[0271] The imaging unit 12031 is an optical sensor that receives light and outputs an electric signal corresponding to the amount of light of the received light. The imaging unit 12031 can output the electric signal as an image, or can output the electric signal as distance measurement information. Furthermore, the light received by the imaging unit 12031 can be visible light, or can be invisible light such as infrared light.

[0272] The vehicle interior information detection unit 12040 detects information inside the vehicle. For example, the vehicle interior information detection unit 12040 is connected with a driver state detection unit 12041 that detects the state of the driver. The driver state detection unit 12041 includes, for example, a camera that images the driver, and on the basis of detection information input from the driver state detection unit 12041, the vehicle interior information detection unit 12040 can calculate the degree of fatigue of the driver or the degree of concentration of the driver, or can determine whether the driver is dozing off.

[0273] The microcomputer 12051 can calculate a control target value of the driving force generation device, the steering mechanism, or the brake device on the basis of information about the inside or outside of the vehicle, which is obtained by the vehicle exterior information detection unit 12030 or the vehicle interior information detection unit 12040, and output a control command to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control aimed at realizing functions of an advanced driver assistance system (ADAS) including collision avoidance or impact mitigation of the vehicle, following travel based on an inter-vehicle distance, vehicle speed maintenance travel, vehicle collision warning, or vehicle lane departure warning.

[0274] Furthermore, the microcomputer 12051 can control the driving force generation device, the steering mechanism, or the brake device, or the like, on the basis of information about the vicinity of the vehicle, which is obtained by the vehicle exterior information detection unit 12030 or the vehicle interior information detection unit 12040, to perform cooperative control aimed at realizing automated driving in which the vehicle autonomously travels without relying on the operation of the driver or the like.

[0275] Further, the microcomputer 12051 can output a control command to the body system control unit 12020 on the basis of information about the outside of the vehicle that is obtained by the outside-vehicle information detecting unit 12030. For example, the microcomputer 12051 can control a headlamp in order to perform cooperative control intended to achieve prevention of glare such as switching from a high beam to a low beam, in accordance with the position of a preceding vehicle or an oncoming vehicle detected by the outside-vehicle information detecting unit 12030.

[0276] The sound and image output unit 12052 transmits an output signal of at least one of a sound and an image to an output device that is capable of visually or aurally notifying information to an occupant of the vehicle or the outside of the vehicle. In Figure 29 Examples of the output device include an audio speaker 12061, a display unit 12062, and an instrument panel 12063. The audio speaker 12061, the display unit 12062, and the instrument panel 12063 are shown as the output device in the example. The display unit 12062 can include at least one of a vehicle-mounted display and a head-up display, for example.

[0277] Figure 30 FIG. 13 is a diagram showing an example of a mounting position of the imaging unit 12031.

[0278] In Figure 30 the vehicle 12100 includes imaging units 12101, 12102, 12103, 12104, and 12105 as the imaging unit 12031.

[0279] The imaging units 12101, 12102, 12103, 12104, and 12105 are provided at positions on the front nose, the side mirror, the rear bumper, and the rear door of the vehicle 12100 and at a position on the upper portion of the interior windshield of the vehicle, for example. The imaging unit 12101 provided to the front nose and the imaging unit 12105 provided to the upper portion of the interior windshield of the vehicle mainly acquire images of the front of the vehicle 12100. The imaging units 12102 and 12103 provided to the side mirror mainly acquire images of the side of the vehicle 12100. The imaging unit 12104 provided to the rear bumper or the rear door mainly acquires images of the rear of the vehicle 12100. The images of the front of the vehicle 12100 acquired by the imaging unit 12101 and the imaging unit 12105 are mainly used for detecting a preceding vehicle, a pedestrian, an obstacle, a traffic signal, a traffic sign, or a lane, and the like.

[0280] Note that Figure 30Examples of imaging ranges of the imaging units 12101 to 12104 are shown. The imaging range 12111 represents an imaging range of the imaging unit 12101 provided to the front nose, the imaging ranges 12112 and 12113 respectively represent imaging ranges of the imaging units 12102 and 12103 provided to the side mirrors, and the imaging range 12114 represents an imaging range of the imaging unit 12104 provided to the rear bumper or the rear door. For example, image data captured by the imaging units 12101 to 12104 is superimposed, thereby obtaining an overhead image of the vehicle 12100 viewed from above.

[0281] At least one of the imaging units 12101 to 12104 can have a function of obtaining distance information. For example, at least one of the imaging units 12101 to 12104 can be a stereo camera including a plurality of imaging elements, or can be an imaging element having pixels for phase difference detection.

[0282] For example, the microcomputer 12051 can extract, as a preceding vehicle, a three-dimensional object on a travel path of the vehicle 12100 that travels in substantially the same direction as the vehicle 12100 at a predetermined speed (for example, equal to or greater than 0 km / h) by calculating a distance to each three-dimensional object within the imaging ranges 12111 to 12114 and a change in distance over time (relative speed with respect to the vehicle 12100) on the basis of distance information obtained from the imaging units 12101 to 12104. Furthermore, the microcomputer 12051 can set a vehicle-to-vehicle distance to be maintained by the preceding vehicle in advance, and perform automatic brake control (including follow-up stop control), automatic acceleration control (including follow-up start control), or the like. As described above, cooperative control intended for automated driving, which enables the vehicle to travel autonomously without relying on the operation of the driver or the like, can be performed.

[0283] For example, the microcomputer 12051 can classify and extract three-dimensional object data on three-dimensional objects on the basis of distance information obtained from the imaging units 12101 to 12104 as other three-dimensional objects such as two-wheeled vehicles, standard-size vehicles, large vehicles, pedestrians, and utility poles, and use the classification and extraction results to automatically avoid obstacles. For example, the microcomputer 12051 identifies obstacles around the vehicle 12100 as obstacles that can be visually recognized by a driver of the vehicle 12100 and obstacles that are difficult for the driver of the vehicle 12100 to visually recognize. Then, the microcomputer 12051 determines a collision risk indicating a degree of risk of collision with each obstacle, and in a case where the collision risk is equal to or higher than a set value and there is a possibility of collision, the microcomputer 12051 can output a warning to the driver via the audio speaker 12061 or the display unit 12062, or perform forced deceleration or evasive steering to perform a driving assist for avoiding collision by driving the system control unit 12010.

[0284] At least one of the imaging units 12101 to 12104 can be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can identify a pedestrian by determining whether a pedestrian is present in a captured image of the imaging units 12101 to 12104. For example, such identification of a pedestrian is performed by a program that extracts feature points in a captured image of the imaging units 12101 to 12104 as an infrared camera and by a program that determines whether it is a pedestrian by performing pattern matching processing on a series of feature points representing the outline of an object. When the microcomputer 12051 determines that a pedestrian is present in a captured image of the imaging units 12101 to 12104 and identifies the pedestrian, the sound and image output unit 12052 controls the display unit 12062 so that a rectangular outline line for emphasis is superimposed on the identified pedestrian. Furthermore, the sound and image output unit 12052 can control the display unit 12062 so that an icon or the like representing the pedestrian is displayed at a desired position.

[0285] In the foregoing, an example of a vehicle control system to which the technology according to the present disclosure can be applied has been described. For example, the technology according to the present disclosure can be applied to the imaging unit 12031 in the configuration described above. Specifically, the imaging device 1 according to any one of the first and second embodiments of the present disclosure and modifications thereof can be applied as the imaging unit 12031.

[0286] In this case, the imaging unit 12031 can obtain vehicle information (traveling speed, surrounding brightness, and the like) of the vehicle from the communication network 12001. For example, based on the acquired vehicle information, the imaging unit 12031 can instruct whether to perform readout of pixels in a pixel array included in the imaging unit 12031 in an additive mode or an individual mode. This makes it possible to output image data corresponding to the traveling speed, the brightness, and the like, and power saving is possible.

[0287] (4-1. Application Example of Endoscopic Surgery System)

[0288] As another application example of the imaging device 1 according to the present disclosure, a more specific example in a case where the imaging device 1 is applied to an endoscopic surgery system will be described.

[0289] Figure 31 is a diagram showing an example of a schematic configuration of an endoscopic surgery system to which the technology according to the present disclosure (the present technology) can be applied.

[0290] Figure 31 A state in which a surgeon (doctor) 11131 is performing surgery on a patient 11132 on a patient bed 11133 using an endoscopic surgery system 11000 is shown. As shown, the endoscopic surgery system 11000 includes an endoscope 11100, other surgical tools 11110 such as a gas tube 11111 and an energy treatment tool 11112, a support arm device 11120 that supports the endoscope 11100 thereon, and a cart 11200 on which various devices for endoscopic surgery are installed.

[0291] The endoscope 11100 includes a lens barrel 11101 whose region of a predetermined length from the distal end thereof is inserted into a body cavity of the patient 11132, and a camera head 11102 connected to the proximal end of the lens barrel 11101. In the example shown, although an endoscope 11100 configured as a so-called rigid scope having a rigid lens barrel 11101 is shown, the endoscope 11100 can also be configured as a so-called flexible scope having a flexible lens barrel.

[0292] The lens barrel 11101 has an opening portion at the distal end thereof in which an objective lens is fitted. A light source device 11203 is connected to the endoscope 11100 so as to introduce light generated by the light source device 11203 into the distal end of the lens barrel 11101 through a light guide extending into the inside of the lens barrel 11101, and irradiate it onto an observation target in the body cavity of the patient 11132 through the objective lens. Note that the endoscope 11100 can be a direct vision endoscope or can be an oblique vision endoscope or a side vision endoscope.

[0293] An optical system and an imaging element are disposed inside the camera head 11102, and a reflected light (observation light) from an observation target is collected on the imaging element by the optical system. The observation light is photoelectrically converted by the imaging element, and an electric signal corresponding to the observation light, that is, an image signal corresponding to an observation image is generated. The image signal is transmitted to the CCU 11201 as RAW data.

[0294] The CCU 11201 is realized by, for example, a central processing unit (CPU) and a graphics processing unit (GPU), and comprehensively controls the operation of the endoscope 11100 and the display apparatus 11202. Further, for example, the CCU 11201 receives the image signal from the camera head 11102, and performs various image processing such as development processing (demosaicing processing) on the image signal to display an image based on the image signal.

[0295] The display apparatus 11202 displays an image based on the image signal on which the image processing has been performed by the CCU 11201 under the control of the CCU 11201.

[0296] The light source apparatus 11203 is realized by, for example, a light source such as a light emitting diode (LED), and supplies irradiation light at the time of imaging a surgical site to the endoscope 11100.

[0297] The input apparatus 11204 is an input interface of the endoscope surgery system 11000. The user can input various information or instructions to the endoscope surgery system 11000 through the input apparatus 11204. For example, the user inputs an instruction to change the imaging conditions (type of irradiation light, magnification, focal distance, and the like) of the endoscope 11100, and the like.

[0298] The treatment tool control apparatus 11205 controls the drive of the energy treatment tool 11112 to cauterize or incise tissue, close a blood vessel, and the like. The pneumoperitoneum apparatus 11206 supplies gas into the body cavity of the patient 11132 through the pneumoperitoneum tube 11111 to inflate the body cavity so as to secure the field of view of the endoscope 11100 and secure the working space of the operator. The recorder 11207 can record various information related to the surgery. The printer 11208 can print various information related to the surgery in various formats such as text, image, or graphics.

[0299] Note that the light source device 11203 that supplies irradiation light to the endoscope 11100 when imaging an image of a surgical site can include, for example, a white light source realized by an LED, a laser light source, or a combination thereof. In the case where the white light source is realized by a combination of RGB laser light sources, the output intensity and output timing of each color (each wavelength) can be controlled with high precision, and thus, white balance adjustment of an imaged image can be performed in the light source device 11203. Further, in this case, the observation target is irradiated with a laser beam from each of the RGB laser light sources in a time-division manner and the driving of the imaging element of the camera head 11102 is controlled in synchronization with the irradiation timing so that an image corresponding to each of the RGB can be imaged in a time-division manner. By this method, a color image can also be obtained without providing a color filter in the imaging element.

[0300] Further, the driving of the light source device 11203 can be controlled so as to change the intensity of light to be output every predetermined time. By controlling the driving of the imaging element of the camera head 11102 in synchronization with the timing of the change in light intensity to acquire images in a time-division manner, and by synthesizing the images, an image of a high dynamic range can be generated without the so-called underexposure and overexposure.

[0301] Further, the light source device 11203 can be configured to be able to supply light of a predetermined wavelength band corresponding to special light observation. For example, in special light observation, so-called narrowband imaging in which a narrowband light is irradiated to a body tissue compared to irradiation light (i.e., white light) at the time of ordinary observation is performed, for example, by utilizing the wavelength dependency of light absorption of the body tissue, so that a predetermined tissue such as a blood vessel like a mucosal surface portion is imaged with high contrast. Alternatively, in special light observation, fluorescence observation that obtains an image using fluorescence generated by irradiation of excitation light can be performed. In fluorescence observation, a fluorescence image can be obtained by irradiating a body tissue with excitation light and observing fluorescence from the body tissue (autofluorescence observation), or can be obtained by locally injecting a reagent like indocyanine green (ICG) into a body tissue and irradiating excitation light corresponding to the fluorescence wavelength of the reagent on the body tissue. The light source device 11203 can be configured to be able to supply narrowband light and / or excitation light corresponding to such special light observation.

[0302] Figure 32 is a block diagram showing an example of the functional configuration of the camera head 11102 and the CCU 11201 shown in Figure 31

[0303] ​The camera head 11102 includes a lens unit 11401, an imaging unit 11402, a driving unit 11403, a communication unit 11404, and a camera head control unit 11405. The CCU 11201 includes a communication unit 11411, an image processing unit 11412, and a control unit 11413. The camera head 11102 and the CCU 11201 are communicably connected to each other through a transmission cable 11400.

[0304] The lens unit 11401 is an optical system provided at a connection portion with the lens barrel 11101. The observation light that enters from the distal end of the lens barrel 11101 is guided to the camera head 11102 and introduced into the lens unit 11401. The lens unit 11401 is realized by a combination of a plurality of lenses including a zoom lens and a focus lens.

[0305] The imaging unit 11402 includes an imaging element. The number of imaging elements that constitute the imaging unit 11402 can be one (so-called single board type) or a plurality (so-called multi board type). For example, in a case where the imaging unit 11402 is configured in the multi board type, image signals corresponding to R, G, and B can be generated by the respective imaging elements, and the image signals can be synthesized to obtain a color image. Alternatively, the imaging unit 11402 can include a pair of imaging elements for respectively acquiring right eye image signals and left eye image signals corresponding to three-dimensional (3D) display. By performing three-dimensional (3D) display, the operator 11131 can more accurately grasp the depth of the living tissue of the surgical site. Note that in a case where the imaging unit 11402 is configured in the multi board type, a plurality of lens units 11401 systems can be provided corresponding to the respective imaging elements.

[0306] Furthermore, the imaging unit 11402 does not necessarily have to be provided in the camera head 11102. For example, the imaging unit 11402 can be provided inside the objective lens of the lens barrel 11101 just behind.

[0307] The driving unit 11403 is realized by an actuator, and moves the zoom lens and the focus lens of the lens unit 11401 by a predetermined distance under the control of the camera head control unit 11405. Accordingly, the magnification and the focus of the image taken by the imaging unit 11402 can be appropriately adjusted.

[0308] The communication unit 11404 is realized by a communication device for transmitting and receiving various information to and from the CCU 11201. The communication unit 11404 transmits the image signal acquired from the imaging unit 11402 to the CCU 11201 as RAW data through the transmission cable 11400.

[0309] Further, the communication unit 11404 receives a control signal for controlling driving of the camera head 11102 from the CCU 11201 and supplies the control signal to the camera head control unit 11405. The control signal includes, for example, information on an imaging condition such as information specifying a frame rate of an image to be captured, information specifying an exposure value at the time of capturing an image, and / or information specifying a magnification and a focus of an image to be captured.

[0310] Note that the imaging condition such as the frame rate, the exposure value, the magnification, or the focus can be appropriately specified by the user or can be automatically set by the control unit 11413 of the CCU 11201 based on the acquired image signal. In the latter case, the endoscope 11100 has a so-called auto exposure (AE) function, an auto focus (AF) function, and an auto white balance (AWB) function.

[0311] The camera head control unit 11405 controls driving of the camera head 11102 based on the control signal received from the CCU 11201 through the communication unit 11404.

[0312] The communication unit 11411 is realized by a communication device for transmitting and receiving various information to and from the camera head 11102. The communication unit 11411 receives an image signal transmitted thereto from the camera head 11102 through the transmission cable 11400.

[0313] Further, the communication unit 11411 transmits a control signal for controlling driving of the camera head 11102 to the camera head 11102. The image signal and the control signal can be transmitted by electric communication, optical communication, or the like.

[0314] The image processing unit 11412 performs various image processing on the image signal as RAW data transmitted thereto from the camera head 11102.

[0315] The control unit 11413 performs various controls related to imaging of a surgical site or the like by the endoscope 11100 and display of a captured image obtained by imaging of the surgical site or the like. For example, the control unit 11413 generates a control signal for controlling driving of the camera head 11102.

[0316] Further, the control unit 11413 causes the display apparatus 11202 to display an obtained captured image in which a surgical site or the like is imaged, on the basis of an image signal that has been subjected to image processing by the image processing unit 11412. At this time, the control unit 11413 can recognize various objects in the captured image using various image recognition techniques. For example, the control unit 11413 can detect the shape, color, or the like of the edge of an object included in the captured image, thereby recognizing, for example, a surgical tool such as forceps, a specific living body region, bleeding, fog when the energy treatment tool 11112 is used, or the like. The control unit 11413, when displaying the captured image on the display apparatus 11202, can cause various kinds of surgery support information to be superimposed on the image of the surgical site using the result of the recognition. By displaying the surgery support information in a superimposed manner and presenting it to the surgeon 11131, the burden on the surgeon 11131 can be alleviated and the surgeon 11131 can reliably perform surgery.

[0317] The transmission cable 11400 that connects the camera head 11102 and the CCU 11201 is an electric signal cable that supports electric signal communication, an optical fiber that supports optical communication, or a composite cable thereof.

[0318] Here, in the example shown, wired communication is performed using the transmission cable 11400, but communication between the camera head 11102 and the CCU 11201 can be performed wirelessly.

[0319] In the foregoing, an example of an endoscope surgery system to which the technology according to the present disclosure can be applied has been described. The technology according to the present disclosure can be applied to, for example, the imaging unit 11402 of the endoscope 11100 and the camera head 11102 in the configuration described above. Specifically, the imaging element described above can be applied to the imaging unit 10112.

[0320] The imaging unit 10112 to which the imaging element according to the present disclosure is applied can give an instruction as to whether to perform readout of the pixels included in the pixel array in the imaging unit 12031 in the additive mode in which high-speed readout and high SNR can be performed or the individual mode in which resolution is high. Thus, for example, the surgeon 11131 can obtain a captured image according to the surgical site and the condition of the surgical site. Thus, for example, the surgeon 11131 can continue surgery more reliably, and power saving can be achieved.

[0321] Note that, here, an endoscope surgery system has been described as an example, but the technology according to the present disclosure can also be applied to, for example, a microscope surgery system or the like.

[0322] Note that the effects described in this specification are merely examples. The effects of the present disclosure are not limited to these, and other effects can be obtained.

[0323] Note that the present technology can also have the following configuration.

[0324] (1) An imaging device comprising:

[0325] an imaging unit including a pixel array including a plurality of pixel groups each including N×N pixels (N is an integer of 2 or more), and outputting a pixel signal read out from each pixel; and

[0326] a switching unit that switches a readout mode in which the pixel signal is read out from each pixel of the imaging unit, wherein

[0327] the switching unit switches the readout mode between an addition mode in which the pixel signals read out from the N×N pixels included in the pixel group are added to form one pixel signal, and an individual mode in which each of the pixel signals read out from the N×N pixels included in the pixel group is outputted respectively.

[0328] (2) The imaging device according to the above (1), wherein

[0329] the imaging device is mounted on a vehicle, and

[0330] the switching unit switches the readout mode based on vehicle information acquired from the vehicle.

[0331] (3) The imaging device according to the above (2), wherein

[0332] the switching unit switches the readout mode according to a speed of the vehicle acquired as the vehicle information.

[0333] (4) The imaging device according to the above (2) or (3), wherein

[0334] the switching unit switches the readout mode according to a brightness around the vehicle acquired as the vehicle information.

[0335] (5) The imaging device according to any one of the above (2) to (4), wherein

[0336] in a case where the speed of the vehicle acquired as the vehicle information is equal to or higher than a predetermined value and the brightness around the vehicle acquired as the vehicle information is equal to or higher than a predetermined value, the switching unit switches the readout mode to the individual mode.

[0337] (6) The imaging device according to any one of the above (1) to (5), wherein

[0338] The switching unit switches the readout mode in accordance with a region of a frame image based on the pixel signal.

[0339] (7) The imaging device according to (6) above, wherein

[0340] The switching unit switches the readout mode to the individual mode for a predetermined region of a central portion of the frame image, and switches the readout mode to the addition mode for a region other than the predetermined region of the frame image.

[0341] (8) The imaging device according to any one of (1) to (7) above, wherein

[0342] The switching unit switches the readout mode based on an object included in a frame image based on the pixel signal.

[0343] (9) The imaging device according to (8) above, wherein

[0344] In a case where a specific object is included in the frame image, the switching unit performs switching of the readout mode for a specific region in which the specific object is included in the frame image and a region other than the specific region.

[0345] (10) The imaging device according to (9) above, wherein

[0346] The imaging device is mounted on a vehicle and used, and

[0347] The switching unit switches the readout mode to the individual mode for the specific region including the specific object, the specific object being at least one of a traffic signal, a traffic sign, an oncoming vehicle, or a pedestrian.

[0348] (11) The imaging device according to any one of (1) to (10), wherein

[0349] The switching unit switches the readout mode by controlling reading of the pixels included in the pixel array.

[0350] (12) The imaging device according to any one of (1) to (10) above, wherein

[0351] The switching unit switches the readout mode by controlling image processing of the pixel signal output by the imaging unit.

[0352] (13) The imaging device according to any one of (1) to (12) above, further comprising

[0353] a synthesizing unit that synthesizes a first image based on a first pixel signal from the pixel exposed with a first exposure time and a second image based on a second pixel signal from the pixel exposed with a second exposure time, the second exposure time being after the exposure performed with the first exposure time.

[0354] (14) The imaging device according to any one of (1) to (13) above, wherein

[0355] In the pixel array, the plurality of pixel groups include:

[0356] a first pixel group that receives light transmitted through a first optical filter;

[0357] a second pixel group that receives light transmitted through a second optical filter; and

[0358] a third pixel group that receives light transmitted through a third optical filter, and

[0359] the first pixel group, the second pixel group, and the third pixel group are arranged such that different pixel groups among the first pixel group, the second pixel group, and the third pixel group are adjacent to each other.

[0360] (15) The imaging device according to (14) above, further comprising

[0361] a remosaicking processing unit that performs remosaicking processing that converts each pixel included in each of the first pixel group, the second pixel group, and the third pixel group to a pixel having a characteristic corresponding to a position of each pixel, in a case where the readout mode is the individual mode.

[0362] List of Reference Signs

[0363] 1 imaging device

[0364] 10 imaging unit

[0365] 12 image processing unit

[0366] 13 output processing unit

[0367] 14, 22 control unit

[0368] 15 vehicle information acquisition unit

[0369] 110, 110a pixel array unit

[0370] 201L, 201M, 201S processing switch

[0371] 202L, 202M, 202S pixel adding unit

[0372] 203L, 203M, 203S remosaicing unit

[0373] 204L, 204M, 204S selector

[0374] 220 detecting unit

[0375] 1200, 1200a, 1200b pixel processing unit

[0376] 1201 HDR synthesis processing unit

Claims

1. An imaging device, comprising: An imaging unit, comprising a pixel array, wherein the pixel array comprises multiple pixel groups, each pixel group comprising N×N pixels, where N is an integer greater than or equal to 2, and the imaging unit outputs a pixel signal read from each pixel; and A switching unit switches the readout mode for reading the pixel signal from each pixel of the imaging unit, wherein... The switching unit switches the readout mode between an addition mode and an individual mode. In the addition mode, the pixel signals read from the N×N pixels included in the pixel group are added together to form a single pixel signal. In the individual mode, each of the pixel signals read from the N×N pixels included in the pixel group is output. The imaging device is mounted on the vehicle, and The switching unit switches the readout mode based on vehicle information obtained from the vehicle, including the brightness around the vehicle and the vehicle's speed. When the vehicle speed is equal to or higher than a predetermined speed value and the brightness is equal to or higher than a predetermined brightness value, the switching unit switches the readout mode to the individual mode. When the vehicle speed is lower than the predetermined speed value or the brightness is lower than the predetermined brightness value, the switching unit switches the readout mode to the addition mode.

2. The imaging device according to claim 1, wherein... The switching unit also switches the readout mode according to the region of the frame image based on the pixel signal.

3. The imaging apparatus according to claim 2, wherein... The switching unit switches the readout mode to the individual mode for a predetermined region in the central part of the frame image, and switches the readout mode to the addition mode for the region of the frame image other than the predetermined region.

4. The imaging apparatus according to claim 1, wherein The switching unit also switches the readout mode based on objects included in the frame image based on the pixel signal.

5. The imaging apparatus according to claim 4, wherein When the frame image includes a specific object, the switching unit performs the switching of the readout mode for a specific region in the frame image that includes the specific object and for regions other than the specific region.

6. The imaging apparatus according to claim 5, wherein The switching unit switches the readout mode to the individual mode for the specific area including the specific object, wherein the specific object is at least one of traffic lights, traffic signs, oncoming vehicles, or pedestrians.

7. The imaging apparatus according to any one of claims 1-6, wherein The switching unit also switches the readout mode by controlling the readout of the pixels included in the pixel array.

8. The imaging apparatus according to any one of claims 1-6, wherein The switching unit also switches the readout mode by controlling the image processing of the pixel signal output by the imaging unit.

9. The imaging apparatus according to any one of claims 1-6, further comprising: A compositing unit that composes a first image based on a first pixel signal from the pixel exposed at a first exposure time and a second image based on a second pixel signal from the pixel exposed at a second exposure time, the second exposure time being after the exposure performed at the first exposure time.

10. The imaging apparatus according to any one of claims 1-6, wherein In the pixel array, the plurality of pixel groups include: The first pixel group receives light that passes through the first filter; The second pixel group receives light that passes through the second filter; and The third pixel group receives light transmitted through the third filter, and The first pixel group, the second pixel group, and the third pixel group are arranged such that different pixel groups in the first pixel group, the second pixel group, and the third pixel group are adjacent to each other.

11. The imaging apparatus according to claim 10, further comprising: A re-mosaic processing unit is configured to perform re-mosaic processing when the readout mode is the individual mode, converting each pixel included in each of the first pixel group, the second pixel group, and the third pixel group into a pixel having characteristics corresponding to the position of each pixel.

Images