Solid-state imaging device and electronic apparatus

Abstract

A solid-state imaging device is provided capable of acquiring RGB images, CMY images, and brightness information in a single capture. The solid-state imaging device includes a pixel array section with multiple pixel unit groups arranged therein. Each pixel unit group includes a 2×2 matrix arrangement of pixel units, and each pixel unit includes a 2×2 matrix arrangement of pixels. Each pixel has a photoelectric conversion unit and a color filter formed corresponding to the photoelectric conversion unit. Each pixel unit group is configured such that, in the four pixel units constituting the pixel unit group, the first pixel unit includes an R filter and a C filter as color filters, each of the second and third pixel units includes a G filter and an M filter as color filters, and the fourth pixel unit includes a B filter and a Y filter as color filters.

Description

Technical Field

[0001] This technology relates to solid-state imaging devices and electronic devices. Background Technology

[0002] In the prior art, solid-state imaging devices have been proposed with a configuration that divides one pixel of a Bayer array into multiple pixels (see, for example, Patent Document 1). In the solid-state imaging device disclosed in Patent Document 1, a high-resolution captured image can be obtained by performing full-resolution demosaic processing (a series of processes for performing demosaic processing after remosaic processing), a captured image with excellent signal-to-noise ratio can be obtained by performing merging processing, and a high dynamic range (HDR) captured image can be obtained by changing the exposure conditions in each of the multiple pixels.

[0003] [List of Citations]

[0004] [Patent Literature]

[0005] [Patent Document 1]

[0006] JP 2019-175912 A Summary of the Invention

[0007] [Technical Issues]

[0008] However, in the solid-state imaging device disclosed in Patent Document 1, it is impossible to acquire all the RGB image, CMY image and brightness information of the subject through a single imaging process.

[0009] The purpose of this disclosure is to provide a solid-state imaging device and electronic device capable of acquiring RGB images, CMY images, and brightness information through imaging processing.

[0010] [Solution to the problem]

[0011] The solid-state imaging apparatus disclosed herein includes: (a) a pixel array section, wherein a plurality of pixel unit groups are arranged, each pixel unit group consisting of pixel units arranged in a 2×2 matrix, each pixel unit consisting of pixels arranged in an m×m matrix (m is 2 or a larger natural number), and each pixel includes a photoelectric conversion unit and a filter formed with the photoelectric conversion unit; (b) each pixel unit group includes an R filter that transmits red light and a C filter that transmits cyan light, serving as a color filter in the first pixel unit among the four pixel units constituting the pixel unit group, wherein cyan light and red light have a complementary color relationship; each pixel unit group includes a G filter that transmits green light and an M filter that transmits magenta light, serving as color filters in each of the second and third pixel units, wherein magenta light and green light have a complementary color relationship; and each pixel unit group includes a B filter that transmits blue light and a Y filter that transmits yellow light, serving as a color filter in the fourth pixel unit, wherein yellow light and blue light have a complementary color relationship.

[0012] The electronic device disclosed herein includes (a) a solid-state imaging device, comprising a pixel array section wherein a plurality of pixel unit groups are arranged, the pixel unit groups being composed of pixel units arranged in a 2×2 matrix, the pixel units being composed of pixels arranged in an m×m matrix (m is 2 or a larger natural number), and each pixel including a photoelectric conversion unit and a color filter formed corresponding to the photoelectric conversion unit; wherein each pixel unit group includes an R filter that transmits red light and a C filter that transmits cyan light, and as the color filter in the first pixel unit among the four pixel units constituting the pixel unit group, cyan light and red light have a complementary color relationship; each pixel unit group A pixel unit group includes a G filter that transmits green light and an M filter that transmits magenta light as color filters in each of the second and third pixel units, wherein the magenta light and the green light have a complementary color relationship; and each pixel unit group includes a B filter that transmits blue light and a Y filter that transmits yellow light as color filters in the fourth pixel unit, wherein the yellow light and the blue light have a complementary color relationship; (b) an optical lens that forms an image from the subject onto the imaging surface of the solid-state imaging device; and (c) a signal processing circuit that performs signal processing on the signal output from the solid-state imaging device. Attached Figure Description

[0013] [ Figure 1 ] Figure 1 This is a diagram illustrating the overall configuration of an electronic device according to a first embodiment of the present disclosure.

[0014] [ Figure 2 ] Figure 2This is a diagram showing the overall configuration of a solid-state imaging apparatus according to a first embodiment of the present disclosure.

[0015] [ Figure 3A ] Figure 3A It shows along Figure 2 A diagram showing the cross-sectional configuration of the pixel array portion of line AA in the diagram.

[0016] [ Figure 3B ] Figure 3B It shows along Figure 3A A diagram showing the smallest unit array of the color filter in line BB.

[0017] [ Figure 4 ] Figure 4 This is a diagram showing the smallest unit array of a color filter according to the modified example.

[0018] [ Figure 5 ] Figure 5 This is a diagram showing the transmittance of red and cyan pixels.

[0019] [ Figure 6 ] Figure 6 This is a diagram showing the transmittance of green and magenta pixels.

[0020] [ Figure 7 ] Figure 7 This is a diagram showing the transmittance of the blue and yellow pixels.

[0021] [ Figure 8 ] Figure 8 This is a diagram showing the configuration of the color filter array.

[0022] [ Figure 9 ] Figure 9 This is a diagram showing the arrangement of microlenses according to the modified example.

[0023] [ Figure 10 ] Figure 10 This is a flowchart illustrating the image generation process.

[0024] [ Figure 11 ] Figure 11 This is a diagram showing the captured image generated by the signal processing circuitry.

[0025] [ Figure 12 ] Figure 12 This is a diagram showing the content of the re-mosaic processing.

[0026] [ Figure 13 ] Figure 13 This is a diagram showing the content of the re-mosaic processing.

[0027] [ Figure 14 ] Figure 14 This is a diagram showing the processing details of the merge process.

[0028] [ Figure 15 ] Figure 15 This is a diagram showing the processing details of the merge process.

[0029] [ Figure 16 ] Figure 16 This is a diagram showing the processing details of the merge process.

[0030] [ Figure 17 ] Figure 17 This is a diagram illustrating the processing details of the luminance synthesis process.

[0031] [ Figure 18 ] Figure 18 This is a diagram illustrating the processing details of the luminance synthesis process.

[0032] [ Figure 19 ] Figure 19 This is a diagram illustrating the overall configuration of an electronic device according to a second embodiment of the present disclosure.

[0033] [ Figure 20 ] Figure 20 This is a flowchart illustrating the image generation process.

[0034] [ Figure 21 ] Figure 21 This is a diagram illustrating the processing involved in generating HDR images.

[0035] [ Figure 22 ] Figure 22 This is a diagram illustrating the processing involved in generating HDR images.

[0036] [ Figure 23 ] Figure 23 This is a diagram showing the connection status of the signal lines.

[0037] [ Figure 24 ] Figure 24 This is a diagram showing the connection status of the signal lines.

[0038] [ Figure 25 ] Figure 25 This is a block diagram illustrating an example of a schematic configuration of a vehicle control system.

[0039] [ Figure 26 ] Figure 26 This is an illustration showing an example of the installation location of the vehicle exterior information detection unit and the imaging unit.

[0040] [ Figure 27 ] Figure 27 This is a diagram illustrating an example of a schematic configuration of an endoscope operating system.

[0041] [ Figure 28 ] Figure 28 This is a block diagram illustrating an example of the functional configuration of the camera and CCU. Specific Implementation

[0042] In the following text, reference will be made to Figures 1 to 28 Examples of solid-state imaging apparatus 1 and electronic device according to embodiments of the present disclosure are described below. Embodiments of the present disclosure will be described in the following order. It should be noted that the present disclosure is not limited to the following examples. Furthermore, the effects described in this specification are exemplary and not limiting, and may provide other effects.

[0043] 1. First embodiment: Electronic device

[0044] 1-1 Overall Configuration of Electronic Equipment

[0045] Configuration of 1-2 main parts

[0046] 1-3 Image Generation and Processing

[0047] 2. Second embodiment: Electronic device

[0048] 2-1 Configuration of the main components

[0049] 2-2 Modified Example

[0050] 3. Examples of application to moving bodies

[0051] 4. Examples of applications in endoscope operating systems

[0052] <1. First Embodiment: Electronic Device>

[0053] [1-1. Overall Structure of Electronic Devices]

[0054] An electronic device 100 according to a first embodiment of the present disclosure will be described. As the electronic device 100, various electronic devices such as imaging devices may be employed, for example, digital still cameras and digital video cameras, mobile phones with imaging capabilities, or other devices with imaging capabilities. Figure 1 This is a schematic diagram illustrating the entire electronic device 100 according to a first embodiment of the present disclosure.

[0055] like Figure 1As shown, the electronic device 100 includes a solid-state imaging device 101 (hereinafter also referred to as "solid-state imaging device 1"), an optical lens 102, a shutter device 103, a drive circuit 104, and a signal processing circuit 105. The signal processing circuit 105 includes a mosaic image generation unit 106, a white balance adjustment unit 107, a mode determination unit 108, a re-mosaic processing unit 109, a merging processing unit 110, a brightness value calculation unit 111, and a brightness synthesis unit 112. In the electronic device 100, the optical lens 102 forms an image of the image light (incident light 113) received from the subject onto the imaging surface of the solid-state imaging device 101. The solid-state imaging device 101 converts the amount of incident light 113 into an electrical signal at the pixel unit level and outputs a pixel signal. The signal processing circuit 105 performs signal processing on the pixel signal output from the solid-state imaging device 101. At this time, the shutter device 103 controls the light illumination period and the light blocking period for the solid-state imaging device 101. In addition, the drive circuit 104 provides drive signals for controlling the transmission operation of pixel signals and the shutter operation of the shutter device 103.

[0056] Figure 2 This is a schematic diagram showing the configuration of the solid-state imaging device 1. Figure 2 The solid-state imaging device 1 in the image is a back-illuminated complementary metal-oxide-semiconductor (CMOS) image sensor.

[0057] like Figure 2 As shown, the solid-state imaging device 1 includes a substrate 2, a pixel array 3, a vertical driving circuit 4, a column signal processing circuit 5, a horizontal driving circuit 6, an output circuit 7, and a control circuit 8.

[0058] The pixel array 3 includes a plurality of pixels 9 arranged in a matrix on the substrate 2. Each pixel 9 includes a photoelectric conversion unit 24, a color filter 19 formed to correspond to the photoelectric conversion unit 24, and a microlens 20, such as... Figure 3A and Figure 3B As shown. For pixel 9, four pixels 9 arranged in a 2×2 matrix constitute one pixel unit 10. Furthermore, for pixel unit 10, four pixel units 10 arranged in a 2×2 matrix constitute one pixel unit group 11. That is, the pixel array section 3 is configured such that multiple pixel unit groups 11 are arranged in a matrix.

[0059] It should be noted that although the first embodiment describes an example where a pixel unit 10 consists of pixels 9 arranged in a 2×2 matrix, other configurations are also possible. For example, as Figure 4 As shown, a pixel unit can be composed of pixels 9 arranged in an m×m matrix (where m is a natural number of 2 or greater). Figure 4 The example shown illustrates the case where m is 5 or more.

[0060] The vertical drive circuit 4, composed of, for example, a shift register, selects the desired pixel drive wiring 12, provides pulses for driving the pixel 9 to the selected pixel drive wiring 12, and drives the pixel 9 row by row. That is, the vertical drive circuit 4 sequentially performs selection scanning of the pixels 9 in the pixel array section 3 row by row in the vertical direction, and provides a pixel signal based on the signal charge generated according to the amount of light received in the photoelectric conversion unit 24 of each pixel 9 to the column signal processing circuit 5 through the vertical signal line 13.

[0061] For example, a column signal processing circuit 5 is arranged for each column of pixels 9, and signal processing such as noise removal is performed on the signal output from the pixel 9 corresponding to a row for each pixel column. For example, the column signal processing circuit 5 performs signal processing such as correlated double sampling (CDS) and analog-to-digital (AD) conversion for removing pixel-specific fixed pattern noise.

[0062] For example, the horizontal drive circuit 6, which consists of a shift register, sequentially outputs horizontal scan pulses to the column signal processing circuit 5 to select each of the column signal processing circuits 5 in turn, and outputs the pixel signal (hereinafter also referred to as "pixel value") that has been processed from each of the column signal processing circuits 5 to the horizontal signal line 14.

[0063] Output circuit 7 performs signal processing on the continuously supplied pixel signals (pixel values) and outputs pixel signals from each column signal processing circuit 5 via horizontal signal line 14. Examples of signal processing that can be employed include buffering, black level adjustment, column change correction, and various types of digital signal processing.

[0064] The control circuit 8 generates a clock signal or control signal based on the vertical synchronization signal, horizontal synchronization signal, and master clock signal, which serves as a reference for the operation of the vertical drive circuit 4, column signal processing circuit 5, horizontal drive circuit 6, etc. Furthermore, the control circuit 8 outputs the generated clock signal or control signal to the vertical drive circuit 4, column signal processing circuit 5, horizontal drive circuit 6, etc.

[0065] [1-2. Configuration of the main components]

[0066] Next, we will describe Figure 1 Detailed configuration of solid-state imaging device 1 in the document. Figure 3A This is a diagram showing the cross-sectional configuration of the pixel array section 3 of the solid-state imaging device 1. Figure 3B It shows along Figure 3A A diagram illustrating the smallest unit array of the color filter 19 in line BB. Figure 3A and Figure 3B In this context, a back-illuminated CMOS image sensor is used as a solid-state imaging device 1.

[0067] like Figure 3A and Figure 3B As shown, the solid-state imaging device 1 according to the first embodiment includes a light-receiving layer 18, wherein a substrate 2, an insulating film 15, a light-shielding film 16, and a planarization film 17 are laminated in this order. Furthermore, a light-concentrating layer 21 is formed on the surface of the light-receiving layer 18 on the insulating film 15 side (hereinafter also referred to as "back surface S1"), wherein a color filter 19 and a microlens 20 (on-chip lens) are laminated sequentially. Additionally, a wiring layer 22 and a support substrate 23 are sequentially laminated on the surface of the light-receiving layer 18 on the substrate 2 side (hereinafter also referred to as "surface S2"). It should be noted that the back surface S1 of the light-receiving layer 18 and the back surface of the planarization film 17 are the same surface, and therefore, in the following description, the back surface of the planarization film 17 will be referred to as "back surface S1". Furthermore, the surface S2 of the light-receiving layer 18 and the surface of the substrate 2 are the same surface, and therefore, in the following description, the surface of the substrate 2 will be referred to as "surface S2".

[0068] Substrate 2 is made of, for example, a semiconductor substrate formed of silicon (Si), and forms Figure 1 The pixel array section 3 is shown. In the pixel array section 3, a plurality of photoelectric conversion units 24 formed on the substrate 2 are arranged in a matrix. In the photoelectric conversion units 24, signal charges corresponding to the amount of incident light 113 are generated and accumulated. In addition, pixel separation units 25 are provided between adjacent photoelectric conversion units 24, so that light that has passed through other photoelectric conversion units 24 does not penetrate. An insulating film 15 continuously covers the entire substrate 2 (the entire light-receiving surface side) on the rear surface S1 side. In addition, a light-shielding film 16 is formed in a grid shape in the portion of the insulating film 15 located on the rear surface S3 side (the portion located on the light-receiving surface side), so that the light-receiving surface of each of the plurality of photoelectric conversion units 24 is open.

[0069] Color filters 19 are formed corresponding to each photoelectric conversion unit 24 on the rear surface S1 side (light receiving surface side) of the insulating film 15. That is, one color filter 19 is formed for each photoelectric conversion unit 24 (pixel 9). Thus, the color filters 19 form a color filter array 26 arranged regularly in a matrix. Each color filter 19 is configured to transmit incident light 113 (red light, green light, blue light, cyan light, magenta light, or yellow light) having a specific wavelength and to cause the transmitted light to be incident on the photoelectric conversion unit 24.

[0070] As a color filter 19, an R filter 19 that transmits red light is used. R G filter 19 that transmits green light G B filter 19 transmits blue light B Transmission and R filter 19R C filter 19 transmits red light with a complementary color relationship to cyan light. C Transmission and the G filter 19 G The transmitted green light has a complementary color relationship with magenta light via an M filter 19. M And transmission and the B filter 19 B The transmitted blue light has a complementary color relationship with yellow light via a Y filter 19. Y In other words, such as Figure 5 As shown in the spectral distribution diagram, filter C19 C It is in R filter 19 R The transmittance increases at wavelengths with decreased transmittance, and in the R filter 19 R A color filter with increased transmittance at wavelengths where transmittance decreases. For example... Figure 6 As shown in the spectral distribution diagram, filter M19 M It is in G filter 19 G The transmittance increases at wavelengths with decreased transmittance, and in G filter 19 G A filter with increased transmittance at wavelengths where transmittance decreases. Furthermore, such as... Figure 7 As shown in the spectral distribution diagram, Y filter 19 Y It is in the B filter 19 B The transmittance increases at wavelengths with decreased transmittance, and in the B filter 19... B A filter that has increased transmittance at wavelengths where transmittance is reduced.

[0071] exist Figure 3A and Figure 3B In the diagram, the symbol R represents an R-filter 19. R And similarly, in the following text, the symbol G denotes G filter 19. G The symbol B indicates a B filter 19. B The symbol C represents a C filter 19. C The symbol M indicates an M-filter 19 M And the symbol Y represents the Y filter 19 Y Furthermore, the following description includes R filter 19. R Pixel 9 will be represented as red pixel 9. R Including G filter 19 G Pixel 9 will be represented as green pixel 9 G Including B filter 19 B Pixel 9 will be represented as blue pixel 9 B Including C filter 19 C Pixel 9 will be represented as cyan pixel 9 C Including M filter 19M Pixel 9 will be represented as magenta pixel 9 M And includes a Y filter 19 Y Pixel 9 will be represented as yellow pixel 9. Y .

[0072] In addition, color filter 19 (R filter 19) R G filter 19 G B filter 19 B C filter 19 C M filter 19 M Y filter 19 Y The array mode is configured such that the arrangement is as follows: Figure 3B The 4×4 matrix array of color filters 19 shown is configured as the smallest unit of the array of color filters 19 (hereinafter also referred to as the "smallest unit array"), as follows. Figure 8 As shown, the smallest unit array is set in all pixel unit groups 11 of the pixel array section 3.

[0073] The smallest unit array of color filter 19 is an array in which a portion of a corrected quartile Bayer array is configured such that the R filter 19 R The pixel unit 10 (hereinafter also referred to as "first pixel unit 101") is located in the upper right corner of the four pixel units 10 constituting the pixel unit group 11. G filter 19 G The pixel unit 10 (hereinafter also referred to as "second pixel unit 102") is arranged in the upper left side and the pixel unit 10 (hereinafter also referred to as "third pixel unit 103") in the lower right side, and the B filter 19 B Arranged in such Figure 3B In the lower left pixel unit 10 shown (hereinafter also referred to as "fourth pixel unit 104"). Specifically, in the 2×2 array of pixels 9 constituting the first pixel unit 101 of the quadruple Bayer array, the R filter 19 is located in the upper right and lower left pixels 9. R C filter 19 C Replacement. Furthermore, the G filter 19 in the pixels 9 located on the upper right and lower left sides of the 2×2 array constituting each of the second pixel unit 102 and the third pixel unit 103. G M filter 19 M Replacement. Furthermore, the B filter 19 in pixels 9 located on the upper right and lower left sides of the 2×2 array constituting the fourth pixel unit 104. B Y filter 19 Y replace.

[0074] That is, in the smallest unit array of color filters 19, R filter 19 R Included in one of the pixel pairs (two pixels 9 forming a group) arranged diagonally in the first pixel unit 101, and C filter 19 C It is included in another pixel pair. Similarly, in the following text, G filter 19... G Included in one pixel pair of pixel pairs arranged at diagonal positions in the second pixel unit 102, and M filter 19 M It is included in another pixel pair. Additionally, G filter 19 G Included in one pixel pair of pixel pairs arranged at diagonal positions in the third pixel unit 103, and M filter 19 M It is included in another pixel pair. Additionally, the B filter 19 B Included in one pixel pair of pixels arranged at a diagonal position in the fourth pixel unit 104, and the Y filter 19 Y Included in another pixel pair.

[0075] Utilizing this arrangement of color filters 19 in the smallest unit array, among all color filters 19 included in the first pixel unit 101, the R filter 19 R R-filters with increased transmittance at wavelengths where transmittance decreases 19 R and C filter 19 C Combined with each other, such as Figure 5 As shown, and therefore for all light from the short wavelength side to the long wavelength side, the transmittance is essentially flat. In the following, among all the color filters 19 included in the second pixel unit 102, as in... Figure 6 As shown, it has a G filter 19 G G-filter 19: Transmittance decreases at wavelengths where transmittance increases. G and M filter 19 M They combine with each other, and therefore the transmittance is essentially flat for all light from the short-wavelength side to the long-wavelength side. Furthermore, in all the color filters 19 included in the fourth pixel unit 104, such as... Figure 7 As shown, in filter B 19 B A B filter with increased transmittance at wavelengths where transmittance decreases. B and Y filter 19 Y They combine with each other, and therefore the transmittance is essentially flat for all light from the short wavelength side to the long wavelength side.

[0076] It is important to note that, such as Figure 4As shown, in the case where pixel unit 10 is composed of pixels 9 arranged in an m×m matrix, and m is a natural number greater than or equal to 3 ( Figure 4 (This illustrates the case where m is greater than or equal to 5), where pixels 9 arranged in a checkerboard pattern can be configured to include an R filter 19. R G filter 19 G and B filter 19 B Specifically, the first pixel unit 101 is configured such that some pixels 9, which are positioned in a checkerboard pattern among a plurality of pixels 9 arranged in an m×m matrix, include an R filter 19. R And the remaining pixels 9 include C filter 19 C Similarly, in the following text, the second pixel unit 102 is configured such that some pixels 9, among a plurality of pixels 9 arranged in an m×m matrix and positioned relative to each other in a checkerboard pattern, include a G filter 19. G And the remaining pixels 9 include M filter 19 M Furthermore, the third pixel unit 103 is configured such that some pixels 9, which are positioned relative to each other in a checkerboard pattern among a plurality of pixels 9 arranged in an m×m matrix, include a G filter 19. G And the remaining pixels 9 include M filter 19 M Furthermore, the fourth pixel unit 104 is configured such that some pixels 9, which are positioned relative to each other in a checkerboard pattern among a plurality of pixels 9 arranged in an m×m matrix, include a B filter 19. B And the remaining pixels 9 include a Y filter 19 Y .

[0077] In this way, the solid-state imaging device 1 according to the first embodiment is configured such that, except for the R filter 19 R G filter 19 G and B filter 19 B In addition, the color filter 19 also includes a C filter 19 C M filter 19 M and Y filter 19 Y Therefore, by using an R filter 19 R Red pixel 9 R Including G filter 19 G Green pixel 9 G And including B filter 19 B Blue pixel 9 B The pixel signals can generate an RGB image (in Figure 14 (RGB mosaic image 38). Furthermore, this can be achieved by using a C filter 19. C Cyan pixel 9 C Including M filter 19 Mmagenta pixel 9 M And including Y filter 19 Y Yellow pixel 9 Y The pixel signals are used to generate CMY images. Figure 15 (CMY mosaic image 39 in the image). Furthermore, brightness information can be generated by adding the pixel values ​​of pixels 9 in the same pixel unit 10. Figure 16 (brightness image 41 in the image). Therefore, RGB images, CMY images, and brightness information can be acquired in a single imaging process. RGB images are easily imaged in dark environments due to the low sensitivity of color filter 19; however, due to the small overlap of the spectral sensitivity of color filter 19, color reproduction is good. CMY images have poor color reproduction due to the large overlap of the spectral sensitivity of color filter 19; however, due to the low sensitivity of color filter 19, CMY images are easier to image in dark environments compared to RGB images. Furthermore, by using six colors—a combination of RGB and CMY images—color reproduction can be further improved.

[0078] Furthermore, in the solid-state imaging device 1 according to the first embodiment, the R filter 19 is configured such that... R In the case where it is included in one of the pixel pairs arranged at the diagonal position in the first pixel unit 101, and, as Figure 14 As shown, in other pixel pairs, the image pixel 31 in the mosaic image 30 corresponding to the arrangement of color filters 19 has only red color information. R The center of gravity and through the image pixels 31 that only have red color information R The resulting image has 37 pixels. R The centers of gravity become equal to each other, therefore, it is not necessary to process the red image pixels 31 in the grouping process. R The center of gravity is corrected. Furthermore, such as... Figure 14 and Figure 15 As shown, for image pixel 31 which only has green color information G 31 image pixels containing only blue color information B 31 image pixels containing only cyan color information C 31 image pixels containing only magenta color information M And image pixels 31 that only have yellow color information Y This is also true. Therefore, it is possible to obtain more appropriate RGB and CMY images. Figure 14 and Figure 15 In the image, the symbol R represents an image pixel 31 that only has red color information. R(Hereinafter also referred to as "red image pixel"), and similarly, hereafter, the symbol G represents an image pixel that has only green color information. G (Hereinafter also referred to as "green image pixels"), the symbol B represents image pixels that have only blue color information. B (Hereinafter also referred to as "blue image pixels"), the symbol C represents an image pixel that has only cyan color information. C (Hereinafter also referred to as "cyan image pixel"), the symbol M represents an image pixel 31 that has only magenta color information. M (Hereinafter also referred to as "magenta image pixels"), and the symbol Y represents image pixels 31 that have only yellow color information. Y (Hereinafter also referred to as “yellow image pixels”).

[0079] It should be noted that although the first embodiment illustrates an example of setting the upper left and lower right pixel units 10 as the second pixel unit 102 and the third pixel unit 103, other structures can also be used. For example, a structure could be used where the second pixel unit 102 and the third pixel unit 103 are located in the upper right and lower left pixel units 10, or where the second pixel unit 102 and the third pixel unit 103 are located within the upper left and lower left pixel units 10, or where the second pixel unit 102 and the third pixel unit 103 are located in the upper right and lower right pixel units 10. Furthermore, for example, a structure could be used where the first pixel unit 101 is located on the lower side and the fourth pixel unit 104 is located on the upper side. That is, it may only be necessary to configure each pixel unit group 11 to include the R filter 19. R and C filter 19 C As the color filter 19 in the first pixel unit 101 of the four pixel units 10 constituting pixel unit group 11, the G filter 19 G and M filter 19 M Color filter 19 is included in the second pixel unit 102 and the third pixel unit 103, and B filter 19 B and Y filter 19 Y Color filter 19 is included as the fourth pixel unit 104.

[0080] Microlenses 20 are formed corresponding to each photoelectric conversion unit 24 on the rear surface S4 side (light receiving surface side) of the color filter 19. That is, one microlens 20 is formed for each photoelectric conversion unit 24 (pixel 9). Thus, the microlenses 20 form a microlens array 27 arranged in a matrix. Each microlens 20 is configured to collect image light (incident light 113) from the object and guide the collected incident light 113 through the color filter 19 to the vicinity of the rear surface (light receiving surface) of the photoelectric conversion unit 24.

[0081] Note that although the first embodiment describes an example of forming a microlens 20 for a photoelectric conversion unit 24, other configurations are also possible. For example, in the green pixel 9 G When used as phase difference pixels, two green pixels arranged in a 1×2 matrix can be used. G Arranged as Figure 9 The configuration illustrated herein, and for the two green pixels 9 arranged G (Phase difference pixels) form a microlens 20. With this configuration, two green pixels 9 sharing a single microlens 20 can be detected. G The phase difference between captured images (phase difference pixels).

[0082] A wiring layer 22 is formed on the surface S2 side of the substrate 2 and is configured to include an interlayer insulating film 28 and wiring 29 laminated in multiple layers with the interlayer insulating film 28 in between. The wiring layer 22 drives the pixel transistors constituting the pixel 9 through the multilayer wiring 29.

[0083] A support substrate 23 is formed on the surface of the wiring layer 22 opposite to the side facing the substrate 2. The support substrate 23 is a substrate used to ensure the strength of the substrate 2 during the manufacturing stage of the solid-state imaging device 1. For example, silicon (Si) can be used as the material of the support substrate 23.

[0084] [1-3. Image Generation and Processing]

[0085] Next, the image generation process performed by the signal processing circuit 105 (mosaic image generation unit 106, white balance adjustment unit 107, mode determination unit 108, re-mosaic processing unit 109, merging processing unit 110, brightness value calculation unit 111, brightness synthesis unit 112) will be described.

[0086] When performing image generation processing, the mosaic image generation unit 106 first bases its image on, for example, the mosaic image generation unit 106. Figure 10 In step S101 shown, from red pixel 9 R Green pixel 9 G Blue pixel 9 B Cyan pixel 9C Magenta pixel 9 M and yellow pixel 9 Y The output pixel signal (pixel value) is generated and as follows Figure 11 The mosaic image 30 corresponding to the array of color filters 19 shown.

[0087] Subsequently, the processing proceeds to step S102, where the white balance adjustment unit 107 estimates the color temperature of the light source based on the pixel values ​​of the image pixels 31 of the mosaic image 30 generated in step S101, and adjusts the white balance of the mosaic image 30 based on the estimated color temperature.

[0088] Subsequently, the process proceeds to step S103, where the mode determination unit 108 determines whether the imaging mode selected by the user of the electronic device 100 is a high-resolution mode or a high signal-to-noise ratio mode. Furthermore, if the determination result is a high-resolution mode, the process proceeds to step S104. On the other hand, if the determination result is a high signal-to-noise ratio mode, the process proceeds to step S105.

[0089] In step S104, the re-mosaic processing unit 109 re-mosaicizes the mosaic image 30 that underwent white balance correction in step S102. In the regeneration process, such as... Figure 12 As shown, the red, green, and blue image pixels 31 that make up the mosaic image 30 R 31 G and 31 B The pixel array is converted into a Bayer array, thereby generating a 32-pixel image of red, green, and blue pixels from the Bayer array. R 32 G and 32 B The resulting RGB mosaic image 33. Furthermore, as... Figure 13 As shown, the cyan, magenta, and yellow image pixels are 31. C 31 M and 31 Y The pixel array is transformed into a predetermined array, thereby generating an image of cyan, magenta, and yellow pixels from the predetermined array. C 34 M and 34 Y The resulting CMY mosaic image 35. Figure 13 This is shown by using cyan, magenta, and yellow image pixels 34 C 34 M and 34 Y Replace 32 red, green, and blue image pixels of the Bayer array. R 32 G and 32 B The resulting pixel array is used as a predetermined array. Figure 12 Enlarged portions of mosaic image 30 and RGB mosaic image 33 are shown. Furthermore, Figure 13 Enlarged portions of mosaic image 30 and CMY mosaic image 35 are shown.

[0090] On the other hand, in step S105, the merging processing unit 110 performs merging processing on the mosaic image 30 whose white balance has been corrected in step S102, and then the processing proceeds to step S106. In the merging processing, as... Figure 14 As shown, in each pixel group 361 corresponding to the first pixel unit 101 within the mosaic image 30, the red image pixel 31 is... R The pixel values ​​are added together to form one red image pixel of 37. R The pixel value. Similarly, below, the green image has 31 pixels. G The pixel value is added in each of the pixel group 362 corresponding to the second pixel unit 102 to form a green image pixel 37. G The pixel value. Additionally, the green image has 31 pixels. G The pixel values ​​are added in each pixel group 363 corresponding to the third pixel unit 103 to form a green image pixel 37. G The pixel value. Furthermore, in each of the pixel groups 364 corresponding to the fourth pixel unit 104, the blue image pixel 31 is... B The pixel values ​​are added together to form a blue image of 37 pixels. B The pixel value. Thus, a red image with 37 pixels is generated. R (Image pixels with only red color information 37) R ), Green image pixels 37 G (Image pixels with only green color information 37) G ), Blue image pixels 37 G (Image pixels with only blue color information 37) B The RGB mosaic image 38 is composed of 30 and 31 pixels. The number of pixels in the RGB mosaic image 38 is one-quarter of the number of pixels in the mosaic image 30. Figure 14 The magnified portions of mosaic image 30 and RGB mosaic image 38 are shown.

[0091] Furthermore, in the merge process, such as Figure 15 As shown, the cyan image has 31 pixels. C The pixel value is added to each of the pixel group 361 corresponding to the first pixel unit 101 in the mosaic image 30 to form the cyan image pixel 38. C A pixel value, wherein the white balance has been corrected for the mosaic image in step S102. Similarly, in the following text, magenta image pixel 31...M The pixel values ​​are added in each of the pixel groups 362 corresponding to the second pixel unit 102 to form a magenta image pixel 38. M The pixel value. Furthermore, in each pixel group 363 corresponding to the third pixel unit 103, a magenta image pixel 31 is added. M The pixel values ​​are used to form a magenta image of 38 pixels. M The pixel value. Furthermore, in each pixel group 364 corresponding to the fourth pixel unit 104, the yellow image pixel 31 Y The pixel values ​​are added together to form a yellow image of 38 pixels. Y The pixel values. Thus, a cyan image of 38 pixels is generated. C (Image pixels with only cyan color information 38) C ), magenta image pixels 38 M (Image pixels with only magenta color information 38) M ) and yellow image pixels 38 Y (Image pixels with only yellow color information 38) Y The CMY mosaic image 39 is composed of 30 and 31 pixels. The number of pixels in the CMY mosaic image 39 is one-quarter of the number of pixels in the mosaic image 30. Figure 15 The magnified portions of mosaic image 30 and CMY mosaic image 39 are shown.

[0092] Furthermore, in the merging process, the brightness value calculation unit 111 will calculate the image pixels 31 with red and cyan color information in each pixel group 361 corresponding to the first pixel unit 101 in the mosaic image 30. R and 31 C The pixel values ​​are added together to calculate, for example, Figure 16 The image pixel 40 shown is a brightness value, to which the white balance of the mosaic image 30 was corrected in step S102. Similarly, image pixels 31 in each of the pixel groups 362 corresponding to the second pixel unit 102 have green and magenta color information. G and 31 M The pixel value is used to calculate the brightness value of an image pixel 40. Additionally, image pixels 31 contain color information for green and magenta. G and 31 M The pixel values ​​are added in each of the pixel groups 363 corresponding to the third pixel unit 103 to calculate the brightness value of an image pixel 40. Furthermore, in each of the pixel groups 364 corresponding to the fourth pixel unit 104, the image pixel 31 with blue and yellow color information is added... B and 31 MThe pixel values ​​are summed to calculate the luminance value of image pixel 40. Therefore, a luminance image 41 is generated that represents only the luminance value of image pixel 40. The number of pixels in luminance image 41 is one-quarter the number of pixels in mosaic image 30. Figure 16 The magnified portions of mosaic image 30 and brightness image 41 are shown.

[0093] Note that the following configuration can be used: the value obtained by multiplying the pixel values ​​of the mosaic image 30 by a correction coefficient is used as the pixel value for calculating the brightness value. The mosaic image is the image whose white balance has been corrected in step S102. Specifically, the pixel values ​​R, G, B, Cy, Mg, and Ye of the image pixel 31 based on the mosaic image 30 and the predetermined correction coefficient a are used. 11 a 21 a 12 a 22 a 13 and a 23 The brightness values ​​Br1, Br2 and Br3 of the image pixel 40 of the brightness image 41 are calculated according to the following formula (1).

[0094]

[0095] In formula (1) above, R is the number of red image pixels (31). R The pixel value, G is the green image pixel value (31). G The pixel value, B is the blue image pixel 31. B The pixel value, Cy is the cyan image pixel value (31). C The pixel value, Mg is the magenta image pixel 31 M The pixel value, and Ye is the yellow image pixel 31. Y The pixel values. Additionally, Br1 is derived from the red and cyan image pixels 31. R and 31 C The brightness values ​​are obtained from the pixel values ​​R and Cy, and Br2 is obtained from the green and magenta image pixels 31. G and 31 M The brightness values ​​are obtained from the pixel values ​​G and Mg, and Br3 is obtained from the blue and yellow image pixels 31. B and 31 Y The brightness values ​​obtained from pixel values ​​B and Ye. Used as a correction factor a. 11 a 21 a 12 a 22 a 13 and a 23 By setting different values ​​for each color temperature of the light source, the relationship Br1 = Br2 = Br3 is established. Therefore, as... Figure 5 , Figure 6 and Figure 7 As shown, the difference between the brightness values ​​Br1, Br2 and Br3 can be reduced.

[0096] Next, the process proceeds to step S106, where the brightness compositing unit 112 performs brightness compositing processing on the RGB mosaic image 38 or CMY mosaic image 39 generated in step S105. In the brightness compositing processing, firstly, based on the pixel values ​​of the image pixels 31 of the mosaic image 30 generated in step S101, it is determined whether the object is bright. Furthermore, as... Figure 17 As shown, when the object is determined to be bright, the brightness value of the brightness image 41 is combined with the RGB mosaic image 38 to generate a composite image 42. By combining the RGB mosaic image 38 and the brightness image 41, both high color reproduction and high resolution can be achieved. Furthermore, unlike a compound eye camera, which includes, for example, a solid-state imaging device for obtaining the RGB mosaic image 38 and a solid-state imaging device for obtaining the brightness image 41, parallax correction is not necessary.

[0097] In the synthesis of RGB mosaic image 38 and luminance image 41, firstly, based on the pixel values ​​R, G, B of pixel 37 of RGB mosaic image 38 and a predetermined coefficient b... 11 b 21 b31, b 12 b 22 b32, a 13 a 23 a 33 The Y1 signal, Cb signal, and Cr signal of the pixel are calculated according to the following formula (2). Subsequently, based on the calculated Y1 signal and the luminance value (Y2 signal) of the image pixel 40 of the luminance image 41, the luminance signal Y of the pixel 43 of the composite image 42 is calculated according to the following formula (3). That is, the Y1 signal is corrected using the Y2 signal to set it as the luminance signal Y. In this case, the Cb signal and Cr signal calculated in the above formula (2) are used as the color information of the pixel 43.

[0098]

[0099] Y = αY1 + (1-α)Y2...(3)

[0100] Here, the signal value Y1CbCr is obtained by two's complement processing of adjacent pixels, and therefore there is a tendency for the resolution (equivalent to the Y1 signal) to degrade. On the other hand, the resolution of the Y2 signal hardly degrades. Therefore, according to the above formula (3), a high-resolution image can be obtained.

[0101] On the other hand, if the subject is determined to be dark, such as Figure 18As shown, the CMY mosaic image 39 is converted into an RGB mosaic image 44 using a color conversion matrix. Next, the luminance image 41 is combined with the converted RGB mosaic image 44 to generate a composite image 45. By combining the luminance image 41 with the RGB mosaic image 44, both high signal-to-noise ratio and high resolution can be achieved. Furthermore, unlike the compound eye camera described above, parallax correction is not necessary.

[0102] As described above, in the solid-state imaging apparatus 1 according to the first embodiment of the present disclosure, each pixel unit group 11 is configured such that the R filter 19 R and C filter 19 C The color filter 19 and G filter 19 are included in the first pixel unit 101 of the four pixel units 10 constituting the pixel unit group 11. G and M filter 19 M Color filter 19 is included in the second pixel unit 102 and the third pixel unit 103, and B filter 19 B and Y filter 19 Y The color filter 19 is included as the fourth pixel unit 104. Therefore, by using the red pixel 9 R Green pixel 9 G Blue pixel 9 B The pixel signals can generate RGB images. Furthermore, cyan pixels 9... C Magenta pixel 9 M and yellow pixel 9 Y The pixel signals are used to generate a CMY image. Furthermore, brightness information can be generated by adding the pixel values ​​of pixels 9 in the same pixel unit 10. For this purpose, a solid-state imaging device 1 capable of acquiring RGB images, CMY images, and brightness information in a single imaging process can be provided.

[0103] <2. Second Embodiment: Electronic Device>

[0104] [Configuration of the main components]

[0105] Next, an electronic device 100 according to a second embodiment of the present disclosure will be described. Figure 19 This is a diagram illustrating the overall configuration of the electronic device 100 according to the second embodiment. Furthermore, Figure 20 This is a flowchart illustrating the image generation process of the first embodiment. Figure 19 and Figure 20 In, corresponding to Figure 1 and Figure 10 The parts of the drawings are given the same reference numerals and symbols, and their descriptions will not be repeated.

[0106] The electronic device 100 according to the second embodiment differs from the electronic device of the first embodiment in that the signal processing circuit 105 includes, for example, Figure 19 The HDR image generation unit 114 shown, and the image generation process includes step S201 instead of... Figure 20 shown Figure 10 Step S106 in the process.

[0107] In step S201, the HDR image generation unit 114 performs HDR image generation processing based on the RGB mosaic image 38 and CMY mosaic image 39 generated in step S105 (see...). Figure 14 and Figure 15 In HDR image generation processing, such as Figure 21 and Figure 22 As shown, a color conversion matrix is ​​used to convert the CMY mosaic image 39 into an RGB mosaic image 46. Subsequently, the converted RGB mosaic image 46 and the RGB mosaic image 38 generated in step S105 are combined to generate an HDR image 47. As a method for combining the RGB mosaic image 46 obtained from the CMY mosaic image 39 and the RGB mosaic image 38 generated in step S105, for example, a method of adding the pixel values ​​of the RGB mosaic image 46 to the pixel values ​​of the RGB mosaic image 38 for each image pixel 48 can be used.

[0108] As described above, the electronic device 100 according to the second embodiment of this disclosure is configured to convert a CMY mosaic image 39 into an RGB mosaic image 46, and to synthesize the converted RGB mosaic image 46 and the RGB mosaic image 38 generated by the merging processing unit 110 to generate an HDR image 47 (high dynamic range image). Therefore, in addition to the RGB image, CMY image, and brightness information, an HDR image 47 can also be generated through an imaging process.

[0109] [2-2. Modified Example]

[0110] It should be noted that in the electronic device 100 according to the first and second embodiments of this disclosure, an example has been described in which the merging process is digitally performed by a signal processing circuit 105 disposed outside the solid-state imaging device 1; however, other configurations may also be employed. For example, a configuration in which the merging process is performed analogically when reading pixel signals from the pixel 9 of the solid-state imaging device 1 may be used. Specifically, as... Figure 23As shown, a configuration can be adopted in which the signal lines 50 connected to the floating diffusion 49 of the pixel 9 constituting the pixel unit 10, and the signal lines 50 connected to the floating diffusion 49 of the pixel 9 of the same color, are electrically connected to each other, and the pixel signals of the pixels 9 of the same color are added together and output to the CDS circuit 51. Therefore, the result of adding the pixel values ​​of the pixels 9 of the same color can be obtained.

[0111] In addition, such as Figure 24 As shown, a configuration can be adopted in which the signal lines 50 connected to the floating diffusers 49 of the pixels 9 constituting the pixel unit 10 are electrically coupled to each other, and the pixel signals of the plurality of pixels 9 are added together and output to the CDS circuit 51. Therefore, it is possible to obtain Figure 16 The brightness values ​​(brightness information) of image pixels 40 in the brightness image 41 shown in the figure.

[0112] <3. Examples of application to moving bodies>

[0113] The technology disclosed herein (the Technology) can be applied to a variety of products. For example, the Technology disclosed herein can be implemented as a device installed on any type of mobile body (such as automobiles, electric vehicles, hybrid electric vehicles, motorcycles, bicycles, personal mobility devices, airplanes, drones, ships, and robots).

[0114] Figure 25 This is a block diagram illustrating an example of a schematic configuration of a vehicle control system, which is an example of a mobile body control system to which the techniques of this disclosure can be applied.

[0115] The vehicle control system 12000 includes multiple electronic control units interconnected via a communication network 12001. Figure 25 In the example shown, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an external information detection unit 12030, an internal information detection unit 12040, and an integrated control unit 12050. Furthermore, as part of the functional configuration of the integrated control unit 12050, a microcomputer 12051, an audio / visual output unit 12052, and an in-vehicle network interface (I / F) 12053 are shown.

[0116] The drive system control unit 12010 controls the operation of devices related to the vehicle's drive system according to various programs. For example, the drive system control unit 12010 functions as a drive force generating device that generates drive force for the vehicle, such as an internal combustion engine or drive electric motor, a drive force transmission mechanism that transmits drive force to the wheels, a steering mechanism that adjusts the vehicle's steering angle, and a braking device that generates braking force for the vehicle.

[0117] The body system control unit 12020 controls the operation of various devices equipped in the vehicle body according to various programs. For example, the body system control unit 12020 is used as a control device for keyless entry systems, smart key systems, power window devices, or various lights such as headlights, taillights, brake lights, turn signals, or fog lights. In this case, radio waves transmitted from a portable device that serves as a substitute key, or signals from various switches, can be input to the body system control unit 12020. The body system control unit 12020 receives these radio wave or signal inputs and controls the vehicle's door locking devices, power window devices, lights, etc.

[0118] The exterior information detection unit 12030 detects information about the exterior of the vehicle on which the vehicle control system 12000 is installed. For example, the imaging unit 12031 is connected to the exterior information detection unit 12030. The exterior information detection unit 12030 causes the imaging unit 12031 to capture images of the exterior of the vehicle and receives the captured images. In addition, the exterior information detection unit 12030 can also perform object detection processing and distance detection processing for people, cars, obstacles, signs, and text on the road based on the received images.

[0119] Imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal corresponding to the amount of light received. Imaging unit 12031 can also output the electrical signal as an image and ranging information. In addition, the light received by imaging unit 12031 can be visible light or invisible light such as infrared.

[0120] The in-vehicle information detection unit 12040 detects information inside the vehicle. For example, a driver state detection unit 12041 for detecting the driver's state is connected to the in-vehicle information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera that captures images of the driver, and the in-vehicle information detection unit 12040 can calculate the driver's fatigue level or concentration level based on the detection information input from the driver state detection unit 12041, or determine whether the driver is dozing off.

[0121] The microcomputer 12051 can calculate control target values ​​for the drive force generating device, steering mechanism, or braking device based on information acquired by the external information detection unit 12030 or the internal information detection unit 12040, and output control commands to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control to realize the functions of an advanced driver assistance system (ADAS), including vehicle collision avoidance, impact mitigation, distance-based driving, speed maintenance driving, vehicle collision warning, and lane departure warning.

[0122] In addition, the microcomputer 12051 controls the drive force generating device, steering mechanism, braking device, etc., based on the information about the vehicle's surroundings obtained by the external information detection unit 12030 or the internal information detection unit 12040, thereby enabling coordinated control for autonomous driving and other functions without relying on the driver's operation.

[0123] Furthermore, the microcomputer 12051 can output control commands to the vehicle system control unit 12020 based on the external information acquired by the external information detection unit 12030. For example, the microcomputer 12051 can perform coordinated control for glare reduction, such as switching from high beam to low beam, by controlling the headlights according to the position of the vehicle in front or oncoming vehicles detected by the external information detection unit 12030.

[0124] The sound and image output unit 12052 sends an output signal of at least one of sound and image to an output device capable of visually or audibly notifying the occupants of the vehicle or the outside of the vehicle. Figure 25 In the example shown, audio speaker 12061, display unit 12062, and dashboard 12063 are shown as output devices. For example, display unit 12062 may include at least one of an onboard display and a head-up display.

[0125] Figure 26 This is a diagram showing an example of the mounting location of the imaging unit 12031.

[0126] exist Figure 26 In the vehicle 12100, imaging units 12101, 12102, 12103, 12104, and 12105 are used as imaging units 12031.

[0127] Imaging units 12101, 12102, 12103, 12104, and 12105 are disposed within the interior of a vehicle, such as vehicle 12100, at locations including the front nose, side mirrors, rear bumper, rear door, and the upper part of the windshield. Imaging unit 12101, disposed at the front nose, and imaging unit 12105, disposed at the upper part of the windshield, primarily acquire images of the front of vehicle 12100. Imaging units 12102 and 12103, disposed at the side mirrors, primarily acquire images of the lateral sides of vehicle 12100. Imaging unit 12104, disposed at the rear bumper or rear door, primarily acquires images of the rear of vehicle 12100. The front view images acquired by imaging units 12101 and 12105 are primarily used for the detection of vehicles, pedestrians, obstacles, traffic signals, traffic signs, lanes, etc.

[0128] It should be noted that Figure 26An example of the imaging range of imaging units 12101 to 12104 is shown. Imaging range 12111 indicates the imaging range of imaging unit 12101 located at the front nose, imaging ranges 12112 and 12113 indicate the imaging ranges of imaging units 12102 and 12103 located at the side mirrors, respectively, and imaging range 12114 indicates the imaging range of imaging unit 12104 located at the rear bumper or rear door. For example, by overlaying the image data captured by imaging units 12101 to 12104, a top-down image of the vehicle 12100 can be obtained.

[0129] At least one of the imaging units 12101 to 12104 may have the function of acquiring distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera configured with a plurality of imaging elements, or may be an imaging element having pixels for phase difference detection.

[0130] For example, the microcomputer 12051 is particularly capable of extracting the nearest three-dimensional object on the path of the vehicle 12100, that is, a three-dimensional object traveling in approximately the same direction as the vehicle 12100 at a predetermined speed (e.g., 0 km / h or higher). Based on distance information obtained from imaging units 12101 to 12104, it acquires the distance between the microcomputer and each three-dimensional object in the imaging range 12111 to 12114 and the time change of that distance (relative speed relative to the vehicle 12100), and uses this information as the vehicle ahead. Furthermore, the microcomputer 12051 can set a pre-set vehicle-to-vehicle distance in front of the vehicle and can execute automatic braking control (including follow-stop control) or automatic acceleration control (including follow-start control). In this way, coordinated control can be performed for autonomous driving and other applications that do not rely on driver operation.

[0131] For example, microcomputer 12051 can classify and extract three-dimensional object data related to three-dimensional objects into two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, and other three-dimensional objects (such as utility poles) based on distance information obtained from imaging units 12101 to 12104, and use this three-dimensional object data for automatic obstacle avoidance. For example, microcomputer 12051 identifies obstacles near vehicle 12100 as obstacles that the driver of vehicle 12100 can visually recognize and obstacles that are difficult to visually recognize. In addition, microcomputer 12051 can determine a collision risk indicating the degree of risk of collision with each obstacle, and when the collision risk has a value equal to or greater than a set value and there is a possibility of collision, output a warning to the driver via audio speaker 12061 or display unit 12062, and execute forced deceleration or evasive steering via drive system control unit 12010, so that it can perform driving assistance for collision avoidance.

[0132] At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared light. For example, the microcomputer 12051 can identify a pedestrian by determining whether a pedestrian is present in the image captured by the imaging units 12101 to 12104. Such pedestrian identification is performed, for example, by extracting feature points from the image captured by the imaging units 12101 to 12104, which are infrared cameras, and by performing pattern matching processing on a series of feature points representing the outline of a subject and determining whether the subject is a pedestrian. When the microcomputer 12051 determines that a pedestrian is present in the captured image of the imaging units 12101 to 12104 and identifies the pedestrian, the sound image output unit 12052 controls the display unit 12062 to overlap the identified pedestrian and display it with a square outline for emphasis. In addition, the sound image output unit 12052 can control the display unit 12062 to display an icon indicating a pedestrian, etc., at a desired location.

[0133] Examples of vehicle control systems to which the technology according to this disclosure can be applied have been described above. The technology of this disclosure can be applied to the imaging unit 12031, etc., in the above configuration. Specifically, Figure 1 and Figure 2 Solid-state imaging devices 101 and 1 and Figure 1 The signal processing circuit 105 can be applied to the imaging unit 12031. By applying the technology according to this disclosure to the imaging unit 12031, a clearer captured image can be obtained, and thus driver fatigue can be reduced.

[0134] <4. Examples of applications in endoscope operating systems>

[0135] For example, the technology according to this disclosure (the technology) can be applied to an endoscope operating system.

[0136] Figure 27 This is a diagram illustrating an example of a schematic configuration of an endoscope operating system to which the technology (the technology) according to this disclosure can be applied.

[0137] Figure 27 The diagram shows an operator (doctor) 11131 using the endoscope operating system 11000 to operate on a patient 11132 on a bed 11133. As shown, the endoscope operating system 11000 consists of an endoscope 11100, an insufflation tube 11111, an electrical processing device 11112, and other processing devices 11110, a support arm device 11120 supporting the endoscope 11100, and a trolley 11200 equipped with various devices for endoscopic surgery.

[0138] Endoscope 11100 comprises a lens barrel 11101 and a camera 11102 connected to the base of the lens barrel 11101. A region of the lens barrel 11101 with a predetermined length from its tip is inserted into the body cavity of the patient 11132. Although the illustrated example shows an endoscope 11100 configured as a so-called rigid endoscope with a rigid lens barrel 11101, endoscope 11100 can be configured as a so-called flexible endoscope with a flexible lens barrel.

[0139] An opening for inserting an objective lens is provided at the top of the lens barrel 11101. A light source device 11203 is connected to the endoscope 11100, and the light generated by the light source device 11203 is guided by a light guide extending inside the lens barrel 11101 to the tip of the lens barrel and radiated towards the object of observation in the body cavity of the patient 11132 via the objective lens. Alternatively, the endoscope 11100 can also be a direct-viewing endoscope, a stereoscopic endoscope, or a side-viewing endoscope.

[0140] An optical system and imaging element are housed inside the camera 11102, and reflected light (observation light) from the observed target is converged onto the imaging element by the optical system. The imaging element performs photoelectric conversion on the observation light to generate an electrical signal corresponding to the observation light, i.e., an image signal corresponding to the observed image. The image signal is sent as RAW data to the camera control unit (CCU) 11201.

[0141] The CCU 11201 consists of a central processing unit (CPU), a graphics processing unit (GPU), etc., and comprehensively controls the operation of the endoscope 11100 and the display device 11202. Furthermore, the CCU 11201 receives image signals from the camera 11102 and performs various types of image processing on the image signals for displaying images based on the image signals, such as image processing (de-mosaic processing).

[0142] The display device 11202 displays an image based on the image signal that has undergone image processing by the CCU 11201 under the control of the CCU 11201.

[0143] The light source device 11203 is composed of a light source such as a light-emitting diode (LED) and supplies illumination light to the endoscope 11100 when photographing the surgical area.

[0144] Input device 11204 is the input interface of endoscope operating system 11000. Users can input various information or commands into endoscope operating system 11000 via input device 11204. For example, users can input commands to change the imaging conditions of endoscope 11100 (type of illumination light, magnification, focal length, etc.).

[0145] The treatment device control unit 11205 controls the drive of the energized processing instrument 11112 used for cauterizing, cutting, and sealing blood vessels. To ensure the field of vision of the endoscope 11100 and to ensure the operator's operating space, the pneumoperitoneum device 11206 inflates the body cavity of the patient 11132 by introducing gas through the pneumoperitoneum tube 11111. The recorder 11207 is a device capable of recording various types of information related to the operator. The printer 11208 is a device capable of printing various types of information related to the operator in various formats such as text, images, and graphics.

[0146] The light source device 11203 that supplies radiation light to the endoscope 11100 for imaging the surgical site can, for example, be a white light source composed of LEDs, laser light sources, or combinations thereof. When a white light source is formed by a combination of RGB laser sources, the output intensity and timing of each color (each wavelength) can be controlled with high precision, and thus, the light source device 11203 adjusts the white balance of the captured image. Furthermore, in this case, the lasers from each of the individual RGB laser sources are irradiated onto the observation target in a time-division manner, and the driving of the imaging element of the camera 11102 is controlled synchronously with the irradiation timing, so that images corresponding to each of the RGB can also be captured in a time-division manner. According to this method, color images can be obtained without setting filters on the imaging element.

[0147] Furthermore, the drive of the light source device 11203 can be controlled to change the intensity of the output light at predetermined time intervals. The drive of the imaging element of the camera 11102 is controlled synchronously with the timing of the change in light intensity, thereby acquiring images in a time-division manner, and the images are synthesized, thereby enabling the generation of so-called images with high dynamic range without underexposure or overexposure.

[0148] Furthermore, the light source device 11203 may also have a structure capable of supplying light in a specified wavelength band corresponding to special light observation. In special light observation, a so-called narrowband light observation (narrowband light observation) is performed by emitting light with a narrower frequency band than the illumination light (i.e., white light) used in normal observation, utilizing the wavelength dependence of light absorption by biological tissues. Alternatively, in special light observation, fluorescence observation can also be performed, acquiring an image by emitting fluorescence generated by excitation light. Fluorescence observation can be performed by emitting excitation light into body tissue and observing fluorescence from the body tissue (autofluorescence observation), or by locally injecting a reagent such as indocyanine green (ICG) into body tissue and emitting excitation light corresponding to the fluorescence wavelength of the reagent into the body tissue to obtain a fluorescence image. The light source device 11203 may be configured to supply narrowband light and / or excitation light corresponding to such special light observation.

[0149] Figure 28 It is shown Figure 27 A block diagram showing an example of the functional configuration of camera 11102 and CCU 11201.

[0150] Camera 11102 includes a lens unit 11401, an imaging unit 11402, a driving unit 11403, a communication unit 11404, and a camera control unit 11405. CCU 11201 includes a communication unit 11411, an image processing unit 11412, and a control unit 11413. Camera 11102 and CCU 11201 are connected to each other via a transmission cable 11400, enabling them to communicate with each other.

[0151] Lens unit 11401 is an optical system disposed at the portion for connection with lens barrel 11101. Observation light obtained from the top of lens barrel 11101 is guided to camera 11102 and incident on lens unit 11401. Lens unit 11401 is composed of a combination of multiple lenses, including zoom lenses and focusing lenses.

[0152] Imaging unit 11402 is composed of imaging elements. The imaging elements constituting imaging unit 11402 can be a single element (so-called single-plate type) or multiple elements (so-called multi-plate type). When imaging unit 11402 is configured as a multi-plate type, for example, an image signal corresponding to RGB is generated by the imaging elements, and a color image can be obtained by synthesizing the image signals. Alternatively, imaging unit 11402 can be configured to include a pair of imaging elements for acquiring image signals for the right and left eyes corresponding to three-dimensional (3D) display. During 3D display, the surgical operator 11131 can more accurately grasp the depth of biological tissue within the surgical area. Here, when imaging unit 11402 is configured as a multi-plate type, multiple lens units 11401 can be provided according to the imaging elements.

[0153] Furthermore, the imaging unit 11402 need not be located within the camera 11102. For example, the imaging unit 11402 can be located inside the lens barrel 11101 immediately after the objective lens.

[0154] The drive unit 11403 is composed of an actuator and, under the control of the camera control unit 11405, moves the zoom lens and focusing lens of the lens unit 11401 a predetermined distance along the optical axis. This allows for appropriate adjustment of the magnification and focus of the image captured by the imaging unit 11402.

[0155] The communication unit 11404 is composed of a communication device for sending various information to or receiving various information from the CCU 11201. The communication unit 11404 transmits the image signal obtained from the imaging unit 11402 as RAW data to the CCU 11201 via the transmission cable 11400.

[0156] Furthermore, the communication unit 11404 receives control signals from the CCU 11201 for controlling the camera 11102 and supplies the control signals to the camera control unit 11405. The control signals include, for example, information about imaging conditions, such as information for specifying the frame rate of the image, information for specifying the exposure value during imaging, and / or information for specifying the magnification and focus of the image.

[0157] Furthermore, imaging conditions (such as frame rate, exposure value, magnification, and focus as mentioned above) can be appropriately specified by the user or can be automatically set by the control unit 11413 of the CCU 11201 based on the acquired image signal. In the latter case, so-called automatic exposure (AE), automatic focus (AF), and automatic white balance (AWB) functions are provided in the endoscope 11100.

[0158] The camera control unit 11405 controls the driving of the camera 11102 based on the control signals received from the CCU 111201 via the communication unit 11404.

[0159] The communication unit 11411 comprises a communication device for sending and receiving various types of information to and from the camera 11102. The communication unit 11411 receives image signals transmitted from the camera 11102 via a transmission cable 11400.

[0160] In addition, the communication unit 11411 sends control signals to the camera 11102 to control the camera 11102. The image signal or control signal can be transmitted via electrical communication, optical communication, etc.

[0161] The image processing unit 11412 performs various image processing operations on the image signal, which is RAW data sent from the camera 11102.

[0162] The control unit 11413 performs various controls regarding the taking of images of the surgical site using the endoscope 11100 and the display of the images obtained by taking images of the surgical site. For example, the control unit 11413 generates control signals for controlling the drive of the camera 11102.

[0163] Furthermore, the control unit 11413 causes the display device 11202 to display the captured image obtained by imaging the surgical area, etc., based on the image signal that has undergone image processing by the image processing unit 11412. In this case, the control unit 11413 can use various image recognition technologies to identify various objects in the captured image. For example, by detecting the edge shape, color, etc. of objects contained in the captured image, the control unit 11413 can identify surgical instruments such as forceps, specific biological parts, bleeding, and fogging when the power-operated processing instrument 11112 is used. When the control unit 11413 causes the display device 11202 to display the captured image, various types of surgical support information can be overlaid and displayed with the image of the surgical area using the recognition results. When surgical assistance information is overlaid, displayed, and presented to the operator 11131, the burden on the operator 11131 can be reduced, and the operator 11131 can operate reliably.

[0164] The transmission cable 11400 connecting the camera 11102 and the CCU 11201 is an electrical signal cable that supports electrical signal communication, an optical fiber that supports optical communication, or a composite cable thereof.

[0165] In the example shown in the accompanying drawings, communication is performed wired using transmission cable 11400, but communication between camera 11102 and CCU 11201 can be performed wirelessly.

[0166] Examples of endoscope operating systems that can be applied according to the technology disclosed herein have been described above. The technology disclosed herein can be applied, for example, to the imaging unit 11402 of the camera 11102, the image processing unit 11412 of the CCU 111201, etc., as described above. Specifically, Figure 1 and Figure 2 The solid-state imaging devices 101 and 1 can be applied to the imaging unit 10402, and Figure 1 The signal processing circuit 105 can be applied to the image processing unit 11412. By applying the technology according to this disclosure to the imaging unit 10402 and the image processing unit 11412, a clearer image of the surgical part can be obtained, and therefore, the surgeon can reliably confirm the surgical part.

[0167] Although the endoscopic operating system has been described here as an example, the techniques according to this disclosure can be applied to others, such as microscopic operating systems.

[0168] This technology can also be configured as follows.

[0169] (1) A solid-state imaging device, comprising:

[0170] The pixel array section has multiple pixel unit groups arranged in it. Each pixel unit group consists of pixel units arranged in a 2×2 matrix. Each pixel unit consists of pixels arranged in an m×m matrix (where m is 2 or a larger natural number). Each pixel includes a photoelectric conversion unit and a color filter formed to correspond to the photoelectric conversion unit.

[0171] Each pixel unit group includes an R filter that transmits red light and a C filter that transmits cyan light, serving as the color filter in the first pixel unit among the four pixel units constituting the pixel unit group. Cyan light and red light are complementary colors. Each pixel unit group also includes a G filter that transmits green light and an M filter that transmits magenta light, serving as the color filters in the second and third pixel units. Magenta light and green light are complementary colors. Furthermore, each pixel unit group includes a B filter that transmits blue light and a Y filter that transmits yellow light, serving as the color filter in the fourth pixel unit. Yellow light and blue light are complementary colors.

[0172] (2) The solid-state imaging device according to (1),

[0173] Where m is a natural number of 3 or greater;

[0174] The first pixel unit is configured such that some pixels among a plurality of pixels arranged in an m×m matrix and positioned in a checkerboard pattern include R filters, and the remaining pixels include C filters.

[0175] The second pixel unit is configured such that among a plurality of pixels arranged in an m×m matrix, some pixels positioned in a checkerboard pattern include G filters, and the remaining pixels include M filters; and

[0176] The third pixel unit is configured such that some pixels among a plurality of pixels arranged in an m×m matrix and positioned in a checkerboard pattern include a B filter, and the remaining pixels include a Y filter.

[0177] (3) The solid-state imaging device according to (1),

[0178] Where m = 2;

[0179] The first pixel unit is configured such that one pixel pair arranged diagonally includes an R filter, and the other pixel pair includes a C filter.

[0180] The second pixel unit is configured such that one pixel pair arranged diagonally includes a G filter, and the other pixel pair includes an M filter; and

[0181] The third pixel unit is configured such that one pixel pair arranged diagonally includes a B filter, and the other pixel pair includes an M filter.

[0182] (4) An electronic device comprising:

[0183] A solid-state imaging device includes a pixel array section, in which multiple pixel unit groups are arranged. Each pixel unit group consists of pixel units arranged in a 2×2 matrix, and each pixel unit consists of pixels arranged in an m×m matrix (where m is 2 or a larger natural number). Each pixel includes a photoelectric conversion unit and a color filter formed corresponding to the photoelectric conversion unit. Each pixel unit group includes an R filter that transmits red light and a C filter that transmits cyan light, serving as the color filter in the first pixel unit among the four pixel units constituting the pixel unit group. Cyan light and red light are complementary colors. Each pixel unit group also includes a G filter that transmits green light and an M filter that transmits magenta light, serving as the color filters in the second and third pixel units. Magenta light and green light are complementary colors. Furthermore, each pixel unit group includes a B filter that transmits blue light and a Y filter that transmits yellow light, serving as the color filter in the fourth pixel unit. Yellow light and blue light are complementary colors.

[0184] An optical lens forms an image from the light emitted by the subject onto the imaging surface of a solid-state imaging device; and

[0185] The signal processing circuit performs signal processing on the signal output from the solid-state imaging device.

[0186] (5) The electronic device according to (4),

[0187] The signal processing circuit includes a merging processing unit, which generates an RGB mosaic image by: adding the pixel values ​​of the image pixels included in each pixel group corresponding to the first pixel unit in the mosaic image corresponding to the arrangement of color filters obtained from the signal, so as to set the pixel value to the pixel value of an image pixel with only red color information; adding the pixel values ​​of the image pixels with only green color information in each pixel group corresponding to the second and third pixel units, so as to set the pixel value to the pixel value of an image pixel with only green color information; and adding the pixel values ​​of the image pixels with only blue color information in each pixel group corresponding to the fourth pixel unit, so as to set the pixel value to the pixel value of an image pixel with only blue color information.

[0188] (6) The electronic device according to (4) or (5),

[0189] The signal processing circuit includes a merging processing unit, which generates a CMY mosaic image by: adding the pixel values ​​of image pixels in each pixel group corresponding to the first pixel unit in the mosaic image corresponding to the arrangement of color filters obtained from the signal, which have only cyan color information, to set the pixel value to the pixel value of an image pixel with only cyan color information; adding the pixel values ​​of image pixels in each pixel group corresponding to the second and third pixel units, which have only magenta color information, to set the pixel value to the pixel value of an image pixel with only magenta color information; and adding the pixel values ​​of image pixels in each pixel group corresponding to the fourth pixel unit, which have only yellow color information, to set the pixel value to the pixel value of an image pixel with only yellow color information.

[0190] (7) The electronic device according to any one of (4) to (6),

[0191] The signal processing circuit includes a brightness value calculation unit. The brightness value calculation unit calculates the brightness value of an image pixel by adding the pixel values ​​of image pixels with red and cyan color information in each pixel group corresponding to the first pixel unit in the mosaic image corresponding to the array of color filters obtained from the signal, adding the pixel values ​​of image pixels with green and magenta color information in each pixel group corresponding to the second and third pixel units, and adding the pixel values ​​of image pixels with blue and yellow color information in each pixel group corresponding to the fourth pixel unit.

[0192] (8) The electronic device according to (7),

[0193] The brightness value calculation unit uses the value obtained by multiplying the pixel value of the mosaic image by a correction coefficient set for each color temperature of the light source as the pixel value to be used to calculate the brightness value.

[0194] (9) The electronic device according to (7) or (8),

[0195] The signal processing circuit includes:

[0196] The merging processing unit generates an RGB mosaic image by: adding the pixel values ​​of image pixels with only red color information in each pixel group corresponding to the first pixel unit in the mosaic image corresponding to the arrangement of color filters obtained from the signal, to set the pixel value to the pixel value of an image pixel with only red color information; adding the pixel values ​​of image pixels with only green color information in each pixel group corresponding to the second and third pixel units, to set the pixel value to the pixel value of an image pixel with only green color information; and adding the pixel values ​​of image pixels with only blue color information in each pixel group corresponding to the fourth pixel unit, to set the pixel value to the pixel value of an image pixel with only blue color information; and

[0197] The brightness synthesis unit synthesizes RGB mosaic images and brightness values.

[0198] (10) The electronic device according to (7) or (8),

[0199] The signal processing circuit includes:

[0200] The merging processing unit generates a CMY mosaic image by: adding the pixel values ​​of image pixels in each pixel group corresponding to the first pixel unit in the mosaic image corresponding to the arrangement of color filters obtained from the signal, which have only cyan color information, to set the pixel value to the pixel value of an image pixel having only cyan color information; adding the pixel values ​​of image pixels in each pixel group corresponding to the second and third pixel units, which have only magenta color information, to set the pixel value to the pixel value of an image pixel having only magenta color information; and adding the pixel values ​​of image pixels in each pixel group corresponding to the fourth pixel unit, which have only yellow color information, to set the pixel value to the pixel value of an image pixel having only yellow color information; and

[0201] The brightness compositing unit converts the CMY mosaic image into an RGB mosaic image, and then combines the converted RGB mosaic image with the brightness value.

[0202] (11) The electronic device according to (5),

[0203] The merging processing unit generates RGB mosaic images and CMY mosaic images by: adding the pixel values ​​of image pixels with only cyan color information in each pixel group corresponding to the first pixel unit in the mosaic image corresponding to the arrangement of color filters obtained from the signal, so as to set the pixel value to the pixel value of an image pixel with only cyan color information; adding the pixel values ​​of image pixels with only magenta color information in each pixel group corresponding to the second and third pixel units, so as to set the pixel value to the pixel value of an image pixel with only magenta color information; and adding the pixel values ​​of image pixels with only yellow color information in each pixel group corresponding to the fourth pixel unit, so as to set the pixel value to the pixel value of an image pixel with only yellow color information; and

[0204] The signal processing circuit includes an HDR image generation unit, which converts a CMY mosaic image into an RGB mosaic image and then combines the converted RGB mosaic image with the RGB mosaic image generated by the merging processing unit to generate a high dynamic range image.

[0205] [List of Reference Numbers]

[0206] 1 Solid-state imaging device

[0207] 2 substrate

[0208] 3-pixel array

[0209] 4 Vertical drive circuit

[0210] 5-column signal processing circuits

[0211] 6. Horizontal drive circuit

[0212] 7 Output Circuit

[0213] 8. Control Circuit

[0214] 9 pixels

[0215] 10-pixel unit

[0216] 11 pixel unit group

[0217] 12-pixel driving wiring

[0218] 13 Vertical signal lines

[0219] 14 horizontal signal lines

[0220] 15 Insulating film

[0221] 16 shading film

[0222] 17. Flat membrane

[0223] 18. Optical receiving layer

[0224] 19 Color Filters

[0225] 20 microlenses

[0226] 21 Concentrating Layer

[0227] 22 Wiring Layer

[0228] 23 Supporting substrate

[0229] 24 Photoelectric conversion units

[0230] 25-pixel separation unit

[0231] 26-color filter array

[0232] 27 microlens array

[0233] 28-layer interlayer insulation film

[0234] 29. Wiring

[0235] 30 mosaic images

[0236] 31 image pixels

[0237] 32 image pixels

[0238] 33 RGB mosaic image

[0239] 34 image pixels

[0240] 35 CMY mosaic images

[0241] Pixel groups 361, 362, 363, and 364

[0242] 37 image pixels

[0243] 38 image pixels

[0244] 39 CMY mosaic images

[0245] 40 image pixels

[0246] 41 Brightness Image

[0247] 42 Composite Image

[0248] 43 Image pixels

[0249] 44 RGB mosaic image

[0250] 45 Composite Image

[0251] 46 RGB mosaic image

[0252] 47 HDR images

[0253] 48 image pixels

[0254] 100 Electronic devices

[0255] 101 Solid-State Imaging Device

[0256] 102 Optical Lenses

[0257] 103 Shutter mechanism

[0258] 104 drive circuit

[0259] 105 Signal Processing Circuit

[0260] 106 Mosaic Image Generation Units

[0261] 107 White Balance Adjustment Units

[0262] 108 Pattern Determination Unit

[0263] 109 Mosaic Processing Units

[0264] 110 Merging Processing Unit

[0265] 111 Brightness value calculation unit

[0266] 112 Luminance Combining Units

[0267] 113 Incident light

[0268] 114 HDR image generation units

Claims

1. A solid-state imaging device, comprising: A pixel array section, wherein a plurality of pixel unit groups are arranged in the pixel array section, the pixel unit group is composed of pixel units arranged in a 2×2 matrix, the pixel unit is composed of pixels arranged in an m×m matrix, wherein m is a natural number of 2 or greater, and the pixel includes a photoelectric conversion unit and a color filter formed to correspond to the photoelectric conversion unit; Each pixel unit group comprises an R filter for transmitting red light and a C filter for transmitting cyan light, serving as the color filter in the first pixel unit among the four pixel units constituting the pixel unit group. The cyan light and red light are complementary colors. Each pixel unit group also comprises a G filter for transmitting green light and an M filter for transmitting magenta light, serving as the color filters in the second and third pixel units. The magenta light and green light are complementary colors. Furthermore, each pixel unit group comprises a B filter for transmitting blue light and a Y filter for transmitting yellow light, serving as the color filter in the fourth pixel unit. The yellow light and blue light are complementary colors. Specifically, in all color filters of the first pixel unit, the R filter and the C filter are combined to ensure that the transmittance is flat for all light from the short wavelength side to the long wavelength side; in all color filters of the second and third pixel units, the G filter and the M filter are combined to ensure that the transmittance is flat for all light from the short wavelength side to the long wavelength side; and in all color filters of the fourth pixel unit, the B filter and the Y filter are combined to ensure that the transmittance is flat for all light from the short wavelength side to the long wavelength side.

2. The solid-state imaging device according to claim 1, in, m is a natural number of 3 or greater; The first pixel unit is configured such that some pixels among a plurality of pixels arranged in an m×m matrix and positioned in a checkerboard pattern include the R filter, and the remaining pixels include the C filter. The second pixel unit is configured such that some pixels among a plurality of pixels arranged in an m×m matrix and positioned in a checkerboard pattern include the G filter, and the remaining pixels include the M filter. and The third pixel unit is configured such that some pixels among a plurality of pixels arranged in an m×m matrix and positioned in a checkerboard pattern include the B filter, and the remaining pixels include the Y filter.

3. The solid-state imaging device according to claim 1, in, m=2； The first pixel unit is configured such that one pixel pair arranged diagonally includes the R filter, and the other pixel pair includes the C filter; The second pixel unit is configured such that one pixel pair arranged diagonally includes the G filter, and the other pixel pair includes the M filter; and The third pixel unit is configured such that one pixel pair arranged diagonally includes the B filter, and the other pixel pair includes the M filter.

4. An electronic device, comprising: A solid-state imaging device includes a pixel array section, in which multiple pixel unit groups are arranged. Each pixel unit group consists of pixel units arranged in a 2×2 matrix, and each pixel unit consists of pixels arranged in an m×m matrix, where m is a natural number of 2 or greater. Each pixel includes a photoelectric conversion unit and a color filter formed corresponding to the photoelectric conversion unit. Each pixel unit group includes an R filter that transmits red light and a C filter that transmits cyan light, serving as the color filter in the first pixel unit among the four pixel units constituting the pixel unit group. The cyan light and red light have a complementary color relationship. Each pixel unit group also includes a G filter that transmits green light and an M filter that transmits magenta light, serving as the color filters in each of the second and third pixel units. The red light and the green light are complementary colors; and each pixel unit group in the pixel unit group includes a B filter that transmits blue light and a Y filter that transmits yellow light, serving as color filters in the fourth pixel unit, where the yellow light and the blue light are complementary colors; wherein, in all color filters in the first pixel unit, the R filter and the C filter are combined to make the transmittance flat for all light from the short wavelength side to the long wavelength side; in all color filters in the second and third pixel units, the G filter and the M filter are combined to make the transmittance flat for all light from the short wavelength side to the long wavelength side; in all color filters in the fourth pixel unit, the B filter and the Y filter are combined to make the transmittance flat for all light from the short wavelength side to the long wavelength side; An optical lens forms an image from the light emitted by the subject onto the imaging surface of the solid-state imaging device; and The signal processing circuit performs signal processing on the signal output from the solid-state imaging device.

5. The electronic device according to claim 4, in, The signal processing circuit includes a merging processing unit that generates an RGB mosaic image by: adding the pixel values ​​of image pixels included in each pixel group corresponding to the first pixel unit in the mosaic image corresponding to the arrangement of color filters obtained from the signal, so as to set the pixel value to the pixel value of an image pixel having only red color information; adding the pixel values ​​of image pixels having only green color information in each pixel group corresponding to the second pixel unit and the third pixel unit, so as to set the pixel value to the pixel value of an image pixel having only green color information; and adding the pixel values ​​of image pixels having only blue color information in each pixel group corresponding to the fourth pixel unit, so as to set the pixel value to the pixel value of an image pixel having only blue color information.

6. The electronic device according to claim 4, in, The signal processing circuit includes a merging processing unit that generates a CMY mosaic image by: adding the pixel values ​​of image pixels in each pixel group corresponding to the first pixel unit that have only cyan color information in the mosaic image corresponding to the arrangement of color filters obtained from the signal, to set the pixel value to the pixel value of an image pixel with only cyan color information; adding the pixel values ​​of image pixels in each pixel group corresponding to the second pixel unit and the third pixel unit that have only magenta color information, to set the pixel value to the pixel value of an image pixel with only magenta color information; and adding the pixel values ​​of image pixels in each pixel group corresponding to the fourth pixel unit that have only yellow color information, to set the pixel value to the pixel value of an image pixel with only yellow color information.

7. The electronic device according to claim 4, in, The signal processing circuit includes a brightness value calculation unit, which: adds the pixel values ​​of image pixels with red and cyan color information in each pixel group corresponding to the first pixel unit in the mosaic image corresponding to the array of color filters obtained from the signal, to calculate the brightness value of an image pixel; adds the pixel values ​​of image pixels with green and magenta color information in each pixel group corresponding to the second pixel unit and the third pixel unit, to calculate the brightness value of an image pixel; and adds the pixel values ​​of image pixels with blue and yellow color information in each pixel group corresponding to the fourth pixel unit, to calculate the brightness value of an image pixel.

8. The electronic device according to claim 7, in, The brightness value calculation unit uses the value obtained by multiplying the pixel value of the mosaic image by a correction coefficient set for each color temperature of the light source as the pixel value to be used to calculate the brightness value.

9. The electronic device according to claim 7, in, The signal processing circuit includes: The merging processing unit generates an RGB mosaic image by: adding the pixel values ​​of image pixels with only red color information in each pixel group corresponding to the first pixel unit in the mosaic image corresponding to the arrangement of color filters obtained from the signal, to set the pixel value to the pixel value of an image pixel with only red color information; adding the pixel values ​​of image pixels with only green color information in each pixel group corresponding to the second pixel unit and the third pixel unit, to set the pixel value to the pixel value of an image pixel with only green color information; and adding the pixel values ​​of image pixels with only blue color information in each pixel group corresponding to the fourth pixel unit, to set the pixel value to the pixel value of an image pixel with only blue color information; and A brightness synthesis unit synthesizes the RGB mosaic image and the brightness value.

10. The electronic device according to claim 7, in, The signal processing circuit includes: A merging processing unit generates a CMY mosaic image by: adding the pixel values ​​of image pixels in each pixel group corresponding to the first pixel unit in the mosaic image corresponding to the arrangement of color filters obtained from the signal, such that the pixel value is set to the pixel value of an image pixel having only cyan color information; adding the pixel values ​​of image pixels in each pixel group corresponding to the second pixel unit and the third pixel unit, such that the pixel value is set to the pixel value of an image pixel having only magenta color information; and adding the pixel values ​​of image pixels in each pixel group corresponding to the fourth pixel unit, such that the pixel value is set to the pixel value of an image pixel having only yellow color information; and... A brightness synthesis unit converts the CMY mosaic image into an RGB mosaic image, and then synthesizes the converted RGB mosaic image with the brightness value.

11. The electronic device according to claim 5, in, The merging processing unit generates RGB mosaic images and CMY mosaic images by: adding the pixel values ​​of image pixels in each pixel group corresponding to the first pixel unit in the mosaic image corresponding to the arrangement of color filters obtained from the signal, to set the pixel value to the pixel value of an image pixel with only cyan color information; adding the pixel values ​​of image pixels in each pixel group corresponding to the second pixel unit and the third pixel unit, to set the pixel value to the pixel value of an image pixel with only magenta color information; and adding the pixel values ​​of image pixels in each pixel group corresponding to the fourth pixel unit, to set the pixel value to the pixel value of an image pixel with only yellow color information; and The signal processing circuit includes an HDR image generation unit, which converts the CMY mosaic image into an RGB mosaic image and combines the converted RGB mosaic image with the RGB mosaic image generated by the merging processing unit to generate a high dynamic range image.

Images