Solid-state imaging device and electronic device

Abstract

Provided is a solid-state imaging device capable of acquiring an RGB image, a CMY image, and luminance information in a single capture. The solid-state imaging device comprises a pixel array portion in which a plurality of pixel unit groups are arrayed, each pixel unit group comprising a 2×2 matrix arrangement of pixel units, each pixel unit comprising a 2×2 matrix arrangement of pixels each having a photoelectric conversion portion and a color filter formed in correspondence to the photoelectric conversion portion. Each of the pixel unit groups is configured such that, of the four pixel units composing the pixel unit group, a first pixel unit includes an R-filter and a C-filter as color filters, each of second and third pixel units includes a G-filter and an M-filter as color filters, and a fourth pixel unit includes a B-filter and a Y-filter as color filters.

Description

[Technical field]

[0001] The present technology relates to a solid-state imaging device and an electronic device. [Background technology]

[0002] Conventionally, a solid-state imaging device has been proposed in which one pixel of a Bayer array is divided into multiple pixels (see, for example, Patent Document 1). In the solid-state imaging device described in Patent Document 1, a high-resolution captured image can be obtained by performing full-resolution demosaic processing (a series of processes in which demosaic processing is performed after re-mosaic processing), and a captured image with an excellent signal-to-noise ratio can be obtained by performing binning processing. Furthermore, a captured image with a high dynamic range (HDR) can be obtained by changing the exposure conditions for each of the multiple pixels. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-175912 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the solid-state imaging device described in Patent Document 1 cannot obtain an RGB image, a CMY image, and luminance information of a subject in a single imaging operation. An object of the present disclosure is to provide a solid-state imaging device and electronic device capable of acquiring an RGB image, a CMY image, and luminance information in a single imaging operation. [Means for solving the problem]

[0005] The solid-state imaging device of the present disclosure (a) comprises a pixel array section in which a plurality of pixel unit groups are arranged in a 2 x 2 matrix, each of which is made up of pixel units each of which is made up of an m x m (m is a natural number greater than or equal to 2) matrix of pixels having a photoelectric conversion section and a color filter formed corresponding to the photoelectric conversion section, and (b) each of the pixel unit groups includes, among four pixel units constituting the pixel unit group, an R filter that transmits red light and a C filter that transmits cyan light that is complementary to the red light, as color filters in the first pixel unit, a G filter that transmits green light and an M filter that transmits magenta light that is complementary to the green light, as color filters in the second and third pixel units, and a B filter that transmits blue light and a Y filter that transmits yellow light that is complementary to the blue light, as color filters, in the fourth pixel unit.

[0006] The electronic device of the present disclosure includes (a) a pixel array section in which a plurality of pixel unit groups are arranged in a 2×2 matrix, each pixel unit being made up of an m×m (m is a natural number of 2 or more) matrix of pixels each having a photoelectric conversion unit and a color filter formed corresponding to the photoelectric conversion unit, and each pixel unit group is made up of a first pixel unit out of four pixel units constituting the pixel unit group, the first pixel unit being provided with an R filter that transmits red light and a C filter that transmits cyan light, which is complementary to the red light, as color filters. the second and third pixel units each including, as color filters, a G filter that transmits green light and an M filter that transmits magenta light, which is a complementary color to the green light, and the fourth pixel unit includes, as color filters, a B filter that transmits blue light and a Y filter that transmits yellow light, which is a complementary color to the blue light; (b) an optical lens that forms an image of light from a subject on an imaging surface of the solid-state imaging device; and (c) a signal processing circuit that performs signal processing on a signal output from the solid-state imaging device. [Brief description of the drawings]

[0007] [Figure 1]1 is a diagram showing an overall configuration of an electronic device according to a first embodiment of the present disclosure. [Diagram 2] 1 is a diagram showing an overall configuration of a solid-state imaging device according to a first embodiment of the present disclosure. [Figure 3A] 3 is a diagram showing a cross-sectional configuration of a pixel array portion taken along line AA in FIG. 2. FIG. [Figure 3B] 3B is a diagram showing a minimum unit array of color filters when cut along line BB in FIG. 3A. FIG. [Figure 4] 13 is a diagram showing a minimum unit array of a color filter according to a modified example. FIG. [Diagram 5] FIG. 13 is a diagram showing the transmittance of a red pixel and a cyan pixel. [Figure 6] FIG. 13 is a diagram showing the transmittance of a green pixel and a magenta pixel. [Figure 7] FIG. 4 is a diagram showing the transmittance of a blue pixel and a yellow pixel. [Figure 8] FIG. 2 is a diagram showing a configuration of a color filter array. [Figure 9] 13A and 13B are diagrams illustrating an arrangement of microlenses according to a modified example. [Figure 10] 13 is a flowchart showing an image generating process. [Figure 11] FIG. 4 is a diagram showing a captured image generated by a signal processing circuit. [Figure 12] FIG. 13 is a diagram showing the processing contents of re-mosaic processing. [Figure 13] FIG. 13 is a diagram showing the processing contents of re-mosaic processing. [Figure 14] FIG. 13 is a diagram showing the processing contents of a binning process. [Figure 15] FIG. 13 is a diagram showing the processing contents of a binning process. [Figure 16] FIG. 13 is a diagram showing the processing contents of a binning process. [Figure 17] FIG. 11 is a diagram showing the processing contents of a luminance synthesis process. [Figure 18] FIG. 11 is a diagram showing the processing contents of a luminance synthesis process. [Figure 19] FIG. 11 is a diagram showing an overall configuration of an electronic device according to a second embodiment of the present disclosure. [Figure 20] 13 is a flowchart showing an image generating process. [Figure 21] FIG. 11 is a diagram showing the processing contents of HDR image generation processing. [Figure 22] FIG. 11 is a diagram showing the processing contents of HDR image generation processing. [Diagram 23] FIG. 4 is a diagram showing a connection state of signal lines. [Figure 24] FIG. 4 is a diagram showing a connection state of signal lines. [Diagram 25] 1 is a block diagram showing an example of a schematic configuration of a vehicle control system; [Figure 26] 4 is an explanatory diagram showing an example of the installation positions of an outside-vehicle information detection unit and an imaging unit; FIG. [Figure 27] 1 is a diagram illustrating an example of a schematic configuration of an endoscopic surgery system. [Figure 28] 2 is a block diagram showing an example of a functional configuration of a camera head and a CCU. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] An example of a solid-state imaging device 1 and an electronic device according to an embodiment of the present disclosure will be described below with reference to FIGS. 1 to 28. The embodiments of the present disclosure will be described in the following order. Note that the present disclosure is not limited to the following examples. Also, the effects described in this specification are merely examples and are not limiting, and other effects may also be present. 1. First embodiment: Electronic device 1-1 Overall configuration of electronic devices 1-2 Main components 1-3 Image generation processing 2. Second embodiment: Electronic device 2-1 Main components 2-2 Variations 3. Application examples for mobile objects 4. Application example to endoscopic surgery system

[0009] 1. First embodiment: Electronic device [1-1 Overall configuration of electronic devices] An electronic device 100 according to a first embodiment of the present disclosure will be described. As the electronic device 100, various electronic devices such as imaging devices such as digital still cameras and digital video cameras, mobile phones with imaging functions, or other devices with imaging functions can be adopted. Fig. 1 is a schematic diagram showing the whole of the electronic device 100 according to the first embodiment of the present disclosure. As shown in FIG. 1, the electronic device 100 includes a solid-state imaging device 101 (hereinafter, also referred to as "solid-state imaging device 1"), an optical lens 102, a shutter device 103, a driving circuit 104, and a signal processing circuit 105. The signal processing circuit 105 includes a mosaic image generating unit 106, a white balance adjusting unit 107, a mode determining unit 108, a re-mosaic processing unit 109, a binning processing unit 110, a luminance value calculating unit 111, and a luminance combining unit 112. In the electronic device 100, the optical lens 102 forms an image light (incident light 113) from a subject on an imaging surface of the solid-state imaging device 101, the solid-state imaging device 101 converts the amount of the incident light 113 into an electrical signal on a pixel-by-pixel basis and outputs the pixel signal, and the signal processing circuit 105 performs signal processing on the pixel signal output from the solid-state imaging device 101. At this time, the shutter device 103 controls a light irradiation period and a light blocking period for the solid-state imaging device 101. In addition, the drive circuit 104 supplies a drive signal that controls the transfer operation of the pixel signals and the shutter operation of the shutter device 103 . Fig. 2 is a schematic diagram showing the configuration of the solid-state imaging device 1. The solid-state imaging device 1 in Fig. 2 is a back-illuminated CMOS (Complementary Metal Oxide Semiconductor) image sensor. As shown in FIG. 2, the solid-state imaging device 1 includes a substrate 2, a pixel array section 3, a vertical drive circuit 4, a column signal processing circuit 5, a horizontal drive circuit 6, an output circuit 7, and a control circuit 8. The pixel array section 3 has a plurality of pixels 9 arranged in a matrix on the substrate 2. As shown in Figs. 3A and 3B, each of the pixels 9 has a photoelectric conversion section 24, a color filter 19 formed corresponding to the photoelectric conversion section 24, and a microlens 20. Four pixels 9 arranged in a 2 x 2 matrix form one pixel unit 10. The pixel units 10 also form one pixel unit group 11, each of which is made up of four pixel units 10 arranged in a 2 x 2 matrix. That is, the pixel array section 3 is configured with a plurality of pixel unit groups 11 arranged in a matrix. In the first embodiment, an example in which one pixel unit 10 is configured with pixels 9 arranged in a 2×2 matrix has been shown, but other configurations can also be adopted. For example, as shown in Fig. 4, one pixel unit 10 may be configured with pixels 9 arranged in an m×m matrix (m is a natural number of 2 or more). Fig. 4 illustrates an example in which m is 5 or more.

[0010] The vertical drive circuit 4 is configured, for example, with a shift register, selects a desired pixel drive wiring 12, supplies a pulse for driving the pixels 9 to the selected pixel drive wiring 12, and drives each pixel 9 row by row. That is, the vertical drive circuit 4 sequentially selects and scans each pixel 9 of the pixel array unit 3 row by row in the vertical direction, and supplies a pixel signal based on a signal charge generated in the photoelectric conversion unit 24 of each pixel 9 according to the amount of received light to the column signal processing circuit 5 via the vertical signal line 13.

[0011] The column signal processing circuit 5 is arranged, for example, for each column of pixels 9, and performs signal processing such as noise removal for each pixel column on signals output from one row of pixels 9. For example, the column signal processing circuit 5 performs signal processing such as CDS (Correlated Double Sampling) and AD (Analog Digital) conversion for removing fixed pattern noise specific to each pixel. The horizontal drive circuit 6 is, for example, constituted by a shift register, and sequentially outputs horizontal scanning pulses to the column signal processing circuits 5, selecting each of the column signal processing circuits 5 in turn, and outputting pixel signals (hereinafter also referred to as "pixel values") that have been subjected to signal processing from each of the column signal processing circuits 5 to the horizontal signal line 14.

[0012] The output circuit 7 performs signal processing on pixel signals (pixel values) sequentially supplied from each of the column signal processing circuits 5 through the horizontal signal line 14, and outputs the processed signals. As the signal processing, for example, buffering, black level adjustment, column variation correction, and various types of digital signal processing can be adopted. The control circuit 8 generates clock signals and control signals that serve as a reference for the operation of the vertical drive circuit 4, the column signal processing circuit 5, the horizontal drive circuit 6, etc., based on the vertical synchronization signal, the horizontal synchronization signal, and the master clock signal. Then, the control circuit 8 outputs the generated clock signals and control signals to the vertical drive circuit 4, the column signal processing circuit 5, the horizontal drive circuit 6, etc.

[0013] [1-2 Main components] Next, a detailed structure of the solid-state imaging device 1 in Fig. 1 will be described. Fig. 3A is a diagram showing a cross-sectional configuration of a pixel array section 3 of the solid-state imaging device 1. Fig. 3B is a diagram showing a minimum unit array of a color filter 19 when cut along line BB in Fig. 3A. In Figs. 3A and 3B, a back-illuminated CMOS image sensor is used as the solid-state imaging device 1.

[0014] As shown in FIG. 3A and FIG. 3B, the solid-state imaging device 1 of the first embodiment includes an absorption layer 18 formed by laminating a substrate 2, an insulating film 15, a light-shielding film 16, and a planarization film 17 in this order. A light-collecting layer 21 formed by laminating a color filter 19 and a microlens 20 (on-chip lens) in this order is formed on a surface of the absorption layer 18 on the insulating film 15 side (hereinafter also referred to as a "rear surface S1"). A wiring layer 22 and a support substrate 23 are laminated in this order on a surface of the absorption layer 18 on the substrate 2 side (hereinafter also referred to as a "front surface S2"). Since the rear surface S1 of the absorption layer 18 and the rear surface of the planarization film 17 are the same surface, the rear surface of the planarization film 17 will also be referred to as the "rear surface S1" in the following description. Since the front surface S2 of the absorption layer 18 and the front surface of the substrate 2 are the same surface, the front surface of the substrate 2 will also be referred to as the "front surface S2" in the following description.

[0015] The substrate 2 is configured by, for example, a semiconductor substrate made of silicon (Si), and forms the pixel array section 3 shown in Fig. 1. In the pixel array section 3, a plurality of photoelectric conversion sections 24 formed on the substrate 2 are arranged in a matrix. In the photoelectric conversion sections 24, signal charges according to the amount of incident light 113 are generated and accumulated. In addition, pixel separation sections 25 are arranged between adjacent photoelectric conversion sections 24 to prevent light that has passed through other photoelectric conversion sections 24 from entering. The insulating film 15 continuously covers the entire back surface S1 side (the entire light-receiving surface side) of the substrate 2. In addition, the light-shielding film 16 is formed in a lattice shape on a part of the back surface S3 side of the insulating film 15 (a part of the light-receiving surface side) so that the light-receiving surfaces of the multiple photoelectric conversion units 24 are open.

[0016] The color filters 19 are formed on the back surface S1 side (light receiving surface side) of the insulating film 15 in correspondence with each photoelectric conversion unit 24. That is, one color filter 19 is formed for one photoelectric conversion unit 24 (pixel 9). As a result, the color filters 19 are regularly arranged in a matrix to form a color filter array 26. Each of the color filters 19 transmits light of a specific wavelength (red light, green light, blue light, cyan light, magenta light, yellow light) of the incident light 113 and allows the transmitted light to enter the photoelectric conversion unit 24. The color filter 19 is an R filter 19 that transmits red light. R , G filter 19 that transmits green light G , B filter 19 that transmits blue light B , R filter 19 R C filter 19 transmits cyan light, which is complementary to the red light that is transmitted by the C , G Filter 19 G The M filter 19 transmits magenta light, which is complementary to the green light that is transmitted by the M , B filter 19 B Y filter 19 transmits yellow light, which is complementary to the blue light that is transmitted by Y In other words, the C filter 19 C As shown in Figure 5, in the spectral distribution diagram, R filter 19 R The transmittance of the R filter is smaller at wavelengths where the transmittance of the R filter is larger. R It is a color filter that has a high transmittance at wavelengths where the transmittance of the other wavelengths is low. M As shown in Figure 6, the G filter 19 G The transmittance of the G filter is smaller at wavelengths where the transmittance of G The Y filter 19 is a color filter that has a high transmittance at wavelengths where the transmittance of the Y filter 19 is low. Y As shown in Fig. 7, in the spectral distribution diagram, the B filter 19 B The transmittance of the B filter is small at wavelengths where the transmittance of the B The color filter has a high transmittance at a wavelength where the transmittance of the other wavelengths is low.

[0017] In FIG. 3A and FIG. 3B, the symbol R indicates the R filter 19. R Similarly, the symbol G indicates the G filter 19 G The symbol B indicates the B filter 19 B The symbol C indicates the C filter 19 C The symbol M indicates the M filter 19 M The symbol Y indicates the Y filter 19 Y In the following description, the R filter 19 R Pixel 9 including red pixel 9 R and G filter 19 G Pixel 9 including green pixel 9 G and B filter 19 B Pixel 9 including blue pixel 9 B and C filter 19 C Pixel 9 containing cyan pixel 9 C and M filter 19 M Pixel 9 containing magenta pixel 9 M and Y filter 19 Y Pixel 9 containing yellow pixel 9 Y This is expressed as:

[0018] In addition, color filter 19 (R filter 19 R , G Filter 19 G , B filter 19 B , C Filter 19 C , M Filter 19 M , Y filter 19 Y ) is configured such that an arrangement of color filters 19 arranged in a 4 × 4 matrix as shown in FIG. 3B is the minimum unit of an arrangement of color filters 19 (hereinafter also referred to as the "minimum unit array"), and as shown in FIG. 8, this minimum unit array is arranged in all pixel unit groups 11 of the pixel array section 3. As shown in FIG. 3B, the minimum unit array of the color filters 19 is an R filter 19 in the upper right pixel unit 10 (hereinafter, also referred to as the “first pixel unit 101”) of the four pixel units 10 constituting the pixel unit group 11. RA G filter 19 is disposed in the upper left pixel unit 10 (hereinafter also referred to as the “second pixel unit 102”) and the lower right pixel unit 10 (hereinafter also referred to as the “third pixel unit 103”). G The pixel unit 10 on the lower left side (hereinafter also referred to as the "fourth pixel unit 104") has a B filter 19. B Specifically, the R filters 19 of the upper right and lower left pixels 9 of the 2×2 pixels 9 constituting the first pixel unit 101 of the four-division Bayer array are R C Filter 19 C In addition, the G filters 19 of the upper right and lower left pixels 9 of the 2×2 pixels 9 constituting each of the second pixel unit 102 and the third pixel unit 103 are replaced with G M Filter 19 M In addition, the B filters 19 of the upper right and lower left pixels 9 of the 2×2 pixels 9 constituting the fourth pixel unit 104 are replaced with B Y filter 19 Y has been replaced by.

[0019] That is, in the minimum unit array of the color filters 19, of the pixel pairs (two pixels 9 forming one set) arranged at diagonal positions in the first pixel unit 101, the R filter 19 is disposed in one pixel pair. R and the other pixel pair includes a C filter 19 C Similarly, a G filter 19 is included in one of the pixel pairs arranged at diagonal positions in the second pixel unit 102. G and the other pixel pair includes M filter 19 M In addition, a G filter 19 is included in one of the pixel pairs arranged at diagonal positions in the third pixel unit 103. G and the other pixel pair includes M filter 19 M In addition, one of the pixel pairs arranged at diagonal positions in the fourth pixel unit 104 includes a B filter 19. B and the other pixel pair includes a Y filter 19 YContains: By arranging the color filters 19 in such a minimum unit array, the entire color filter 19 included in the first pixel unit 101 has an R filter 19 as shown in FIG. R and R filter 19 R C filter 19, which has a higher transmittance at wavelengths where the transmittance of C 6, the transmittance is almost flat for all light from the short wavelength side to the long wavelength side. G And G Filter 19 G M filter 19, which has a higher transmittance at wavelengths where the transmittance of M 7, the transmittance is almost flat for all light from the short wavelength side to the long wavelength side. B and B filter 19 B Y filter 19, which has a higher transmittance at wavelengths where the transmittance of Y Since the above is combined, the transmittance is almost flat for all light from the short wavelength side to the long wavelength side.

[0020] As shown in FIG. 4, when the pixel unit 10 is composed of pixels 9 arranged in an m×m matrix, where m is a natural number of 3 or more (FIG. 4 illustrates an example in which m is 5 or more), the pixels 9 arranged in a checkerboard pattern are used as an R filter 19. R , G Filter 19 G and B filter 19 B Specifically, in the first pixel unit 101, among the multiple pixels 9 arranged in an m×m matrix, some of the pixels 9 arranged in a checkerboard pattern are R filters 19. R The remaining pixels 9 are C filters 19 C Similarly, in the second pixel unit 102, among the multiple pixels 9 arranged in an m×m matrix, some of the pixels 9 arranged in a checkerboard pattern are G filters 19. GThe remaining pixels 9 are M filters 19 M In addition, in the third pixel unit 103, among the multiple pixels 9 arranged in an m×m matrix, some of the pixels 9 arranged in a checkerboard pattern include a G filter 19. G The remaining pixels 9 are M filters 19 M In addition, in the fourth pixel unit 104, among the multiple pixels 9 arranged in an m×m matrix, some of the pixels 9 arranged in a checkerboard pattern are B filters 19. B The remaining pixels 9 are Y filters 19 Y The configuration includes:

[0021] In this manner, in the solid-state imaging device 1 according to the first embodiment, the R filter 19 is used as the color filter 19. R , G Filter 19 G and B filter 19 B Plus C filter 19 C , M Filter 19 M , Y filter 19 Y Therefore, the R filter 19 R 9 red pixels including R , G Filter 19 G 9 green pixels including G , and B filter 19 B 9 blue pixels including B By using the pixel signals, an RGB image (RGB mosaic image 38 in FIG. 14) can be generated. C 9 cyan pixels including C , M Filter 19 M 9 magenta pixels including M , and Y filter 19 Y Yellow pixels including 9 YBy using these pixel signals, a CMY image (CMY mosaic image 39 in FIG. 15) can be generated. Furthermore, by adding up the pixel values ​​of each pixel 9 in the same pixel unit 10, luminance information (luminance image 41 in FIG. 16) can be generated. Therefore, an RGB image, a CMY image, and luminance information can be obtained in one imaging. The RGB image is weak when photographed in a dark place because the sensitivity of the color filter 19 is low, but the color reproducibility is good because the overlap of the spectral sensitivities of the color filters 19 is small. The CMY image is poor in color reproducibility because the overlap of the spectral sensitivities of the color filters 19 is large, but it is better at photographing in a dark place than the RGB image because the sensitivity of the color filter 19 is low. In addition, by combining the RGB image and the CMY image to use six colors, the color reproducibility can be further improved.

[0022] Also, as in the solid-state imaging device 1 according to the first embodiment, an R filter 19 is provided in one of the pixel pairs arranged at diagonal positions in the first pixel unit 101. R and the other pixel pair includes a C filter 19 C In the case where the pixel 31 has only red color information in the mosaic image 30 corresponding to the arrangement of the color filters 19, as shown in FIG. R and the image pixel 31 having only the red color information. R 37 image pixels generated by binning to R Since the center of gravity of the red image pixel 31 is the same as that of the red image pixel 31 in the binning process, R 14 and 15, the image pixel 31 having only green color information does not need to be corrected. G , image pixel 31 having only blue color information B , image pixel 31 having only cyan color information C , image pixel 31 having only magenta color information M , and image pixel 31 having only yellow color information Y 14 and 15, the symbol R denotes an image pixel 31 having only red color information (hereinafter, also referred to as a "red image pixel") RSimilarly, hereinafter, the symbol G denotes an image pixel having only green color information (hereinafter, also referred to as a "green image pixel") 31 G and B denotes an image pixel having only blue color information (hereinafter also referred to as a "blue image pixel") 31 B and symbol C denotes an image pixel having only cyan color information (hereinafter also referred to as a "cyan image pixel") 31 C M denotes an image pixel having only magenta color information (hereinafter also referred to as a "magenta image pixel") 31 M and the symbol Y denotes an image pixel having only yellow color information (hereinafter also referred to as a "yellow image pixel") 31 Y Shows.

[0023] In the first embodiment, an example is shown in which the pixel units 10 on the upper left and lower right sides are the second pixel unit 102 and the third pixel unit 103, but other configurations can also be adopted. For example, a configuration in which the second pixel unit 102 and the third pixel unit 103 are arranged in the pixel units 10 on the upper right and lower left sides, in the pixel units 10 on the upper left and lower left sides, or in the pixel units 10 on the upper right and lower right sides can also be adopted. In addition, for example, a configuration in which the first pixel unit 101 is arranged on the lower side and the fourth pixel unit 104 is arranged on the upper side can also be adopted. That is, in each of the pixel unit groups 11, the R filter 19 is provided in the first pixel unit 101 of the four pixel units 10 constituting the pixel unit group 11. R and C Filter 19 C and a G filter 19 in each of the second pixel unit 102 and the third pixel unit 103. G and M Filter 19 M as a color filter 19, and the fourth pixel unit 104 includes a B filter 19 B and Y filter 19 Y As the color filter 19,

[0024] The microlenses 20 are formed on the back surface S4 side (light receiving surface side) of the color filter 19 in correspondence with each photoelectric conversion unit 24. That is, one microlens 20 is formed for one photoelectric conversion unit 24 (pixel 9). As a result, the microlenses 20 are regularly arranged in a matrix to form a microlens array 27. Each of the microlenses 20 is configured to collect image light (incident light 113) from an object and guide the collected incident light 113 to the vicinity of the back surface (light receiving surface) of the photoelectric conversion unit 24 via the color filter 19. In the first embodiment, an example in which one microlens 20 is formed for one photoelectric conversion unit 24 has been described, but other configurations may be adopted. For example, G When using the green pixel as a phase difference pixel, as shown in FIG. G The two green pixels 9 G A single microlens 20 may be formed for each green pixel 9 (phase difference pixel). According to this configuration, two green pixels 9 share one microlens 20. G It is possible to detect a phase difference between captured images (phase difference pixels).

[0025] The wiring layer 22 is formed on the surface S2 side of the substrate 2, and includes an interlayer insulating film 28 and wiring 29 stacked in multiple layers with the interlayer insulating film 28 interposed therebetween. The wiring layer 22 drives pixel transistors that configure each pixel 9 via the multiple layers of wiring 29. The support substrate 23 is formed on the surface of the wiring layer 22 opposite to the surface facing the substrate 2. The support substrate 23 is a substrate for ensuring the strength of the substrate 2 during the manufacturing stage of the solid-state imaging device 1. The support substrate 23 may be made of, for example, silicon (Si).

[0026] [1-3 Image generation processing] Next, the image generation processing executed by the signal processing circuit 105 (mosaic image generation unit 106, white balance adjustment unit 107, mode determination unit 108, re-mosaic processing unit 109, binning processing unit 110, luminance value calculation unit 111, and luminance synthesis unit 112) will be described. When the image generation process is started, first, in step S101, the mosaic image generation unit 106 generates a mosaic image of the red pixel 9 R , green pixel 9 G , blue pixel 9 B , cyan pixel 9 C , magenta pixel 9 M and yellow pixel 9 Y Based on the pixel signals (pixel values) output from the pixel filter 19, a mosaic image 30 corresponding to the arrangement of the color filters 19 is generated as shown in FIG.

[0027] Next, the process proceeds to step S102, where the white balance adjustment unit 107 estimates the color temperature of the light source based on the pixel values ​​of each image pixel 31 of the mosaic image 30 generated in step S101, and adjusts the white balance of the mosaic image 30 based on the estimated color temperature. Next, the process proceeds to step S103, where the mode determination unit 108 determines whether the shooting mode selected by the user of the electronic device 100 is the high resolution mode or the high S / N ratio mode. If it is determined that the shooting mode is the high resolution mode, the process proceeds to step S104. On the other hand, if it is determined that the shooting mode is the high S / N ratio mode, the process proceeds to step S105.

[0028] In step S104, the re-mosaic processing unit 109 performs a re-mosaic process on the mosaic image 30 whose white balance has been corrected in step S102. In the re-mosaic process, as shown in FIG. 12, the red, green, and blue image pixels 31 constituting the mosaic image 30 are R , 31 G , 31 B The pixel array is converted to a Bayer array, and the red, green and blue image pixels 32 of the Bayer array are R , 32 G , 32 BAs shown in FIG. 13, a RGB mosaic image 33 consisting of cyan, magenta and yellow image pixels 31 is generated. C , 31 M , 31 Y The pixel array is converted into a predetermined array, and the cyan, magenta and yellow image pixels 34 in the predetermined array are C , 34 M , 34 Y In FIG. 13, a CMY mosaic image 35 consisting of red, green and blue image pixels 32 in a Bayer array is generated as the predetermined array. R , 32 G , 32 B 34 C , 34 M , 34 Y 12 shows an enlarged view of a portion of the mosaic image 30 and the RGB mosaic image 33. Also, FIG. 13 shows an enlarged view of a portion of the mosaic image 30 and the CMY mosaic image 35.

[0029] On the other hand, in step S105, the binning processing unit 110 performs binning processing on the mosaic image 30 whose white balance has been corrected in step S102, and then the process proceeds to step S106. In the binning processing, as shown in FIG. 14, the red image pixels 31 in each pixel group 361 corresponding to the first pixel unit 101 in the mosaic image 30 are binned. R Add the pixel values ​​of 37 to create one red image pixel. R Similarly, in each of the pixel groups 362 corresponding to the second pixel unit 102, the pixel value of the green image pixel 31 is set to 1. G Add the pixel values ​​of 37 to create one green image pixel. G In addition, in each of the pixel groups 363 corresponding to the third pixel unit 103, the green image pixel 31 G Add the pixel values ​​of 37 to create one green image pixel. G In addition, in each of the pixel groups 364 corresponding to the fourth pixel unit 104, the pixel value of the blue image pixel 31 B Add the pixel values ​​of 37 to create one blue image pixel.B This sets the pixel value of the red image pixel 37 R (Image pixel 37 has only red color information.) R ), green image pixel 37 G (Image pixel 37 has only green color information.) G ), blue image pixel 37 G (Image pixel 37 has only blue color information.) B ) is generated. The number of pixels in the RGB mosaic image 38 is ¼ of the number of pixels in the mosaic image 30. FIG. 14 shows an enlarged portion of the mosaic image 30 and the RGB mosaic image 38.

[0030] In the binning process, as shown in FIG. 15, in the mosaic image 30 in which the white balance has been corrected in step S102, the cyan image pixels 31 in the pixel group 361 corresponding to the first pixel unit 101 are C The pixel values ​​of the cyan image pixel 38 are added together to produce C Similarly, in each of the pixel groups 362 corresponding to the second pixel unit 102, the magenta image pixel 31 M The pixel values ​​of magenta image pixel 38 are added together to produce M In addition, in each of the pixel groups 363 corresponding to the third pixel unit 103, the magenta image pixel 31 M The pixel values ​​of magenta image pixel 38 are added together to produce M In addition, in each of the pixel groups 364 corresponding to the fourth pixel unit 104, the yellow image pixel 31 Y Add the pixel values ​​of 38 to create one yellow image pixel. Y This results in a pixel value of cyan image pixel 38 C (Image pixel 38 has only cyan color information) C ), magenta image pixel 38 M (Image pixel 38 has only magenta color information) M ), yellow image pixel 38 Y (Image pixel 38 has only yellow color information.) Y) is generated. The number of pixels of the CMY mosaic image 39 is 1 / 4 of the number of pixels of the mosaic image 30. FIG. 15 shows an enlarged portion of the mosaic image 30 and the CMY mosaic image 39.

[0031] In the binning process, the luminance value calculation unit 111 calculates the image pixels 31 having red and cyan color information in each of the pixel groups 361 corresponding to the first pixel unit 101 in the mosaic image 30 whose white balance has been corrected in step S102, as shown in FIG. R , 31 C The pixel values ​​of the image pixels 31 having the color information of green and magenta in the pixel group 362 corresponding to the second pixel unit 102 are added together to calculate the luminance value of one image pixel 40. G , 31 M The pixel values ​​of the image pixels 31 having the color information of green and magenta in the pixel group 363 corresponding to the third pixel unit 103 are added together to calculate the luminance value of one image pixel 40. G , 31 M The pixel values ​​of the image pixels 31 having blue and yellow color information in each of the pixel groups 364 corresponding to the fourth pixel unit 104 are added together to calculate the luminance value of one image pixel 40. B , 31 M The pixel values ​​of these are added together to calculate the luminance value of one image pixel 40. This generates a luminance image 41 that represents only the luminance values ​​of each image pixel 40. The number of pixels in the luminance image 41 is ¼ the number of pixels in the mosaic image 30. FIG. 16 shows an enlarged portion of the mosaic image 30 and the luminance image 41.

[0032] In addition, the pixel values ​​used to calculate the luminance value may be calculated by multiplying the pixel values ​​of the mosaic image 30 whose white balance has been corrected in step S102 by a correction coefficient. 11 , a 21 , a 12 , a 22 , a 13 , a23 Based on this, the luminance values ​​Br1, Br2, and Br3 of each image pixel 40 in the luminance image 41 are calculated according to the following formula (1).

number

[0033] Next, the process proceeds to step S106, where the luminance synthesis unit 112 performs a luminance synthesis process on the RGB mosaic image 38 or the CMY mosaic image 39 generated in step S105. In the luminance synthesis process, first, it is determined whether the subject is bright based on the pixel value of each image pixel 31 of the mosaic image 30 generated in step S101. If it is determined that the subject is bright, as shown in FIG. 17, the luminance value of the luminance image 41 is synthesized with the RGB mosaic image 38 to generate a synthesized image 42. By synthesizing the luminance image 41 with the RGB mosaic image 38, it is possible to achieve both high color reproducibility and high resolution. Also, unlike a compound eye camera having, for example, a solid-state imaging device for obtaining the RGB mosaic image 38 and a solid-state imaging device for obtaining the luminance image 41 separately, no parallax correction is required.

[0034] In the synthesis of the RGB mosaic image 38 and the luminance image 41, first, the pixel values ​​R, G, and B of each pixel 37 of the RGB mosaic image 38 are calculated based on a predetermined coefficient b 11 , b 21 , b 31 , b 12 , b 22 , b 32 , a 13 , a 23 , a 33 Based on this, the Y1 signal, Cb signal, and Cr signal of each pixel are calculated according to the following formula (2). Next, based on the calculated Y1 signal and the luminance value (Y2 signal) of each image pixel 40 of the luminance image 41, the luminance signal Y of each pixel 43 of the composite image 42 is calculated according to the following formula (3). That is, the Y1 signal is corrected with the Y2 signal to obtain the luminance signal Y. At this time, the Cb signal and Cr signal calculated according to the above formula (2) are used as color information of each pixel 43.

number

[0035] On the other hand, if it is determined that the subject is dark, the CMY mosaic image 39 is converted into an RGB mosaic image 44 using a color conversion matrix, as shown in Fig. 18. Next, the converted RGB mosaic image 44 is composited with a luminance image 41 to generate a composite image 45. By compositely combining the luminance image 41 with the RGB mosaic image 44, it is possible to achieve both a high S / N ratio and high resolution. Also, unlike the compound eye camera described above, for example, no parallax correction is required.

[0036] As described above, in the solid-state imaging device 1 according to the first embodiment of the present disclosure, each of the pixel unit groups 11 includes an R filter 19 in the first pixel unit 101 of the four pixel units 10 constituting the pixel unit group 11. R and C Filter 19 C and a G filter 19 in each of the second and third pixel units 102 and 103. G and M Filter 19 M and the fourth pixel unit 104 includes a B filter 19 as a color filter 19. B and Y Filter 19 Y Therefore, the red pixel 9 R , green pixel 9 G and blue pixel 9 B By using the pixel signals, an RGB image can be generated. C , magenta pixel 9 M and yellow pixel 9 Y By using these pixel signals, a CMY image can be generated. Furthermore, luminance information can be generated by adding up the pixel values ​​of the pixels 9 in the same pixel unit 10. Therefore, it is possible to provide a solid-state imaging device 1 that can acquire an RGB image, a CMY image, and luminance information in a single imaging operation.

[0037] 2. Second embodiment: Electronic device [2-1 Configuration of main parts] Next, an electronic device 100 according to a second embodiment of the present disclosure will be described. Fig. 19 is a diagram showing an overall configuration of the electronic device 100 according to the second embodiment. Fig. 20 is a flowchart showing the image generation process of the first embodiment. In Figs. 19 and 20, parts corresponding to Figs. 1 and 10 are given the same reference numerals and duplicated explanations will be omitted.

[0038] The electronic device 100 according to the second embodiment differs from the first embodiment in that, as shown in FIG. 19, the signal processing circuit 105 includes an HDR image generation unit 114, and, as shown in FIG. 20, the image generation process has step S201 instead of step S106 in FIG. 10. In step S201, the HDR image generating unit 114 performs HDR image generation processing based on the RGB mosaic image 38 and the CMY mosaic image 39 (see Figs. 14 and 15) generated in step S105. In the HDR image generation processing, a color conversion matrix is ​​used to convert the CMY mosaic image 39 into an RGB mosaic image 46 as shown in Figs. 21 and 22. Then, the converted RGB mosaic image 46 and the RGB mosaic image 38 generated in step S105 are synthesized to generate an HDR image 47. As a method of synthesizing the RGB mosaic image 46 obtained from the CMY mosaic image 39 and the RGB mosaic image 38 generated in step S105, for example, a method of adding the pixel value of the RGB mosaic image 46 and the pixel value of the RGB mosaic image 38 for each image pixel 48 can be adopted.

[0039] As described above, the electronic device 100 according to the second embodiment of the present disclosure converts the CMY mosaic image 39 into the RGB mosaic image 46, and synthesizes the converted RGB mosaic image 46 with the RGB mosaic image 38 generated by the binning processing unit 110 to generate an HDR image 47 (high dynamic range image). Therefore, in addition to the RGB image, the CMY image, and luminance information, the HDR image 47 can be generated with a single shooting operation.

[0040] [2-2 Modified Example] In the electronic device 100 according to the first and second embodiments of the present disclosure, the binning process is digitally performed by the signal processing circuit 105 outside the solid-state imaging device 1, but other configurations can be adopted. For example, a configuration in which the binning process is performed analogically when reading out pixel signals from the pixels 9 of the solid-state imaging device 1 may be adopted. Specifically, as shown in FIG. 23, a configuration in which, among the signal lines 50 connected to the floating diffusions 49 of the pixels 9 constituting the pixel unit 10, the signal lines 50 connected to the floating diffusions 49 of the pixels 9 of the same color are electrically connected, and the pixel signals of the pixels 9 of the same color are added and output to the CDS circuit 51 can be adopted. This makes it possible to obtain the addition result of the pixel values ​​of the pixels 9 of the same color.

[0041] 24, a configuration can be adopted in which signal lines 50 connected to the floating diffusions 49 of the pixels 9 constituting the pixel unit 10 are electrically connected, and pixel signals of the multiple pixels 9 are added and output to a CDS circuit 51. This makes it possible to obtain the luminance value (luminance information) of each image pixel 40 of the luminance image 41 shown in FIG.

[0042] 3. Examples of applications to moving objects The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure may be realized as a device mounted on any type of moving body such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility device, an airplane, a drone, a ship, a robot, etc.

[0043] FIG. 25 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a moving object control system to which the technology according to the present disclosure can be applied.

[0044] The vehicle control system 12000 includes a plurality of electronic control units connected via a communication network 12001. In the example shown in Fig. 25, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside-vehicle information detection unit 12030, an inside-vehicle information detection unit 12040, and an integrated control unit 12050. Also, as functional configurations of the integrated control unit 12050, a microcomputer 12051, an audio / video output unit 12052, and an in-vehicle network I / F (interface) 12053 are illustrated.

[0045] The drivetrain control unit 12010 controls the operation of devices related to the drivetrain of the vehicle in accordance with various programs. For example, the drivetrain control unit 12010 functions as a control device for a drive force generating device for generating a drive force of the vehicle, such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating a braking force of the vehicle.

[0046] The body system control unit 12020 controls the operation of various devices mounted on the vehicle body according to various programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as head lamps, back lamps, brake lamps, turn signals, and fog lamps. In this case, radio waves or signals of various switches transmitted from a portable device that replaces a key may be input to the body system control unit 12020. The body system control unit 12020 receives the input of these radio waves or signals and controls the door lock device, power window device, lamps, and the like of the vehicle.

[0047] The outside-vehicle information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000. For example, an imaging unit 12031 is connected to the outside-vehicle information detection unit 12030. The outside-vehicle information detection unit 12030 causes the imaging unit 12031 to capture an image outside the vehicle and receives the captured image. The outside-vehicle information detection unit 12030 may perform object detection processing or distance detection processing for a person, a car, an obstacle, a sign, or characters on a road surface, based on the received image.

[0048] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal according to the amount of light received. The imaging unit 12031 can output the electrical signal as an image, or as distance measurement information. The light received by the imaging unit 12031 may be visible light or invisible light such as infrared light.

[0049] The in-vehicle information detection unit 12040 detects information inside the vehicle. For example, a driver state detection unit 12041 that detects the state of the driver is connected to the in-vehicle information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera that captures an image of the driver, and the in-vehicle information detection unit 12040 may calculate the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 12041, or may determine whether the driver is dozing.

[0050] The microcomputer 12051 can calculate control target values ​​for the driving force generating device, the steering mechanism, or the braking device based on the information inside and outside the vehicle acquired by the outside-of-vehicle information detection unit 12030 or the inside-vehicle information detection unit 12040, and output a control command to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control aimed at realizing the functions of an ADAS (Advanced Driver Assistance System), including vehicle collision avoidance or impact mitigation, following driving based on the inter-vehicle distance, vehicle speed maintenance driving, vehicle collision warning, vehicle lane departure warning, etc.

[0051] In addition, the microcomputer 12051 can perform cooperative control for the purpose of autonomous driving, which allows the vehicle to travel autonomously without relying on the driver's operation, by controlling the driving force generating device, steering mechanism, braking device, etc. based on information about the surroundings of the vehicle acquired by the outside vehicle information detection unit 12030 or the inside vehicle information detection unit 12040.

[0052] Furthermore, the microcomputer 12051 can output a control command to the body system control unit 12020 based on the information outside the vehicle acquired by the outside-vehicle information detection unit 12030. For example, the microcomputer 12051 can control the headlamps according to the position of a preceding vehicle or an oncoming vehicle detected by the outside-vehicle information detection unit 12030, and perform cooperative control for the purpose of preventing glare, such as switching from high beams to low beams.

[0053] The audio / video output unit 12052 transmits at least one output signal of audio and video to an output device capable of visually or audibly notifying information to passengers in the vehicle or the outside of the vehicle. In the example of Fig. 25, an audio speaker 12061, a display unit 12062, and an instrument panel 12063 are exemplified as the output device. The display unit 12062 may include at least one of an on-board display and a head-up display, for example.

[0054] FIG. 26 is a diagram showing an example of the installation position of the imaging unit 12031.

[0055] In FIG. 26, a vehicle 12100 has imaging units 12101, 12102, 12103, 12104, and 12105 as an imaging unit 12031.

[0056] The imaging units 12101, 12102, 12103, 12104, and 12105 are provided at positions such as the front nose, side mirrors, rear bumper, back door, and the upper part of the windshield inside the vehicle cabin of the vehicle 12100. The imaging unit 12101 provided at the front nose and the imaging unit 12105 provided at the upper part of the windshield inside the vehicle cabin mainly acquire images of the front of the vehicle 12100. The imaging units 12102 and 12103 provided at the side mirrors mainly acquire images of the sides of the vehicle 12100. The imaging unit 12104 provided at the rear bumper or back door mainly acquires images of the rear of the vehicle 12100. The images of the front acquired by the imaging units 12101 and 12105 are mainly used to detect a preceding vehicle, a pedestrian, an obstacle, a traffic light, a traffic sign, a lane, or the like.

[0057] 26 shows an example of the imaging ranges of the imaging units 12101 to 12104. Imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose, imaging ranges 12112 and 12113 indicate the imaging ranges of the imaging units 12102 and 12103 provided on the side mirrors, respectively, and imaging range 12114 indicates the imaging range of the imaging unit 12104 provided on the rear bumper or back door. For example, an overhead image of the vehicle 12100 viewed from above is obtained by superimposing the image data captured by the imaging units 12101 to 12104.

[0058] At least one of the imaging units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera made up of a plurality of imaging elements, or may be an imaging element having pixels for detecting a phase difference.

[0059] For example, the microcomputer 12051 can extract, as a preceding vehicle, a three-dimensional object that is the closest three-dimensional object on the travel path of the vehicle 12100 and travels at a predetermined speed (for example, 0 km / h or more) in approximately the same direction as the vehicle 12100, by calculating the distance to each three-dimensional object in the imaging ranges 12111 to 12114 and the change over time of this distance (relative speed with respect to the vehicle 12100) based on the distance information obtained from the imaging units 12101 to 12104. Furthermore, the microcomputer 12051 can set a vehicle distance to be secured in advance in front of the preceding vehicle, and perform automatic brake control (including follow-up stop control) and automatic acceleration control (including follow-up start control). In this way, cooperative control can be performed for the purpose of automatic driving that travels autonomously without relying on the driver's operation.

[0060] For example, the microcomputer 12051 classifies and extracts three-dimensional object data on three-dimensional objects, such as two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects, based on the distance information obtained from the imaging units 12101 to 12104, and can use the data for automatic obstacle avoidance. For example, the microcomputer 12051 distinguishes obstacles around the vehicle 12100 into obstacles that are visible to the driver of the vehicle 12100 and obstacles that are difficult to see. Then, the microcomputer 12051 determines a collision risk indicating the degree of danger of collision with each obstacle, and when the collision risk is equal to or exceeds a set value and there is a possibility of collision, the microcomputer 12051 can provide driving assistance for collision avoidance by outputting an alarm to the driver via the audio speaker 12061 or the display unit 12062, or by performing forced deceleration or avoidance steering via the drive system control unit 12010.

[0061] At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether or not a pedestrian is present in the captured images of the imaging units 12101 to 12104. The recognition of such a pedestrian is performed, for example, by a procedure of extracting feature points in the captured images of the imaging units 12101 to 12104 as infrared cameras, and a procedure of performing pattern matching processing on a series of feature points that indicate the contour of an object to determine whether or not the object is a pedestrian. When the microcomputer 12051 determines that a pedestrian is present in the captured images of the imaging units 12101 to 12104 and recognizes the pedestrian, the audio / image output unit 12052 controls the display unit 12062 to superimpose a rectangular contour line for emphasis on the recognized pedestrian. The audio / image output unit 12052 may also control the display unit 12062 to display an icon or the like indicating a pedestrian at a desired position.

[0062] An example of a vehicle control system to which the technology according to the present disclosure can be applied has been described above. The technology according to the present disclosure can be applied to the imaging unit 12031 among the configurations described above. Specifically, the solid-state imaging devices 101, 1 in FIG. 1 and FIG. 2 and the signal processing circuit 105 in FIG. 1 can be applied to the imaging unit 12031. By applying the technology according to the present disclosure to the imaging unit 12031, a captured image that is easier to see can be obtained, thereby reducing the driver's fatigue.

[0063] 4. Application example to endoscopic surgery system The technology according to the present disclosure (the present technology) may be applied to, for example, an endoscopic surgery system.

[0064] FIG. 27 is a diagram showing an example of a schematic configuration of an endoscopic surgery system to which the technology according to the present disclosure (the present technology) can be applied.

[0065] 27 shows a state in which an operator (doctor) 11131 is performing surgery on a patient 11132 on a patient bed 11133 using an endoscopic surgery system 11000. As shown in the figure, the endoscopic surgery system 11000 is composed of an endoscope 11100, other surgical tools 11110 such as an insufflation tube 11111 and an energy treatment tool 11112, a support arm device 11120 that supports the endoscope 11100, and a cart 11200 on which various devices for endoscopic surgery are mounted.

[0066] The endoscope 11100 is composed of a lens barrel 11101, a region of a predetermined length from the tip of which is inserted into a body cavity of a patient 11132, and a camera head 11102 connected to the base end of the lens barrel 11101. In the illustrated example, the endoscope 11100 is configured as a so-called rigid scope having a rigid lens barrel 11101, but the endoscope 11100 may be configured as a so-called flexible scope having a flexible lens barrel.

[0067] An opening into which an objective lens is fitted is provided at the tip of the lens barrel 11101. A light source device 11203 is connected to the endoscope 11100, and light generated by the light source device 11203 is guided to the tip of the lens barrel by a light guide extending inside the lens barrel 11101, and is irradiated via the objective lens toward an observation target in the body cavity of the patient 11132. The endoscope 11100 may be a direct-viewing endoscope, an oblique-viewing endoscope, or a side-viewing endoscope.

[0068] An optical system and an image sensor are provided inside the camera head 11102, and reflected light (observation light) from an observation target is collected on the image sensor by the optical system. The observation light is photoelectrically converted by the image sensor to generate an electrical signal corresponding to the observation light, i.e., an image signal corresponding to an observation image. The image signal is transmitted to a camera control unit (CCU) 11201 as RAW data.

[0069] The CCU 11201 is configured with a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc., and performs overall control of the operations of the endoscope 11100 and the display device 11202. Furthermore, the CCU 11201 receives an image signal from the camera head 11102, and performs various types of image processing on the image signal, such as development processing (demosaic processing), for displaying an image based on the image signal.

[0070] Under the control of the CCU 11201, the display device 11202 displays an image based on an image signal that has been subjected to image processing by the CCU 11201.

[0071] The light source device 11203 is composed of a light source such as an LED (Light Emitting Diode), and supplies the endoscope 11100 with irradiation light when photographing an operation site or the like.

[0072] The input device 11204 is an input interface for the endoscopic surgery system 11000. A user can input various information and instructions to the endoscopic surgery system 11000 via the input device 11204. For example, the user inputs an instruction to change the imaging conditions (type of irradiated light, magnification, focal length, etc.) of the endoscope 11100.

[0073] The treatment tool control device 11205 controls the driving of the energy treatment tool 11112 for cauterizing tissue, incising, sealing blood vessels, etc. The insufflation device 11206 sends gas into the body cavity of the patient 11132 via the insufflation tube 11111 to inflate the body cavity for the purpose of securing the field of view of the endoscope 11100 and securing the working space of the surgeon. The recorder 11207 is a device capable of recording various information related to the surgery. The printer 11208 is a device capable of printing various information related to the surgery in various formats such as text, image, or graph.

[0074] The light source device 11203 that supplies irradiation light to the endoscope 11100 when photographing the surgical site can be composed of a white light source composed of, for example, an LED, a laser light source, or a combination of these. When the white light source is composed of a combination of RGB laser light sources, the output intensity and output timing of each color (each wavelength) can be controlled with high precision, so that the white balance of the captured image can be adjusted in the light source device 11203. In this case, it is also possible to capture images corresponding to each of the RGB colors in a time-division manner by irradiating the observation object with laser light from each of the RGB laser light sources in a time-division manner and controlling the driving of the image sensor of the camera head 11102 in synchronization with the irradiation timing. According to this method, a color image can be obtained without providing a color filter to the image sensor.

[0075] The light source device 11203 may be controlled to change the intensity of the light it outputs at predetermined time intervals. The driving of the image sensor of the camera head 11102 may be controlled in synchronization with the timing of the change in the light intensity to obtain images in a time-division manner, and the images may be synthesized to generate an image with a high dynamic range that is free of so-called blackout and whiteout.

[0076] The light source device 11203 may be configured to supply light of a predetermined wavelength band corresponding to the special light observation. In the special light observation, for example, by utilizing the wavelength dependency of light absorption in body tissue, a narrow band light is irradiated compared to the irradiated light (i.e., white light) during normal observation, and a predetermined tissue such as blood vessels on the mucous membrane surface is photographed with high contrast, so-called narrow band imaging is performed. Alternatively, in the special light observation, a fluorescent observation may be performed in which an image is obtained by fluorescence generated by irradiating an excitation light. In the fluorescent observation, it is possible to irradiate an excitation light to a body tissue and observe the fluorescence from the body tissue (autofluorescence observation), or to locally inject a reagent such as indocyanine green (ICG) into the body tissue and irradiate the body tissue with excitation light corresponding to the fluorescent wavelength of the reagent to obtain a fluorescent image. The light source device 11203 may be configured to supply narrow band light and / or excitation light corresponding to such special light observation.

[0077] FIG. 28 is a block diagram showing an example of the functional configuration of the camera head 11102 and the CCU 11201 shown in FIG.

[0078] The camera head 11102 has a lens unit 11401, an imaging unit 11402, a drive unit 11403, a communication unit 11404, and a camera head control unit 11405. The CCU 11201 has a communication unit 11411, an image processing unit 11412, and a control unit 11413. The camera head 11102 and the CCU 11201 are connected to each other by a transmission cable 11400 so as to be able to communicate with each other.

[0079] The lens unit 11401 is an optical system provided at the connection portion with the lens barrel 11101. Observation light taken in from the tip of the lens barrel 11101 is guided to the camera head 11102 and enters the lens unit 11401. The lens unit 11401 is configured by combining a plurality of lenses including a zoom lens and a focus lens.

[0080] The imaging unit 11402 is composed of an imaging element. The imaging element constituting the imaging unit 11402 may be one (so-called single-plate type) or multiple (so-called multi-plate type). When the imaging unit 11402 is composed of a multi-plate type, for example, each imaging element may generate an image signal corresponding to each of RGB, and a color image may be obtained by combining the image signals. Alternatively, the imaging unit 11402 may be configured to have a pair of imaging elements for acquiring image signals for the right eye and the left eye corresponding to 3D (Dimensional) display. By performing 3D display, the surgeon 11131 can more accurately grasp the depth of the biological tissue in the surgical site. Note that when the imaging unit 11402 is composed of a multi-plate type, multiple lens units 11401 may be provided corresponding to each imaging element.

[0081] Furthermore, the imaging unit 11402 does not necessarily have to be provided in the camera head 11102. For example, the imaging unit 11402 may be provided inside the lens barrel 11101, immediately behind the objective lens.

[0082] The driving unit 11403 is configured by an actuator, and moves the zoom lens and focus lens of the lens unit 11401 by a predetermined distance along the optical axis under the control of the camera head control unit 11405. This allows the magnification and focus of the image captured by the imaging unit 11402 to be appropriately adjusted.

[0083] The communication unit 11404 is configured by a communication device for transmitting and receiving various information to and from the CCU 11201. The communication unit 11404 transmits the image signal obtained from the imaging unit 11402 to the CCU 11201 via the transmission cable 11400 as RAW data.

[0084] Furthermore, the communication unit 11404 receives a control signal for controlling the driving of the camera head 11102 from the CCU 11201, and supplies the control signal to the camera head control unit 11405. The control signal includes information on the imaging conditions, such as information specifying the frame rate of the captured image, information specifying the exposure value at the time of capturing the image, and / or information specifying the magnification and focus of the captured image.

[0085] The image capturing conditions such as the frame rate, exposure value, magnification, and focus may be appropriately specified by a user, or may be automatically set by the control unit 11413 of the CCU 11201 based on an acquired image signal. In the latter case, the endoscope 11100 is equipped with a so-called AE (Auto Exposure) function, AF (Auto Focus) function, and AWB (Auto White Balance) function.

[0086] The camera head control unit 11405 controls the driving of the camera head 11102 based on a control signal received from the CCU 11201 via the communication unit 11404.

[0087] The communication unit 11411 is configured with a communication device for transmitting and receiving various information to and from the camera head 11102. The communication unit 11411 receives an image signal transmitted from the camera head 11102 via the transmission cable 11400.

[0088] Furthermore, the communication unit 11411 transmits, to the camera head 11102, a control signal for controlling the driving of the camera head 11102. The image signal and the control signal can be transmitted by electrical communication, optical communication, or the like.

[0089] The image processing unit 11412 performs various types of image processing on the image signal, which is RAW data sent from the camera head 11102 .

[0090] The control unit 11413 performs various controls related to imaging of the surgical site etc. by the endoscope 11100 and display of the captured image obtained by imaging the surgical site etc. For example, the control unit 11413 generates a control signal for controlling the driving of the camera head 11102.

[0091] Further, the control unit 11413 causes the display device 11202 to display the captured image showing the surgical site, etc., based on the image signal that has been image-processed by the image processing unit 11412. At this time, the control unit 11413 may recognize various objects in the captured image using various image recognition techniques. For example, the control unit 11413 can recognize surgical tools such as forceps, specific living body parts, bleeding, mist when the energy treatment tool 11112 is used, etc., by detecting the shape and color of the edge of an object included in the captured image. When the control unit 11413 causes the display device 11202 to display the captured image, it may use the recognition result to superimpose various types of surgery support information on the image of the surgical site. By superimposing and presenting the surgery support information to the surgeon 11131, the burden on the surgeon 11131 can be reduced and the surgeon 11131 can proceed with the surgery reliably.

[0092] The transmission cable 11400 connecting the camera head 11102 and the CCU 11201 is an electrical signal cable for communication of electrical signals, an optical fiber for optical communication, or a composite cable of these.

[0093] Here, in the illustrated example, communication is performed wired using a transmission cable 11400, but communication between the camera head 11102 and the CCU 11201 may also be performed wirelessly.

[0094] An example of an endoscopic surgery system to which the technology according to the present disclosure can be applied has been described above. The technology according to the present disclosure can be applied to the imaging unit 11402 of the camera head 11102, the image processing unit 11412 of the CCU 11201, and the like, among the configurations described above. Specifically, the solid-state imaging devices 101 and 1 in FIG. 1 and FIG. 2 can be applied to the imaging unit 10402, and the signal processing circuit 105 in FIG. 1 can be applied to the image processing unit 11412. By applying the technology according to the present disclosure to the imaging unit 10402 and the image processing unit 11412, a clearer image of the surgical site can be obtained, so that the surgeon can reliably confirm the surgical site.

[0095] Although an endoscopic surgery system has been described here as an example, the technology disclosed herein may also be applied to other systems, such as a microsurgery system.

[0096] The present technology can be configured as follows. (1) a pixel array section in which a plurality of pixel unit groups are arranged in a 2×2 matrix, the pixel unit being formed in an m×m (m is a natural number of 2 or more) matrix of pixel units, each pixel unit having a photoelectric conversion section and a color filter formed corresponding to the photoelectric conversion section; Each of the pixel unit groups includes, as the color filters, an R filter that transmits red light and a C filter that transmits cyan light that is complementary to the red light in a first pixel unit of the four pixel units constituting the pixel unit group, a G filter that transmits green light and an M filter that transmits magenta light that is complementary to the green light in each of the second and third pixel units, and a B filter that transmits blue light and a Y filter that transmits yellow light that is complementary to the blue light in each of the fourth pixel units. Solid-state imaging device. (2) m is a natural number greater than or equal to 3, the first pixel unit includes a part of the pixels arranged in a checkerboard pattern among the plurality of pixels arranged in the m×m matrix, the part of the pixels including the R filter, and the remaining part of the pixels including the C filter; the second pixel unit includes a part of the pixels arranged in a checkerboard pattern among the plurality of pixels arranged in the m×m matrix, the part of the pixels including the G filters, and the remaining part of the pixels including the M filters; In the third pixel unit, some of the pixels arranged in a checkerboard pattern among the plurality of pixels arranged in the m×m matrix include the B filters, and the remaining pixels include the Y filters. The solid-state imaging device according to (1) above. (3) m=2, the first pixel unit includes a pair of pixels arranged at diagonal positions, one of which includes the R filter and the other of which includes the C filter; the second pixel unit includes a pair of pixels arranged at diagonal positions, one of which includes the G filter and the other of which includes the M filter; The third pixel unit has a pair of pixels arranged at diagonal positions, one of which includes the B filter and the other of which includes the M filter. The solid-state imaging device according to (1) above. (4) a pixel array section in which a plurality of pixel unit groups are arranged in a 2×2 matrix, each pixel unit being configured as a pixel unit in which pixels having a photoelectric conversion section and a color filter formed corresponding to the photoelectric conversion section are arranged in an m×m (m is a natural number equal to or greater than 2) matrix, and each pixel unit group is configured as a pixel unit group including, as the color filters, an R filter that transmits red light and a C filter that transmits cyan light that is complementary to the red light, in a first pixel unit of four pixel units constituting the pixel unit group, a G filter that transmits green light and an M filter that transmits magenta light that is complementary to the green light, in each of a second and a third pixel unit, and a B filter that transmits blue light and a Y filter that transmits yellow light that is complementary to the blue light, in a fourth pixel unit, an optical lens that forms an image of image light from a subject on an imaging surface of the solid-state imaging device; a signal processing circuit for performing signal processing on a signal output from the solid-state imaging device; electronic equipment. (5) The signal processing circuit includes a binning processing unit that generates an RGB mosaic image by adding pixel values ​​of image pixels having only red color information in each of the pixel groups corresponding to the first pixel unit, of a mosaic image corresponding to the array of the color filters obtained from the signal, to obtain a pixel value of a single image pixel having only red color information, adding pixel values ​​of image pixels having only green color information in each of the pixel groups corresponding to the second and third pixel units, to obtain a pixel value of a single image pixel having only green color information, and adding pixel values ​​of image pixels having only blue color information in each of the pixel groups corresponding to the fourth pixel unit, to obtain a pixel value of a single image pixel having only blue color information. The electronic device described in (4). (6) The signal processing circuit includes a binning processing unit that generates a CMY mosaic image by adding pixel values ​​of image pixels having only cyan color information in each of the pixel groups corresponding to the first pixel unit, of a mosaic image corresponding to the array of the color filters obtained from the signal, to obtain a pixel value of an image pixel having only cyan color information, adding pixel values ​​of image pixels having only magenta color information in each of the pixel groups corresponding to the second and third pixel units, to obtain a pixel value of an image pixel having only magenta color information, and adding pixel values ​​of image pixels having only yellow color information in each of the pixel groups corresponding to the fourth pixel unit, to obtain a pixel value of an image pixel having only yellow color information. The electronic device according to (4) or (5). (7) The signal processing circuit includes a binning processing unit that calculates a luminance value of one image pixel by adding pixel values ​​of image pixels having color information of red and cyan in each of the pixel groups corresponding to the first pixel unit in a mosaic image corresponding to the array of the color filters obtained from the signal, calculates a luminance value of one image pixel by adding pixel values ​​of image pixels having color information of green and magenta in each of the pixel groups corresponding to the second and third pixel units, and calculates a luminance value of one image pixel by adding pixel values ​​of image pixels having color information of blue and yellow in each of the pixel groups corresponding to the fourth pixel unit. The electronic device according to any one of (4) to (6). (8) The binning processing unit uses a numerical value obtained by multiplying a pixel value of the mosaic image by a correction coefficient set for each color temperature of a light source as a pixel value used for calculating the luminance value. The electronic device according to (7) above. (9) The signal processing circuit includes: a binning processing unit that generates an RGB mosaic image by adding pixel values ​​of image pixels having only red color information in each of the pixel groups corresponding to the first pixel unit in a mosaic image corresponding to the color filter array obtained from the signal to obtain a pixel value of a single image pixel having only red color information, adding pixel values ​​of image pixels having only green color information in each of the pixel groups corresponding to the second and third pixel units to obtain a pixel value of a single image pixel having only green color information, and adding pixel values ​​of image pixels having only blue color information in each of the pixel groups corresponding to the fourth pixel unit to obtain a pixel value of a single image pixel having only blue color information; a luminance synthesis unit that synthesizes the luminance value with the RGB mosaic image. The electronic device according to (7) or (8). (10) The signal processing circuit includes: a binning processing unit that generates a CMY mosaic image by adding pixel values ​​of image pixels having only cyan color information in each of the pixel groups corresponding to the first pixel unit in a mosaic image corresponding to the color filter array obtained from the signal to obtain a pixel value of a single image pixel having only cyan color information, adding pixel values ​​of image pixels having only magenta color information in each of the pixel groups corresponding to the second and third pixel units to obtain a pixel value of a single image pixel having only magenta color information, and adding pixel values ​​of image pixels having only yellow color information in each of the pixel groups corresponding to the fourth pixel unit to obtain a pixel value of a single image pixel having only yellow color information; a luminance synthesis unit that converts the CMY mosaic image into an RGB mosaic image and then synthesizes the luminance value with the converted RGB mosaic image. The electronic device according to (7) or (8). (11) the binning processing unit generates the RGB mosaic image, and adds pixel values ​​of image pixels having only cyan color information in each of the pixel groups corresponding to the first pixel unit in the mosaic image corresponding to the array of the color filters obtained from the signal to obtain a pixel value of a single image pixel having only cyan color information, adds pixel values ​​of image pixels having only magenta color information in each of the pixel groups corresponding to the second and third pixel units to obtain a pixel value of a single image pixel having only magenta color information, and adds pixel values ​​of image pixels having only yellow color information in each of the pixel groups corresponding to the fourth pixel unit to obtain a pixel value of a single image pixel having only yellow color information, thereby generating a CMY mosaic image; The signal processing circuit includes an HDR image generating unit that converts the CMY mosaic image into an RGB mosaic image, and combines the converted RGB mosaic image with the RGB mosaic image generated by the binning processing unit to generate a high dynamic range image. The electronic device described in (5) above. [Explanation of symbols]

[0097] 1...solid-state imaging device, 2...substrate, 3...pixel array section, 4...vertical drive circuit, 5...column signal processing circuit, 6...horizontal drive circuit, 7...output circuit, 8...control circuit, 9...pixel, 10...pixel unit, 11...pixel unit group, 12...pixel drive wiring, 13...vertical signal line, 14...horizontal signal line, 15...insulating film, 16...light-shielding film, 17...planarizing film, 18...light-receiving layer, 19...color filter, 20...microlens, 21...light-collecting layer, 22...wiring layer, 23...support substrate, 24...photoelectric conversion section, 25...pixel separation section, 26...color filter array, 27...microlens array, 28...interlayer insulating film, 29...wiring, 30...mosaic image, 31...image pixel, 32...image pixel, 33...RGB mosaic image, 34...image pixel, 35...CMY Mosaic image, 361, 362, 363, 364...pixel group, 37...image pixel, 38...image pixel, 39...CMY mosaic image, 40...image pixel, 41...luminance image, 42...synthetic image, 43...image pixel, 44...RGB mosaic image, 45...synthetic image, 46...RGB mosaic image, 47...HDR image, 48...image pixel, 100...electronic device, 101...solid-state imaging device, 102...optical lens, 103...shutter device, 104...drive circuit, 105...signal processing circuit, 106...mosaic image generating unit, 107...white balance adjustment unit, 108...mode determination unit, 109...re-mosaic processing unit, 110...binning processing unit, 111...luminance value calculation unit, 112...luminance synthesis unit, 113...incident light, 114...HDR image generating unit

Claims

1. a pixel array section in which a plurality of pixel unit groups are arranged in a 2×2 matrix, the pixel unit being formed in an m×m (m is a natural number of 2 or more) matrix of pixel units, the pixel unit including a photoelectric conversion unit and a color filter formed corresponding to the photoelectric conversion unit; Each of the pixel unit groups includes, as the color filters, an R filter that transmits red light and a C filter that transmits cyan light that is complementary to the red light in a first pixel unit of the four pixel units constituting the pixel unit group, a G filter that transmits green light and an M filter that transmits magenta light that is complementary to the green light in each of the second and third pixel units, and a B filter that transmits blue light and a Y filter that transmits yellow light that is complementary to the blue light in each of the fourth pixel units. Solid-state imaging device.

2. m is a natural number equal to or greater than 3, the first pixel unit includes a part of the pixels arranged in a checkerboard pattern among the plurality of pixels arranged in the m×m matrix, the part of the pixels including the R filter and the remaining part of the pixels including the C filter; the second pixel unit includes a part of the pixels arranged in a checkerboard pattern among the plurality of pixels arranged in the m×m matrix, the part of the pixels including the G filters, and the remaining part of the pixels including the M filters; In the third pixel unit, some of the pixels arranged in a checkerboard pattern among the plurality of pixels arranged in the m×m matrix include the B filter, and the remaining pixels include the Y filter. The solid-state imaging device according to claim 1 .

3. m=2, the first pixel unit includes a pair of pixels arranged at diagonal positions, one of which includes the R filter and the other of which includes the C filter; the second pixel unit includes a pair of pixels arranged at diagonal positions, one of which includes the G filter and the other of which includes the M filter; The third pixel unit has a pair of pixels arranged at diagonal positions, one of which includes the B filter and the other of which includes the M filter. The solid-state imaging device according to claim 1 .

4. a pixel array section in which a plurality of pixel unit groups are arranged in a 2×2 matrix, each pixel unit being configured as a pixel unit in which pixels having a photoelectric conversion section and a color filter formed corresponding to the photoelectric conversion section are arranged in an m×m (m is a natural number equal to or greater than 2) matrix, and each pixel unit group is configured as a pixel unit in which a first pixel unit of four pixel units constituting the pixel unit group includes, as the color filters, an R filter that transmits red light and a C filter that transmits cyan light that is complementary to the red light, a second pixel unit and a third pixel unit include, as the color filters, a G filter that transmits green light and an M filter that transmits magenta light that is complementary to the green light, and a fourth pixel unit includes, as the color filters, a B filter that transmits blue light and a Y filter that transmits yellow light that is complementary to the blue light; an optical lens that forms an image of image light from a subject on an imaging surface of the solid-state imaging device; a signal processing circuit for performing signal processing on a signal output from the solid-state imaging device; electronic equipment.

5. The signal processing circuit includes a binning processing unit that generates an RGB mosaic image by adding pixel values ​​of image pixels having only red color information in each of the pixel groups corresponding to the first pixel unit, of a mosaic image corresponding to the array of the color filters obtained from the signal, to obtain a pixel value of an image pixel having only red color information, adding pixel values ​​of image pixels having only green color information in each of the pixel groups corresponding to the second and third pixel units, to obtain a pixel value of an image pixel having only green color information, and adding pixel values ​​of image pixels having only blue color information in each of the pixel groups corresponding to the fourth pixel unit, to obtain a pixel value of an image pixel having only blue color information.

5. The electronic device according to claim 4.

6. The signal processing circuit includes a binning processing unit that generates a CMY mosaic image by adding pixel values ​​of image pixels having only cyan color information in each of the pixel groups corresponding to the first pixel unit, of a mosaic image corresponding to the array of the color filters obtained from the signal, to obtain a pixel value of an image pixel having only cyan color information, adding pixel values ​​of image pixels having only magenta color information in each of the pixel groups corresponding to the second and third pixel units, to obtain a pixel value of an image pixel having only magenta color information, and adding pixel values ​​of image pixels having only yellow color information in each of the pixel groups corresponding to the fourth pixel unit, to obtain a pixel value of an image pixel having only yellow color information.

5. The electronic device according to claim 4.

7. The signal processing circuit includes a luminance value calculation unit that calculates a luminance value of one image pixel by adding pixel values ​​of image pixels having color information of red and cyan in each of the pixel groups corresponding to the first pixel unit in a mosaic image corresponding to the array of the color filters obtained from the signal, calculates a luminance value of one image pixel by adding pixel values ​​of image pixels having color information of green and magenta in each of the pixel groups corresponding to the second and third pixel units, and calculates a luminance value of one image pixel by adding pixel values ​​of image pixels having color information of blue and yellow in each of the pixel groups corresponding to the fourth pixel unit.

5. The electronic device according to claim 4.

8. The luminance value calculation unit uses a numerical value obtained by multiplying a pixel value of the mosaic image by a correction coefficient set for each color temperature of a light source as a pixel value used for calculating the luminance value.

8. The electronic device according to claim 7.

9. The signal processing circuit includes: a binning processing unit that generates an RGB mosaic image by adding pixel values ​​of image pixels having only red color information in each of the pixel groups corresponding to the first pixel unit in a mosaic image corresponding to the color filter array obtained from the signal to obtain a pixel value of a single image pixel having only red color information, adding pixel values ​​of image pixels having only green color information in each of the pixel groups corresponding to the second and third pixel units to obtain a pixel value of a single image pixel having only green color information, and adding pixel values ​​of image pixels having only blue color information in each of the pixel groups corresponding to the fourth pixel unit to obtain a pixel value of a single image pixel having only blue color information; a luminance synthesis unit that synthesizes the luminance value with the RGB mosaic image.

8. The electronic device according to claim 7.

10. The signal processing circuit includes: a binning processing unit that generates a CMY mosaic image by adding pixel values ​​of image pixels having only cyan color information in each of the pixel groups corresponding to the first pixel unit in a mosaic image corresponding to the color filter array obtained from the signal to obtain a pixel value of a single image pixel having only cyan color information, adding pixel values ​​of image pixels having only magenta color information in each of the pixel groups corresponding to the second and third pixel units to obtain a pixel value of a single image pixel having only magenta color information, and adding pixel values ​​of image pixels having only yellow color information in each of the pixel groups corresponding to the fourth pixel unit to obtain a pixel value of a single image pixel having only yellow color information; a luminance synthesis unit that converts the CMY mosaic image into an RGB mosaic image and then synthesizes the luminance value with the converted RGB mosaic image.

8. The electronic device according to claim 7.

11. the binning processing unit generates the RGB mosaic image, and adds pixel values ​​of image pixels having only cyan color information in each of the pixel groups corresponding to the first pixel unit out of the mosaic image corresponding to the array of the color filters obtained from the signal to obtain a pixel value of a single image pixel having only cyan color information, adds pixel values ​​of image pixels having only magenta color information in each of the pixel groups corresponding to the second and third pixel units to obtain a pixel value of a single image pixel having only magenta color information, and adds pixel values ​​of image pixels having only yellow color information in each of the pixel groups corresponding to the fourth pixel unit to obtain a pixel value of a single image pixel having only yellow color information, thereby generating a CMY mosaic image; The signal processing circuit includes an HDR image generating unit that converts the CMY mosaic image into an RGB mosaic image and synthesizes the converted RGB mosaic image with the RGB mosaic image generated by the binning processing unit to generate a high dynamic range image.

6. The electronic device according to claim 5.

Images