Imaging devices and electronic devices

Abstract

The objective of the present invention is to improve the color accuracy of an imaging device. [Solution] This imaging element is provided with a plurality of pixels for acquiring first information, which is information relating to three primary colors, and second information, which is information relating to at least two colors different from the three primary colors, and which includes at least one color from among the complementary colors of the three primary colors.

Description

[Technical Field]

[0001] The present disclosure relates to an imaging element and an electronic device. [Background technology]

[0002] In devices including solid-state imaging modules, it is common to use three-color filters as the imaging element. In recent years, techniques have been widely developed in which a different color filter is arranged in addition to the three-color, e.g., RGB, color filters. However, it is difficult to say that such color filter arrangements have been found to be satisfactory from the viewpoint of image composition.

[0003] With the development of electronic devices such as smartphones, images are frequently shared on social networking services (SNS), etc. When sharing images in this way, not only faithful color reproduction but also color rendition is required, and there is a growing demand for increased freedom in color adjustment and the ability to change color creation according to object recognition results.

[0004] For devices such as smartphones, for example, under-display cameras (UDCs) have been developed, which place a camera under the display surface. However, this type of configuration poses a problem: the camera receives light that has passed through the display, resulting in a significant decrease in blue sensitivity due to the materials used. Furthermore, one of the challenges with organic photoelectric conversion films is that extracting RGB vertically from a single pixel requires a two-layer structure of organic photoelectric conversion films or photodiodes, which is inefficient in terms of cost and other factors. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-045917 Summary of the Invention [Problem to be solved by the invention]

[0006] In the embodiments of the present disclosure, an imaging device and an electronic device are provided that include color filters of different colors in addition to color filters of three colors (for example, RGB) and improve the accuracy of image reconstruction. [Means for solving the problem]

[0007] According to one embodiment, the imaging element includes a plurality of pixels that acquire first information, which is information on three primary colors, and second information, which is information on at least two colors different from the three primary colors and includes at least one of the complementary colors of the three primary colors.

[0008] The three primary colors may be R (red), G (green), and B (blue), and the complementary colors may be Cy (cyan), Ye (yellow), and Mg (magenta).

[0009] The second information may include Cy and Ye information.

[0010] The second information may include information on Cy, Ye, and Mg.

[0011] The second information may include at least one of white and emerald information.

[0012] The pixel may output information of at least two colors from among the first information and the second information.

[0013] The pixel may include a plurality of divided pixels, and information of one color of the first information and the second information may be acquired for each of the divided pixels.

[0014] Of the color information included in the first information and the second information, information on similar colors may be acquired together.

[0015] The information on similar colors may include information on at least one of R and Mg, B and Cy, G and Ye, R and Ye, or R and infrared light.

[0016] The first information and the second information may be information obtained by different light receiving elements at the same time.

[0017] The second information may include information on at least one of the three primary colors.

[0018] The first information and the second information may be acquired at different resolutions.

[0019] At least one of the pixels may acquire information using an organic photoelectric conversion film.

[0020] The first information may be acquired in an organic photoelectric conversion film, and the second information may be acquired in a photodiode via the organic photoelectric conversion film, or the second information may be acquired in an organic photoelectric conversion film, and the first information may be acquired in a photodiode via the organic photoelectric conversion film.

[0021] The electronic device may include any of the imaging elements described above, and may have a still image mode and a video mode, and may combine the first information and the second information at different times or in different blocks in the still image mode and the video mode.

[0022] At least one of the pixels may acquire information about light in the infrared region.

[0023] A filter capable of removing infrared light may be provided in at least a part of the imaging element included in the pixel that acquires information about light in the infrared region.

[0024] The frame images output from the pixels may be synthesized, and the number of the pixels to be added for each frame may be controlled at the timing of synthesis.

[0025] According to one embodiment, an electronic device includes the above-described image sensor, and performs object identification, biometric identification, or light source estimation based on spectral information acquired from the image sensor.

[0026] Image processing or parameters of the image processing may be controlled based on the target identified by the object identification or the biometric identification.

[0027] The image pickup device may include a display that displays an image, and any one of the imaging elements described above, located so as to overlap the display.

[0028] According to one embodiment, an electronic device includes a compound eye camera including at least one of the imaging elements described above, and at least one of the imaging elements does not include an infrared rejection filter.

[0029] According to one embodiment, the imaging element may include a first pixel group that acquires the first information and a second pixel group that acquires the second information, and the second pixel group may be pixels that are more sensitive than the pixels in the first pixel group that acquire G information.

[0030] The second pixel group may include at least a pixel that acquires white information.

[0031] Information obtained by the first group of pixels may be used to interpolate information obtained by the second group of pixels.

[0032] The imaging device may include a still image mode for acquiring a still image and a moving image mode for acquiring a moving image, and may combine the first pixel group and the second pixel group using different methods in the still image mode and the moving image mode.

[0033] The still image mode and the moving image mode may be combined at different timings.

[0034] The still image mode and the moving image mode may be synthesized using different system blocks.

[0035] In the moving image mode, the pixels belonging to the first pixel group may have a lower resolution than the pixels belonging to the second pixel group.

[0036] The color information acquired from the second pixel group may be corrected based on color information statistics or a light source estimation result acquired from the first pixel group.

[0037] The pixels belonging to the first pixel group and the nearest pixels belonging to the second pixel group may be spaced apart by a distance of 10 mm or less.

[0038] The nearest pixels may be spaced a distance of 50mm to 80mm apart.

[0039] Either the first pixel group or the second pixel group may be configured to include divided pixels obtained by dividing pixels belonging to the first pixel group or the second pixel group.

[0040] A white pixel may be provided in the divided pixel.

[0041] The optical parameters of the paths of the light acquired in the first pixel group and the second pixel group may be different between the first pixel group and the second pixel group.

[0042] The second pixel group may include pixels having higher sensitivity than the pixels of the first pixel group that obtain the information B.

[0043] The pixels having higher sensitivity than the pixels acquiring the B information may be pixels acquiring white or cyan color information.

[0044] The imaging device may recalculate three primary color information from the acquired image information by arithmetic processing, for pixel information acquired in five or more colors, and output rearranged information.

[0045] The imaging device may calculate pixel values ​​or statistical values ​​of information including at least complementary color information from the acquired image information.

[0046] The imaging device may calculate the statistical values ​​for information on the three primary colors and the complementary color information from the acquired image information.

[0047] The imaging device may calculate the statistical value from the acquired image information using the complementary color information without including information about the three primary colors.

[0048] The imaging device may control the amount of exposure for pixels that receive light of the three primary colors and for pixels that receive light of complementary colors, respectively.

[0049] The exposure amount may be controlled by controlling the shutter time.

[0050] The exposure amount may be controlled by controlling the gain.

[0051] According to one embodiment, a solid-state imaging device includes at least one of the imaging elements described above.

[0052] According to one embodiment, an electronic device includes an imaging element having a plurality of pixels that acquire first information, which is information on three primary colors, and second information, which is information on at least two colors different from the three primary colors and includes at least one color that is a complementary color of the three primary colors. [Brief explanation of the drawings]

[0053] [Figure 1] FIG. 1 is a block diagram showing an example of a solid-state imaging device according to an embodiment. [Figure 2] FIG. 1 is a block diagram showing an example of a solid-state imaging device according to an embodiment. [Figure 3] A diagram showing the relationship between color and the spectrum. [Figure 4] FIG. 2 is a diagram showing the relationship between pixels and light receiving elements according to an embodiment. [Figure 5] Cross section AA of Figure 4. [Figure 6] FIG. 1 is a diagram illustrating a pixel and a pixel array according to an embodiment. [Figure 7] FIG. 2 is a diagram showing an arrangement of colors received by a pixel according to an embodiment. [Figure 8] FIG. 4 is a diagram showing the colors of light received at a predetermined timing in a pixel according to an embodiment. [Figure 9] FIG. 4 is a diagram showing the colors of light received at a predetermined timing in a pixel according to an embodiment. [Figure 10] FIG. 2 is a diagram showing types of light received in a pixel according to an embodiment. [Figure 11] FIG. 1 is a block diagram showing an example of a solid-state imaging device according to an embodiment. [Figure 12] FIG. 2 is a diagram showing an arrangement of colors received by a pixel according to an embodiment. [Figure 13] FIG. 2 is a diagram showing an arrangement of colors received by a pixel according to an embodiment. [Figure 14] FIG. 2 is a diagram showing an arrangement of colors received by a pixel according to an embodiment. [Figure 15] FIG. 2 is a diagram showing an arrangement of colors received by a pixel according to an embodiment. [Figure 16] FIG. 4 is a diagram showing the colors of light received by a pixel according to an embodiment. [Figure 17] FIG. 4 is a diagram showing the colors of light received by a pixel according to an embodiment. [Figure 18] 1A and 1B are diagrams showing an example of pixel formation according to an embodiment; [Figure 19] 1A and 1B are diagrams showing an example of pixel formation according to an embodiment; [Figure 20] 1A and 1B are diagrams showing an example of pixel formation according to an embodiment; [Figure 21] 1A and 1B are diagrams showing an example of pixel formation according to an embodiment; [Figure 22] 1A and 1B are diagrams showing an example of pixel formation according to an embodiment; [Figure 23] 1A and 1B are diagrams showing an example of pixel formation according to an embodiment; [Figure 24] 1A and 1B are diagrams showing an example of pixel formation according to an embodiment; [Figure 25] 1A and 1B are diagrams showing an example of pixel formation according to an embodiment; [Figure 26] FIG. 2 is a diagram showing an example of a substrate configuration according to an embodiment. [Figure 27] FIG. 2 is a diagram showing an example of a substrate configuration according to an embodiment. [Figure 28] FIG. 2 is a diagram showing an example of a substrate configuration according to an embodiment. [Figure 29] FIG. 1 is a schematic diagram illustrating an example of an electronic device including a solid-state imaging device according to an embodiment. [Figure 30] FIG. 1 is a schematic diagram illustrating an example of an electronic device including a solid-state imaging device according to an embodiment. [Figure 31] FIG. 1 is a block diagram showing an example of a solid-state imaging device according to an embodiment. [Figure 32] FIG. 4 illustrates a spectrum constituting the input of a pre-linear matrix unit according to an embodiment. [Figure 33] FIG. 2 illustrates spectra constituting a rearranged RGB signal according to one embodiment. [Figure 34] FIG. 1 is a diagram illustrating an electronic device according to an embodiment. [Figure 35] FIG. 2 is a diagram showing an arrangement of colors received by a pixel according to an embodiment. [Figure 36] FIG. 2 is a diagram showing an arrangement of colors received by a pixel according to an embodiment. [Figure 37A] FIG. 2 is a diagram showing the interior of the vehicle from the rear to the front. [Figure 37B] FIG. 2 is a diagram showing the interior of the vehicle from the diagonally rear to the diagonally front of the vehicle. [Figure 38A] FIG. 10 is a front view of a digital camera as a second application example of the electronic device. [Figure 38B] Rear view of a digital camera. [Figure 39A] FIG. 10 is an external view of an HMD, which is a third application example of the electronic device. [Figure 39B] Appearance of smart glasses. [Figure 40] FIG. 10 is an external view of a TV as a fourth application example of an electronic device. [Figure 41] FIG. 10 is an external view of a smartphone, which is a fifth application example of an electronic device. DETAILED DESCRIPTION OF THE INVENTION

[0054] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. The drawings are used for explanation purposes, and the shape, size, and size ratio of each component in an actual device do not necessarily have to be the same as those shown in the drawings. Furthermore, since the drawings are simplified, components necessary for implementation other than those shown in the drawings are also assumed to be appropriately provided.

[0055] 1 is a diagram showing an example of a block diagram of a solid-state imaging device according to an embodiment. The solid-state imaging device 10 includes a light receiving unit 100, a storage unit 102, a control unit 104, a signal processing unit 106, and an image processing unit 108. The solid-state imaging device 10 is a device that appropriately processes light received by the light receiving unit 100, converts it into image information, recognition information, etc., and outputs the information.

[0056] The light receiving unit 100 receives light from the outside and outputs a signal based on the intensity of the received light.

[0057] The storage unit 102 stores data required by each component of the solid-state imaging device 10 or data output from each component. The storage unit 102 is configured to include any suitable temporary or non-temporary storage medium such as memory or storage.

[0058] The control unit 104 controls the light receiving unit 100, etc. The control unit 104 may perform control based on, for example, input from a user, or may perform control based on preset conditions. The control unit 104 may also perform control based on outputs from the signal processing unit 106, image processing unit 108, etc.

[0059] The signal processing unit 106 appropriately processes and outputs the signal output by the light receiving unit 100. For example, the signal processing unit 106 performs processing to convert the analog signal output by the light receiving unit 100 into a digital signal. In addition to this, the signal processing unit 106 may perform other processing such as signal clamping. For example, the signal processing unit 106 converts the received analog signal into a digital image signal and outputs the image information to the image processing unit 108.

[0060] The image processing unit 108 performs predetermined image processing on the converted information from the signal processing unit 106. The image processing unit 108 performs, for example, noise removal processing, various filtering processing, and the like, to appropriately process the image information.

[0061] Although the signal processing unit 106 and the image processing unit 108 are described separately for convenience, they may be provided as a single signal processing unit. As another example, each process may be configured with a smaller unit (circuit) rather than two units (circuits). The signal processing unit 106 and / or the image processing unit 108 generate information and images for various purposes based on the signal output from the imaging element.

[0062] Some or all of the components of the solid-state imaging device 10 described above may be implemented as dedicated digital or analog circuits. In the case of dedicated circuits, they may be configured, for example, by an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). They may also be implemented by general-purpose processing circuits such as a CPU (Central Processing Unit).

[0063] FIG. 2 is a block diagram showing a more detailed, non-limiting example of the configuration of the solid-state imaging device 10 shown in FIG.

[0064] The light receiving unit 100 includes, for example, a lens 110, an infrared ray removal filter (hereinafter, IRCF 112), and an image sensor 114. Light appropriately refracted by the lens 110 enters the image sensor 114 via the IRCF 112, and the image sensor 114 outputs an analog signal based on the intensity of the received light. The IRCF 112 is not an essential component. Furthermore, an optical system other than the lens 110 may be included.

[0065] The image sensor 114 includes, for example, an organic photoelectric conversion film or a photodiode. The image sensor 114 may include a color filter so that each image sensor 114 acquires light of an appropriate spectrum. If the image sensor 114 includes an organic photoelectric conversion film, each image sensor 114 may output an analog signal based on the intensity of light of an appropriate spectrum, rather than including a filter.

[0066] The signal processing unit 106 or the image processing unit 108 includes, for example, an A / D conversion unit 120, a clamping unit 122, a linear matrix unit 124, a gamma correction unit 126, a light source estimation unit 130, an object recognition unit 132, and a luminance chroma signal generation unit 140.

[0067] In this way, the functions do not have to be clearly divided between the signal processing unit 106 and the image processing unit 108. For example, the signal processing unit 106 may process the signals themselves, such as the A / D conversion unit 120, and the image processing unit 108 may perform image-related processing, such as the gamma correction unit 126.

[0068] 2 is a diagram shown as an example, and the signal processing unit 106 or the image processing unit 108 may perform further signal processing or image processing rather than only performing the processing shown in the figure. Conversely, not all of the components shown in Fig. 2 are essential, and components may be omitted as appropriate based on the processing to be performed.

[0069] The A / D conversion section 120 converts the analog signal output from the image sensor 114 into a digital signal and outputs the digital signal. Note that the A / D conversion section 120 may be built into the image sensor.

[0070] The clamping unit 122 subtracts the black level from the digital signal output by the A / D conversion unit 120 .

[0071] The linear matrix unit 124 reproduces the color of the photographed target by combining the digital signals output by the clamp unit 122 for each color.

[0072] The gamma correction unit 126 performs gamma correction processing on the digital signal output by the linear matrix unit 124 .

[0073] On the other hand, the light source estimation unit 130 estimates the light source environment, the scene, etc. from the multi-color color information that is the digital signal output by the clamp unit 122.

[0074] The object recognition unit 132 recognizes the object indicated by the digital signal in the environment estimated by the light source estimation unit 130. For this recognition, for example, a trained neural network model may be used.

[0075] The luminance chrominance signal generation unit 140 reconstructs an image based on the image signal gamma-corrected by the gamma correction unit 126 and the recognition result output by the object recognition unit 132. For example, the luminance chrominance signal generation unit 140 may manipulate color information based on the object recognized by the object recognition unit 132 to generate an image having colors suitable for the recognized object.

[0076] The input / output I / F 150 outputs the image data generated by the luminance chroma signal generating unit 140 to the outside.

[0077] In this way, the image captured as the subject is appropriately processed and output. Here, the arrangement of the colors of light received within the image sensor 114 will be described in detail below.

[0078] Figure 3 is a graph showing the spectra of typical R (red), G (green), and B (blue) overlaid with the spectra of Cy (cyan), Ye (yellow), and Mg (magenta). As shown in Figure 3, the spectra of the three colors R (dotted line), G (dash-dotted line), and B (dashed line) have spectral valleys near wavelengths of 500 nm and 580 nm. In other words, even when the gains of the three colors are multiplied and added, the light reception of colors with spectral peaks at these positions deteriorates.

[0079] In the present disclosure, to address such spectral valleys, color filters or organic photoelectric conversion films are appropriately arranged to receive light intensities with at least two different spectral characteristics in addition to receiving light of the three colors R, G, and B. The spectral characteristics shown by the solid lines represent the color spectra of cyan, yellow, and magenta, as indicated by Cy, Ye, and Mg, respectively.

[0080] The solid-state imaging device 10 includes pixels that acquire, for example, first information, which is the result of acquiring information on the three primary colors of R, G, and B, and second information, which is the result of acquiring information on at least two colors different from the three primary colors and includes at least one color of Cy, Ye, and Mg, which are complementary colors of the three primary colors, in the image sensor 114. Although the three primary colors are not limited to RGB, RGB is used as the three primary colors and Cy, Ye, and Mg are used as complementary colors in the following description.

[0081] As can be seen from Figure 3, if the color information acquired as the second information is Cy and Ye, it is possible to cover the valleys of the spectrum formed by RGB. Also, color information of the three complementary colors Cy, Ye, and Mg may be used as the second information. Adding Mg allows for more detailed color reproduction.

[0082] In this way, by using other spectral components of the three primary colors, it is possible to cover the visible light range and aim for more faithful color reproduction. For example, when using the three primary colors of RGB, as shown in Figure 3, there are two spectral valleys, so it is desirable to add an element that can capture two or more colors that can cover each valley.

[0083] Alternatively, emerald, which has a similar peak position, may be used instead of Cy. Although not shown, emerald is expressed by a spectrum with a peak between B and G. The spectra of Cy and emerald are also useful for generating negative components in color-matching functions. As another example, an element that receives white light having an overall spectrum may be provided.

[0084] By using more than three colors in this way, it is possible to improve the color reproducibility in the image, and also to improve the accuracy of the light source estimation in the light source estimation unit 130 described above.

[0085] (First embodiment) 4 is a diagram showing an example of a pixel provided in the image sensor 114. In this diagram, a configuration is shown in which a plurality of light receiving elements 200 are provided in the pixel, but the configuration is not limited to this, and the color combinations described below can be similarly applied even in other configurations, such as a one-to-one correspondence between the pixels 20 and the light receiving elements 200. Note that while this diagram mainly shows the arrangement of the light receiving elements 200 in the pixel, it is assumed that signal lines, light-shielding walls, and other elements not shown are provided appropriately.

[0086] The pixel 20 includes, for example, four light receiving elements 200a, 200b, 200c, and 200d. Each of the light receiving elements 200a to 200d includes an independent photodiode and an organic photoelectric conversion film. Filters of different colors may be provided on these imaging elements, or some of them may be provided with filters of the same color. In the case of an organic photoelectric conversion film, the light receiving elements themselves may be configured to receive light having different color spectra or the same color spectrum.

[0087] As an example, the light receiving elements 200a and 200d may receive light having a spectrum of Mg, and the light receiving elements 200b and 200c may receive light having a spectrum of R. Such a configuration example will be described in more detail below in light of various variations.

[0088] 5 is a cross-sectional view taken along the line AA in FIG. 4. In this figure, the light receiving element 200 includes, for example, photodiodes. An insulating film may be provided between these photodiodes. Such a segmented photodiode may be, for example, a semiconductor device such as that disclosed in Japanese Patent Application Laid-Open No. 2013-045917, or may be a semiconductor device of another type.

[0089] A filter 202a that transmits light that matches the spectrum received by the light receiving element 200a is provided above the light receiving element 200a. Similarly, a filter 202b is provided above the light receiving element 200b. According to the above example, the filter 202a is a filter that transmits Mg light, and the filter 202b is a filter that transmits R light.

[0090] The light receiving elements 200a and 200b convert the light received by each element into an analog signal and output the analog signal to an appropriate location such as a floating diffusion within the pixel 20. The light receiving elements 200a and 200b may output the analog signal at appropriate timings, or may output the analog signals at the same timing.

[0091] FIG. 6 is a diagram showing an example of a pixel array in which pixels are provided. As shown in FIG. 6, a plurality of pixels 20 are provided in an array. The pixels provided in the array form a pixel array 204. This pixel array 204 is provided in the light receiving unit 100, and receives light, converts it into an analog signal based on the characteristics of each light receiving element 200, and outputs the analog signal. Each pixel 20 may be provided with a light receiving element 200 as a divided pixel as shown in FIG. 5, or, as described above, the following configuration may be appropriately expressed by obtaining a predetermined color for each pixel 20 without using divided pixels.

[0092] In this way, the pixel 20 may include a light receiving element 200 that forms divided pixels, and each divided pixel may receive light that has passed through a different filter. Acquisition of color using this configuration will be described below. As described above, as several examples, reception of two colors of light in the pixel 20 will first be described. These two colors are preferably similar colors. Here, similar colors are, for example, combinations of R and Mg, G and Ye, or B and Cy. Other combinations include G and emerald, or B and emerald. As can be seen from the spectrum, these color combinations can also be defined as colors that have a common wavelength component of a predetermined value or more.

[0093] 7 is a diagram showing an example of a combination of colors received by a pixel 20 according to one embodiment. In the diagram, the colors are designated as R, G, B, Mg, Ye, and Cy, as in the specification, for red, green, blue, magenta, yellow, and cyan, respectively.

[0094] 7 shows a combination of four pixels 20. In the upper left pixel 20, the upper left and lower right light receiving elements 200 receive Mg color light, and the upper right and lower left image capturing elements 114 receive R color light. As described above, R and Mg are similar colors.

[0095] In the upper right and lower left pixels 20, the upper left and lower right light receiving elements 200 receive Ye color light, and the upper right and lower left light receiving elements 200 receive G color light. As described above, G and Ye are similar colors.

[0096] In the lower right pixel 20, the upper left and lower right light receiving elements 200 receive light of color Cy, and the upper right and lower left light receiving elements 200 receive light of color B. As described above, B and Cy are similar colors.

[0097] In this way, the light receiving elements 200 (or filters 202) are arranged so as to receive light of a similar color combination for each pixel 20. By arranging similar colors in the same pixel 20, it is possible to reduce the influence of color mixing compared to when dissimilar colors are arranged in the same pixel 20.

[0098] Conversely, although not shown, similar colors may be included in other pixels 20. In this way, color resolution can be improved by arranging similar colors in other pixels 20. In this way, the color distribution within a pixel 20 can be changed appropriately depending on the environment and application.

[0099] As shown in the figure, the R pixels 20, the G pixels 20, and the B pixels 20 may be arranged in a Bayer array. The Bayer array is given as a non-limiting example, and the array may be other arrays such as a checkerboard array or an RGBW array.

[0100] In the pixels 20 having a light receiving array as shown in Fig. 7, an analog signal can be output in bulk from each pixel 20. For example, the upper left pixel 20 in Fig. 7 can receive light that is a mixture of R and Mg and output an analog signal based on the intensity of the received light. Similarly, the upper right and lower left pixels 20 can output an analog signal corresponding to light that is a mixture of G and Ye, and the lower right pixel 20 can output an analog signal corresponding to light that is a mixture of B and Cy.

[0101] In this way, by collectively acquiring signals from the divided pixels (light receiving elements 200) within the pixel 20, it is possible to reduce readout noise and prevent a decrease in frame rate. For example, in a scene with high brightness, light may be received collectively for each pixel 20 in order to reduce readout noise. Furthermore, by collectively reading out signals, analog signals can be acquired at different exposure amounts between frames, making it possible to perform HDR compositing (High Dynamic Range rendering).

[0102] On the other hand, it is also possible to change the timing of reading out the divided pixels within the pixel 20.

[0103] 8 and 9 are diagrams showing an example in which the readout timing is set to different timings within the pixel 20. In FIG.

[0104] In the solid-state imaging device 10, for example, at a certain reading time, the pixel 20 acquires an analog signal from the intensity of light in the complementary colors of Mg, Ye, and Cy, as shown in Fig. 8. At the next reading time, the solid-state imaging device 10 acquires an analog signal from the intensity of light in the primary colors of R, G, and B, as shown in Fig. 9. That is, the light receiving elements 200a and 200d in Fig. 4 may output analog signals to the floating diffusion at the same timing, and the light receiving elements 200b and 200c may output analog signals to the floating diffusion at the same timing.

[0105] In this way, the timing of reading can be shifted. For example, this reading may be performed for each frame. While the frame rate is lower than when reading all pixels 20 at once as described above, the solid-state imaging device 10 can acquire information on five or more colors (six colors in this embodiment).

[0106] Acquiring spectral information for five or more colors allows for more faithful reproduction of color matching functions, improving color reproducibility and enabling the creation of highly dramatic images. Using multiple colors improves the accuracy of light source estimation, scene determination, and object identification, making it possible to more vividly express the colors of specific objects / living organisms. For example, increasing the saturation of a certain food item can help create an image that retains its freshness.

[0107] In this way, object recognition and biometric recognition are possible based on the acquired spectral information. The solid-state imaging device 10 can also change control parameters for image reconstruction based on the results of biometric recognition, etc. The control parameters may be, for example, parameters related to digital signal processing, such as a filter kernel in image processing or a parameter in gamma correction, or parameters such as a weighting parameter for combining frames or the number of frames to be combined. Furthermore, the parameters may be parameters used when combining the first information and the second information.

[0108] In this embodiment, the acquired color information can be processed by the linear matrix unit 124, the luminance chroma signal generation unit 140, etc. This makes it possible to improve the saturation appropriately for an object, etc. In other words, in image creation using the solid-state imaging device 10 according to this embodiment, not only can the overall color saturation be increased, but more appropriate improvement in saturation can be achieved using information on five or more colors.

[0109] In the above, color information is acquired at the same timing or at different timings for all pixels 20, but this is not limited to this. For example, a frame may be divided into three frames, and in one frame, each pixel 20 may be read out at the same timing, and in the other two frames, each pixel 20 may be read out at different timings as shown in Figures 8 and 9. For example, when three frames are combined in a scene where bright and dark areas are mixed, image information for bright areas may be generated from information read out at different timings with reduced exposure, and image information for dark areas may be generated from information read out all at once with priority given to SNR (Signal to Noise Ratio).

[0110] In the above, Ye is used to improve color reproducibility, but this is not limiting. For example, the pixel 20 that acquires G information does not need to acquire Ye information. Also, as described above, emerald may be used instead of Ye or Cy. Also, the complementary color provided in the same pixel 20 as R may be Ye.

[0111] Furthermore, as mentioned above, it is desirable to obtain analog signals based on the intensities of five or more colors of light, but this is not limiting and information may be obtained using RGBW. For example, the complementary color division pixels in Figure 7 may be white pixels. However, for the reasons mentioned above, it is more desirable to use complementary color and primary color systems, which are similar colors.

[0112] By using the pixel 20 according to this embodiment, it becomes possible to acquire color information of five or more colors with high accuracy in a monocular camera. Furthermore, even when there is an object having a diagonal pattern, it becomes possible to acquire either RGB or CMY information across the pattern. According to this embodiment, the solid-state imaging device 10 can interpolate the acquired information and aggregate it into, for example, RGB information. Therefore, it becomes possible to appropriately reconstruct an image even for an object having a diagonal pattern.

[0113] In the above description, the light receiving elements 200a and 200d and the light receiving elements 200b and 200c in Fig. 4 are described as receiving light of the same color in each combination, but this is not limited to this. For example, the light receiving elements 200a and 200b and the light receiving elements 200c and 200d may receive light of the same color. As another example, the light receiving elements 200a and 200c and the light receiving elements 200b and 200d may receive light of the same color.

[0114] (Variation) In the above description, the light receiving element 200 provided in the pixel 20 acquires visible light information, but the present invention is not limited to this.

[0115] FIG. 10 is a diagram showing a non-limiting example of a pixel 20 according to a modification of the first embodiment. In the pixel 20 that receives R light, a divided pixel labeled IR is shown. This IR indicates a light receiving element 200 that receives infrared light. Hereinafter, when referring to color, IR may be included. In other words, IR may be considered a color similar to R.

[0116] 2 may be omitted. In order to omit the IRCF 112, a configuration may be adopted in which an IRCF is individually provided in each of the light receiving elements 200 other than the light receiving element 200 that receives IR color light.

[0117] 10, a pixel 20 may include a light receiving element 200 as a divided pixel that receives IR light, a color similar to R. By receiving IR light, it becomes possible to obtain information that is difficult to obtain within the visible light spectrum. As a result, for example, the estimation and recognition performance of the light source estimation unit 130 and the object recognition unit 132 can be improved.

[0118] (Second embodiment) Although the above has been described in terms of arrangements that are useful in monocular cameras, more diverse arrangements are possible when using compound cameras.

[0119] 11 is an example of a block diagram of a solid-state imaging device 10 according to this embodiment. The solid-state imaging device 10 includes a first light receiving section 100A and a second light receiving section 100B.

[0120] The first light receiving unit 100A includes, for example, a lens 110A, an IRCF 112A, and an image sensor 114A, and the second light receiving unit 100B includes, for example, a lens 110B, an IRCF 112B, and an image sensor 114B. For example, first information is acquired in the first light receiving unit 100A, and second information is acquired in the second light receiving unit 100B.

[0121] Each imaging element is provided with an A / D conversion unit and a clamp unit.

[0122] The image memory unit 134 stores the outputs from the clamp units 122A and 122B corresponding to the first light receiving unit 100A and the second light receiving unit 100B, respectively.

[0123] The two-lens synthesis unit 136 converts the information acquired from the image sensors 114A and 114B into appropriate data. For example, the two-lens synthesis unit 136 synthesizes information acquired by the separate image sensors 114A and 114B at the same time.

[0124] Various techniques can be used for this composition. For example, when the solid-state imaging device 10 is capturing a video, simple color information may be obtained as a thinned output based on data acquired from an imaging element receiving RGB light, and color correction may be performed using the thinned information based on data acquired from an imaging element receiving complementary color light. Alternatively, bright and dark areas may be determined, and an image may be generated based on information acquired from an RGB imaging element and a complementary color imaging element, respectively. Furthermore, the two-eye composition unit 136 may be configured to perform processing on the cloud rather than within the solid-state imaging device 10.

[0125] Based on the output synthesized by the two-lens synthesis unit 136, the solid-state imaging device 10 performs the same processing as that of each component in FIG. 2, thereby obtaining and outputting an image signal and a video signal.

[0126] 12 is a diagram showing an outline of the pixels 20 in each camera. The image sensor 114A is provided with light receiving elements that receive light of the three primary colors R, G, and B, and the image sensor 114B is provided with light receiving elements that receive light of the complementary colors Mg, Ye, and Cy. Unlike the configuration of the pixels 20 shown in FIG. 4 etc., each pixel may be configured to receive light of one color.

[0127] By concentrating light receiving elements 200 with similar light receiving sensitivities in each image sensor 114, it becomes possible to perform suitable exposure control. For example, when the light intensity in the surrounding environment is weak (dark), an image can be generated based on the output of image sensor 114B, which is a complementary color system with high light receiving sensitivity, and color correction can be performed using the output of image sensor 114A, which is a primary color system with low light receiving sensitivity. Conversely, when it is bright, color correction can be performed using the output of image sensor 114A, which is a complementary color system with high light receiving sensitivity, for an image based on the output of image sensor 114B, which is a primary color system with low light receiving sensitivity.

[0128] Exposure control can also be performed by setting different exposure amounts for primary colors and complementary colors, for example. This exposure control can be performed by changing the shutter time or the gain. By making these two controls variable, for example, in an RGB+CMY array, by setting a short accumulation time for RGB and a long accumulation time for CMY, it becomes possible to capture dark areas of the subject with a large exposure amount in CMY and bright areas with RGB. This makes it possible to obtain HDR images.

[0129] In the above, a short accumulation time can be interpreted as a low gain, and a long accumulation time can be interpreted as a high gain. In other words, control can be performed so that RGB is acquired at a low gain and CMY is acquired at a high gain. In this case, too, it is possible to acquire an image with a high dynamic range.

[0130] That is, the first information with low sensitivity and the second information with high sensitivity can be mutually interpolated. In addition to this, color correction may be appropriately performed in the two-eye synthesis unit 136 according to the characteristics of the color of the received light.

[0131] Fig. 13 is a diagram showing another arrangement of light receiving colors when a twin-lens camera is used. As shown in Fig. 13, the image sensor 114B that acquires the second information may be configured to acquire G light, which is one of the three primary colors.

[0132] When using twin image sensors 114, it is necessary to correct parallax when combining the outputs from the respective image sensors 114. The twin image synthesis unit 136, for example, combines the colors of the respective pixels 20 while performing pattern matching. When performing pattern matching in this way, if the color difference becomes large, the accuracy of matching may decrease.

[0133] In contrast to this, by configuring pixel 20 as shown in Figure 13, by mixing the G light reception information into the second information, it becomes possible to obtain the G light reception information as information common to the first information, thereby improving the accuracy of pattern matching.

[0134] Fig. 14 is a diagram showing yet another arrangement of light receiving colors when a twin-lens camera is used. As shown in Fig. 14, the image sensor 114A that acquires the first information may be configured to acquire W (white) light in addition to the three primary colors.

[0135] With this configuration, it is possible to acquire information with high sensitivity for the RGB color light acquired as the first information. Note that the W pixel 20 may be provided as the pixel 20 on the second information side, i.e., the image sensor 114B side.

[0136] Fig. 15 is a diagram showing yet another arrangement of light receiving colors when a twin-lens camera is used. As shown in Fig. 15, the image sensor 114A that acquires the first information may be configured to acquire IR (infrared) light in addition to the three primary colors.

[0137] Receiving IR light can improve the accuracy of light source estimation, object recognition, etc. As in the above-described embodiment, it is desirable that the image sensor 114 that receives IR light is not provided with the IRCF 112. In this case, a filter that removes or absorbs IR light may be individually implemented in the pixels 20 other than the pixels 20 that receive IR light.

[0138] The solid-state imaging device 10 shown in the block diagram of Fig. 11 can be configured for the imaging element 114 configured as shown in Fig. 12 to Fig. 15. As described above, when the imaging element 114 receives IR light, the IRCF 112A is not inserted.

[0139] Compared to the block diagram of FIG. 2 for a monocular camera, a configuration for combining information from two cameras has been added. As described above, the two-camera combining unit 136 performs, for example, pattern matching and then combines the outputs from the two image sensors 114A and 114B. In the configurations of FIGS. 12 to 15, outputs of six to seven colors are appropriately combined. At least one of these multiple colors may be used to perform object recognition or the like using a trained neural network model (not shown). This neural network model may be optimized, for example, by a deep learning technique.

[0140] 12 to 15 are given as examples and are not intended to be limiting. Arrays other than those based on the Bayer array shown in these figures may also be used as long as they can appropriately achieve the effects described above.

[0141] As described above, according to this embodiment, even in the case of a twin-lens system, it is possible to appropriately reconstruct an image output from the solid-state imaging device 10 by receiving light of five or more colors.

[0142] This embodiment can also be applied to a compound eye with three or more eyes.

[0143] In the case of a compound eye, the size of each optical system can be set to any size. The resolution of the image sensor 114 and the number of pixels 20 can also be set to any value for each image sensor 114. In other words, the first information and the second information can be acquired at different resolutions.

[0144] In this way, by changing the size, resolution, etc. of the optical system in the compound eye, it becomes possible to obtain images with different resolutions, etc. from the three primary color system and the complementary color system, respectively, and by using this information mutually, it is possible to achieve higher resolution or improved color reproducibility.

[0145] By setting different resolutions for the first information and the second information, the solid-state imaging device 10 may capture still images and moving images in separate imaging modes. For example, in moving image mode, the complementary color system may be driven normally to reduce power consumption during readout so that the three primary color system is used as an auxiliary signal for the complementary color system, and color information acquired from the three primary color system may be transmitted to the complementary color system to assist in image creation. For example, the three primary color system may be used as an auxiliary pixel for pixel addition (pixel binning) in the complementary color system. This can be similarly applied even when segmented pixels are used.

[0146] Furthermore, composition may be performed at different times and in different blocks in still image mode and video mode. For example, if real-time processing of video is difficult due to processing speed issues, composition may be performed over time after capture. In this case, image reconstruction processing for video may be performed in the signal processing unit 106 and image processing unit 108, or the video may be uploaded to the cloud and image reconstruction processing may be performed on a high-performance server or the like.

[0147] Furthermore, the image sensor 114 may be configured in a compound eye with two or more eyes using the same divided pixels as in the above-described single-eye embodiment. In this case, the applications of the multiple cameras can be changed. For example, one camera can be used as a wide-angle camera and the other as a normal camera, and the cameras can be used for different purposes. Of course, by using data acquired from these multiple types of cameras, it is possible to acquire high-resolution images and improve color reproducibility.

[0148] (Third embodiment) In the above-described embodiment, a color filter or an organic photoelectric conversion film is provided for each color, but this is not limiting. For example, in a monocular camera, light of three primary colors and complementary colors may be received using an organic photoelectric conversion film and a photodiode.

[0149] 16 shows an example of a pixel 20 that uses an organic photoelectric conversion film and a photodiode as a light receiving element. The pixel 20R_Cy is configured to include an organic photoelectric conversion film 206R that receives R light and a photodiode 208Cy that receives Cy light. Although not shown, an optical system such as an on-chip lens may be provided on the side of the pixel 20R_Cy where external light is incident.

[0150] In the pixel 20R_Cy having this configuration, the organic photoelectric conversion film 206R receives R light and outputs an analog signal based on the intensity of the R light. The light passing through the organic photoelectric conversion film 206R becomes Cy light, which is a complementary color with the R component removed, and is input to the photodiode 208Cy. Therefore, the photodiode 208Cy receives Cy light. As a result, the photodiode 208Cy receives Cy light passing through the organic photoelectric conversion film 206R and outputs an analog signal corresponding to this Cy light.

[0151] Similarly, a pixel 20G_Mg may be configured by combining an organic photoelectric conversion film 206G that receives G light and a photodiode 208Mg that receives Mg light. Also, a pixel 20B_Ye may be configured by combining an organic photoelectric conversion film 206B that receives B light and a photodiode 208Ye that receives Ye light.

[0152] 17 is a diagram showing another example of the pixel configuration according to this embodiment. The pixel 20Cy_R includes an organic photoelectric conversion film 206Cy that receives Cy light and a photodiode 208R that receives R light.

[0153] The pixel 20Cy_R receives light of Cy through the organic photoelectric conversion film 206Cy and outputs an analog signal based on the intensity of the light of Cy. The light that passes through the organic photoelectric conversion film 206Cy becomes R light, which is a complementary color with the Cy component removed, and is input to the photodiode 208R. Therefore, the photodiode 208R receives R light. As a result, the photodiode 208R receives R light that passes through the organic photoelectric conversion film 206Cy and outputs an analog signal corresponding to this R light.

[0154] 16, a pixel 20Mg_G may be configured by combining an organic photoelectric conversion film 206Mg that receives Mg light and a photodiode 208G that receives G light. Also, a pixel 20Ye_B may be configured by combining an organic photoelectric conversion film 206Ye that receives Ye light and a photodiode 208B that receives B light.

[0155] In the examples of FIGS. 16 and 17, the filter 202 is not provided on the photodiode 208, but an appropriate filter 202 may be provided between the organic photoelectric conversion film 206 and the photodiode 208.

[0156] As described above, according to this embodiment, it is possible for pixels arranged at the same position to appropriately receive light of complementary colors. With this configuration, it is possible for the monocular camera to appropriately receive light of the first information and the second information for each pixel 20. In this case, since there is no misalignment in the light receiving positions between the first information and the second information, it is possible to appropriately improve the color reproducibility from each piece of information in image reconstruction without performing pattern matching or the like.

[0157] (Fourth embodiment) In the second and third embodiments, information on the three primary colors and their complementary colors is used, but as mentioned above, the color combinations are not limited to these. For example, the second information may be the color Ye or emerald. By receiving light of emerald, it is possible to improve the accuracy of generating negative components of the color matching function. Furthermore, light of W may be received.

[0158] (Fifth embodiment) Various applications of the pixels and divided pixels described in the above embodiments will be described below. In the following examples, some color combinations are listed as non-limiting examples.

[0159] 18 is a diagram showing an example of dividing the pixel 20. Each pixel 20 receives light of R, G, B, Mg, Ye, and Cy.

[0160] 18, the pixel 20 may be divided into two regions in a predetermined direction, and each divided region is provided with a light receiving element 200, such as a photodiode.

[0161] Fig. 19 is a diagram showing an example of on-chip lens formation in the example of Fig. 18. By dividing in this way and providing an on-chip lens for each pixel as shown in Fig. 19, it is also possible to acquire parallax information for each pixel.

[0162] Fig. 20 is a diagram showing an example of the formation of on-chip lenses in the example of Fig. 18. An elliptical on-chip lens 210 may be provided for each divided pixel. In this case, it is possible to improve the resolution.

[0163] Fig. 21 is a diagram showing another example of dividing a pixel 20. As shown in Fig. 21, a light receiving element 200 that receives W light may be provided in the same pixel that includes a G light receiving element 200 (divided pixel). Furthermore, instead of W, the light receiving element 200 may be a light receiving element of a color such as emerald, as described above. In this way, some divided pixels may be combined in a manner different from the example shown in Fig. 7, etc.

[0164] Fig. 22 is a diagram showing another example of dividing the pixels 20. As shown in Fig. 22, the pixels 20 that receive G light and the pixels 20 that receive Ye light may be separated. By adopting such a configuration, it becomes possible for the pixels 20 that receive G and Ye light to acquire analog signals at the same timing, thereby improving the frame rate.

[0165] FIG. 23 is a diagram showing another example of dividing a pixel 20. As shown in FIG. 23, a configuration may be adopted in which complementary colors are provided in different pixels 20. In this case, for example, as shown in the figure, a configuration may be adopted in which a pixel 20 has a divided pixel of R and Ye, a pixel 20 has a divided pixel of B and Ye, a pixel 20 has a divided pixel of G and Mg, and a pixel 20 has a divided pixel of G and Cy, so that many divided pixels of G and Ye are arranged. By forming pixels in this manner, every pixel 20 has a divided pixel that receives green light, which has a high light intensity received by the human eye, thereby improving color reproducibility, etc.

[0166] Fig. 24 shows an example of the arrangement of on-chip lenses in the case of Fig. 22. As shown in Fig. 24, when divided pixels that receive light of different colors are provided within the pixel 20, an on-chip lens 210 of a size applicable to the divided pixel may be provided. On the other hand, when light of the same color is received in the pixel 20, a configuration may be adopted in which an on-chip lens 210 that covers the pixel 20 is provided.

[0167] Of course, at least one of G and Ye may have a configuration including two elliptical on-chip lenses as shown in FIG.

[0168] By using on-chip lenses with different shapes in this way, it is possible to obtain a phase difference from the analog signal obtained in the pixel 20.

[0169] FIG. 25 is a diagram showing another example of dividing a pixel 20. As shown in FIG. 25, the pixel 20 may be divided into 3 × 3 divided pixels. In this case, the three primary colors with low sensitivity may be assigned to five pixels, and the complementary colors may be assigned to the remaining four pixels. By allocating in this manner, it is possible to achieve a certain degree of balance in sensitivity after adding pixel values. Furthermore, the division is not limited to 3 × 3, and the pixel may be divided into more divided pixels, such as 4 × 4 or 5 × 5.

[0170] The addition of pixel values ​​may be performed at the timing when an analog signal is acquired by floating diffusion, as described in the above embodiment, or, as another example, may be realized by an analog circuit or a digital circuit after A / D conversion.

[0171] When using split pixels, the output values ​​from the split pixels acquired for each frame may be added together. In such cases, the number of pixels to be added may be changed for each frame. This can also be applied to each pixel 20 instead of each split pixel. Controlling in this way also leads to the generation of images with higher reproducibility based on the frame rate.

[0172] (Sixth embodiment) In this embodiment, a semiconductor chip including an image sensor 114 will be briefly described.

[0173] Fig. 26 is a diagram showing an example of a substrate having an image sensor 114. The substrate 30 has a pixel region 300, a control circuit 302, and a logic circuit 304. As shown in Fig. 26, the pixel region 300, the control circuit 302, and the logic circuit 304 may be configured to be provided on the same substrate 30.

[0174] The pixel region 300 is a region in which, for example, the pixels 20 in each of the above-described embodiments are provided. Of the signal processing unit 106, A / D conversion and the like may be provided appropriately in this pixel region 300, or may be provided in another region (not shown) on the substrate 30. The control circuit 302 includes the control unit 104 in FIG. 1. The logic circuit includes, for example, a circuit after A / D conversion of the signal processing unit 106 and an image processing unit 108. Furthermore, at least a portion of the signal processing unit 106 and the image processing unit 108 may be implemented not on this chip but on a separate signal processing chip provided in a location separate from the substrate 30, or may be implemented in a separate processor, etc.

[0175] 27 is a diagram showing another example of a substrate including an image sensor 114. The substrates include a first substrate 32 and a second substrate 34. The first substrate 32 and the second substrate 34 have a stacked structure, and can transmit and receive signals to and from each other appropriately via connecting portions such as via holes. For example, the first substrate 32 may include a pixel region 300 and a control circuit 302, and the second substrate 34 may include a logic circuit 304.

[0176] 28 is a diagram showing another example of a substrate including an image sensor 114. The substrates include a first substrate 32 and a second substrate 34. The first substrate 32 and the second substrate 34 have a stacked structure, and can transmit and receive signals to and from each other appropriately via connecting portions such as via holes. For example, the first substrate 32 may include a pixel region 300, and the second substrate 34 may include a control circuit 302 and a logic circuit 304.

[0177] 27 and 28, the stacked substrates may be connected to each other by via holes as described above, or may be connected by a method such as a micro-dump, etc. These substrates can be stacked by any method such as CoC (Chip on Chip), CoW (Chip on Wafer), or WoW (Wafer on Wafer).

[0178] Seventh embodiment 29 is a diagram showing an example of an electronic device 1. As shown in this diagram, the electronic device 1 may be, for example, a smartphone or a tablet terminal. In this case, when the electronic device 1 includes a display as a display unit that displays images or videos, a solid-state imaging device 10 that captures an image of light passing through the display may be provided so as to overlap the display.

[0179] By configuring the pixel 20 as described in each of the above-described embodiments, it is possible to appropriately receive light transmitted through the display. For example, a material that absorbs a lot of blue light, such as polyimide, may be used for the display of a smartphone or the like. Even in such a case, by providing the solid-state imaging device 10 capable of receiving light of five or more colors including at least the three primary colors and their complementary colors, it is possible to receive light having an appropriate spectrum, thereby improving the color reproducibility of the image.

[0180] Although the solid-state imaging device 10 is provided below the display in the above description, this is not limiting. For example, a smartphone or the like may be provided with the solid-state imaging device 10 described in each of the above embodiments as a so-called outer camera, independent of the display.

[0181] FIG. 30 is a diagram showing another example of the electronic device 1. The electronic device 1 may be, for example, an xR terminal such as VR (Virtual Reality), AR (Augmented Reality), or MR (Mixed Reality). The solid-state imaging device 10 may be used as a camera mounted on such an xR terminal. Using the solid-state imaging device 10 as a camera can improve sensitivity and color reproducibility, thereby further increasing the immersive experience felt by the user.

[0182] In addition, object identification can also be used in a UI (User Interface) that recognizes objects that the user is interested in.

[0183] Furthermore, it can also be used for healthcare sensing in wearable devices. For example, vital information such as blood oxygen saturation can be obtained by looking at the R and IR sensitivity ratio. The obtained vital information can be saved as a database and used for healthcare, etc., along with accurate daily facial color. As another example, a neural network model can be updated each time vital information is obtained, and this neural network model can be used for healthcare, etc. This application can also be applied to medical devices such as capsule endoscopes.

[0184] According to each of the above-mentioned embodiments, the solid-state imaging device 10 can achieve high sensitivity by using complementary colors and high color reproducibility by combining complementary colors and primary colors. In addition, in some embodiments, it is also possible to combine primary colors and complementary colors without misalignment of the centers of gravity.

[0185] For example, when the electronic device 1 is a smartphone or the like and images are shared on social media or the like, not only faithful color reproduction but also color rendition may be required. Even in such a case, according to the embodiment of the present disclosure, it is possible to increase the degree of freedom in color adjustment and change the color creation depending on the background recognition result.

[0186] Furthermore, in systems and devices that have a camera under a display, as shown in Figure 29, the amount of light may be blocked by the display panel, etc., and sensitivity to blue light in particular may be significantly reduced. However, according to the embodiments of the present disclosure, these problems can also be solved.

[0187] Furthermore, by using an organic photoelectric conversion film, primary colors and complementary colors can be obtained in the vertical direction (for example, the height direction of the drawings in Figures 16 and 17), which makes it possible to suppress adverse effects such as degradation of resolution caused by adding complementary colors.

[0188] (Eighth embodiment) In the above embodiment, the reproducibility of colors and the like has been described, but this can also be applied to improve the accuracy of light source estimation and object recognition, and correct image flicker.

[0189] 31 is a block diagram showing another example of the solid-state imaging device 10. The solid-state imaging device 10 further includes a pre-linear matrix unit 160, an RGB rearrangement unit 162, a sensor output unit 164, and a statistical value acquisition unit 166 in addition to the configuration shown in FIG.

[0190] Fig. 32 is a diagram showing the acquired RGB information and CMY information in the form of spectra. In this embodiment, for example, a configuration is used in which light receiving elements receive light of six colors, RGB+CMY, as shown in Fig. 32, but the number of colors is not limited to these six.

[0191] As in the above-described embodiment, the light receiving element receives light and outputs an analog signal based on the intensity, and this analog signal is converted into a digital signal by the A / D conversion unit 120. The clamp unit 122 corrects the black level.

[0192] The pre-linear matrix unit 160 mixes the first information and the second information based on the information output from the clamp unit 122, and obtains, for example, information on RGB pixel values ​​that differ from the RGB information received by the light receiving elements. In this way, the pre-linear matrix unit 160 recalculates the RGB information through a predetermined arithmetic process.

[0193] The RGB rearrangement unit 162 rearranges the signals output by the pre-linear matrix unit 160 .

[0194] FIG. 33 is a diagram showing color matching functions in the RGB color system. The RGB information output by the pre-linear matrix unit 160 and the RGB rearrangement unit 162 has improved SNR or color reproducibility compared to the first information and the second information. Alternatively, this RGB information may be RGB information as shown in FIG. 33, which is converted into spectral output that approximates color matching functions that are considered ideal for color reproducibility using complementary color information. That is, prior to other processing, the pre-linear matrix unit 160 generates RGB color information with improved SNR and the like from color information of five or more colors. Then, based on this information, the solid-state imaging device 10 performs subsequent processing.

[0195] For example, a signal processor downstream of the image sensor 114 is generally implemented to handle information on the three colors RGB. In such cases, outputting a digital signal as an RGB signal, as in this embodiment, makes it possible to support many downstream signal processors. Furthermore, as described above, this RGB signal is a signal with improved SNR characteristics, and therefore it is possible to use a general signal processing circuit while utilizing data with better color reproducibility.

[0196] The sensor output unit 164 outputs the image signal rearranged by the RGB rearrangement unit 162 .

[0197] The statistical value acquisition unit 166 acquires statistical values ​​of color information for five or more colors. The acquired information is used, for example, to improve the accuracy of light source estimation and object recognition. For example, the light source estimation unit 130 can reproduce the spectrum of the light source with higher accuracy based on the statistical values ​​of five or more colors, and acquire information such as the environment based on the reproduced spectrum. The object recognition unit 132 can also improve the recognition accuracy of objects, living organisms, etc. based on these statistical values.

[0198] Specifically, the statistical value acquisition unit 166 divides the field of view into regions and outputs the average value for each color for each region. For example, it outputs the average value of pixel values ​​within a specific region. The information obtained in this way is detailed color information using information on six colors, so it can be used for light source estimation and color correction. Color correction refers to the general process of adjusting colors when creating an optimal image in a downstream processor, for example.

[0199] Although statistical values ​​of six colors are acquired, RGB information can also be output from the RGB rearrangement unit 162 as normal imaging data, so statistical values ​​of only CMY may be output.

[0200] Furthermore, although the region division has been described as including multiple pixels within a region, this is not limited thereto, and a region may be one pixel. For example, while RGB information is converted into RGB information that has been subjected to processing such as SNR by pre-linear matrix processing, CMY is lost at this stage, so the CMY information may be output by the statistical value acquisition unit 166, and the information may be transmitted to the light source estimation unit 130 or the like at a subsequent stage.

[0201] The outputs of the sensor output unit 164 and the statistical value acquisition unit 166 are output as output information of the image sensor 114 to, for example, the light source estimation unit 130. Subsequent processing may be the same as in the above-described embodiment. As another example, white balance adjustment and linear matrix control may be performed after light source estimation. Furthermore, the light source estimation, white balance adjustment, and linear matrix processing may also be performed within the image sensor 114.

[0202] As described above, according to this embodiment, it is possible to output more information to a downstream processor than in the previous embodiment. By using more information, it becomes easier to improve the SNR and add processing to improve color reproduction. In this way, it is possible to increase the SNR, improve the accuracy of light source estimation, and improve the accuracy of object and living body recognition.

[0203] In the solid-state imaging device 10 having this configuration, the processes of acquiring the first information and acquiring the second information shown in Figures 8 and 9 are executed, and information of five or more colors is converted into RGB information with a high SNR. The control unit 104 can also change the shutter speed used in acquiring the first information and the second information. By changing the shutter speed, flicker in the RGB image acquired by the pre-linear matrix unit 160 can be corrected.

[0204] For example, in Figures 8 and 9, if there is a light source or the like that lights up periodically, and this period is the same as or close to the period for switching between the state of Figure 8 and the state of Figure 9, flicker will occur throughout the image. Furthermore, even when using a general CMOS (Complementary MOS) element or the like, flicker may occur due to causes similar to rolling shutter distortion. Even in such cases, by acquiring information at different timings and different shutter speeds in the same pixel 20, it is possible to shift these timings. As a result, it is possible to suppress flicker in the image, such as flicker caused by a light source such as an LED.

[0205] (Ninth embodiment) 29 is an under-display camera (UDC) in which a camera is disposed under a display, and a more detailed description will be given of a case in which the solid-state imaging device 10 is provided in the electronic device 1. In such a case, the accuracy of imaging can be improved by utilizing the compound-eye camera in the above-described embodiment.

[0206] FIG. 34 is a diagram showing the arrangement of the electronic device 1 and the imaging elements 114 of the solid-state imaging device 10 according to this embodiment. The solid-state imaging device 10 includes imaging elements 114A and 114B. When capturing images for use in VR (Virtual Reality) / AR (Augmented Reality), for example, the distance between the centers of these imaging elements is preferably 50 to 80 mm, which allows for maintaining the same parallax as the human eye. However, this is not limited to this, and the imaging elements may be arranged closer or farther apart. For example, the distance between the nearest pixels of each imaging element may be 10 mm or less.

[0207] 35 shows an example of the color arrangement of pixels 20 in each of image sensors 114A and 114B. As described above, solid-state imaging device 10 in this embodiment includes image sensor 114A that acquires first information with low sensitivity, and image sensor 114B that acquires second information with high sensitivity. Hereinafter, pixels 20 included in image sensor 114A will also be referred to as a first pixel group, and pixels 20 included in image sensor 114B will also be referred to as a second pixel group.

[0208] The pixels 20 belonging to the first pixel group are arranged in, for example, a Bayer array of RGB, and the pixels 20 belonging to the second pixel group are arranged in, for example, an RGW array.

[0209] In the case of a still image, highly accurate color reproduction can be achieved by acquiring the first information and the second information from all pixels 20 and performing demosaic and color synthesis processing. On the other hand, in the case of a moving image, the information acquired from the image sensor 114A is compressed into an RGB Bayer array as indicated by the arrow. By adding the color information of the compressed low-sensitivity first information to the high-sensitivity second information, color reproducibility can be improved while maintaining a high frame rate.

[0210] In this case, the amount of information acquired as the first information can be reduced, which makes it possible to reduce power consumption compared to acquiring information at the same resolution as a still image. As an example, the sensitivity of at least one of the R, G, and W pixels 20 in the second pixel group may be set higher than the sensitivity of the G pixels 20 in the first pixel group. By setting it in this manner, the sensitivity of the second information can be made higher than the sensitivity of the first information.

[0211] Furthermore, instead of the sensitivity to G, the sensitivity may be set higher than the sensitivity to R or the sensitivity to B. For example, as will be described later, by providing the second pixel group with pixels 20 of Cy or W having a higher sensitivity than the sensitivity to B in the first pixel group, it is possible to improve the sensitivity to blue even when the image sensor 114 is provided under a display such as that shown in FIG.

[0212] In this way, different synthesis methods can be used when shooting moving images and when shooting still images, which solves the problems of power consumption when shooting moving images and real-time performance.

[0213] 34, both image sensors 14 (sensors) are the same size, but the sensor on the higher sensitivity side, i.e., the sensor on the image sensor 114B side to which the second pixel group belongs, may be larger. For example, if changing the sensor sizes in this way makes it difficult to use both sensors, the control of the solid-state image sensor 10 may be switched so that only the sensor on the higher sensitivity side is used.

[0214] In the signal processing unit 106 or the image processing unit 108, the first information acquired from the first pixel group is added (weighted addition) to the second information acquired from the second pixel group to improve color reproducibility. If the first pixel group has a lower resolution than the second pixel group, demosaic processing may be performed based on the analog signal acquired from the first pixel group to generate the first information, and this first information and second information may be added. This applies to both still images and moving images.

[0215] The same is true when reducing the amount of information in the first pixel group in a moving image. In this case, too, demosaicing the information obtained from the first pixel group allows interpolation of the second information, enabling the reconstruction of an image with improved resolution and color reproducibility. Furthermore, adding the information reduces the effect of noise, contributing to an improvement in SNR.

[0216] In the above, the high sensitivity side has an RGW arrangement, but it can also have a complementary color system as in the previous embodiment. For example, it can have the same configuration as the image sensor 114B shown in Figures 12 to 15.

[0217] 36 is a diagram showing another example of the pixel arrangement in the image sensor 114B. As shown in the figure, the second pixel group may have a complementary color system +W form. As in the above, the solid-state image sensor 10 adds first information obtained by appropriately converting the signal acquired from the first pixel group to second information acquired from the second pixel group and outputs the result, thereby achieving at least one of the effects of color reproducibility, high resolution, and high SNR.

[0218] The IR component may be acquired based on the information acquired from the W pixel 20. As in the above, acquiring the IR component can also contribute to improving the accuracy of light source estimation and object recognition.

[0219] Furthermore, similar to the above-described embodiment, the timing of combining the first information and the second information can be changed between when shooting moving images and when shooting still images. Similarly, the block that performs processing when shooting moving images and the block that performs processing when shooting still images can be configured differently. As described above, when shooting moving images, data can be acquired and reconstructed in a higher-performance server device or the like. In other words, a configuration can be adopted in which post-processing is performed in different system blocks when shooting moving images and still images.

[0220] In this embodiment, too, statistical values ​​of color information may be obtained, as shown in Fig. 31. These statistical values ​​may be used to correct the color information of the second information. As another example, color statistics may be obtained from signals obtained from the first pixel group, and the second information obtained from the second pixel group may be corrected based on these statistical values.

[0221] As in the previous embodiment, the pixels may be color-coded as divided pixels rather than by pixel 20. The divided pixels may be applied to only one of the first pixel group and the second pixel group, or may be applied to both. In other words, the divided pixels may be configured so that at least one of them receives W light.

[0222] Also, one pixel or divided pixel in the array may be configured to exclusively capture IR light. The use of IR light is the same as described above. In this case, it is desirable that the image sensor 114 that captures IR light does not have an IRCF, as in the above-described embodiment. Furthermore, elements other than the pixel 20 or divided pixel that captures IR light may be configured to have a separate IRCF.

[0223] The optical systems of the first pixel group and the second pixel group may be different. For example, by making the optical size of the second pixel group larger than that of the first pixel group, it becomes possible to obtain more detailed signals on the high sensitivity side and to drive the device at higher speeds and with less power consumption on the low sensitivity side.

[0224] (Application Examples of the Electronic Device 1 or Solid-State Imaging Device 10 According to the Present Disclosure) (First application example) The electronic device 1 or solid-state imaging device 10 according to the present disclosure can be used for various purposes. Figures 37A and 37B are diagrams showing the internal configuration of a vehicle 360, which is a first application example of an electronic device 1 including a solid-state imaging device 10 according to the present disclosure. Figure 37A is a diagram showing the interior of the vehicle 360 ​​from the rear to the front of the vehicle 360, and Figure 37B is a diagram showing the interior of the vehicle 360 ​​from diagonally rear to diagonally front of the vehicle 360.

[0225] Vehicle 360 ​​of FIGS. 37A and 37B includes a center display 361, a console display 362, a head-up display 363, a digital rearview mirror 364, a steering wheel display 365, and a rear entertainment display 366.

[0226] Center display 361 is disposed on dashboard 367 in a position facing driver's seat 368 and passenger seat 369. FIG. 37 shows an example of center display 361 having a horizontally elongated shape extending from driver's seat 368 to passenger seat 369, but the screen size and location of center display 361 are arbitrary. Center display 361 can display information detected by various sensors. As a specific example, center display 361 can display an image captured by an image sensor, a distance image to obstacles in front of or on the side of the vehicle measured by a ToF sensor, and the body temperature of a passenger detected by an infrared sensor. Center display 361 can be used to display, for example, at least one of safety-related information, operation-related information, a life log, health-related information, authentication / identification-related information, and entertainment-related information.

[0227] The safety-related information includes information such as detection of drowsiness, distraction, child mischief, whether a seatbelt is fastened, and whether a passenger has been abandoned. This information is detected, for example, by a sensor placed on the back side of the center display 361. The operation-related information is obtained by detecting gestures related to passenger operations using a sensor. The detected gestures may include operations of various equipment within the vehicle 360. For example, operations of the air conditioning system, navigation system, AV system, lighting system, etc. are detected. The life log includes life logs of all passengers. For example, the life log includes a record of each passenger's actions while on board. By acquiring and storing the life log, the passenger's condition at the time of an accident can be confirmed. The health-related information is obtained by detecting the passenger's body temperature using a temperature sensor and inferring the passenger's health condition based on the detected body temperature. Alternatively, the passenger's face may be captured using an image sensor and the passenger's health condition may be inferred from the facial expression in the captured image. Furthermore, the passenger may be spoken to by an automated voice and the passenger's health condition may be inferred based on the passenger's responses. Authentication / identification-related information includes functions such as a keyless entry function that uses sensors to perform facial recognition, and a function that automatically adjusts seat height and position using facial recognition. Entertainment-related information includes functions such as a function that uses sensors to detect operation information of AV equipment by passengers, and a function that uses sensors to recognize passengers' faces and provides content appropriate for the passengers via AV equipment.

[0228] Console display 362 can be used to display, for example, life log information. Console display 362 is disposed near a shift lever 371 on a center console 370 between a driver's seat 368 and a passenger seat 369. Information detected by various sensors can also be displayed on console display 362. Furthermore, console display 362 may display an image of the vehicle's surroundings captured by an image sensor, or an image showing the distance to an obstacle around the vehicle.

[0229] The head-up display 363 is virtually displayed behind a windshield 372 in front of the driver's seat 368. The head-up display 363 can be used to display, for example, at least one of safety-related information, operation-related information, a life log, health-related information, authentication / identification-related information, and entertainment-related information. Because the head-up display 363 is often virtually disposed in front of the driver's seat 368, it is suitable for displaying information directly related to the operation of the vehicle 360, such as the speed of the vehicle 360 ​​and the remaining fuel (battery) level.

[0230] The digital rearview mirror 364 can not only display the rear of the vehicle 360, but also the status of passengers in the rear seats. Therefore, by placing a sensor on the back side of the digital rearview mirror 364, it can be used to display life log information, for example.

[0231] The steering wheel display 365 is disposed near the center of the steering wheel 373 of the vehicle 360. The steering wheel display 365 can be used to display, for example, at least one of safety-related information, operation-related information, a life log, health-related information, authentication / identification-related information, and entertainment-related information. In particular, because the steering wheel display 365 is located near the driver's hands, it is suitable for displaying life log information such as the driver's body temperature, and for displaying information related to the operation of AV equipment, air conditioning equipment, etc.

[0232] The rear entertainment display 366 is attached to the back side of the driver's seat 368 and the passenger seat 369, and is intended for viewing by rear seat passengers. The rear entertainment display 366 can be used to display, for example, at least one of safety-related information, operation-related information, a life log, health-related information, authentication / identification-related information, and entertainment-related information. In particular, since the rear entertainment display 366 is located directly in front of the rear seat passengers, information related to the rear seat passengers is displayed on the rear entertainment display 366. For example, the rear entertainment display 366 may display information related to the operation of the AV equipment or air conditioning equipment, or may display the results of measuring the body temperature of the rear seat passengers using a temperature sensor.

[0233] As described above, by arranging a sensor on the back side of the electronic device 1, the distance to surrounding objects can be measured. Optical distance measurement methods can be broadly divided into passive and active types. Passive methods measure distance by receiving light from an object without projecting light from the sensor onto the object. Passive methods include the lens focusing method, the stereo method, and the monocular vision method. Active methods measure distance by projecting light onto an object and receiving the light reflected from the object with a sensor. Active methods include the optical radar method, the active stereo method, the photometric stereo method, the moire topography method, and the interferometry method. The electronic device 1 according to the present disclosure can be applied to any of these distance measurement methods. By using a sensor arranged on the back side of the electronic device 1 according to the present disclosure, the above-mentioned passive or active distance measurement can be performed.

[0234] (Second application example) The electronic device 1 including the solid-state imaging device 10 according to the present disclosure can be applied not only to various displays used in vehicles but also to displays mounted on various electronic devices.

[0235] Fig. 38A is a front view of digital camera 310, which is a second application example of electronic device 1, and Fig. 38B is a rear view of digital camera 310. Digital camera 310 in Figs. 38A and 38B shows an example of a single-lens reflex camera with interchangeable lens 121, but the present invention is also applicable to cameras in which lens 121 is not interchangeable.

[0236] 38A and 38B, when the photographer holds the grip 313 of the camera body 311, looks through the electronic viewfinder 315, decides on the composition of the shot, adjusts the focus, and presses the shutter, the photographed data is saved in the camera's internal memory. As shown in Fig. 38B, the rear side of the camera is provided with a monitor screen 316 that displays photographed data, live images, etc., and the electronic viewfinder 315. In addition, a sub-screen that displays setting information such as shutter speed and exposure value may be provided on the top surface of the camera.

[0237] By arranging the sensor on the back side of the monitor screen 316, electronic viewfinder 315, sub-screen, etc. used in the camera, it can be used as the electronic device 1 according to the present disclosure.

[0238] (Third application example) The electronic device 1 according to the present disclosure can also be applied to a head-mounted display (hereinafter referred to as an HMD). The HMD can be used for VR, AR, MR (Mixed Reality), SR (Substitutional Reality), or the like.

[0239] Fig. 39A is an external view of an HMD 320, which is a third application example of the electronic device 1. The HMD 320 in Fig. 39A has a mounting member 322 for being worn over a person's eyes. This mounting member 322 is secured by hooking it onto a person's ear, for example. A display device 321 is provided inside the HMD 320, and the wearer of the HMD 320 can view 3D images and the like on this display device 321. The HMD 320 is equipped with, for example, a wireless communication function and an acceleration sensor, and can switch the 3D images and the like displayed on the display device 321 according to the posture, gestures, and the like of the wearer.

[0240] Alternatively, a camera may be provided in the HMD 320 to capture an image of the wearer's surroundings, and an image obtained by combining the image captured by the camera with an image generated by a computer may be displayed on the display device 321. For example, a camera may be placed on the back side of the display device 321, which is viewed by the wearer of the HMD 320, to capture an image of the area around the wearer's eyes, and the captured image may be displayed on another display provided on the outer surface of the HMD 320, allowing people around the wearer to grasp the wearer's facial expressions and eye movements in real time.

[0241] Various types of HMD 320 are possible. For example, as shown in FIG. 39B , the electronic device 1 according to the present disclosure can also be applied to smart glasses 340 that display various information on glasses 344. The smart glasses 340 in FIG. 39B include a main body 341, an arm 342, and a lens barrel 343. The main body 341 is connected to the arm 342. The main body 341 is detachable from the glasses 344. The main body 341 incorporates a control board and a display unit for controlling the operation of the smart glasses 340. The main body 341 and the lens barrel are connected to each other via the arm 342. The lens barrel 343 emits image light emitted from the main body 341 via the arm 342 toward lenses 345 of the glasses 344. This image light enters the human eye through the lens 345. A person wearing the smart glasses 340 of FIG. 39B can visually recognize not only the surrounding situation but also various pieces of information emitted from the lens barrel portion 343, just like with regular glasses.

[0242] (Fourth Application Example) The electronic device 1 according to the present disclosure can also be applied to a television device (hereinafter referred to as a TV). Recent TVs tend to have as small a frame as possible from the viewpoints of miniaturization and design. For this reason, if a camera for capturing images of viewers is installed in the TV, it is desirable to place the camera on the back side of the display panel 331 of the TV.

[0243] FIG. 40 is an external view of a TV 330 which is a fourth application example of the electronic device 1. The TV 330 in FIG. 40 has a minimized frame, with almost the entire front side being the display area. The TV 330 has a built-in sensor such as a camera for capturing images of the viewer. The sensor in FIG. 40 is disposed on the back side of a portion of the display panel 331 (for example, the area enclosed by the dashed line). The sensor may be an image sensor module, or various sensors such as a face authentication sensor, a distance measurement sensor, or a temperature sensor may be used, and multiple types of sensors may be disposed on the back side of the display panel 331 of the TV 330.

[0244] As described above, according to the electronic device 1 of the present disclosure, the image sensor module can be placed on top of the back side of the display panel 331, which eliminates the need to place a camera or the like in the frame, allowing the TV 330 to be made smaller, and there is no risk of the frame compromising the design.

[0245] (5th Application Example) The electronic device 1 according to the present disclosure can also be applied to smartphones and mobile phones. FIG. 41 is an external view of a smartphone 350, which is a fifth application example of the electronic device 1. In the example of FIG. 41, the display surface 2z extends to nearly the outer size of the electronic device 1, and the width of the bezel 2y around the display surface 2z is set to a few millimeters or less. Typically, a front camera is mounted in the bezel 2y. However, in FIG. 41, as shown by the dashed line, an image sensor module 9 functioning as a front camera is disposed on the back side of the display surface 2z, for example, in the approximate center thereof. By providing the front camera on the back side of the display surface 2z in this way, there is no need to place the front camera in the bezel 2y, and the width of the bezel 2y can be narrowed.

[0246] The above-described embodiment may be modified as follows.

[0247] (1) a plurality of pixels that acquire first information that is information about three primary colors and second information that is information about at least two colors different from the three primary colors and includes at least one color that is a complementary color of the three primary colors; An imaging element comprising:

[0248] (2) The three primary colors are R (red), G (green), and B (blue), The complementary colors are Cy (cyan), Ye (yellow), and Mg (magenta). The imaging element according to (1).

[0249] (3) The second information includes Cy and Ye information. The imaging element according to (2).

[0250] (4) the second information includes information on Cy, Ye, and Mg; The imaging element according to (2).

[0251] (5) The second information includes at least one of white and emerald information. An imaging element according to any one of (1) to (4).

[0252] (6) The pixel outputs information of at least two colors out of the first information and the second information. An imaging element according to any one of (1) to (5).

[0253] (7) The pixel comprises a plurality of divided pixels; acquiring information on one color from the first information and the second information for each of the divided pixels; The imaging element according to (6).

[0254] (8) acquiring information on similar colors from the color information included in the first information and the second information at the same time; The imaging element according to (7).

[0255] (9) The information on similar colors includes at least one of information on R and Mg, B and Cy, G and Ye, R and Ye, or R and infrared light; The imaging element according to (8).

[0256] (10) The first information and the second information are information acquired by different light receiving elements at the same time. The imaging element according to (1).

[0257] (11) The second information includes at least one of the three primary color information. The imaging element according to (10).

[0258] (12) The first information and the second information are acquired at different resolutions. The imaging element according to (10) or (11).

[0259] (13) At least one pixel among the pixels acquires information using an organic photoelectric conversion film. An imaging element according to any one of (1) to (12).

[0260] (14) the first information is acquired in an organic photoelectric conversion film, and the second information is acquired in a photodiode via the organic photoelectric conversion film; Or, the second information is acquired in an organic photoelectric conversion film, and the first information is acquired in a photodiode via the organic photoelectric conversion film; The imaging element according to (13).

[0261] (15) The imaging device according to any one of (1) to (14), It has a still image mode and a video mode, The first information and the second information are combined at different timings or in different blocks in the still image mode and the moving image mode. electronic equipment.

[0262] (16) At least one of the pixels acquires information about light in the infrared region. An imaging element according to any one of (1) to (15).

[0263] (17) a filter capable of removing infrared light is provided in at least a part of the image sensor included in the pixel that acquires information on light in the infrared region; The imaging element according to (16).

[0264] (18) A frame image output from the pixels is synthesized, Controlling the number of pixels to be added for each frame at the timing of synthesis. An imaging element according to any one of (1) to (17).

[0265] (19) The imaging device according to any one of (1) to (18), An electronic device that performs object identification, biometric identification, or light source estimation based on the spectral information acquired from the imaging element.

[0266] (20) Controlling image processing or parameters of the image processing based on the target identified by the object identification or the biometric identification. (19) The electronic device described in (19).

[0267] (twenty one) a display for displaying an image; an imaging element according to any one of (1) to (18) at a position overlapping the display; An electronic device comprising:

[0268] (twenty two) A compound eye camera having at least one imaging element according to any one of (1) to (18), At least one of the imaging elements does not include an infrared rejection filter. electronic equipment.

[0269] (twenty three) a first pixel group that acquires the first information; a second pixel group that acquires the second information; Equipped with the second pixel group is a pixel having a higher sensitivity than the pixel in the first pixel group that acquires G information; An imaging element according to any one of (1) to (18).

[0270] (twenty four) the second pixel group includes at least a pixel that acquires white information; The imaging element according to (23).

[0271] (twenty five) interpolating information obtained by the second pixel group using information obtained by the first pixel group; The imaging element according to (23) or (24).

[0272] (26) a still image mode for acquiring a still image and a moving image mode for acquiring a moving image, and the first pixel group and the second pixel group are combined in different methods in the still image mode and the moving image mode; The imaging element according to any one of (23) to (25).

[0273] (27) The still image mode and the moving image mode are combined at different timings. The imaging element according to (26).

[0274] (28) The still image mode and the moving image mode are synthesized using different system blocks. The imaging element according to (26) or (27).

[0275] (29) In the moving image mode, the resolution of the pixels belonging to the first pixel group is set to be lower than the resolution of the pixels belonging to the second pixel group. The imaging element according to any one of (23) to (28).

[0276] (30) correcting color information acquired from the second pixel group based on color information statistics or illuminant estimation results acquired from the first pixel group; The imaging element according to any one of (23) to (29).

[0277] (31) The pixels belonging to the first pixel group and the pixels belonging to the second pixel group are arranged so that the nearest pixels are spaced apart by a distance of 10 mm or less. The imaging element according to any one of (23) to (30).

[0278] (32) the nearest pixels are spaced apart at a distance of 50 mm to 80 mm; The imaging element according to (31).

[0279] (33) Either the first pixel group or the second pixel group is configured to include divided pixels obtained by dividing pixels belonging to the first pixel group or the second pixel group. The imaging element according to any one of (23) to (32).

[0280] (34) A white pixel is provided in the divided pixel. The imaging element according to (33).

[0281] (35) For light acquired in the first pixel group and the second pixel group, optical parameters of the path of the light are different in each group. The imaging element according to any one of (23) to (34).

[0282] (36) the second pixel group includes pixels having higher sensitivity than the pixels of the first pixel group that acquire information B; The imaging element according to any one of (23) to (35).

[0283] (37) The pixels having higher sensitivity than the pixels for acquiring the B information are pixels for acquiring white or cyan color information. The imaging element according to (36).

[0284] (38) For pixel information acquired with five or more colors, three primary color information is recalculated from the acquired image information by arithmetic processing, and rearranged information is output. An imaging element according to any one of (1) to (37).

[0285] (39) Calculating pixel values ​​or statistical values ​​of information including at least complementary color information from the acquired image information. The imaging element according to (38).

[0286] (40) calculating the statistical values ​​for the information on the three primary colors and the information on the complementary colors from the acquired image information; The imaging element according to (39).

[0287] (41) calculating the statistical value from the complementary color information without including information on the three primary colors from the acquired image information; The imaging element according to (39).

[0288] (42) The exposure amount is controlled separately for the pixels that receive the three primary colors and the pixels that receive the complementary colors. An imaging element according to any one of (1) to (41).

[0289] (43) The exposure amount is controlled by controlling a shutter time. The imaging element according to (41).

[0290] (44) The exposure amount is controlled by controlling a gain. The imaging element according to (42) or (43).

[0291] (45) An imaging device according to any one of (1) to (44), Solid-state imaging device.

[0292] (46) an imaging element having a plurality of pixels for acquiring first information which is information on three primary colors and second information which is information on at least two colors different from the three primary colors and which includes at least one color among complementary colors of the three primary colors; An electronic device comprising:

[0293] The aspects of the present disclosure are not limited to the above-described embodiments and include various conceivable modifications, and the effects of the present disclosure are not limited to the above-described contents. The components in each embodiment may be appropriately combined and applied. In other words, various additions, modifications, and partial deletions are possible within the scope of the conceptual idea and intent of the present disclosure, which is derived from the content defined in the claims and their equivalents. [Explanation of symbols]

[0294] 1: Electronic equipment, 10: Solid-state imaging device, 100: Light receiving part, 102: Memory section, 104: Control section, 106: signal processing section; 108: Image processing unit, 110: Lens, 112: IRCF, 114: Image sensor, 120: A / D conversion section, 122: Clamping part, 124: Linear matrix section, 126: Gamma correction section, 130: Light source estimation section, 132: Object recognition section, 134: Image memory section, 136: Two-lens synthesis part, 140: Luminance chrominance signal generator; 150: Input / output I / F, 160: Pre-linear matrix section, 162: RGB rearrangement part, 164: Sensor output section, 166: Statistics acquisition unit, 20: pixels, 200: Light receiving element, 202: Filter, 204: pixel array; 206: Organic photoelectric conversion film, 208: Photodiode, 210: On-chip lens, 30: Substrate, 32: First board, 34: Second board, 300: pixel area, 302: control circuit, 304: Logic circuits,

Claims

1. a plurality of pixels, each of which acquires first information and second information; the first information is information of any one of R (red), G (green), or B (blue), the second information is information of any one of Cy (cyan), Ye (yellow), or Mg (magenta), The combination of the first information and the second information acquired for each pixel is R and Mg, R and Ye, G and Ye, G and Cy, or B and Cy, Selected from either Image sensor.

2. a plurality of pixels, each of which acquires first information and second information; the first information is information of any one of R (red), G (green), or B (blue), the second information is any one of Cy (cyan), Ye (yellow), Mg (magenta), W (white), E (emerald), or IR (infrared light), The combination of the first information and the second information acquired for each pixel is R and Mg, R and Ye, G and Ye, G and Cy, B and Cy, R and IR, G and W, G and E, or B and E, Selected from either Image sensor.

3. The pixel comprises a plurality of divided pixels; acquiring information on one color from the first information and the second information for each of the divided pixels; 3. The imaging device according to claim 1 or 2.

4. acquiring color information of the first information and the second information together; 4. The imaging device according to claim 3.

5. The first information and the second information are acquired at different resolutions.

5. The imaging device according to claim 4.

6. At least one pixel among the pixels acquires information using an organic photoelectric conversion film.

6. The imaging device according to claim 1.

7. the first information is acquired in an organic photoelectric conversion film, and the second information is acquired in a photodiode via the organic photoelectric conversion film; Or, the second information is acquired in an organic photoelectric conversion film, and the first information is acquired in a photodiode via the organic photoelectric conversion film; 7. The imaging device according to claim 6.

8. An imaging device comprising: an imaging element according to any one of claims 1 to 7; It has a still image mode and a video mode, The first information and the second information are combined at different timings or in different blocks in the still image mode and the moving image mode. electronic equipment.

9. At least one pixel is provided for acquiring IR information, and an IR removal filter is provided in at least a part of a light receiving element included in the at least one pixel other than the area for acquiring IR information. An imaging device according to claim 2 or any one of claims 3 to 7 which cites claim 2.

10. An imaging device comprising: an imaging element according to any one of claims 1 to 9; An electronic device that performs object identification, biometric identification, or light source estimation based on the spectral information acquired from the imaging element.

11. Controlling image processing or parameters related to the image processing based on the target identified by the object identification or the biometric identification.

11. The electronic device according to claim 10.

12. For image information acquired using five or more colors, three primary color information is recalculated from the acquired image information by arithmetic processing, and rearranged information is output.

10. The imaging device according to claim 1, wherein the first and second electrodes are arranged parallel to each other.

13. In addition to the image data rearranged into the three primary colors, information including at least complementary colors is output as either pixel values ​​or statistical values.

13. The imaging device according to claim 12.

14. controlling the amount of exposure light in the region where the first information is acquired and the region where the second information is acquired, respectively; 14. The imaging device according to claim 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, or 13.

15. a display for displaying an image; an imaging element according to any one of claims 1 to 7, 9, and 12 to 14, located at a position overlapping the display; An electronic device comprising:

16. a layer formed of polyimide on the light incident surface side of the imaging element; 16. The electronic device according to claim 15.

17. A plurality of pixels each acquiring first information and second information, the first information is information of any one of R (red), G (green), or B (blue), the second information is information of any one of Cy (cyan), Ye (yellow), or Mg (magenta), The combination of the first information and the second information acquired for each pixel is R and Mg, R and Ye, G and Ye, G and Cy, or B and Cy, an imaging element having a plurality of pixels selected from any one of An electronic device comprising:

18. A plurality of pixels each acquiring first information and second information, the first information is information of any one of R (red), G (green), or B (blue), the second information is any one of Cy (cyan), Ye (yellow), Mg (magenta), W (white), E (emerald), or IR (infrared light), The combination of the first information and the second information acquired for each pixel is R and Mg, R and Ye, G and Ye, G and Cy, B and Cy, R and IR, G and W, G and E, or B and E, an imaging element having a plurality of pixels selected from any one of An electronic device comprising:

Images