Electronic equipment and imaging control method

Abstract

In a configuration that adds signals read from multiple adjacent pixels and outputs the result, the present invention provides an electronic device and imaging control method that can utilize the signals from these multiple pixels more efficiently. [Solution] The electronic device according to this disclosure includes an imaging unit having a pixel array in which a plurality of pixels, each outputting a pixel signal according to the light they receive, are arranged in a two-dimensional grid; a conversion unit that performs a conversion process to convert the pixel signals within a range into digital pixel data; a signal processing unit that performs signal processing including white balance processing on the pixel data; and a setting unit that changes the range according to the gain in the white balance processing.

Description

[Technical Field]

[0001] This disclosure relates to electronic equipment and imaging control methods. [Background technology]

[0002] In image sensors, a technique is known to obtain a large Full Well Capacity (FWC) by adding the signals read from multiple adjacent pixels and outputting them, and by providing a single On-Chip Lens (OCL) common to these multiple pixels. For example, a single OCL is provided common to four pixels arranged in a 2x2 horizontal x 2 vertical configuration, each equipped with a color filter of the same color (hereinafter referred to as a 2x2 OCL configuration). [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-136890 [Overview of the project] [Problems that the invention aims to solve]

[0004] In the 2x2 OCL configuration described above, significant differences in sensitivity between the same color pixels may occur in the four pixels of the same color whose outputs are added together, due to variations in the characteristics of each pixel. The resulting FWC, obtained by adding the outputs of these four pixels, will lose linearity if even one of the four pixels has a saturated output. Therefore, the resulting FWC must use only the linear region, which means that the FWC corresponding to the difference in sensitivity between the same colors will be lost.

[0005] Therefore, the present disclosure aims to provide an electronic device and an imaging control method that can more efficiently utilize the signals from a plurality of pixels that are arranged adjacent to each other, in a configuration that adds the signals read from a plurality of pixels and outputs the result. [Means for solving the problem]

[0006] The electronic device according to this disclosure includes an imaging unit having a pixel array in which a plurality of pixels, each outputting a pixel signal in accordance with the light they receive, are arranged in a two-dimensional grid; a conversion unit that performs a conversion process to convert the pixel signals within a range into digital pixel data; a signal processing unit that performs signal processing including white balance processing on the pixel data; and a setting unit that changes the range according to the gain in the white balance processing. [Brief explanation of the drawing]

[0007] [Figure 1] Block diagram shows an example configuration of an electronic device applicable to this disclosure. [Figure 2] This is a schematic diagram showing an example of a pixel array applicable to this disclosure. [Figure 3] This block diagram shows an example configuration of an imaging unit applicable to this disclosure. [Figure 4] This figure shows an example configuration of a pixel block applicable to each embodiment. [Figure 5] This is a schematic diagram illustrating an example where one OCL is provided for each pixel block using existing technology. [Figure 6] This is a schematic diagram illustrating the degradation of the FWC due to differences in sensitivity between identical colors using existing technologies. [Figure 7] This is a schematic diagram showing an example of a pixel array including a 1×1OCL configuration applicable to the embodiment. [Figure 8A] This is a schematic diagram showing an example relationship between charge quantity Q and illuminance L for each color before performing white balance adjustment processing. [Figure 8B] This is a schematic diagram showing an example relationship between charge quantity Q and illuminance L for each color after white balance adjustment processing. [Figure 9] This is an example functional block diagram illustrating the functions of the electronic device according to the embodiment. [Figure 10] This is an example flowchart illustrating the process according to the embodiment. [Figure 11A] It is a schematic diagram that generally shows the processing until the charge photoelectrically converted by the photoelectric conversion element according to the embodiment is converted into a digital value. [Figure 11B] It is a schematic diagram that generally shows the processing until the charge photoelectrically converted by the photoelectric conversion element according to the embodiment is converted into a digital value. [Figure 12] It is a schematic diagram showing an example of the OCL arrangement according to the first modification of the embodiment. [Figure 13] It is a schematic diagram showing an example of the OCL arrangement according to the second modification of the embodiment. [Figure 14] It is a schematic diagram showing an example of the pixel configuration according to the third modification of the embodiment. [Figure 15] It is a schematic diagram showing an example of the pixel configuration according to the fourth modification of the embodiment. [Figure 16] It is a schematic diagram showing an example of a color arrangement applicable to the fifth modification of the embodiment.

Mode for Carrying Out the Invention

[0008] Hereinafter, embodiments of the present disclosure will be described in detail based on the drawings. In the following embodiments, the same parts are denoted by the same reference numerals, and redundant explanations are omitted.

[0009] Hereinafter, embodiments of the present disclosure will be described in the following order. 1. Overview of the present disclosure 2. Regarding the existing technology 2-1. Regarding the waste of FWC 2-2. Regarding high resolution 2-3. Regarding the processing when high resolution is achieved by the existing technology 3. Embodiments of the present disclosure 3-1. Configuration according to the embodiment 3-2. Processing according to the embodiment 4. First modification of the embodiment 5. Second modification of the embodiment 6. Third modification of the embodiment 7. Fourth modification of the embodiment 8. Fifth Modification of the Embodiment

[0010] (1. Summary of this disclosure) First, an overview of the technology of this disclosure will be given. Figure 1 is a block diagram showing the configuration of an example of an electronic device applicable to this disclosure. In Figure 1, the electronic device 1 includes a sensor 10, an optical unit 11, an application processor (AP) 20, a frame memory 21, and an output unit 30. The electronic device 1 shown in Figure 1 is configured as an imaging device.

[0011] The optical unit 11 includes, for example, multiple lenses and has a focusing mechanism, and guides the incident light to the sensor 10. The sensor 10 includes an imaging unit having a pixel array in which multiple pixels, each outputting a pixel signal according to the light it receives, are arranged in a two-dimensional grid, and an ADC (Analog to Digital Converter) that converts the pixel signals output from the pixel array in analog format into pixel data in digital format.

[0012] In sensor 10, the pixel array is composed of pixel blocks made up of multiple pixels arranged adjacent to each other, and the pixel signals of each of the multiple pixels contained in the pixel block are added together and output. By adding the pixel signals of multiple pixels in this way, the full well capacity (FWC) for the pixel signals can be increased. Here, full well capacity refers to the effective range when the pixel signal is converted to AD by the ADC.

[0013] Pixel blocks are arranged in a two-dimensional grid in a pixel array. Each pixel within a pixel block may be configured to selectively receive light in a specific wavelength range of a particular color using a predetermined optical element. In this case, the optical element may be a color filter.

[0014] Figure 2 is a schematic diagram showing an example of a pixel array applicable to this disclosure. In the example in Figure 2, a pixel block 110 is composed of four pixels 11001 to 11004 arranged in a 2x2 horizontal x 2 vertical pattern. Thus, the four pixels 11001, 11002, 11003, and 11004 that make up the pixel block 110 are arranged adjacent to each other.

[0015] Hereafter, a horizontal 2x2 pixel array will be described as a 2x2 array, as appropriate. That is, when n is an integer greater than or equal to 1, a horizontal n pixel x vertical n pixel array will be described as an nxn array. Also, when there is no need to distinguish between pixels 11001 to 11004, each pixel 11001 to 11004 will be described as pixel 1100.

[0016] Furthermore, in Figure 2, the upper left and lower right pixel blocks 110 are each provided with a color filter that selectively transmits light in the green (G) wavelength band for each pixel 11001 to 11004. Similarly, in Figure 2, the upper right and lower left pixel blocks 110 are each provided with a color filter that selectively transmits light in the red (R) and blue (B) wavelength bands for each pixel 11001 to 11004.

[0017] In the following, unless otherwise specified, "light in the green (G) wavelength band" will be described as "green light" or "G-colored light," and "a pixel equipped with a color filter that selectively transmits light in the green (G) wavelength band" will be described as "a green pixel" or "G-colored pixel." This applies not only to cases where a color filter is used, but also to configurations where a pixel selectively receives light in a specific wavelength range using methods other than a color filter.

[0018] In the example in Figure 2, each pixel block 110 is arranged such that no pixel blocks 110 of the same color are adjacent to each other. This arrangement on a pixel-by-pixel basis is called a Bayer array, but for convenience, this arrangement on a pixel block 110 basis, as in Figure 2, is called an extended Bayer array. Also, in the example in Figure 2, each pixel block 110 consists of 4 pixels (2 horizontally x 2 vertically), but this is not limited to this example, and each pixel block 110 may be composed of more pixels, such as 3 horizontally x 3 vertically.

[0019] In sensor 10, the pixel array outputs pixel signals of R, G, and B colors in response to incident light. The R, G, and B colored pixel signals output from the pixel array are converted into R, G, and B colored pixel data by the ADC, respectively. Sensor 10 outputs the R, G, and B colored pixel data converted by the ADC to AP20.

[0020] AP20 includes, for example, a CPU (Central Processing Unit) and an ISP (Image Signal Processor), memory such as ROM (Read Only Memory) and RAM (Random Access Memory), and peripheral circuits such as various interfaces. However, AP20 may implement ISP functionality using the CPU without using an ISP.

[0021] AP20 stores the pixel data for each of the R, G, and B colors output from the sensor 10 into a frame memory 21 connected to AP20. When one frame's worth of pixel data is stored in the frame memory 21, AP20 automatically performs white balance adjustment processing based on the pixel data for that frame stored in the frame memory 21. For example, AP20 performs white balance adjustment processing by controlling the levels of the pixel data for the other colors (R and / or B) based on the brightest G color pixel data among the R, G, and B color pixel data.

[0022] In the following, the pixel data for one frame stored in the frame memory 21 will be referred to as image data.

[0023] AP20 may further perform focus control and exposure control based on image data. For example, AP20 may control the focus mechanism of the optical unit 11 based on a focus control value calculated based on image data. Alternatively, AP20 may control the exposure of each pixel in the pixel array within the sensor 10 based on an exposure control value calculated based on image data. AP20 applies the various image processing described above to the image data based on the pixel data output from the sensor 10, and outputs the processed image data to the output unit 30.

[0024] The output unit 30 outputs the image data supplied from AP20 to the outside of the electronic device 1. The output unit 30 may include, for example, a display control unit and a display device, and may display a screen on the display device corresponding to the image data supplied from AP20. As a result, the electronic device 1 can display a screen based on the captured image captured by the sensor 10 in real time using the output unit 30. However, the output unit 30 may also include an interface for outputting the image data supplied from AP20 to the outside.

[0025] Figure 3 is a block diagram showing the configuration of an example imaging unit applicable to this disclosure. In Figure 3, the imaging unit 100 includes a pixel array 111, a vertical scanning unit 112, an AD (Analog to Digital) conversion unit 130, a pixel signal line 116, a vertical signal line 117, a control unit 119, and a signal processing unit 120.

[0026] The pixel array 111 includes a pixel block 110, each of which has a photoelectric conversion unit, for example, a photodiode, that performs photoelectric conversion on received light. In the pixel array 111, the multiple pixels are arranged in a two-dimensional grid in the horizontal (row) and vertical (column) directions, and accordingly, each pixel block 110 is arranged in a two-dimensional grid. In the pixel array 111, the row-direction arrangement of the pixel blocks 110 is called a line. In this pixel array 111, a frame of image (image data) is formed by the pixel signals read from a predetermined number of lines.

[0027] Furthermore, for each row and column of each pixel block 110 in the pixel array 111, one or more pixel signal lines 116 are connected, and vertical signal lines 117 are connected to each column.

[0028] The end of the pixel signal line 116 that is not connected to the pixel array 111 is connected to the vertical scanning unit 112. The vertical scanning unit 112 transmits control signals, such as drive pulses for reading pixel signals from pixels, to the pixel array 111 via the pixel signal line 116, in accordance with the control of the control unit 119, which will be described later. The end of the vertical signal line 117 that is not connected to the pixel array 111 is connected to the AD conversion unit 130. The pixel signals read from pixels are transmitted to the AD conversion unit 130 via the vertical signal line 117.

[0029] The AD conversion unit 130 includes an AD converter (ADC) 131 provided for each vertical signal line 117, a reference signal generation unit 114, and a horizontal scanning unit 115. The AD converter 131 is a column AD converter that performs AD conversion processing for each column of the pixel array 111. The AD converter 131 performs AD conversion processing on the pixel signals supplied from the pixel block 110 via the vertical signal line 117 and generates two digital values ​​for correlated double sampling (CDS) processing to reduce noise. Specific examples of the configuration and processing of the AD converter 131 will be described later.

[0030] Each AD converter 131 supplies the two digital values ​​it generates to the signal processing unit 120. The signal processing unit 120 performs CDS processing based on the two digital values ​​supplied from each AD converter 131 to generate pixel data using digital signals. The pixel data generated by the signal processing unit 120 is output from the imaging unit 100 to, for example, AP20.

[0031] The pixel data output from the signal processing unit 120 is sequentially stored in, for example, the frame memory 21 in the AP20. When the AP20 has stored one frame's worth of pixel data in the frame memory 21, it reads the stored pixel data from the frame memory 21 as one frame of image data.

[0032] The reference signal generation unit 114 generates ramp signals (RAMP) that each AD converter 131 uses to convert pixel signals into two digital values, based on the ADC control signal input from the control unit 119. The ramp signal (RAMP) is a signal whose level (voltage value) decreases at a constant rate over time, or a signal whose level decreases in a stepwise manner. The reference signal generation unit 114 supplies the generated ramp signals (RAMP) to each AD converter 131. The reference signal generation unit 114 is configured using, for example, a DA (Digital to Analog) conversion circuit.

[0033] The horizontal scanning unit 115, under the control of the control unit 119, performs a selective scan to select each AD converter 131 in a predetermined order, thereby sequentially outputting each digital value temporarily held by each AD converter 131 to the signal processing unit 120. The horizontal scanning unit 115 is configured using, for example, a shift register or an address decoder.

[0034] The control unit 119 controls the operation of the vertical scanning unit 112, the AD conversion unit 130, the reference signal generation unit 114, and the horizontal scanning unit 115. The control unit 119 generates various drive signals that serve as a reference for the operation of the vertical scanning unit 112, the AD conversion unit 130, the reference signal generation unit 114, and the horizontal scanning unit 115. For example, based on a vertical synchronization signal or external trigger signal supplied from an external source and a horizontal synchronization signal, the control unit 119 generates control signals that the vertical scanning unit 112 supplies to each pixel block 110 via the pixel signal line 116. The control unit 119 supplies the generated control signals to the vertical scanning unit 112.

[0035] The vertical scanning unit 112, based on control signals supplied from the control unit 119, supplies various signals, including drive pulses, to the pixel signal lines 116 of the selected row of the pixel array 111, line by line, to each pixel block 110, causing each pixel block 110 to output a pixel signal to the vertical signal line 117. The vertical scanning unit 112 is configured using, for example, a shift register or an address decoder.

[0036] The imaging unit 100 configured in this way is a column AD type CMOS (Complementary Metal Oxide Semiconductor) image sensor in which AD converters 131 are arranged in rows.

[0037] Figure 4 shows an example configuration of a pixel block 110 applicable to each embodiment. In Figure 4, the pixel block 110 includes, for example, four photoelectric conversion elements 11011 to 11014, each consisting of a PN junction photodiode, four trigger transistors 11021 to 11024, each being an N-type MOS (Metal Oxide Semiconductor) transistor, and similarly, a reset transistor 1103, an amplification transistor 1104, and a selection transistor 1105, each being an N-type MOS transistor. The pixel signal line 116 connected to the pixel block 110 supplies a reset pulse RST, a transfer pulse TRG, and a selection signal SEL, respectively.

[0038] In the pixel block 110, the cathodes of each photoelectric conversion element 11011 to 11014 are connected to ground, and the anodes are connected to the drains of each trigger transistor 11021 to 11024, respectively. The sources of each trigger transistor 11021 to 11024 are connected to the FD (Floating Diffusion). A common transfer pulse TRG is supplied to the gates of each trigger transistor 11021 to 11024. Each trigger transistor 11021 to 11024 is turned on (closed) when the transfer pulse TRG is high, and turned off (open) when the transfer pulse TRG is low. For example, when trigger transistor 11021 is turned on, the charge output from photoelectric conversion element 11011 is supplied to the FD.

[0039] In cases where it is not necessary to distinguish between the photoelectric conversion elements 11011 to 11014, each photoelectric conversion element 11011 to 11014 will be described as photoelectric conversion element 1101.

[0040] The FD stores the charge supplied from each photoelectric conversion element 11011 to 11014. The FD generates a voltage corresponding to the amount of charge stored.

[0041] The source of the reset transistor 1103 is connected to the FD. The power supply VDD for the pixel block 110 is connected to the drain of the reset transistor 1103. The gate of the reset transistor 1103 is supplied with a reset pulse RST. The reset transistor 1103 turns on when the reset pulse RST is high and turns off when the reset pulse RST is low.

[0042] The gate of the amplification transistor 1104 is connected to FD. The power supply VDD is connected to the drain of the amplification transistor 1104, and the drain of the selection transistor 1105 is connected to its source. The source of the selection transistor 1105 is connected to the vertical signal line (VSL) 117. The selection signal SEL is supplied to the gate of the selection transistor 1105. The selection transistor 1105 is turned on when the selection signal SEL is high and turned off when it is low.

[0043] The control of reading out pixel signals from pixel block 110 will be briefly explained. Reading out pixel signals from pixel block 110 is performed by transferring the charge accumulated in each photoelectric conversion element 11011~11014 due to exposure to the FD, and then converting the transferred charge into a voltage in the FD. The voltage converted from the charge in the FD is output to the vertical signal line (VSL) 117 via the amplification transistor 1104.

[0044] More specifically, in the pixel block 110, during exposure, the transfer pulse TRG is set to a low state, and the connection between each photoelectric conversion element 11011~11014 and the FD is set to an off (open) state, allowing each photoelectric conversion element 11011~11014 to accumulate charge generated in response to the incident light by photoelectric conversion. After exposure is complete, the FD (amplifier transistor 1104) and the vertical signal line 117 are connected in accordance with the selection signal SEL supplied via the pixel signal line 1106. Furthermore, in accordance with the reset pulse RST supplied via the pixel signal line 1106, the reset transistor 1103 briefly connects the FD to the power supply voltage VDD or black level voltage supply line, thereby resetting the FD. The vertical signal line 117 outputs the voltage of the FD reset level (let's call it voltage A).

[0045] Subsequently, the transfer pulse TRG supplied via the pixel signal line 1106 is set to high, and each trigger transistor 11021 to 11024 is turned on. This turns on (closes) the connection between each photoelectric conversion element 11011 to 11014 and the FD, and the charge accumulated in each photoelectric conversion element 11011 to 11014 is transferred to the FD. A voltage (let's call it voltage B) corresponding to the amount of charge in the FD is output to the vertical signal line 117.

[0046] The pixel signals supplied from the pixel block 110 via the vertical signal line 117 are subjected to AD conversion processing by the AD converter 131, generating two digital values ​​(values ​​corresponding to voltage A and voltage B, respectively) for correlated double sampling (CDS) processing to reduce noise.

[0047] The AD converter 131 supplies the two generated digital values ​​to the signal processing unit 120. The signal processing unit 120 performs CDS processing based on the two digital values ​​supplied from the AD converter 131 to generate pixel data in digital format. The pixel data generated by the signal processing unit 120 is output to the outside of the imaging unit 100.

[0048] (2. Regarding existing technologies) Here, we will describe the existing technologies related to this disclosure.

[0049] (2-1. Regarding the damage to the FWC) In a pixel array, an on-chip lens (hereinafter referred to as OCL) is generally provided for each pixel to improve light-gathering efficiency. When a pixel block 110 is composed of pixels 11001 to 11004 in a 2x2 array, as explained using Figure 2, an OCL is provided for each pixel block 110, as illustrated in Figure 5. Figure 5 is a schematic diagram showing an example in which one OCL 150 is provided for each pixel block 110 according to existing technology. In the example in Figure 5, one OCL 150 is provided in common for pixels 11001 to 11004 of the same color in the pixel block 110. This configuration, in which one OCL is provided in common for four pixels in a 2x2 array, is called a 2x2 OCL configuration.

[0050] Existing 2x2 OCL configurations have the problem of large sensitivity differences (inter-color sensitivity differences) between pixels 11001-11004 of the same color, where a common OCL 150 is provided. This sensitivity difference arises not only from individual differences in each pixel 11001-11004, but also from factors such as the bias of the light center when light is incident on the light-receiving surface at an oblique angle, and the bias of the OCL 150's position relative to the center of the pixel block 110. The sensitivity difference between pixels 11001-11004 of the same color can be as high as 30%.

[0051] Furthermore, the pixel block 110 generates a pixel signal by adding the charges accumulated in each photoelectric conversion element 11011 to 11014 using a common FD. Therefore, if even one of the photoelectric conversion elements 11011 to 11014 becomes saturated, the linearity of the added charges is lost, resulting in a loss of the FWC due to the difference in sensitivity between identical colors.

[0052] Figure 6 is a schematic diagram illustrating the degradation of the freewave grid (FWC) due to sensitivity differences between identical colors using existing technologies. Figure 6 assumes four identical pixels 11001 to 11004, each contained within a single pixel block 110 and sharing a common OCL (Optical Color Block). In Figure 6, the vertical axis represents the amount of charge Q accumulated in the FD of the pixel block 110, and the horizontal axis represents the illuminance L.

[0053] Each of the photoelectric conversion elements 11011 to 11014 has a charge quantity Q. single It is assumed that the system reaches a saturation state. Furthermore, if the photoelectric conversion elements 1101 are classified as high, medium, and low sensitivity from highest to lowest, then one of the photoelectric conversion elements 11011 to 11014 is assumed to have high sensitivity, another to have low sensitivity, and the remaining two to have medium sensitivity. Here, sensitivity refers to the degree of charge accumulation in the photoelectric conversion element 1101 in relation to illuminance.

[0054] As shown by characteristic curve 140a in Figure 6, the photoelectric conversion element 1101 with high sensitivity is assumed to reach a saturated state at illuminance L1. The photoelectric conversion element 1101 with medium sensitivity is assumed to reach a saturated state at illuminance L2, which is higher than illuminance L1 (characteristic curve 140b), and the photoelectric conversion element 1101 with low sensitivity is assumed to reach a saturated state at illuminance L3, which is even higher than illuminance L2 (characteristic curve 140c).

[0055] The charge Q is the sum of the charges of each photoelectric conversion element 11011 to 11014 using FD. add As shown in characteristic curve 141, the linearity increases up to illuminance L1, at which the high-sensitivity photoelectric conversion element 1101 saturates. On the other hand, as the high-sensitivity photoelectric conversion element 1101 saturates at illuminance L1, the linearity is lost, as shown by the bend in characteristic curve 141 at illuminance L1. Furthermore, as the charge of the medium-sensitivity and low-sensitivity photoelectric conversion elements 1101 increases and saturates, the linearity is lost, as shown by the bend in characteristic curve 141 at illuminances L2 and L3.

[0056] As shown in Figure 6, the charge characteristic with respect to illuminance after summing the charges of each photoelectric conversion element 11011 to 11014 loses its linearity if even one of the photoelectric conversion elements 11011 to 11014 becomes saturated. Therefore, the region in which it can be used as an FWC is the charge amount Q at illuminance L1 where the highly sensitive photoelectric conversion element 1101 becomes saturated. Linear It falls within the range of the corresponding pixel signal level Q. Therefore, the charge quantity Q Linear The FWC (Free Flow Control) for the same-color sensitivity difference corresponding to pixel signal levels exceeding a certain threshold will be impaired. Furthermore, improving the FWC is difficult simply by improving the potential of the photoelectric conversion element 1101.

[0057] (2-2. Regarding high-resolution enhancement) To achieve higher resolution, one method is to use a 1×1OCL configuration in which, among the R, G, and B color pixel blocks 110 of the extended Bayer array, only the brightest G color pixel block 110 has an OCL for every pixel 1100. Figure 7 is a schematic diagram showing an example of a pixel array including a 1×1OCL configuration that can be applied to the embodiment.

[0058] As shown in Figure 7, in a pixel block 110 containing G-colored pixels 11001 to 11004, an OCL 151 is provided for each of the G-colored pixels 11001 to 11004. On the other hand, in each pixel block 110 containing R-colored and B-colored pixels 11001 to 11004, one OCL 150 is provided in common for pixels 11001 to 11004.

[0059] This configuration suppresses the bias of the light center in each G-color pixel 11001-11004 when light is incident on the light-receiving surface from an oblique angle, thereby improving resolution and also suppressing the difference in sensitivity between the same color pixels 11001-11004 of the G-color.

[0060] Even with the configuration shown in Figure 7, the pixel blocks 110 for colors B and R are in a 2x2 OCL configuration, and the FWC degradation explained using Figures 5 and 6 remains unresolved.

[0061] (2-3. Processing when increasing resolution using existing technologies) The process according to the embodiment will be explained using Figures 8A and 8B.

[0062] Figure 8A is a schematic diagram showing an example relationship between charge amount Q and illuminance L for each color before performing white balance adjustment processing. For green (G color), the green pixel block 110 is configured as 1×1OCL, and the variation in the characteristics of the accumulated charge amount with respect to illuminance among each pixel 11001 to 11004 of the pixel block 110 is suppressed. Therefore, the change in the accumulated charge amount of the green pixel 1100 with respect to illuminance is as shown in characteristic line 142, where at illuminance L1, each pixel 11001 to 11004 has a charge amount Q GREEN Linearity is maintained until saturation occurs. Therefore, the region in which the green pixel block 110 can be used as FWC is the region of charge Q GREEN This falls within the range of the corresponding pixel signal level.

[0063] On the other hand, for red (R color) and blue (B color), since each pixel block 110 has a 2×2 OCL configuration, similar to the example in FIG. 6, the photoelectric conversion element 1101 with high sensitivity is at illuminance L1, the photoelectric conversion element 1101 with medium sensitivity is at an illuminance L2 higher than illuminance L1, and the photoelectric conversion element 1101 with low sensitivity is at an illuminance L3 even higher than illuminance L2, and they are each in a saturated state (characteristic lines 140a’, 140b’, 140c’).

[0064] In the pixel blocks 110 of red (R color) and blue (B color), the charge amount Q obtained by adding the charges of each photoelectric conversion element 11011 - 11014 with FD, as shown by the characteristic line 141’, loses linearity at illuminances L1, L2, and L3 where the photoelectric conversion elements 1101 with high sensitivity, medium sensitivity, and low sensitivity are saturated. Therefore, the region that can be used as FWC is within the range of pixel signal levels corresponding to the charge amount Q at illuminance L1 where the photoelectric conversion element 1101 with high sensitivity is saturated. Linear and becomes within the range corresponding to the pixel signal level according to the charge amount Q. Therefore, the FWC of the sensitivity difference between the same colors corresponding to pixel signal levels exceeding the charge amount Q Linear will be discarded.

[0065] FIG. 8B is a schematic diagram showing an example of the relationship between the charge amount Q and the illuminance L for each color after performing the white balance adjustment process. In the white balance adjustment process, generally, based on the pixel signal level of the brightest color (for example, green), the gain (white balance gain) for the pixel signal levels of other colors (for example, red and / or blue) is adjusted. By this gain adjustment, the pixel signal levels of red and / or blue change, and the apparent value of the charge amount Q Linear changes. If the value of the gain for other colors with respect to the reference value exceeds 1, the pixel signal levels of other colors will saturate at an illuminance lower than the pixel signal level of the reference value color, and the FWC of the sensitivity difference between the same colors will be discarded.

[0066] In the example of FIG. 8B, the pixel signal levels (added values) of other colors after the white balance adjustment process are at an illuminance L4 lower than illuminance L1, and the charge amount Q GREENThe saturation signal level of the corresponding reference color has been reached. In this case, the FWC (Free Flow Control) of the inter-color sensitivity difference corresponding to the amount by which the pixel signal levels of the other colors exceed the saturation signal level of the reference color is lost.

[0067] Furthermore, depending on the captured scene, the white balance gain value may be close to 1, rendering the white balance adjustment process ineffective. Therefore, it is difficult to relax the individual adjustments required for the FWC, which are fixed values ​​in existing technologies.

[0068] (3. Embodiments of the Disclosure) Next, embodiments of the present disclosure will be described.

[0069] (3.1. Configuration according to the embodiment) First, the configuration according to the embodiment of this disclosure will be described in more detail. Figure 9 is an example functional block diagram for illustrating the functions of the electronic device 1 according to the embodiment. In Figure 9, the electronic device 1 according to the embodiment includes a sensor 10, an AP 20, and a frame memory 21. Note that in Figure 9, the optical unit 11 is omitted compared to the configuration shown in Figure 1.

[0070] The sensor 10 includes a pixel unit 101, an FD 102, an ADC 103, a signal processing unit 104, an I / F 105, and a register control unit 106. Referring to Figures 1 to 4, the pixel unit 101 may correspond to each photoelectric conversion element 1101 in each pixel included in the pixel array 111, and the FD 102 may correspond to the FD in a pixel block 110 consisting of multiple pixels arranged adjacent to each other. The ADC 103 may correspond to each AD converter 131 included in the AD conversion unit 130. The signal processing unit 104 may correspond to the signal processing unit 120.

[0071] The pixel unit 101 accumulates charge corresponding to the light received during the exposure period. The charge accumulated in the pixel unit 101 is read out from the pixel unit 101 and transferred to the FD102, where it is also stored. The FD102 adds and stores the charges read out from the photoelectric conversion elements 11011 to 11014 of each pixel included in the pixel block 110. The charge accumulated in the FD102 is converted into an electrical signal when read out from the FD102 and transferred to the ADC103 as a pixel signal.

[0072] The ADC103, following the control of the register control unit 106, performs AD conversion processing on the pixel signal transferred from the FD102, converting the pixel signal into digital pixel data and outputting it. At this time, the ADC103 sets the level range (ADC range) of the pixel signal to be subjected to AD conversion processing, following the control of the register control unit 106.

[0073] The register control unit 106 controls the registers that set the AD conversion parameters in the ADC 103. For example, the register control unit 106 sets the range of pixel signal levels (ADC range) that the ADC 103 targets for AD conversion processing. The register control unit 106 may be a microprocessor or it may be a register itself.

[0074] The signal processing unit 104 performs predetermined signal processing, such as noise reduction processing including CDS processing, on the pixel data output from the ADC 103. The signal processing unit 104 outputs the processed pixel data to the AP20 via the I / F 105. The I / F 105 may be, for example, a MIPI (Mobile Industry Processor Interface).

[0075] AP20 includes an auto white balance (AWB) unit 201, an autofocus (AF) unit 202, and an auto exposure (AE) unit 203. AP20 sequentially stores pixel data supplied from the sensor 10 into the frame memory 21. When one frame's worth of pixel data is stored in the frame memory 21, AP20 reads the stored pixel data for one frame from the frame memory 21 as image data.

[0076] The AWB unit 201 automatically performs white balance adjustment processing (auto white balance) based on the image data read from the frame memory 21. For example, the AWB unit 201 calculates a gain (white balance gain) for the pixel data of the red and / or blue pixels 1100 contained in the image data, using the pixel data of each green pixel 1100 contained in the image data as a reference. Based on the calculated gain, the AWB unit 201 corrects the level of the pixel data and performs white balance adjustment processing. The AWB unit 201 may store the image data with the white balance adjusted based on the white balance gain in the frame memory 21.

[0077] Furthermore, the AWB unit 201 may calculate the range in which the ADC 103 performs AD conversion processing based on the white balance gain used in the white balance adjustment process. The AWB unit 201 passes a value indicating the calculated range to the register control unit 106. The register control unit 106 may, for example, change the range in which the ADC 103 performs AD conversion of the pixel signals of the red and blue pixels 1100 according to the value indicating the range passed from the AWB unit 201.

[0078] The AWB unit 201 may also pass only the white balance gain to the register control unit 106. In this case, the register control unit 106 may calculate the range in which the ADC 103 performs AD conversion processing based on the white balance gain passed from the AWB unit 201.

[0079] In AP20, the AF unit 202 performs focus adjustment processing (autofocus) based on image data. The AF unit 202 may, for example, calculate the focus position based on the phase difference of two adjacent pixels 1100 in each pixel block 110 of the image data. The AF unit 202 outputs a focus value indicating the calculated focus position to an optical unit 11 (not shown). The optical unit 11 controls the focus mechanism according to the focus value supplied from the AF unit 202.

[0080] In AP20, the AE unit 203 performs exposure adjustment processing (auto exposure) to control exposure based on image data. The AE unit 203 may, for example, calculate the analog gain when the ADC 103 performs AD conversion processing based on the image data. The AE unit 203 passes the calculated value indicating the analog gain to the register control unit 106. The register control unit 106 may set an analog gain control value as a parameter for AD conversion in the ADC 103 based on the value indicating the analog gain passed from the AE unit 203.

[0081] For example, the AE unit 203 may calculate exposure control information (shutter speed, aperture value, etc.) for controlling exposure based on image data. The AE unit 203 outputs the calculated exposure control information to the pixel unit 101 (pixel array 111). The pixel unit 101 controls the exposure of each pixel 1100 according to the exposure control information supplied from the AE unit 203. In addition, the AE unit 203 may change the range when the ADC 103 performs AD conversion on the pixel signals of the green, red, and blue pixels 1100, for example, if the AWB unit 201 changes the range when the ADC 103 performs AD conversion on the pixel signals of the red and blue pixels 1100, causing a large change in the brightness of the image due to the image data.

[0082] The auto white balance processing by the AWB unit 201, the autofocus processing by the AF unit 202, and the auto exposure processing by the AE unit 203 are sometimes collectively referred to as "3A adjustment processing."

[0083] AP20 may output image data stored in frame memory 21 to output unit 30. For example, AP20 may output image data stored in frame memory 21 after white balance adjustment processing has been performed by AWB unit 201 to output unit 30.

[0084] As described above, the output unit 30 may include a display control unit and a display device. Alternatively, the output unit 30 may output image data to a display device including a display control unit and a display device. The electronic device 1 according to the embodiment may be an electronic device having a shooting function (e.g., a camera). However, the electronic device 1 according to the embodiment may be a device capable of displaying captured information in real time. Examples of such devices include HMDs (Head Mounted Displays) and AR (Augmented Reality) glasses that have a pass-through function that superimposes content images onto captured images and displays them in real time. Furthermore, the electronic device 1 according to the embodiment may be an in-vehicle camera (dashcam).

[0085] The AWB unit 201, AF unit 202, and AE unit 203 described above may be configured, for example, by executing the imaging control program according to this embodiment on the CPU included in AP20. However, they may also be implemented by a hardware circuit in which the AWB unit 201, AF unit 202, and AE unit 203 work in cooperation with each other.

[0086] In AP20, for example, the CPU executes the imaging control program according to the embodiment, thereby configuring the above-mentioned AWB unit 201, AF unit 202, and AE unit 203 as modules in the main memory area of ​​the RAM of AP20. This imaging control program can be acquired from an external source via a communication network through communication via a communication I / F (not shown), or acquired from a storage medium connected to a data I / F (not shown), and installed on AP20.

[0087] In the embodiments of this disclosure, in the configuration shown in Figure 9, the range of the ADC that performs AD conversion on the pixel signal, i.e., the FWC, is changed according to the result of the white balance adjustment process based on the image data. More specifically, in this disclosure, in the white balance adjustment process, if the gain of other pixels (e.g., red and / or blue pixels) with respect to the brightest reference pixel (e.g., a green pixel) exceeds 1, the range of the ADC is expanded according to that gain. This makes it possible to suppress the degradation of the FWC of the same-color sensitivity difference and improve the FWC.

[0088] In other words, in the embodiments of this disclosure, the white balance gain in the white balance adjustment process is fed back to the sensor 10, thereby enabling the maximization of the FWC according to the scene.

[0089] (3-2. Processing according to the embodiment) Next, the processing according to the embodiment will be described in more detail.

[0090] Figure 10 is an example flowchart illustrating the process according to the embodiment.

[0091] In step S10, the AP20 automatically performs the 3A adjustment process on the image data supplied from the sensor 10 and stored in the frame memory 21 using the AWB unit 201, AF unit 202, and AE unit 203. However, in step S10, only the white balance adjustment process by the AWB unit 201 may be performed.

[0092] In the next step S11, AP20 determines whether the red and / or blue gains resulting from the white balance adjustment process of the AWB unit 201 exceed a threshold. For example, AP20 determines whether the red and / or blue gains relative to the green gain exceed 1.

[0093] If AP20 determines that the gain of red and / or blue relative to the gain of green is 1 or less (step S11, "No"), it returns to step S10. On the other hand, if AP20 determines that the gain of red and / or blue relative to the gain of green is greater than 1 (step S11, "Yes"), it moves the process to step S12. As an example, in the state shown in Figure 8B, the gain for red and / or blue is greater than the gain for green, and it is determined that the gain of red and / or blue relative to the gain of green is greater than 1.

[0094] Note that the threshold in step S11 is not limited to 1; other values ​​greater than 1, such as 1.1, may also be used.

[0095] In step S12, AP20 maximizes the individual adjustment value of the FWC. More specifically, for example, AP20 uses the AWB unit 201 to calculate the range over which ADC103 performs AD conversion processing for pixel signal levels corresponding to the amount of charge Q accumulated in the FD, based on the red and / or blue gains.

[0096] The range calculation for the AD conversion process in step S12 will be explained in more detail. Figures 11A and 11B are schematic diagrams illustrating the process from the photoelectric conversion of the charge by the photoelectric conversion element to its conversion into a digital value, according to the embodiment.

[0097] Figure 11A schematically shows the flow of charge and pixel signals in sensor 10. For example, in each pixel 11001 to 11004 (not shown) in the green pixel block 110, the charge converted photoelectrically by the photoelectric conversion elements 11011 to 11014 (not shown) is transferred to the FD. The charges transferred from each photoelectric conversion element 11011 to 11014 to the FD are accumulated and added in the FD. The charges accumulated in the FD are read out as pixel signals. The ADC 103 performs AD conversion processing on the pixel signals read out from the FD within a set range and outputs pixel data.

[0098] Figure 11B shows the flow of charge and pixel signals in sensor 10, with charge quantity Q S This is a schematic representation focusing on the following. In the example shown in the figure, in the pixel block 110, each photoelectric conversion element 11011 to 11014 (also referred to as G1, G2, G3, and G4 respectively) is formed individually and has variations in saturation charge amount and sensitivity. For example, as shown in the figure, each photoelectric conversion element 11011 to 11014 has an accumulated charge amount Q S The amount of charge Q that is stored varies. In the example shown in the figure, photoelectric conversion elements 11011 and 11013 store more charge than photoelectric conversion elements 11012 and 11014. S There are few.

[0099] In FD, the charges read from these photoelectric conversion elements 11011 to 11014 are added together to form a charge quantity Q. add The charge is accumulated. The variation of the four photoelectric conversion elements 11011~11014 in the pixel block 110 differs for each pixel block 110 and for each sensor 10. Therefore, in FD, the linearity of the charge amount Q with respect to illuminance L is maintained for the charge amount Q. Linear Therefore, the charge quantity Q add The region up to this point (the region of characteristic variation) becomes an unstable region, and the range of pixel signal levels corresponding to the total amount of charge accumulated in the FD (the full range) is not considered part of the effective range in the AD conversion processing of ADC103. Therefore, ADC103 targets a region narrower than the full range of the FD for AD conversion processing.

[0100] In this disclosure, for example, when the sensitivity variation of the brightest color (green) pixel 1100 is suppressed by a 1×1 OCL configuration, variations in other colors (red and / or blue) are tolerated, and the characteristic variation region of the FD is considered smaller. By considering the characteristic variation region of the FD to be smaller, the range of the AD conversion processing by the ADC 103 is expanded. That is, the range of the AD conversion processing by the ADC 103 is adjusted according to the result of the white balance adjustment processing by the AWB unit 201. For example, if the gain for the pixel data of the red and / or blue pixel 1100 is increased, the range of the AD conversion processing for the pixel signal of that pixel 1100 is expanded, and the individual adjustment value of the FWC is maximized.

[0101] Returning to Figure 10, after the processing in step S12, the process moves to step S13. In step S13, the AP20 optimizes the exposure settings according to the FWC maximized in step S12 using the AE unit 203. For example, as the AD conversion processing range is expanded by the processing in step S12, the brightness becomes relatively lower. In this case, the AP20 optimizes the exposure settings using the AE unit 203 to perform brighter exposure according to the AD conversion processing range maximized in step S12. The AP20 may optimize the exposure settings by controlling, for example, the exposure time (shutter speed) in the pixel unit 101 using the AE unit 203.

[0102] For example, AP20 may automatically optimize exposure settings using the AE function inherent in AE unit 203. As an example, AP20 may use AE unit 203 to control the analog gain when ADC103 performs AD conversion processing on the pixel signal. Alternatively, AP20 may use AE unit 203 to change the exposure ratio (such as the shutter speed ratio) according to the ratio of the expanded range of the AD conversion processing. Furthermore, AP20 may use AE unit 203 to optimize exposure settings by referring to a table that associates the AD conversion processing range with the exposure setting value.

[0103] After processing in step S13, AP20 returns to processing in step S10.

[0104] In the example shown in Figure 10, AP20 makes a binary decision in step S11 to determine whether or not to maximize the individual adjustment value of the FWC based on a threshold for the red and / or blue gains resulting from the white balance adjustment process of the AWB unit 201. However, this is not limited to this example. For example, AP20 may continuously control the individual adjustment value of the FWC in accordance with the red and / or blue gains resulting from the white balance adjustment process of the AWB unit 201.

[0105] Thus, in the embodiments of this disclosure, the range of the AD conversion processing by the ADC103 can be expanded according to the result of the white balance adjustment processing. As a result, the amount of charge available in the AD conversion processing increases, which can be expected to expand the dynamic range and improve the signal-to-noise ratio due to the increased amount of charge available. Therefore, by applying the embodiments of this disclosure, in a configuration in which signals read from multiple adjacent pixels are added together and output, the signals from the multiple pixels can be used more efficiently.

[0106] (4. First modified example of the embodiment) Next, a first modification of the embodiment of the present disclosure will be described. The first modification of the embodiment is a modification relating to a combination of multiple pixels 1100 in which an OCL is commonly provided in the pixel block 110.

[0107] In the embodiment described above, as shown in Figure 7, in the pixel block 110 of other colors (e.g., red and blue) relative to the brightest color, one OCL 150 was provided in common for four pixels 11001 to 11004. In contrast, in the first modified embodiment, the pixel block 110 of the other color is divided into regions containing two or more pixels 1100, and in each of the divided regions, one OCL is provided in common for multiple pixels 1100 included in that region.

[0108] Figure 12 is a schematic diagram showing an example of an OCL arrangement according to a first modification of the embodiment. In the example in Figure 12, red and blue pixel blocks 110, each containing four pixels 11001 to 11004, are each divided vertically into two sections, and in each divided region, one OCL 152 is provided in common for two pixels 1100, resulting in a 1×2OCL configuration.

[0109] In this way, by dividing the pixel block 110 into regions containing two or more pixels and providing one OCL 152 for each region, it is possible to suppress variations in the characteristics of pixels 11001 to 11004 with respect to incident light, compared to a configuration in which one OCL 150 is provided in common for all pixels 1100 included in the pixel block 110. Therefore, it is possible to achieve a larger FWC.

[0110] Furthermore, the above is not the only option; for example, the green pixel block 110 may be configured as a 1x2 OCL configuration, and the red and blue pixel blocks 110 may be configured as a 2x2 OCL configuration.

[0111] (5. Second modified example of the embodiment) Next, a second modification of the embodiment will be described. The second modification of the embodiment is an example in which a single ring-shaped OCL (referred to as a ring-shaped OCL) is provided for a pixel block 110 which is arranged in a horizontal 2 pixels × vertical 2 pixels pattern and consists of four pixels 11001 to 11004 of the brightest color.

[0112] Figure 13 is a schematic diagram showing an example of OCL arrangement according to a second modification of the embodiment. In Figure 13, section (a) is a view of each pixel block 110 from the light-receiving surface side, and section (b) is a view of the A-A' cross section shown in section (a). As shown in section (a) of Figure 13, one ring-shaped OCL 153 is provided for a pixel block 110 consisting of four green pixels 11001 to 11004, such that the center of the hole in the ring is located in the center of the pixel block 110. Also, as shown in section (b) of Figure 13, the ring-shaped OCL 153 has a convex shape at the center of the width of the ring.

[0113] With this configuration, each of the four divided portions of the ring-shaped OCL 153 covers each of the pixels 11001 to 11004 of the pixel block 110, making it possible to obtain the same effect as a 1x1 OCL configuration. Furthermore, since one ring-shaped OCL 153 is provided in common for the four pixels 11001 to 11004 of the pixel block 110, it is expected that characteristic variations will be suppressed more effectively compared to a 1x1 OCL configuration.

[0114] (6. Third Modification of the Embodiment) Next, a third modification of the embodiments of this disclosure will be described. The third modification of the embodiments is an example in which a color splitter is used instead of an OCL. The color splitter (metasurface) is formed, for example, by pillar-shaped structures with different refractive indices. The color splitter separates light by wavelength by utilizing the difference in refractive index.

[0115] Figure 14 is a schematic diagram showing an example of a pixel configuration according to a third modification of the embodiment. In Figure 14, section (a) shows an example of a cross-section of a part of the pixel array 111 perpendicular to the light-receiving surface.

[0116] In section (a) of Figure 14, each pixel 1100 includes a photoelectric conversion element 1101, a pixel separation unit 156 that separates adjacent pixels 1100, a light shielding unit 155 that limits the incident light to the photoelectric conversion element 1101, and a color filter 154.

[0117] As shown in section (b) of Figure 14, the color filter 154 is provided in accordance with the extended Bayer array described above, with color filters 154R, 154G, and 154B for each pixel block 110, selectively transmitting light in the red, green, and blue wavelength bands, respectively, such that adjacent pixel blocks 110 have different colors. When it is not necessary to distinguish between color filters 154R, 154G, and 154B, they are referred to simply as color filter 154.

[0118] Furthermore, a color splitter 300 is provided on the light-receiving surface side of the pixel 1100. In a configuration combining OCL and a color filter, the color filter controls light transmission and absorption for light incident from the same coordinate. In contrast, the color splitter 300 separates and focuses light in the required wavelength range from a wider range, i.e., light in a wavelength range similar to that of the color filter 154 directly below it, in the upper layer of the color filter 154. By applying a configuration using the color splitter 300, the amount of light absorbed by the color filter 154 can be reduced, and the sensitivity of the pixel 1100 can be maximized.

[0119] In the examples in sections (a) and (b) of Figure 14, the color splitter 300 focuses light in the red and blue wavelength ranges to the center of red and blue pixel blocks 110, where the pixels 1100 are arranged in a 2x2 horizontal x 2 vertical pattern, respectively. This makes it possible to reduce the difference in sensitivity between the same colors in the red and blue pixel blocks 110. On the other hand, the color splitter 300 focuses light in the green wavelength range to the center of each green pixel 1100. This makes it possible to reduce the difference in sensitivity between the same colors in the green pixel block 110.

[0120] In the configuration of this third modified example of the embodiment, the same configuration as the embodiment can be achieved by feeding back the result of the white balance adjustment process to the AD conversion process by the ADC103, in the same manner as the embodiment. Furthermore, the third modified example of the embodiment can obtain the same effects as the embodiment.

[0121] (7. Fourth Modification of the Embodiment) Next, a fourth modification of the embodiment will be described. The fourth modification of the embodiment is an example in which an organic photoelectric conversion film is used instead of a color filter. The organic photoelectric conversion film is capable of selectively performing photoelectric conversion for light in a specific wavelength band.

[0122] Figure 15 is a schematic diagram showing an example of a pixel configuration according to a fourth modification of the embodiment. Figure 15 shows an example of a cross-section of pixel 1100 perpendicular to the light-receiving surface.

[0123] In each pixel 1100, an OCL 151 is provided on the light-receiving surface side. Below the OCL 151, an organic photoelectric conversion film 160B that selectively converts light in the blue wavelength band into photoelectric energy and an organic photoelectric conversion film 160G that selectively converts light in the green wavelength band into photoelectric energy are stacked, and a photoelectric conversion element 1101 is provided below the organic photoelectric conversion film 160G. In each pixel 1100, the organic photoelectric conversion films 160B and 160G are separated from adjacent pixels 1100 by a separation layer and an electrode 161.

[0124] In this configuration, incident light for pixel 1100 is incident on organic photoelectric conversion film 160B via OCL 151. Organic photoelectric conversion film 160B selectively converts light in the blue wavelength band of the incident light into photoelectric form and stores the charge generated by this photoelectric conversion. Of the light incident on organic photoelectric conversion film 160B, light in wavelength bands other than the blue wavelength band is transmitted through organic photoelectric conversion film 160B and incident on organic photoelectric conversion film 160G. Organic photoelectric conversion film 160G selectively converts light in the green wavelength band of the incident light into photoelectric form and stores the charge generated by this photoelectric conversion. Of the light incident on organic photoelectric conversion film 160G, light in wavelength bands other than the green wavelength band is transmitted through organic photoelectric conversion film 160G and incident on photoelectric conversion element 1101.

[0125] The light incident on the photoelectric conversion element 1101 is light obtained by photoelectric conversion of blue and green wavelength bands by the organic photoelectric conversion films 160B and 160G, and is substantially red wavelength band light. The photoelectric conversion element 1101 photoelectrically converts this red wavelength band light and stores the charge obtained by the photoelectric conversion of said light.

[0126] Thus, the pixel 1100, which includes organic photoelectric conversion films 160B and 160G and a photoelectric conversion element 1101, accumulates blue, green, and red charges, respectively, which are obtained by photoelectric conversion of light in the blue, green, and red wavelength bands, respectively.

[0127] When OCL151 is fabricated with the same curvature, the ease with which light is bent is wavelength-dependent. In the example shown in Figure 15, OCL151 is designed so that incident light is focused at the position of the organic photoelectric conversion film 160G, as shown as light 162 in Figure 15. This makes it possible to achieve the highest sensitivity to green light and suppress characteristic variations between pixels 1100.

[0128] On the other hand, if the design is optimized for green light, it will not be optimized for blue or red light. In this case, reflection and absorption of light will occur in surrounding structures such as the separation tank and electrode 161, which can be a factor in the difference in sensitivity between blue and red light of the same color.

[0129] In this fourth modified example of the embodiment, the same configuration as the embodiment can be achieved by taking advantage of the small variation among the 1100 pixels for a specific color (e.g., green) and feeding back the result of the white balance adjustment process to the AD conversion process by the ADC103, in the same manner as in the embodiment. As a result, the fourth modified example of the embodiment can obtain the same effects as in the embodiment.

[0130] (8. Fifth modified example of the embodiment) Next, a fifth modification of the embodiment of the present disclosure will be described. In the embodiment described above, the color arrangement in each pixel 1100 was described as an RGB arrangement of red, green, and blue, but the color arrangement is not limited to an RGB arrangement. The fifth modification of the embodiment is an example in which the color arrangement in each pixel 1100 is an arrangement different from the RGB arrangement that allows for white balance adjustment processing.

[0131] Figure 16 is a schematic diagram showing an example of a color arrangement applicable to a fifth modified example of the embodiment. In each example in Figure 16, a color filter is applied as an optical element to the pixel 1100, thereby enabling selective reception of light in a specific wavelength band.

[0132] In Figure 16, section (a) is an example of an RYB array, where yellow (Y) is used as the brightest color instead of green in the RGB array. In the example in section (a), the top left is a red (R) pixel block 110 and the bottom right is a blue (B) pixel block 110, with one OCL 150 common to each of the four pixels 1100 contained in each pixel block 110, resulting in a 2×2OCL configuration. In addition, the top right and bottom left are yellow (Y) pixel blocks 110, with one OCL 151 provided for each of the four pixels 1100 contained in each pixel block 110, resulting in a 1×1OCL configuration.

[0133] In Figure 16, section (b) is an example of a CMY array using yellow (Y) as the brightest color, and cyan (C) and magenta (M) as the other colors. In the example of section (b), the upper left is a magenta (M) pixel block 110 and the lower right is a cyan (C) pixel block 110, with one OCL 150 common to each of the four pixels 1100 contained in each pixel block 110, resulting in a 2×2OCL configuration. Also, the upper right and lower left are yellow (Y) pixel blocks 110, with one OCL 151 provided for each of the four pixels 1100 contained in each pixel block 110, resulting in a 1×1OCL configuration.

[0134] For the arrays in sections (a) and (b), the processing for the RGB array described in the embodiment can be applied directly. In the example of section (b), for example, AP20 performs white balance adjustment processing by adjusting the gain for the magenta and / or cyan pixel signal levels with reference to the pixel signal level of the yellow (Y) pixel block 110 using the AWB unit 201 (Figure 10, step S10). If AP20 determines that the gain for magenta and / or cyan relative to yellow exceeds a threshold (Figure 10, step S11, "Yes"), it maximizes the individual adjustment value of the FWC (Figure 10, step S12).

[0135] In Figure 16, section (c) shows an example where emerald green (E) is added to an RGB array. Emerald green (emerald green) is effective in realizing color matching functions and improving color reproduction. In the example in section (c), the red (R) pixel block 110 of the RGB array contains a mixture of red pixels 1100 and emerald green (E) pixels 1100. In addition, an OCL 151 is provided for each pixel 1100 in the pixel block 110, resulting in a 1×1OCL configuration.

[0136] In the arrangement of section (c), if a common OCL 150 is provided for four pixels 1100 in a pixel block 110 where red pixels 1100 and emerald (E) pixels 1100 are mixed, different colors will be mixed and focused onto each photoelectric conversion element 1101 of the pixel block 110. Therefore, the pixel block 110 needs to be 1×1OCL.

[0137] In the arrangement of section (c), the sensitivity difference between the same colors becomes small for green, red, and emerald, which are each configured as 1×1OCL. Therefore, in step S11 of Figure 10, the determination process may be performed using only the result of the white balance adjustment process on the pixel signal level of the blue pixel block 110 among the red, emerald, and blue pixel blocks 110.

[0138] In Figure 16, section (d) is an example of an RGBCMY array. In the example of section (d), the top-left pixel block 110 has two red pixels 1100 and two magenta pixels 1100 arranged alternately, while the top-right and bottom-left pixel blocks 110 have two green pixels 1100 and two yellow pixels 1100 arranged alternately. The bottom-right pixel block 110 has two blue pixels 1100 and two cyan pixels 1100 arranged alternately. Each pixel 1100 is also provided with an OCL 151, resulting in a 1×1OCL configuration.

[0139] As described in the third modification of the above embodiment, the light collection by the OCL 151 can be optimized for a specific color. In the example of section (d), for example, the OCL 151 provided on the green pixel 1100 may be designed to have minimal reflection and absorption of green light (e.g., curvature, height). By designing the OCL 151 in this way, the same-color sensitivity difference for green can be made the smallest compared to other colors. This makes it possible to maximize the individual adjustment value of the FWC based on the white balance gain of other colors, with the pixel signal level of the brightest color (in this case, green) as described in the embodiment.

[0140] In the case of this RGBCMY array, the cyan, magenta, and yellow pixels 1100 are highly sensitive and saturate at lower light levels than the red, green, and blue pixels 1100. Therefore, white balance adjustment may be performed based on the red and / or blue pixel signal levels, similar to the RGB array.

[0141] In Figure 16, section (e) is an example of an RGBW array, in which a white (W) pixel is added to the RGB array. The white pixel 1100 may be realized, for example, by providing a color filter that transmits white light (light in the visible light wavelength range) to the pixel 1100, or by not providing a color filter to the pixel 1100.

[0142] In the example in section (e), the top-left pixel block 110 consists of alternating two red pixels 1100 and two white pixels 1100, while the top-right and bottom-left pixel blocks 110 each consist of alternating two green pixels 1100 and two white pixels 1100. The bottom-right pixel block 110 consists of alternating two blue pixels 1100 and two white pixels 1100. Each pixel 1100 is also provided with an OCL 151, resulting in a 1×1OCL configuration.

[0143] In this RGBW array, the white pixels 1100 are highly sensitive and saturate at lower light levels than the red, green, and blue pixels 1100. Therefore, the white balance adjustment process may be performed based on the red and / or blue pixel signal levels, similar to the RGB array. This makes it possible to maximize the individual adjustment values ​​of the FWC based on the white balance gains of the other colors (red and / or blue) relative to the pixel signal level of the brightest color (in this case, green), as described in the embodiment.

[0144] Furthermore, the effects described herein are merely illustrative and not limiting, and other effects may also occur.

[0145] Furthermore, this technology can also be configured as follows. (1) An imaging unit having a pixel array in which multiple pixels, each outputting a pixel signal according to the light they receive, are arranged in a two-dimensional grid, A conversion unit that performs a conversion process to convert the pixel signals within the range into digital pixel data, A signal processing unit that performs signal processing, including white balance processing, on the aforementioned pixel data, A setting unit that changes the range according to the gain in the white balance processing, Equipped with, electronic equipment. (2) The aforementioned pixel array is Multiple pixel blocks, each containing adjacent pixels, are arranged in a two-dimensional grid, and each pixel in the pixel block selectively receives light of the same wavelength band. The electronic device described in (1) above. (3) The setting unit is, For a first pixel among the plurality of pixels that selectively receives light in a first wavelength band and serves as the reference for the white balance processing, a second pixel among the plurality of pixels that selectively receives light in a second wavelength band different from the first wavelength band, and if the gain of the second pixel, which is less sensitive than the first pixel, exceeds 1, the range is changed according to the gain. The electronic device described in (1) or (2) above. (4) The aforementioned plurality of pixels are The sensitivity difference between the first pixels, which selectively receive light in a first wavelength band from among the plurality of pixels and serve as the reference for the white balance processing, Among the plurality of pixels, light in a second wavelength band different from the first wavelength band is selectively received, and the sensitivity difference between the second pixels, which is less sensitive than the first pixel, is smaller. The electronic device described in any of (1) to (3) above. (5) The aforementioned pixel array is A single on-chip lens is provided for the first pixel, For a pixel block containing a plurality of the second pixels arranged adjacent to each other, the number of on-chip lenses is provided that is less than the number of the second pixels included in the pixel block. The electronic device described in (4) above. (6) The aforementioned pixel array is A single on-chip lens is provided in common to the plurality of second pixels in the aforementioned pixel block. The electronic device described in (5) above. (7) The aforementioned pixel array is For a pixel block containing a plurality of the first pixels arranged adjacent to each other, an on-chip lens is provided in a number of first pixels that is less than the number of first pixels in the pixel block and is common to two or more of the first pixels. A pixel block containing a plurality of the aforementioned second pixels arranged adjacent to each other is provided with a single on-chip lens common to the plurality of the aforementioned second pixels. The electronic device described in (4) above. (8) The aforementioned pixel array is For a pixel block containing four of the aforementioned first pixels, an on-chip lens having a ring shape with a convex central portion is provided such that the central portion of the ring shape is located in the center of the pixel block. The electronic device described in (4) above. (9) The aforementioned pixel array is A color filter is provided for each of the aforementioned plurality of pixels, and a color splitter is further provided on the light-receiving surface side of the color filter. The electronic device described in any of (1) through (8) above. (10) The aforementioned pixel array is Each of the aforementioned plurality of pixels, A first photoelectric conversion unit that performs photoelectric conversion according to the light in a first wavelength band and a second wavelength band, and the light in wavelength bands other than the first and second wavelength bands, included in the incident light. A second photoelectric conversion unit is provided on the incident side of the first photoelectric conversion unit and performs photoelectric conversion according to the light in the first wavelength band contained in the incident light, A third photoelectric conversion unit is provided on the incident side of the second photoelectric conversion unit, which performs photoelectric conversion according to the light in the second wavelength band contained in the incident light, An optical element for focusing the incident light at the position of the second photoelectric conversion unit is provided on the incident surface of each of the plurality of pixels. The electronic device described in (1) above. (11) The aforementioned pixel array is Each contains four pixels arranged adjacent to each other, and sets of four adjacent pixel blocks are repeated. Of the four pixel blocks mentioned above, The first pixel block and the second pixel block have optical elements that selectively transmit light in the wavelength band of the first color, The third pixel block has an optical element that selectively transmits light in the wavelength band of the second color, The fourth pixel block has an optical element that selectively transmits light in the wavelength band of the third color, The first pixel block and the second pixel block are arranged so that they are not adjacent to each other. The electronic device described in any of (1) to (10) above. (12) The first color is green, the second color is blue, and the third color is red. The electronic device described in (11) above. (13) The first color is yellow, the second color is cyan, and the third color is magenta. The electronic device described in (11) above. (14) The aforementioned pixel array is Each contains four pixels arranged adjacent to each other, and sets of four adjacent pixel blocks are repeated. Of the four pixel blocks mentioned above, The first pixel block and the second pixel block have optical elements that selectively transmit light in the green wavelength band. The third pixel block has an optical element that selectively transmits light in the blue wavelength band, The fourth pixel block is provided with an optical element that selectively transmits light in the red wavelength band and an optical element that selectively transmits light in the emerald wavelength band, adjacent to each other for four pixels included in the fourth pixel block. The first pixel block and the second pixel block are arranged so that they are not adjacent to each other. The electronic device described in any of (1) to (10) above. (15) The aforementioned pixel array is Each contains four pixels arranged adjacent to each other, and sets of four adjacent pixel blocks are repeated. Of the four pixel blocks mentioned above, The first and second pixel blocks are arranged such that two optical elements selectively transmit light in the green wavelength band and two optical elements selectively transmit light in the yellow wavelength band are arranged so that no optical elements transmitting light in the same wavelength band are adjacent to each other. The third pixel block is arranged such that two optical elements that selectively transmit light in the blue wavelength band and two optical elements that selectively transmit light in the cyan wavelength band are not adjacent to each other. The fourth pixel block is arranged such that two optical elements that selectively transmit light in the red wavelength band and two optical elements that selectively transmit light in the magenta wavelength band are arranged so that no optical elements that transmit light in the same wavelength band are adjacent to each other. The first pixel block and the second pixel block are arranged so that they are not adjacent to each other. The electronic device described in any of (1) to (10) above. (16) The aforementioned pixel array is Each contains four pixels arranged adjacent to each other, and sets of four adjacent pixel blocks are repeated. Of the four pixel blocks mentioned above, The first and second pixel blocks are arranged such that two optical elements that selectively transmit light in the green wavelength band and two optical elements that transmit light in the white wavelength band are arranged so that no optical elements that transmit light in the same wavelength band are adjacent to each other. The third pixel block is arranged such that two optical elements that selectively transmit light in the blue wavelength band and two optical elements that transmit light in the white wavelength band are not adjacent to any optical elements that transmit light in the same wavelength band. The fourth pixel block is arranged such that two optical elements that selectively transmit light in the red wavelength band and two optical elements that transmit light in the white wavelength band are not adjacent to any optical elements that transmit light in the same wavelength band. The first pixel block and the second pixel block are not adjacent to each other, and the optical elements that transmit light in the white wavelength band are not adjacent to each other. The electronic device described in any of (1) to (10) above. (17) The setting unit is, In accordance with the change in the aforementioned gain, the analog gain for the pixel signal in the conversion process is changed. The electronic device described in any of (1) to (16) above. (18) A display unit that displays a screen based on the image data output by the signal processing unit after the signal processing is performed, Furthermore, The electronic equipment described in (1) through (17) above. (19) A conversion step is performed to convert the pixel signals within a range, output from an imaging unit having a pixel array in which multiple pixels, each outputting a pixel signal according to the light it receives, are arranged in a two-dimensional grid, into digital pixel data. A signal processing step which performs signal processing including white balance processing on the aforementioned pixel data, A setting step of changing the range according to the gain in the white balance processing, Having, Image control method. [Explanation of Symbols]

[0146] 1 Electronic equipment 10 sensors 11 Optics Department 20 AP 21 frame memory 30 Output section 100 Imaging Unit 101 pixel section 102 FD 103 ADC 104,120 Signal Processing Unit 106 Register Control Unit 110 pixel blocks 111 pixel array 130 AD conversion unit 131 AD Converter 150, 151, 152 OCL 153 Ring-shaped OCL 160B,160G Organic photoelectric conversion film 201 AWB Department 202 AF section 203 AE Department 1100, 11001, 11002, 11003, 11004 pixels 1101, 11011, 11012, 11013, 11014 Photoelectric conversion elements

Claims

1. An imaging unit having a pixel array in which multiple pixels, each outputting a pixel signal according to the light they receive, are arranged in a two-dimensional grid, A conversion unit that performs a conversion process to convert the pixel signals within the range into digital pixel data, A signal processing unit that performs signal processing, including white balance processing, on the aforementioned pixel data, A setting unit that changes the range according to the gain in the white balance processing, Equipped with, electronic equipment.

2. The aforementioned pixel array is Multiple pixel blocks, each containing adjacent pixels, are arranged in a two-dimensional grid, and each pixel in a pixel block selectively receives light of the same wavelength band. The electronic device according to claim 1.

3. The setting unit is, For a first pixel among the plurality of pixels that selectively receives light in a first wavelength band and serves as the reference for the white balance processing, a second pixel among the plurality of pixels that selectively receives light in a second wavelength band different from the first wavelength band, and if the gain of the second pixel, which is less sensitive than the first pixel, exceeds 1, the range is changed according to the gain. The electronic device according to claim 1.

4. The aforementioned plurality of pixels are The sensitivity difference between the first pixels, which selectively receive light in a first wavelength band from among the plurality of pixels and serve as the reference for the white balance processing, Among the plurality of pixels, light in a second wavelength band different from the first wavelength band is selectively received, and the sensitivity difference between the second pixels, which is less sensitive than the first pixel, is smaller. The electronic device according to claim 1.

5. The aforementioned pixel array is A single on-chip lens is provided for the first pixel, For a pixel block containing a plurality of the second pixels arranged adjacent to each other, an on-chip lens is provided that is less than the number of the second pixels included in the pixel block. The electronic device according to claim 4.

6. The aforementioned pixel array is A single on-chip lens is provided in common to the plurality of second pixels in the aforementioned pixel block. The electronic device according to claim 5.

7. The aforementioned pixel array is For a pixel block containing a plurality of the first pixels arranged adjacent to each other, an on-chip lens is provided in a number of first pixels that is less than the number of first pixels in the pixel block and is common to two or more of the first pixels. A pixel block containing a plurality of the second pixels arranged adjacent to each other is provided with a single on-chip lens common to the plurality of the second pixels. The electronic device according to claim 4.

8. The aforementioned pixel array is For a pixel block containing four of the aforementioned first pixels, an on-chip lens having a ring shape with a convex central portion is provided such that the central portion of the ring shape is located in the center of the pixel block. The electronic device according to claim 4.

9. The aforementioned pixel array is A color filter is provided for each of the aforementioned plurality of pixels, and a color splitter is further provided on the light-receiving surface side of the color filter. The electronic device according to claim 1.

10. The aforementioned pixel array is Each of the aforementioned plurality of pixels, A first photoelectric conversion unit performs photoelectric conversion according to the light in a first wavelength band and a second wavelength band, and the light in wavelength bands other than the first and second wavelength bands, which are included in the incident light. A second photoelectric conversion unit is provided on the incident side of the first photoelectric conversion unit and performs photoelectric conversion according to the light in the first wavelength band contained in the incident light, A third photoelectric conversion unit is provided on the incident side of the second photoelectric conversion unit, which performs photoelectric conversion according to the light in the second wavelength band contained in the incident light, An optical element for focusing the incident light at the position of the second photoelectric conversion unit is provided on the incident surface of each of the plurality of pixels. The electronic device according to claim 1.

11. The aforementioned pixel array is Each contains four pixels arranged adjacent to each other, and sets of four adjacent pixel blocks are repeated. Of the four pixel blocks mentioned above, The first pixel block and the second pixel block have optical elements that selectively transmit light in the wavelength band of the first color, The third pixel block has an optical element that selectively transmits light in the wavelength band of the second color, The fourth pixel block has an optical element that selectively transmits light in the wavelength band of the third color, The first pixel block and the second pixel block are arranged so that they are not adjacent to each other. The electronic device according to claim 1.

12. The first color is green, the second color is blue, and the third color is red. The electronic device according to claim 11.

13. The first color is yellow, the second color is cyan, and the third color is magenta. The electronic device according to claim 11.

14. The aforementioned pixel array is Each contains four pixels arranged adjacent to each other, and sets of four adjacent pixel blocks are repeated. Of the four pixel blocks mentioned above, The first pixel block and the second pixel block have optical elements that selectively transmit light in the green wavelength band. The third pixel block has an optical element that selectively transmits light in the blue wavelength band, The fourth pixel block is provided with an optical element that selectively transmits light in the red wavelength band and an optical element that selectively transmits light in the emerald wavelength band, adjacent to each other for four pixels included in the fourth pixel block. The first pixel block and the second pixel block are arranged so that they are not adjacent to each other. The electronic device according to claim 1.

15. The aforementioned pixel array is Each contains four pixels arranged adjacent to each other, and sets of four adjacent pixel blocks are repeated. Of the four pixel blocks mentioned above, The first pixel block and the second pixel block are arranged such that two optical elements that selectively transmit light in the green wavelength band and two optical elements that selectively transmit light in the yellow wavelength band are arranged so that no optical elements that transmit light in the same wavelength band are adjacent to each other. The third pixel block is arranged such that two optical elements that selectively transmit light in the blue wavelength band and two optical elements that selectively transmit light in the cyan wavelength band are arranged so that no optical elements that transmit light in the same wavelength band are adjacent to each other. The fourth pixel block is arranged such that two optical elements that selectively transmit light in the red wavelength band and two optical elements that selectively transmit light in the magenta wavelength band are arranged so that no optical elements that transmit light in the same wavelength band are adjacent to each other. The first pixel block and the second pixel block are arranged so that they are not adjacent to each other. The electronic device according to claim 1.

16. The aforementioned pixel array is Each contains four pixels arranged adjacent to each other, and sets of four adjacent pixel blocks are repeated. Of the four pixel blocks mentioned above, The first pixel block and the second pixel block are arranged such that two optical elements that selectively transmit light in the green wavelength band and two optical elements that transmit light in the white wavelength band are arranged so that no optical elements that transmit light in the same wavelength band are adjacent to each other. The third pixel block is arranged such that two optical elements that selectively transmit light in the blue wavelength band and two optical elements that transmit light in the white wavelength band are not adjacent to each other. The fourth pixel block is arranged such that two optical elements that selectively transmit light in the red wavelength band and two optical elements that transmit light in the white wavelength band are arranged so that no optical elements that transmit light in the same wavelength band are adjacent to each other. The first pixel block and the second pixel block are not adjacent to each other, and the optical elements that transmit light in the white wavelength band are not adjacent to each other. The electronic device according to claim 1.

17. The setting unit is, In accordance with the change in the aforementioned gain, the analog gain for the pixel signal in the conversion process is changed. The electronic device according to claim 1.

18. A display unit that displays a screen based on the image data output by the signal processing unit after the signal processing is performed, Furthermore, The electronic device according to claim 1.

19. A conversion step is performed to convert the pixel signals within a range, output from an imaging unit having a pixel array in which multiple pixels, each outputting a pixel signal according to the light they receive, are arranged in a two-dimensional grid, into digital pixel data. A signal processing step which performs signal processing including white balance processing on the aforementioned pixel data, A setting step of changing the range according to the gain in the white balance processing, Having, Image capture control method.

Images