Image processing device and image sensor

Abstract

To provide an image processing device and an imaging element that generate a high-dynamic-range image by converting the brightness of a pixel signal using a conversion function which converts the input brightness of the pixel signal into an output brightness.SOLUTION: A brightness conversion method by an imaging element generates an integrated conversion function obtained by combining a plurality of conversion functions that includes a first conversion function which converts the input brightness of a pixel signal into an output brightness and a second conversion function which converts the input brightness into a different output brightness from the foregoing output brightness, and converts the brightness of the pixel signal using the integrated conversion function. The first conversion function converts the input brightness into the output brightness with a first brightness being taken as a reference brightness. The second conversion function converts the input brightness into the output brightness with a second brightness different from the first brightness being taken as a reference brightness. In the brightness distribution of the pixel signals, a first band including the first brightness and a second band including the second brightness are decided, and the combining ratio of the plurality of conversion functions is decided in accordance with the bandwidth of the first band and that of the second band.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to an image processing device and an imaging device. [Background technology]

[0002] Patent Document 1 describes a tone mapping technique for reproducing real-world scenes into high dynamic range images. [Prior art document] [Patent documents] [Non-patent document 1] Erik Reinhard, et al., "Photographic Tone Reproduction for Digital Images", SIGGRAPH '02: Proceedings of the 29th annual conference on Computer graphics and interactive techniques, July 2002, pp. 267-276, https: / / doi.org / 10.1145 / 566570.566575 Summary of the Invention

[0003] A first aspect of the present invention provides an image processing device, comprising a signal processing unit that generates an integrated conversion function by combining a plurality of conversion functions, including a first conversion function that converts an input luminance of a pixel signal to an output luminance, and a second conversion function that converts the input luminance to an output luminance different from the output luminance, and converts the luminance of the pixel signal using the integrated conversion function.

[0004] A second aspect of the present invention provides an image processing device, comprising: a signal processing unit that converts the luminance of a pixel signal using a conversion function that converts the input luminance of the pixel signal to an output luminance, the conversion function having three or more inflection points.

[0005] A third aspect of the present invention provides an image sensor comprising a plurality of pixels that output pixel signals in response to incident light, and a signal processing unit that performs arithmetic processing required for image processing on the pixel signals, wherein the signal processing unit generates an integrated conversion function that combines a plurality of conversion functions, including a first conversion function that converts input luminance to output luminance and a second conversion function that converts the input luminance to an output luminance different from the output luminance, and converts the luminance of the pixel signals using the integrated conversion function.

[0006] The above summary of the invention does not list all of the necessary features of the present invention, and subcombinations of these features may also constitute inventions. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a cross-sectional view of an image sensor 100 according to the present embodiment. [Figure 2] 1 is a diagram illustrating the pixel array and blocks 131 of the imaging chip 113. FIG. [Figure 3] 10 is a circuit diagram corresponding to block 131 of imaging chip 113. FIG. [Figure 4] FIG. 2 is a block diagram showing the functional configuration of the image sensor 100. [Figure 5] FIG. 2 is a block diagram showing the functional configuration of the image sensor 100. [Figure 6] FIG. 1 is a block diagram showing the configuration of an imaging device 500 according to the present embodiment. [Figure 7] FIG. 10 is a diagram illustrating an example of a conversion function. [Figure 8] FIG. 10 is a flowchart illustrating an example of luminance conversion. [Figure 9] 10A and 10B are diagrams illustrating an example of a step of determining a reference luminance and a band from a luminance histogram. [Figure 10A] FIG. 10 is a diagram showing an example of a captured image in which luminance has been converted using a conversion function according to the prior art. [Figure 10B] FIG. 10 is a diagram showing an example of a captured image in which luminance has been converted using a conversion function according to the prior art. [Figure 10C]FIG. 10 is a diagram showing an example of a captured image in which luminance has been converted using an integrated conversion function according to the present embodiment. [Figure 11A] FIG. 10 is a diagram showing an example of an image in which luminance has been converted in consideration of feature points. [Figure 11B] FIG. 10 is a diagram illustrating an example of a saliency map. [Figure 12] 10A and 10B are diagrams illustrating another example of the step of determining the reference luminance and the band from the luminance histogram. DETAILED DESCRIPTION OF THE INVENTION

[0008] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0009] 1 is a cross-sectional view of an image sensor 100 according to this embodiment. The image sensor 100 includes an image sensor chip 113 that outputs pixel signals corresponding to incident light, a signal processing chip 111 that processes the pixel signals, and a memory chip 112 that stores the pixel signals. The image sensor chip 113, signal processing chip 111, and memory chip 112 are stacked and electrically connected to one another by bumps 109, which are conductive connecting portions made of Cu or the like.

[0010] As shown in the figure, incident light is mainly incident in the positive direction of the Z axis, as indicated by the white arrow. In this embodiment, the surface of the imaging chip 113 on which the incident light is incident is referred to as the back surface. As shown by the coordinate axes, the left direction on the paper, perpendicular to the Z axis, is the positive X axis, and the front direction on the paper, perpendicular to the Z axis and the X axis, is the positive Y axis. In the following figures, the coordinate axes are displayed so that the orientation of each figure can be understood, based on the coordinate axes in Figure 1.

[0011] An example of the imaging chip 113 is a back-illuminated MOS image sensor. The PD layer 106 is arranged on the back side of the wiring layer 108. The PD layer 106 has a plurality of PDs (photodiodes) 104 arranged two-dimensionally and transistors 105 provided corresponding to the PDs 104.

[0012] A color filter 102 is provided on the incident light side of the PD layer 106 via a passivation film 103. There are multiple types of color filters 102 that transmit different wavelength ranges, and each has a specific arrangement corresponding to each PD 104. The arrangement of the color filters 102 will be described later. A set of a color filter 102, a PD 104, and a transistor 105 forms one pixel.

[0013] A microlens 101 is provided corresponding to each pixel on the incident light side of the color filter 102. The microlens 101 condenses the incident light toward the corresponding PD 104.

[0014] The wiring layer 108 has wiring 107 that transmits pixel signals from the PD layer 106 to the signal processing chip 111. The wiring 107 may be multi-layered, and may be provided with passive elements and active elements.

[0015] A plurality of bumps 109 serving as connection portions are arranged on the surface of the wiring layer 108. The plurality of bumps 109 are aligned with and joined to a plurality of bumps 109 provided on the opposing surface of the signal processing chip 111, thereby electrically connecting the imaging chip 113 and the signal processing chip 111.

[0016] Similarly, a plurality of bumps 109 serving as connecting portions are arranged on the opposing surfaces of the signal processing chip 111 and the memory chip 112. These bumps 109 are aligned and joined to each other, thereby electrically connecting the signal processing chip 111 and the memory chip 112.

[0017] The imaging element 100 is formed by bonding the imaging chip 113, the signal processing chip 111, and the memory chip 112 in a wafer state before they are separated into chips, and then dicing the bonded wafer.

[0018] When bonding wafers together, plasma is applied to the wafer surfaces using an activation device to activate the bonding surfaces of the wafers. The wafers with activated surfaces are bonded together by hydrogen bonds, van der Waals bonds, covalent bonds, and the like that are formed by contact, forming a laminated substrate. If the two wafers are hydrogen bonded by contact with each other, after forming the laminated substrate, the laminated substrate is placed in a heating device such as an annealing furnace and heated to form covalent bonds between the wafers.

[0019] The term "activation" includes treating the bonding surface of at least one of the substrates so that when the bonding surface of one wafer comes into contact with the bonding surface of another wafer, hydrogen bonds, van der Waals bonds, covalent bonds, etc. are generated, resulting in solid-state bonding without melting. In other words, activation includes generating dangling bonds (unbonded hands) on the wafer surface, thereby making it easier to form bonds.

[0020] More specifically, in an activation device, oxygen gas, which is a process gas, is excited to form plasma in a reduced-pressure atmosphere, and oxygen ions are irradiated onto the surfaces that will become the bonding surfaces of the two substrates. For example, if the wafer is a substrate with an SiO film formed on Si, the irradiation of this oxygen ion breaks the SiO bonds on the wafer surfaces that will become the bonding surfaces during lamination, forming dangling bonds of Si and O. The formation of such dangling bonds on the wafer surfaces is sometimes referred to as activation.

[0021] When a substrate with dangling bonds is exposed to the atmosphere, for example, moisture in the air binds to the dangling bonds, and the substrate surface becomes covered with hydroxyl groups (OH groups). The substrate surface becomes hydrophilic, making it more likely to bond with water molecules. In other words, activation results in the substrate surface becoming hydrophilic. Furthermore, in solid-state bonding, the presence of impurities such as oxides at the bonding interface, as well as defects at the bonding interface, affect the bonding strength. Therefore, cleaning the bonding surface can be considered part of the activation process.

[0022] Furthermore, the wafer may be activated by applying pure water or the like to the hydrophilized surface of the wafer that will become the bonding surface using an apparatus not shown, so that the wafer surface is in a state in which OH groups are attached, i.e., is terminated with OH groups.

[0023] By heating the laminated substrate, the bumps 109 on each of the bonding surfaces of the two wafers are integrated with each other, forming an electrical connection between the wafers. By forming the bumps 109 from a material that melts at a low temperature, such as indium or a tin-silver alloy, the laminated substrate can be reflow-treated at a low temperature of 200°C or less. Alternatively, if the bumps 109 are made of a conductive metal such as copper, they expand during the heat treatment, causing the bumps 109 between the wafers to press together and bond by solid-state diffusion.

[0024] The bonding between the bumps 109 is not limited to Cu bump bonding by solid-phase diffusion, but may also employ micro-bump bonding by solder melting. For example, it is sufficient to provide one bump 109 for each pixel block described below. Therefore, the size of the bumps 109 may be larger than the pitch of the PDs 104. Furthermore, in a peripheral region other than the pixel region where the pixels are arranged, bumps larger than the bumps 109 corresponding to the pixel region may also be provided.

[0025] The signal processing chip 111 has TSVs (through silicon vias) 110 that connect circuits provided on the front and back surfaces of the chip to each other. The TSVs 110 are preferably provided in the peripheral region. The TSVs 110 may also be provided in the peripheral region of the imaging chip 113 and in the memory chip 112. The TSVs 110 may also be used to electrically connect circuits provided on the memory chip 112 and circuits provided on the imaging chip 113.

[0026] In this way, the imaging chip 113 and the signal processing chip 111 are bonded to each other by their opposing surfaces and the bumps 109. The signal processing chip 111 and the memory chip 112 are bonded to each other by their opposing surfaces and connected to each other by the bumps 109 and the TSVs 110 provided on the signal processing chip 111. Note that the signal processing chip 111 and the memory chip 112 may be bonded to each other by their opposing surfaces and connected to each other by at least one of the bumps 109 and the TSVs 110.

[0027] FIG. 2 is a diagram illustrating the pixel array and blocks 131 of the imaging chip 113. In particular, the imaging chip 113 is shown as viewed from the back side. More than 20 million pixels are arranged in a matrix in the pixel region. These pixels are divided into blocks, each containing at least two pixels. In this embodiment, one block consists of 16 pixels, or 4 pixels by 4 pixels. The grid lines in the diagram illustrate the concept of adjacent pixels collectively forming the block 131.

[0028] As shown in the partially enlarged view of the pixel region, block 131 contains four so-called Bayer arrays, arranged vertically and horizontally, each consisting of four pixels: green pixels Gb and Gr, blue pixels B, and red pixels R. The green pixels are pixels that have a green filter as the color filter 102 and receive light in the green wavelength band of incident light. Similarly, the blue pixels are pixels that have a blue filter as the color filter 102 and receive light in the blue wavelength band, and the red pixels are pixels that have a red filter as the color filter 102 and receive light in the red wavelength band.

[0029] 3 is a circuit diagram corresponding to block 131 of imaging chip 113. In the figure, a rectangle surrounded by a dotted line typically represents a circuit corresponding to one pixel. Note that at least some of the transistors described below correspond to transistor 105 in FIG. 1.

[0030] As described above, the block 131 is formed of 16 pixels. The 16 PDs 104 corresponding to the respective pixels are connected to transfer transistors 302, and each gate of each transfer transistor 302 is connected to a TX wiring 307 through which a transfer pulse is supplied. In this embodiment, the TX wiring 307 is commonly connected to the 16 transfer transistors 302.

[0031] The drain of each transfer transistor 302 is connected to the source of the corresponding reset transistor 303, and a so-called floating diffusion FD between the drain of the transfer transistor 302 and the source of the reset transistor 303 is connected to the gate of the amplification transistor 304. The drain of the reset transistor 303 is connected to a Vdd wiring 310 to which a power supply voltage is supplied, and the gate of the reset transistor 303 is connected to a reset wiring 306 to which a reset pulse is supplied. In this embodiment, the reset wiring 306 is commonly connected to the 16 reset transistors 303.

[0032] The drain of each amplification transistor 304 is connected to a Vdd wiring 310 to which a power supply voltage is supplied. The source of each amplification transistor 304 is connected to the drain of a corresponding selection transistor 305. The gate of each selection transistor is connected to a decoder wiring 308 to which a selection pulse is supplied. In this embodiment, the decoder wiring 308 is provided independently for each of the 16 selection transistors 305. The sources of each selection transistor 305 are connected to a common output wiring 309. A load current source 311 supplies a current to the output wiring 309. In other words, the output wiring 309 for the selection transistor 305 is formed by a source follower. The load current source 311 may be provided on the imaging chip 113 side or on the signal processing chip 111 side.

[0033] Here, we will explain the flow from the start of pixel exposure to the output of a pixel signal after exposure is completed. When a reset pulse is applied to the reset transistor 303 via the reset wiring 306 and at the same time a transfer pulse is applied to the transfer transistor 302 via the TX wiring 307, the potentials of the PD 104 and floating diffusion FD are reset and exposure begins.

[0034] When the transfer pulse application is stopped, the PD 104 converts the incident light it receives into electric charges and accumulates them. Then, when the transfer pulse is applied again without the reset pulse being applied, exposure ends. The electric charges accumulated up until the end of exposure are transferred to the floating diffusion FD, and the potential of the floating diffusion FD changes from the reset potential to the signal potential after exposure ends. When a selection pulse is applied to the selection transistor 305 via the decoder wiring 308, the fluctuation in the signal potential of the floating diffusion FD is transmitted to the output wiring 309 via the amplification transistor 304 and the selection transistor 305. As a result, a pixel signal corresponding to the reset potential and the signal potential is output from the unit pixel to the output wiring 309.

[0035] As shown in the figure, in this embodiment, the reset wiring 306 and TX wiring 307 are common to the 16 pixels that form the block 131. That is, the reset pulse and transfer pulse are each applied simultaneously to all 16 pixels. Therefore, all pixels that form the block 131 start and end exposure at the same timing. However, pixel signals corresponding to the accumulated charges are selectively output to the output wiring 309 by sequentially applying selection pulses to the respective selection transistors 305.

[0036] By configuring the circuit based on the block 131 in this way, the exposure time can be controlled for each block 131. Because the exposure time can be controlled for each block, adjacent blocks 131 can output pixel signals with different exposure times. Furthermore, a common exposure time can be set for all blocks 131, and a certain block 131 can be exposed once and output a pixel signal, while an adjacent block 131 can be repeatedly exposed twice and output a pixel signal. This latter type of repeated control of exposure and pixel signal output based on a common unit time is called unit time control. Note that when unit time control is performed, if the start and end points of exposure are synchronized across all blocks 131, the reset wiring 306 may be commonly connected to all reset transistors 303 on the imaging chip 113.

[0037] 4 is a block diagram showing the functional configuration of the image sensor 100. Here, the flow of pixel signals will be particularly described.

[0038] An analog multiplexer 411 sequentially selects the 16 PDs 104 that form the block 131 and outputs the respective pixel signals to the output wiring 309. The multiplexer 411 is formed in the imaging chip 113 together with the PDs 104.

[0039] The pixel signals output via the multiplexer 411 undergo correlated double sampling (CDS) and analog-to-digital (A / D) conversion by a signal processing circuit 412 formed in the signal processing chip 111, which performs CDS and A / D conversion. The A / D conversion converts the input analog pixel signals into 12-bit digital pixel signals. The A / D converted pixel signals are passed to an arithmetic circuit 415, also formed in the signal processing chip 111. The arithmetic circuit 415 performs arithmetic processing on the received pixel signals required for subsequent image processing, and passes the signals to the demultiplexer 413.

[0040] The demultiplexer 413 stores the received pixel signals in pixel memories 414 corresponding to the respective pixels. Each pixel memory 414 has a capacity capable of storing the pixel signals after arithmetic processing has been performed. The demultiplexer 413 and pixel memories 414 are formed on the memory chip 112.

[0041] The arithmetic circuit 415 reads pixel signals to be used for arithmetic processing from the pixel memory 414 via the demultiplexer 413. Alternatively, in accordance with an external transfer request, the arithmetic circuit 415 transfers the pixel signals read from the pixel memory 414 via the demultiplexer 413 to a downstream image processing unit. The arithmetic circuit 415 may be provided in the memory chip 112.

[0042] Also, while the figure shows the flow of pixel signals for one block, in reality these exist for each block and operate in parallel. However, a calculation circuit 415 does not have to exist for each block; for example, one calculation circuit 415 may process sequentially while referring to the values ​​of the pixel memories 414 corresponding to each block in order.

[0043] 5 is a block diagram showing the functional configuration of the image sensor 100. Here, the specific configuration of the signal processing chip 111 and the setting unit 460 provided in the signal processing chip 111 will be mainly described.

[0044] The signal processing chip 111 includes a sensor control unit 441, a synchronization control unit 443, and a signal control unit 444 as distributed control functions, and a drive control unit 420 that controls these control units in an integrated manner. The drive control unit 420 is a control circuit that converts instructions from a system control unit 501 that is responsible for integrated control of the entire imaging device into control signals that can be executed by each control unit and passes them on to each unit.

[0045] The sensor control unit 441 is responsible for controlling the transmission of control pulses related to charge accumulation and charge readout of each pixel, which are sent to the imaging chip 113. Specifically, the sensor control unit 441 controls the start and end of exposure by sending reset pulses and transfer pulses to the target pixels, and outputs pixel signals to the output wiring 309 by sending selection pulses to the readout pixels.

[0046] The synchronization control unit 443 sends a synchronization signal to the imaging chip 113. Each pulse becomes active in the imaging chip 113 in synchronization with the synchronization signal. For example, by adjusting the synchronization signal, random control, thinning control, and the like can be realized, in which only specific pixels belonging to the same block 131 are the control targets.

[0047] The signal control unit 444 is mainly responsible for timing control of the A / D converter 412b. The pixel signals output via the output wiring 309 are input to the CDS circuit 412a and the A / D converter 412b via the multiplexer 411. The A / D converter 412b is controlled by the signal control unit 444 and converts the input pixel signals into digital signals. The converted digital pixel signals are passed to the arithmetic circuit 415, where they are subjected to arithmetic processing. The processed pixel signals are passed to the demultiplexer 413 of the memory chip 112 and stored as digital pixel values ​​in the pixel memories 414 corresponding to each pixel.

[0048] The drive control unit 420 sends a control signal to the sensor control unit 441 in accordance with the exposure conditions for each block 131. The exposure conditions are conditions for changing the brightness of the image to be acquired, such as exposure time, aperture value, ISO sensitivity, etc. The drive control unit 420 may set the exposure conditions for each block 131 in accordance with the luminance distribution of the scene acquired from the system control unit 501, or may acquire the exposure conditions for each block 131 from the system control unit 501.

[0049] In response to a transfer request from the system control unit 501, the drive control unit 420 reads the target pixel signal from the pixel memory 414 via the arithmetic circuit 415 and demultiplexer 413, and transfers the signal to the image processing unit 511 of the system control unit 501. The pixel memory 414 is provided with a data transfer interface that transmits the pixel signal in response to the transfer request. The data transfer interface is connected to a data transfer line that is connected to the image processing unit 511. The data transfer line is formed, for example, by a data bus among the bus lines. In this case, the transfer request from the system control unit 501 to the drive control unit 420 is executed by address specification using the address bus.

[0050] The transmission of pixel signals via the data transfer interface is not limited to the addressing method, and various other methods can be used. For example, a double data rate method can be used, which uses both the rising and falling edges of the clock signal used to synchronize each circuit when transferring data. A burst transfer method can also be used, which transfers data all at once by omitting some steps such as addressing, thereby increasing speed. It is also possible to use a combination of a bus method using lines connecting the control unit, memory unit, and input / output unit in parallel, and a serial method that transfers data one bit at a time in series.

[0051] With this configuration, the image processing unit 511 can receive only the necessary pixel signals, and therefore can complete image processing at high speed, especially when forming a low-resolution image.

[0052] The signal processing chip 111 further includes a setting unit 460 that sets exposure conditions for each block 131. The setting unit 460 acquires the luminance distribution of the scene from the system control unit 501, and sets exposure conditions for each block 131 according to the luminance distribution. The exposure conditions are conditions that change the brightness of the image to be acquired, such as exposure time, aperture value, ISO sensitivity, etc. The setting unit 460 also determines the timing of opening and closing the shutter according to the exposure time of each block 131.

[0053] The setting unit 460 is provided on the same signal processing chip 111 as the drive control unit 420 that controls each control unit, thereby realizing faster processing. Furthermore, instead of providing the setting unit 460 separately from the drive control unit 420, the drive control unit 420 may take on the function of the setting unit 460.

[0054] Furthermore, the setting unit 460 may be provided in the memory chip 112 instead of the signal processing chip 111. When the setting unit 460 is provided in the memory chip 112, it is possible to increase the transmission speed via the bumps 109 and the TSVs 110, while ensuring a larger space in the signal processing chip 111.

[0055] 6 is a block diagram showing the configuration of an image capturing apparatus 500 according to this embodiment. The image capturing apparatus 500 mainly includes an image sensor 100, a photographing lens 520, a system control unit 501, a photometry unit 503, a work memory 504, a recording unit 505, and a display unit 506.

[0056] The photometry unit 503 detects the luminance distribution of a scene prior to a series of shooting sequences for generating image data. The photometry unit 503 includes, for example, an AE sensor with about one million pixels.

[0057] The photographing lens 520 guides a subject light beam incident along the optical axis O to the image sensor 100. The photographing lens 520 is an example of an optical system. The photographing lens 520 is composed of a group of optical lenses, and focuses the subject light beam from the scene near its focal plane. Note that in FIG. 6, the photographing lens 520 is represented by a single virtual lens placed near the pupil. The photographing lens 520 may be an interchangeable lens that can be attached to and detached from the image capturing device 500.

[0058] The image processing unit 511 receives pixel signals from the drive control unit 420 of the image sensor 100. The image processing unit 511 performs various image processing using the work memory 504 as a workspace to generate image data. For example, the image processing unit 511 detects feature points from multiple captured images and synthesizes the multiple images based on the feature points to generate final image data. When generating image data in JPEG file format, the image processing unit 511 performs white balance processing, gamma processing, etc., and then performs compression processing. The generated image data is recorded in the recording unit 505 and converted into a display signal, which is displayed on the display unit 506 for a predetermined time. The image processing unit 511 is an example of a signal processing unit that generates a luminance conversion function, and may be configured as an ASIC independent of the system control unit 501 or may be provided in the memory chip 112.

[0059] The system control unit 501 is responsible for integrated control of the entire imaging device. Upon receiving instructions from a user, the system control unit 501 generates imaging instructions to be sent to the image sensor 100. Furthermore, the calculation unit 512 of the system control unit 501 receives output from the photometry unit 503 and calculates the luminance for each region of the scene. The calculation unit 512 outputs the luminance distribution of the scene to the setting unit 460 described above. The calculation unit 512 also performs various calculations for operating the imaging device 500.

[0060] The calculation unit 512 may perform the functions of the setting unit 460. In this case, the calculation unit 512 sets exposure conditions for each block 131 according to the luminance distribution of the scene. The calculation unit 512 also determines the timing of opening and closing the shutter according to the exposure conditions set for each block 131.

[0061] Fig. 7 is a diagram showing an example of a conversion function. Before explaining Fig. 7, a conversion function used in a tone mapping process that converts the luminance of a real-world scene into the luminance for an image will be explained.

[0062] The conversion function is an increasing function that converts input luminance into a smaller output luminance based on a reference luminance. The conversion function G(x) is a sigmoid function. The conversion function converts input luminance into output luminance based on the reference luminance, the average luminance of pixel signals in the region where the pixel is located, and a predetermined brightness adjustment parameter. For example, the Reinhard conversion function is known as such a conversion function.

number

[0063] The reference luminance Lm(x) of the pixel at coordinate x is expressed by the above formula. The reference luminance is the luminance obtained by scaling the input luminance Lw(x). Here, α is a brightness adjustment parameter, L w,Global is the average brightness of the entire image (overall average brightness). The brightness adjustment parameter α is a value between 0 and 1, and different values ​​can be set depending on the brightness of the entire image. For example, to maintain the image in bright tones, the brightness adjustment parameter α is changed from 0.18 to 0.36 and 0.72, and to maintain the image in dark tones, the brightness adjustment parameter α is changed from 0.18 to 0.09 and 0.045.

[0064] The output luminance Ld(x) is expressed by the following formula:

number

[0065] where L m,Local (x) is the average luminance (local average luminance) in the region where the pixel is located. Putting these equations together, the conversion function that converts input luminance to output luminance is as follows:

number

[0066] Here, G(x) is the conversion function, X(x) is the input luminance of each RGB value, and X'(x) is the output luminance of each RGB value. These formulas for the conversion function G(x) can be summarized as follows:

number

[0067] As can be seen from the above formula, the conversion function G(x) is w,Global , is determined by the local average brightness and the brightness adjustment parameter α. Here, the global average brightness L w,Global Since the local average luminance and the local average luminance are preset values, it can be seen that the only parameter for adjusting the luminance band of the conversion result is the brightness adjustment parameter α.

[0068] Next, referring to FIG. 7, the conversion functions s1, s2, s3, and s123 that convert input luminance Lw to output luminance Ld will be described. The horizontal axis represents the luminance of input luminance Lw, and the vertical axis represents the luminance of output luminance Ld. Each of the conversion functions s1, s2, and s3 is a function that uses one reference luminance as a reference and draws an S-shaped curve with two inflection points before and after the rising edge. The conversion function s2 uses the overall average luminance as the reference luminance. The conversion functions s1 and s3 use luminances lower and higher than the overall average luminance as the reference luminance, respectively, and emphasize luminance bands darker and brighter than the overall average luminance.

[0069] In both conversion functions, when the input luminance Lw is within a certain range centered on the reference luminance, the output luminance Ld functions as an increasing function that varies between 0 and 1, but outside this range the output luminance Ld is compressed to a value of either 0 or 1, and the input luminance Lw is essentially ignored. Figure 7 shows the range in which the input luminance Lw is ignored in conversion function s2.

[0070] On the other hand, the conversion function s123 is an integrated conversion function that combines the conversion functions s1, s2, and s3 according to the following formula.

number

[0071] Ld(x) is the output luminance obtained by combining N conversion functions. m,n (x) is the reference luminance of each conversion function, where n = 1, ..., N (here, N = 3 because three conversion functions s1, s2, and s3 are combined). L m,Local,n (x) is the local average brightness of each transformation function,

number

number

[0072] 7, the conversion function s123 has three or more inflection points because it combines multiple conversion functions with different reference luminances. With such a conversion function s123, a wider range of input luminance Lw can be converted without being compressed, and the luminance distribution of a real-world scene can be reproduced more accurately.

[0073] Fig. 8 is a flow diagram showing an example of brightness conversion. In the description of Fig. 8, the image processing unit 511 is assumed to be the main operator of each step, but this is not limiting. In another example, the setting unit 460 and the arithmetic circuit 415 of the image sensor 100 may be the main operator of each step, and in this case, these may be referred to as a signal processing unit as an integrated functional block.

[0074] Here, an example will be described in which an integrated conversion function is generated from N conversion functions. In step S102, the image processing unit 511 calculates the overall average luminance and the local average luminance of the image according to the luminance distribution of the scene. In step S104, the image processing unit 511 obtains a luminance histogram. The luminance histogram may be a logarithmic histogram.

[0075] The image processing unit 511 may acquire a luminance histogram based on the exposure conditions set for each pixel block 131. In this way, by using the luminance histogram generated to set the exposure conditions for each block 131, it is possible to omit the logarithmic conversion process of the luminance.

[0076] In step S106, the image processing unit 511 determines a reference luminance and a band for each of the N conversion functions. The N reference luminances are different from one another, and may be, for example, an overall average luminance calculated from a luminance histogram, an average luminance higher than the overall average luminance, or an average luminance lower than the overall average luminance. The image processing unit 511 may determine each band so that the median value of the reference luminance becomes the boundary of the band.

[0077] In step S108, the image processing unit 511 determines the combination ratio of the N number of transformation functions. The image processing unit 511 may determine the combination ratio according to the bandwidth of each transformation function.

[0078] The image processing unit 511 may determine the combination ratio of the N conversion functions according to the luminance histogram. For example, the image processing unit 511 calculates a weighting coefficient according to the luminance frequency, and adjusts the combination ratio by multiplying the weighting coefficient by the combination ratio determined according to the bandwidth.

[0079] In step S110, the image processing unit 511 generates an integrated conversion function by combining N conversion functions. The equation for generating the integrated conversion function is as explained in relation to FIG. 7, so the explanation will be omitted here.

[0080] In step S112, the arithmetic circuit 415 converts the luminance of the pixel signal using the integrated conversion function generated by the image processing unit 511.

[0081] Fig. 9 is a diagram illustrating an example of steps for determining a reference luminance and a band from a luminance histogram. Fig. 9 corresponds to steps S104 to S108 of the luminance conversion flow described in Fig. 8, and illustrates an example of generating an integrated conversion function from three conversion functions 1 to 3 (N=3).

[0082] 9 shows the luminance histogram acquired in step S104. In step S106, the image processing unit 511 determines the overall average luminance, the average luminance of luminances lower than the overall average luminance, and the average luminance of luminances higher than the overall average luminance, and calculates the reference luminances L of the conversion functions 1 to 3. w,global,1 , L w,global,2 and L w,global,3 Set to.

[0083] Next, the image processing unit 511 calculates the reference luminance L w,global,1 and the reference luminance L of transformation function 2 w,global,2 The median value of L w,med12 , the reference luminance L of transformation function 1 w,global,1 and the reference luminance L of the transformation function 3 w,global,3 The median value of L w,med13 Determine the intermediate value L w,med12 to the intermediate value L w,med13 The range of the conversion function 1 band 1, the minimum input luminance L w,minto the intermediate value L w,med12 The range of the transformation function 2 is band 2, the intermediate value L w,med13 to maximum input luminance L w,min The range is determined as band 3 of transformation function 3.

[0084] In step S108, the image processing unit 511 determines the combination ratios k1 to k3 of the conversion functions 1 to 3 from the ratio of the bandwidths of the bands 1 to 3.

[0085] 10A and 10B are diagrams showing an example of a captured image whose luminance has been converted using a conversion function according to the prior art. Fig. 10C is a diagram showing an example of a captured image whose luminance has been converted using the integrated conversion function according to this embodiment. Both images show the same outdoor scenery photographed from indoors through a window.

[0086] Figures 10A and 10B show images whose luminance has been converted using a conversion function that uses a single reference luminance as a reference. In Figure 10A, the reference luminance is set with emphasis on dark areas, resulting in blown-out highlights of pixels corresponding to outdoors. In Figure 10B, on the other hand, the reference luminance is set with emphasis on bright areas, resulting in crushed blacks of pixels corresponding to indoors.

[0087] Figure 10C shows an image whose luminance has been converted using an integrated conversion function that combines three conversion functions whose reference luminance is the overall average luminance, the average luminance of luminances lower than the overall average luminance, and the average luminance of luminances higher than the overall average luminance. As shown in Figure 10C, the luminance of bright areas is reduced and the luminance of dark areas is increased, thereby ensuring that the luminance of the actual scene is not locally impaired and the gradation is well reproduced throughout the image.

[0088] FIG. 11A shows an example of an image in which luminance has been converted taking feature points into consideration. FIG. 11A shows the image processing results of an image of a high-rise building photographed from indoors at night. Human vision recognizes the high-rise building as the center of the subject in a dark area. The image processing unit 511 extracts pixels corresponding to the high-rise building from the image as feature points and determines the combination ratio of conversion functions according to the luminance of the feature points. The image processing unit 511 may include a conversion function that uses the luminance of the feature points as a reference luminance among the multiple conversion functions that form the basis of the integrated conversion function. Alternatively, the image processing unit 511 may generate an integrated conversion function that prioritizes the luminance of the feature points by multiplying the combination ratio corresponding to a band including the luminance of the feature points by a high weighting coefficient.

[0089] Fig. 11B shows an example of a saliency map. A saliency map is a mapping of image features such as brightness, color, orientation, and direction, and is used to extract eye-catching features in an image. Fig. 11B shows a saliency map corresponding to the scene in Fig. 11A.

[0090] In Figure 11B, the illuminated windows and obstruction lights on the top floor of the high-rise building at the center of the subject, as well as the area corresponding to the window frame near the photographer, stand out in bright colors, indicating that they are eye-catching features in the scene. In this way, by taking into account factors other than brightness, the brightness of the scene can be reproduced naturally.

[0091] Fig. 12 is a diagram illustrating another example of steps for determining the reference luminance and band from the luminance histogram. Like Fig. 9, Fig. 12 corresponds to steps S104 to S108 of the luminance conversion flow described in Fig. 8. Here, a description of parts common to Fig. 9 will be omitted.

[0092] In FIG. 12, an integrated conversion function is generated from four conversion functions (N=4) including the same three conversion functions 1 to 3 as in FIG. 9, as well as conversion function 4 which uses idiosyncratic luminance as a reference luminance. In step S106, the image processing unit 511 extracts idiosyncratic luminance from the luminance histogram. The idiosyncratic luminance is, for example, luminance that deviates from the luminance distribution but is detected with a significant frequency. The image processing unit 511 converts the extracted idiosyncratic luminance into the reference luminance L of conversion function 4. w,global,4 The explanation of how to determine the band and the combination ratio will be omitted.

[0093] Next, the image processing unit 511 calculates the reference luminance L w,global,1 and the reference luminance L of transformation function 2 w,global,2 The median value of L w,med12 , the reference luminance L of transformation function 1 w,global,1 and the reference luminance L of the transformation function 3 w,global,3 The median value of L w,med13 Determine the intermediate value L w,med12 to the intermediate value L w,med13 The range of the conversion function 1 band 1, the minimum input luminance L w,min to the intermediate value L w,med12 The range of the transformation function 2 is band 2, the intermediate value L w,med13 to maximum input luminance L w,min The range of is determined as band 3 of conversion function 3. The image processing unit 511 generates the integrated conversion function from four conversion functions, including a conversion function that uses the anomalous luminance as the reference luminance, in addition to the multiple conversion functions that form the basis of the integrated conversion function.

[0094] In this way, even if a scene in the center of a dark area includes a point with significantly high luminance, such as an obstacle light on the top floor of a high-rise building as shown in FIG. 11A, the luminance of the scene can be accurately reproduced by performing luminance conversion using an integrated conversion function that takes into account the specific luminance.

[0095] Although the present invention has been described above using embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the claims that such modifications and improvements can also be included within the technical scope of the present invention.

[0096] It should be noted that the execution order of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before," "prior to," etc., and that the processes can be performed in any order unless the output of a previous process is used in a subsequent process. Even if the operational flow in the claims, specifications, and drawings is described using "first," "next," etc. for convenience, this does not mean that the processes must be performed in this order. [Explanation of symbols]

[0097] 100 imaging element, 101 microlens, 102 color filter, 103 passivation film, 104 PD, 105 transistor, 106 PD layer, 107 wiring, 108 wiring layer, 109 bump, 110 TSV, 111 signal processing chip, 112 memory chip, 113 imaging chip, 131 block, 302 transfer transistor, 303 reset transistor, 304 amplification transistor, 305 selection transistor, 306 reset wiring, 307 TX wiring, 308 decoder wiring, 309 output wiring, 310 Vdd wiring, 311 load current source, 411 multiplexer, 412 signal processing circuit, 413 demultiplexer, 414 pixel memory, 415 arithmetic circuit, 420 drive control unit, 441 sensor control unit, 443 synchronization control unit, 444 Signal control unit, 460 setting unit, 500 imaging device, 501 system control unit, 503 photometry unit, 504 work memory, 505 recording unit, 506 display unit, 511 image processing unit, 512 calculation unit, 520 photographing lens

Claims

1. a signal processing unit that generates an integrated conversion function by combining a plurality of conversion functions, the integrated conversion function including a first conversion function that converts an input luminance of a pixel signal into an output luminance, and a second conversion function that converts the input luminance into an output luminance different from the output luminance, and converts the luminance of the pixel signal using the integrated conversion function; The first conversion function converts an input luminance into an output luminance using a first luminance as a reference luminance; the second conversion function converts the input luminance into the output luminance using a second luminance different from the first luminance as a reference luminance; The signal processing unit determines a combination ratio of the plurality of conversion functions according to a bandwidth of a first band including the first luminance and a second band including the second luminance in a luminance distribution of the pixel signal. Image processing device.

2. The signal processing unit determines a combination ratio of the plurality of conversion functions according to a luminance histogram of the pixel signal. The image processing device according to claim 1 .

3. The signal processing unit extracts feature points from the pixel signals and determines a combination ratio of the plurality of transformation functions according to the luminance of the feature points.

3. The image processing device according to claim 1.

4. The signal processing unit sets an average luminance calculated from a luminance histogram of the pixel signal as the first luminance. The image processing device according to claim 1 .

5. the plurality of conversion functions further include a third conversion function that converts an input luminance into an output luminance using a third luminance that is different from the first luminance and the second luminance as a reference luminance; The signal processing unit sets the average luminance of pixel signals having a luminance higher than the first luminance to the second luminance, and sets the average luminance of pixel signals having a luminance lower than the first luminance to the third luminance. The image processing device according to claim 1 .

6. the signal processing unit extracts a peculiar luminance from a luminance distribution of the pixel signal; The plurality of conversion functions further includes a conversion function that uses the anomalous luminance as a reference luminance. The image processing device according to claim 1 .

7. Each transformation function is an increasing function The image processing device according to claim 1 .

8. Each transformation function is a sigmoid function The image processing device according to claim 7 .

9. Each conversion function converts input luminance into output luminance based on the reference luminance, the average luminance of pixel signals for each imaging region, and a predetermined brightness adjustment parameter. The image processing device according to claim 8 .

Images