Imaging device and imaging method

Abstract

Provided is an image capture apparatus comprising: an image capture element including a plurality of pixels that output a pixel signal in accordance with incident light, the plurality of pixels being divided into a plurality of blocks each including at least two pixels and having an exposure condition set therefor; an image capture control unit for changing a light receiving position on the image capture element on the basis of the exposure conditions set for adjacent blocks; and an image processing unit for synthesizing a plurality of images generated at a plurality of light receiving positions.

Description

[Technical Field]

[0001] The present invention relates to an imaging device and an imaging method. [Background technology]

[0002] Patent Document 1 describes an imaging device that divides one screen into a plurality of blocks, detects motion for each block, and controls the exposure time for each block. [Prior art document] [Patent documents] [Patent Document 1] JP 2006-197192 A

[0003] A first aspect of the present invention provides an imaging device comprising: an imaging element having a plurality of pixels that output pixel signals in response to incident light, the plurality of pixels being divided into a plurality of blocks each including at least two pixels, with exposure conditions set for each block, an imaging control unit that changes a light-receiving position on the imaging element based on the exposure conditions set for adjacent blocks, and an image processing unit that combines a plurality of images generated at the plurality of light-receiving positions.

[0004] In a second aspect of the present invention, there is provided an imaging method using an image sensor having a plurality of pixels that output pixel signals in response to incident light, the plurality of pixels being divided into a plurality of blocks each including at least two pixels, and exposure conditions being set for each block. The imaging method includes a step of changing a light-receiving position on the image sensor based on exposure conditions set for adjacent blocks, and a step of combining a plurality of images generated at the plurality of light-receiving positions.

[0005] The above summary of the invention does not list all of the necessary features of the present invention, and subcombinations of these features may also constitute inventions. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is a cross-sectional view of an image sensor 100 according to the present embodiment. [Figure 2] 1 is a diagram illustrating a pixel array and a block 131 of an imaging chip 113. FIG. [Figure 3] FIG. 10 is a circuit diagram corresponding to block 131 of imaging chip 113. [Figure 4] FIG. 2 is a block diagram showing the functional configuration of the image sensor 100. [Figure 5] FIG. 2 is a block diagram showing the functional configuration of the image sensor 100. [Figure 6] FIG. 1 is a block diagram showing the configuration of an imaging device 500 according to the present embodiment. [Figure 7] FIG. 2 is a conceptual diagram of a driving mode. [Figure 8A] FIG. 10 is a conceptual diagram showing the gradient of the noise amount at the block boundary. [Figure 8B] FIG. 10 is a conceptual diagram showing the gradient of the noise amount at the block boundary when imaging is performed in drive mode. [Figure 9] FIG. 10 is a conceptual diagram showing the relationship between exposure time and drive position. [Figure 10A] An example of setting the drive position is shown. [Figure 10B] An example of setting the drive position is shown. [Figure 10C] An example of setting the drive position is shown. [Figure 11A] 10A to 10C are diagrams showing specific examples of images captured according to exposure conditions. [Figure 11B] 11B is a diagram showing a specific example of an image captured by driving the image sensor 100 under the same exposure conditions as those in FIG. 11A. [Figure 12] FIG. 10 is a flowchart showing an example of an imaging method according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0007] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention as claimed. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0008] 1 is a cross-sectional view of an image sensor 100 according to this embodiment. The image sensor 100 includes an image sensor chip 113 that outputs pixel signals corresponding to incident light, a signal processing chip 111 that processes the pixel signals, and a memory chip 112 that stores the pixel signals. The image sensor chip 113, signal processing chip 111, and memory chip 112 are stacked and electrically connected to one another by bumps 109, which are conductive connecting portions made of Cu or the like.

[0009] As shown in the figure, incident light is mainly incident in the positive direction of the Z axis, as indicated by the white arrow. In this embodiment, the surface of the imaging chip 113 on which the incident light is incident is referred to as the back surface. As shown by the coordinate axes, the left direction on the paper, perpendicular to the Z axis, is the positive X axis, and the front direction on the paper, perpendicular to the Z axis and the X axis, is the positive Y axis. In the following figures, the coordinate axes are displayed so that the orientation of each figure can be understood, based on the coordinate axes in Figure 1.

[0010] An example of the imaging chip 113 is a back-illuminated MOS image sensor. The PD layer 106 is arranged on the back side of the wiring layer 108. The PD layer 106 has a plurality of PDs (photodiodes) 104 arranged two-dimensionally and transistors 105 provided corresponding to the PDs 104.

[0011] A color filter 102 is provided on the incident light side of the PD layer 106 via a passivation film 103. There are multiple types of color filters 102 that transmit different wavelength ranges, and each has a specific arrangement corresponding to each PD 104. The arrangement of the color filters 102 will be described later. A set of a color filter 102, a PD 104, and a transistor 105 forms one pixel.

[0012] A microlens 101 is provided corresponding to each pixel on the incident light side of the color filter 102. The microlens 101 condenses the incident light toward the corresponding PD 104.

[0013] The wiring layer 108 has wiring 107 that transmits pixel signals from the PD layer 106 to the signal processing chip 111. The wiring 107 may be multi-layered, and may be provided with passive elements and active elements.

[0014] A plurality of bumps 109 serving as connection portions are arranged on the surface of the wiring layer 108. The plurality of bumps 109 are aligned with and joined to a plurality of bumps 109 provided on the opposing surface of the signal processing chip 111, thereby electrically connecting the imaging chip 113 and the signal processing chip 111.

[0015] Similarly, a plurality of bumps 109 serving as connecting portions are arranged on the opposing surfaces of the signal processing chip 111 and the memory chip 112. These bumps 109 are aligned and joined to each other, thereby electrically connecting the signal processing chip 111 and the memory chip 112.

[0016] The imaging element 100 is formed by bonding the imaging chip 113, the signal processing chip 111, and the memory chip 112 in a wafer state before they are separated into chips, and then dicing the bonded wafer.

[0017] When bonding wafers together, plasma is applied to the wafer surfaces using an activation device to activate the bonding surfaces of the wafers. The wafers with activated surfaces are bonded together by hydrogen bonds, van der Waals bonds, covalent bonds, and the like that are formed by contact, forming a laminated substrate. If the two wafers are hydrogen bonded by contact with each other, after forming the laminated substrate, the laminated substrate is placed in a heating device such as an annealing furnace and heated to form covalent bonds between the wafers.

[0018] The term "activation" includes treating the bonding surface of at least one of the substrates so that when the bonding surface of one wafer comes into contact with the bonding surface of another wafer, hydrogen bonds, van der Waals bonds, covalent bonds, etc. are generated, resulting in solid-state bonding without melting. In other words, activation includes generating dangling bonds (unbonded hands) on the wafer surface, thereby making it easier to form bonds.

[0019] More specifically, in an activation device, oxygen gas, which is a process gas, is excited to form plasma in a reduced-pressure atmosphere, and oxygen ions are irradiated onto the surfaces that will become the bonding surfaces of the two substrates. For example, if the wafer is a substrate with an SiO film formed on Si, the irradiation of this oxygen ion breaks the SiO bonds on the wafer surfaces that will become the bonding surfaces during lamination, forming dangling bonds of Si and O. The formation of such dangling bonds on the wafer surfaces is sometimes referred to as activation.

[0020] When a substrate with dangling bonds is exposed to the atmosphere, for example, moisture in the air binds to the dangling bonds, and the substrate surface becomes covered with hydroxyl groups (OH groups). The substrate surface becomes more likely to bond with water molecules, i.e., more likely to become hydrophilic. In other words, activation results in the substrate surface becoming more likely to become hydrophilic. Furthermore, in solid-state bonding, the presence of impurities such as oxides at the bonding interface, as well as defects at the bonding interface, affect the bonding strength. Therefore, cleaning the bonding surface can be considered part of the activation process.

[0021] Furthermore, the wafer may be activated by applying pure water or the like to the surface of the wafer to be bonded using a hydrophilizing device (not shown). This hydrophilization makes the wafer surface in a state where OH groups are attached, i.e., terminated with OH groups.

[0022] By heating the laminated substrate, the bumps 109 on each of the bonding surfaces of the two wafers are integrated with each other, forming an electrical connection between the wafers. By forming the bumps 109 from a material that melts at a low temperature, such as indium or a tin-silver alloy, the laminated substrate can be reflow-treated at a low temperature of 200°C or less. Alternatively, if the bumps 109 are made of a conductive metal such as copper, they expand during the heat treatment, causing the bumps 109 between the wafers to press together and bond by solid-state diffusion.

[0023] The bonding between the bumps 109 is not limited to Cu bump bonding by solid-phase diffusion, but may also employ micro-bump bonding by solder melting. For example, it is sufficient to provide one bump 109 for each pixel block described below. Therefore, the size of the bumps 109 may be larger than the pitch of the PDs 104. Furthermore, in a peripheral region other than the pixel region where the pixels are arranged, bumps larger than the bumps 109 corresponding to the pixel region may also be provided.

[0024] The signal processing chip 111 has TSVs (through silicon vias) 110 that connect circuits provided on the front and back surfaces of the chip to each other. The TSVs 110 are preferably provided in the peripheral region. The TSVs 110 may also be provided in the peripheral region of the imaging chip 113 and in the memory chip 112. The TSVs 110 may also be used to electrically connect circuits provided on the memory chip 112 and circuits provided on the imaging chip 113.

[0025] In this way, the imaging chip 113 and the signal processing chip 111 are bonded to each other by their opposing surfaces and the bumps 109. The signal processing chip 111 and the memory chip 112 are bonded to each other by their opposing surfaces and connected to each other by the bumps 109 and the TSVs 110 provided on the signal processing chip 111. Note that the signal processing chip 111 and the memory chip 112 may be bonded to each other by their opposing surfaces and connected to each other by at least one of the bumps 109 and the TSVs 110.

[0026] FIG. 2 is a diagram illustrating the pixel array and blocks 131 of the imaging chip 113. In particular, the imaging chip 113 is shown as viewed from the back side. More than 20 million pixels are arranged in a matrix in the pixel region. These pixels are divided into blocks, each containing at least two pixels. In this embodiment, one block consists of 16 pixels, or 4 pixels by 4 pixels. The grid lines in the diagram illustrate the concept of adjacent pixels collectively forming the block 131.

[0027] As shown in the partially enlarged view of the pixel region, block 131 contains four so-called Bayer arrays, arranged vertically and horizontally, each consisting of four pixels: green pixels Gb and Gr, blue pixels B, and red pixels R. The green pixels are pixels that have a green filter as the color filter 102 and receive light in the green wavelength band of incident light. Similarly, the blue pixels are pixels that have a blue filter as the color filter 102 and receive light in the blue wavelength band, and the red pixels are pixels that have a red filter as the color filter 102 and receive light in the red wavelength band.

[0028] 3 is a circuit diagram corresponding to block 131 of imaging chip 113. In the figure, a rectangle surrounded by a dotted line typically represents a circuit corresponding to one pixel. Note that at least some of the transistors described below correspond to transistor 105 in FIG. 1.

[0029] As described above, the block 131 is formed of 16 pixels. The 16 PDs 104 corresponding to the respective pixels are connected to transfer transistors 302, and each gate of each transfer transistor 302 is connected to a TX wiring 307 through which a transfer pulse is supplied. In this embodiment, the TX wiring 307 is commonly connected to the 16 transfer transistors 302.

[0030] The drain of each transfer transistor 302 is connected to the source of the corresponding reset transistor 303, and a so-called floating diffusion FD between the drain of the transfer transistor 302 and the source of the reset transistor 303 is connected to the gate of the amplification transistor 304. The drain of the reset transistor 303 is connected to a Vdd wiring 310 to which a power supply voltage is supplied, and the gate of the reset transistor 303 is connected to a reset wiring 306 to which a reset pulse is supplied. In this embodiment, the reset wiring 306 is commonly connected to the 16 reset transistors 303.

[0031] The drain of each amplification transistor 304 is connected to a Vdd wiring 310 to which a power supply voltage is supplied. The source of each amplification transistor 304 is connected to the drain of a corresponding selection transistor 305. The gate of each selection transistor is connected to a decoder wiring 308 to which a selection pulse is supplied. In this embodiment, the decoder wiring 308 is provided independently for each of the 16 selection transistors 305. The sources of each selection transistor 305 are connected to a common output wiring 309. A load current source 311 supplies a current to the output wiring 309. In other words, the output wiring 309 for the selection transistor 305 is formed by a source follower. The load current source 311 may be provided on the imaging chip 113 side or on the signal processing chip 111 side.

[0032] Here, we will explain the flow from the start of pixel exposure to the output of a pixel signal after exposure is completed. When a reset pulse is applied to the reset transistor 303 via the reset wiring 306 and at the same time a transfer pulse is applied to the transfer transistor 302 via the TX wiring 307, the potentials of the PD 104 and floating diffusion FD are reset and exposure begins.

[0033] When the transfer pulse application is stopped, the PD 104 converts the incident light it receives into electric charges and accumulates them. Then, when the transfer pulse is applied again without the reset pulse being applied, exposure ends. The electric charges accumulated up until the end of exposure are transferred to the floating diffusion FD, and the potential of the floating diffusion FD changes from the reset potential to the signal potential after exposure ends. When a selection pulse is applied to the selection transistor 305 via the decoder wiring 308, the fluctuation in the signal potential of the floating diffusion FD is transmitted to the output wiring 309 via the amplification transistor 304 and the selection transistor 305. As a result, a pixel signal corresponding to the reset potential and the signal potential is output from the unit pixel to the output wiring 309.

[0034] As shown in the figure, in this embodiment, the reset wiring 306 and TX wiring 307 are common to the 16 pixels that form the block 131. That is, the reset pulse and transfer pulse are each applied simultaneously to all 16 pixels. Therefore, all pixels that form the block 131 start and end exposure at the same timing. However, pixel signals corresponding to the accumulated charges are selectively output to the output wiring 309 by sequentially applying selection pulses to the respective selection transistors 305.

[0035] By configuring the circuit based on the block 131 in this way, the exposure time can be controlled for each block 131. Because the exposure time can be controlled for each block, adjacent blocks 131 can output pixel signals with different exposure times. Furthermore, a common exposure time can be set for all blocks 131, and a certain block 131 can be exposed once and output a pixel signal, while an adjacent block 131 can be repeatedly exposed twice and output a pixel signal. This latter type of repeated control of exposure and pixel signal output based on a common unit time is called unit time control. Note that when unit time control is performed, if the start and end points of exposure are synchronized across all blocks 131, the reset wiring 306 may be commonly connected to all reset transistors 303 on the imaging chip 113.

[0036] 4 is a block diagram showing the functional configuration of the image sensor 100. Here, the flow of pixel signals will be particularly described.

[0037] An analog multiplexer 411 sequentially selects the 16 PDs 104 that form the block 131 and outputs the respective pixel signals to the output wiring 309. The multiplexer 411 is formed in the imaging chip 113 together with the PDs 104.

[0038] The pixel signals output via the multiplexer 411 undergo correlated double sampling (CDS) and analog-to-digital (A / D) conversion by a signal processing circuit 412 formed in the signal processing chip 111, which performs CDS and A / D conversion. The A / D conversion converts the input analog pixel signals into 12-bit digital pixel signals. The A / D converted pixel signals are passed to an arithmetic circuit 415, also formed in the signal processing chip 111. The arithmetic circuit 415 performs arithmetic processing on the received pixel signals required for subsequent image processing, and passes the signals to the demultiplexer 413.

[0039] The demultiplexer 413 stores the received pixel signals in pixel memories 414 corresponding to the respective pixels. Each pixel memory 414 has a capacity capable of storing the pixel signals after arithmetic processing has been performed. The demultiplexer 413 and pixel memories 414 are formed on the memory chip 112.

[0040] The arithmetic circuit 415 reads pixel signals to be used for arithmetic processing from the pixel memory 414 via the demultiplexer 413. Alternatively, in accordance with an external transfer request, the arithmetic circuit 415 transfers the pixel signals read from the pixel memory 414 via the demultiplexer 413 to a downstream image processing unit. The arithmetic circuit 415 may be provided in the memory chip 112.

[0041] Also, while the figure shows the flow of pixel signals for one block, in reality these exist for each block and operate in parallel. However, a calculation circuit 415 does not have to exist for each block; for example, one calculation circuit 415 may process sequentially while referring to the values ​​of the pixel memories 414 corresponding to each block in order.

[0042] 5 is a block diagram showing the functional configuration of the image sensor 100. Here, the specific configuration of the signal processing chip 111 and the control by the system control unit 501 will be mainly explained.

[0043] The signal processing chip 111 includes a sensor control unit 441, a synchronization control unit 443, and a signal control unit 444 as distributed control functions, and a drive control unit 420 that controls these control units in an integrated manner. The drive control unit 420 is a control circuit that converts instructions from a system control unit 501 that is responsible for integrated control of the entire imaging device into control signals that can be executed by each control unit and passes them on to each unit.

[0044] The sensor control unit 441 is responsible for controlling the transmission of control pulses related to charge accumulation and charge readout of each pixel, which are sent to the imaging chip 113. Specifically, the sensor control unit 441 controls the start and end of exposure by sending reset pulses and transfer pulses to the target pixels, and outputs pixel signals to the output wiring 309 by sending selection pulses to the readout pixels.

[0045] The synchronization control unit 443 sends a synchronization signal to the imaging chip 113. Each pulse becomes active in the imaging chip 113 in synchronization with the synchronization signal. For example, by adjusting the synchronization signal, random control, thinning control, and the like can be realized, in which only specific pixels belonging to the same block 131 are the control targets.

[0046] The signal control unit 444 is mainly responsible for timing control of the A / D converter 412b. The pixel signals output via the output wiring 309 are input to the CDS circuit 412a and the A / D converter 412b via the multiplexer 411. The A / D converter 412b is controlled by the signal control unit 444 and converts the input pixel signals into digital signals. The converted digital pixel signals are passed to the arithmetic circuit 415, where they are subjected to arithmetic processing. The processed pixel signals are passed to the demultiplexer 413 of the memory chip 112 and stored as digital pixel values ​​in the pixel memories 414 corresponding to each pixel.

[0047] A system control unit 501 of the imaging device 500, which will be described later, functions as an imaging instruction unit that receives instructions from a user and generates imaging instructions to be sent to the image sensor 100. The drive control unit 420 receives exposure conditions set for each block 131 from the imaging control unit 512 of the system control unit 501. The exposure conditions are conditions for changing the brightness of the image to be acquired, such as exposure time, aperture value, ISO sensitivity, etc. The drive control unit 420 sends a control signal to the sensor control unit 441 in accordance with the exposure conditions for each block 131.

[0048] In response to a transfer request from the imaging control unit 512, the drive control unit 420 reads the target pixel signal from the pixel memory 414 via the arithmetic circuit 415 and demultiplexer 413, and transfers the signal to the image processing unit 511 of the system control unit 501. The pixel memory 414 is provided with a data transfer interface that transmits the pixel signal in response to the transfer request. The data transfer interface is connected to a data transfer line that is connected to the image processing unit 511. The data transfer line is formed, for example, by a data bus among the bus lines. In this case, the transfer request from the system control unit 501 to the drive control unit 420 is executed by address specification using the address bus.

[0049] The transmission of pixel signals via the data transfer interface is not limited to the addressing method, and various other methods can be used. For example, a double data rate method can be used, which uses both the rising and falling edges of the clock signal used to synchronize each circuit when transferring data. A burst transfer method can also be used, which transfers data all at once by omitting some steps such as addressing, thereby increasing speed. It is also possible to use a combination of a bus method using lines connecting the control unit, memory unit, and input / output unit in parallel, and a serial method that transfers data one bit at a time in series.

[0050] With this configuration, the image processing unit 511 can receive only the necessary pixel signals, and therefore can complete image processing at high speed, especially when forming a low-resolution image.

[0051] 6 is a block diagram showing the configuration of an image capturing apparatus 500 according to this embodiment. The image capturing apparatus 500 mainly includes an image sensor 100, a photographing lens 520, a system control unit 501, a drive unit 502, a photometry unit 503, a work memory 504, a recording unit 505, and a display unit 506.

[0052] The driving unit 502 drives the image sensor 100 and the photographing lens 520. "Driving" refers to moving the object to be driven on the xy plane where the object to be driven is located at the start of image capture. The driving unit 502 is, for example, an actuator such as a motor or a piezoelectric element. The driving direction by the driving unit 502 is not limited to the xy plane, and the image sensor 100 may be driven to rotate in a direction around the z axis, for example.

[0053] The photometry unit 503 detects the luminance distribution of a scene prior to a series of shooting sequences for generating image data. The photometry unit 503 includes, for example, an AE sensor with about one million pixels.

[0054] The photographing lens 520 guides a subject light beam incident along the optical axis O to the image sensor 100. The photographing lens 520 is an example of an optical system. The photographing lens 520 is composed of a group of optical lenses, and focuses the subject light beam from the scene near its focal plane. Note that in FIG. 6, the photographing lens 520 is represented by a single virtual lens placed near the pupil. The photographing lens 520 may be an interchangeable lens that can be attached to and detached from the image capturing device 500.

[0055] The image processing unit 511 receives pixel signals from the drive control unit 420 of the image sensor 100. The image processing unit 511 performs various image processing using the work memory 504 as a workspace to generate image data. For example, the image processing unit 511 detects feature points from multiple captured images and synthesizes the multiple images based on the feature points to generate final image data. When generating image data in JPEG file format, the image processing unit 511 performs white balance processing, gamma processing, etc., and then performs compression processing. The generated image data is recorded in the recording unit 505 and converted into a display signal to be displayed on the display unit 506 for a predetermined time. The image processing unit 511 may be configured as an ASIC independent of the system control unit 501, or may be provided in the memory chip 112.

[0056] The imaging control unit 512 receives the output of the photometry unit 503 and calculates the luminance for each region of the scene. The imaging control unit 512 sets exposure conditions for each block 131 according to the luminance distribution of the scene. The imaging control unit 512 also determines the timing of opening and closing the shutter according to the exposure conditions set for each block 131.

[0057] The imaging control unit 512 in this example changes the light receiving position on the imaging element 100 based on the exposure conditions set for each block 131. Here, changing the light receiving position on the imaging element 100 means changing the position on the imaging chip 113 where incident light reaches via the photographing lens 520.

[0058] The imaging control unit 512 controls the drive unit 502 to drive at least one of the image sensor 100 and the photographing lens 520, thereby changing the light receiving position on the image sensor 100. The imaging control unit 512 may drive either the image sensor 100 or the photographing lens 520, or may drive both. Which of the image sensor 100 and the photographing lens 520 is to be driven may be set according to a user input.

[0059] The imaging control unit 512 calculates the drive amount based on the exposure conditions set for the adjacent block 131. The method for calculating the drive amount will be described later. Next, the imaging control unit 512 determines the number of drives, drive timing, and corresponding drive positions during the imaging period. Here, exposure time is used as an example of the exposure condition. The number of drives and drive timing may be set based on the exposure time and number of images of the block 131 to which the shortest exposure time is set. The drive position may be determined based on the calculated drive amount and drive number.

[0060] The mode selection unit 513 selects an imaging mode for the imaging device 500. The mode selection unit 513 selects either a drive mode that changes the light receiving position on the imaging element 100 or a standard mode that does not change the light receiving position on the imaging element 100, based on the exposure conditions set for each block 131. The mode selection unit 513 may select a drive mode when the difference between the values ​​related to the exposure times set for adjacent blocks exceeds a threshold, and may select the standard mode when the difference is equal to or less than the threshold.

[0061] When the standard mode is selected, the imaging control unit 512 passes the exposure conditions set for each block 131 to the drive control unit 420. The drive control unit 420 controls the sensor control unit 441 to expose each pixel in accordance with the acquired exposure conditions. On the other hand, when the drive mode is selected, the imaging control unit 512 passes drive information indicating the drive position and drive timing along with the exposure conditions set for each block 131 to the drive control unit 420. The drive control unit 420 controls the sensor control unit 441 to expose each pixel in accordance with the exposure conditions while changing the light receiving position on the image sensor 100 in accordance with the drive information.

[0062] The mode selection unit 513 may be a part of the imaging control unit 512. Alternatively, the imaging control unit 512 and the mode selection unit 513 may be provided in the memory chip 112. In this case, the imaging control unit 512 and the mode selection unit 513 receive information from the system control unit 501, and transfer the results of calculations based on the received information to the drive control unit 420.

[0063] 7 is a conceptual diagram of the drive modes. As an example, a case will be described in which the image sensor 100 is driven from drive position 0 to drive positions 1 to N at drive timings 1 to N. Note that when the photographing lens 520 is driven to drive positions 1 to N, the effect of changing the light receiving position on the image sensor 100 is obtained in the same way as when the image sensor 100 is driven, so a description thereof will be omitted.

[0064] Drive position 0 is a reference position where the image sensor 100 is located when image capturing begins, and drive positions 1 to N are points on the same plane as the xy plane where the image sensor 100 is located at drive position 0. In other words, drive positions 1 to N are relative positions of the image sensor 100 with drive position 0 as the reference. At drive timing 1, the image sensor 100 moves from drive position 0 to drive position 1, and at drive timing 2, the image sensor 100 moves to the next drive position 2. In this way, driving is repeated N times until image capturing ends, and the image sensor 100 moves in order from drive position 0 to N.

[0065] In this way, the position of the image sensor 100 changes for each drive timing, and therefore the light receiving position on the image sensor 100 also changes relatively. For example, if the light receiving position at drive position 0 is a pixel near the boundary of a certain block 131, the light receiving position at a certain drive position may be a pixel in an adjacent block 131.

[0066] 8A is a conceptual diagram showing the gradient of noise amount at block boundaries. Noise generated during exposure affects image quality. The amount of noise generated at each pixel mainly depends on the number of captures during the capture period, so it can be estimated in advance from the exposure conditions set for the block 131 to which the pixel belongs.

[0067] In FIG. 8A, the horizontal axis represents the position of the imaging chip 113 in the x-axis direction, and the vertical axis represents the amount of noise σ generated in a pixel. 2 In the block 131, the amount of noise generated in each pixel in the block 1 and the block 2 adjacent to each other in the x-axis direction is expressed as σ 2 1, σ 22. In each block 131, the exposure conditions set for the pixels are the same, so σ 2 1, σ 2 2 are each constant.

[0068] Here, the block boundary region is defined as a range of distances s in the positive and negative x-axis directions, i.e., distance 2s, centered on the boundary between blocks 1 and 2. When image sensor 100 is not driven, distance 2s is the minimum distance between the pixels of block 1 and block 2, i.e., the distance between the centers of the pixels of block 1 and block 2 that meet at the boundary between the two blocks. In this case, the gradient k of the noise amount in the block boundary region is given by the following equation:

number

[0069] From the above formula, when the distance 2s is constant, the noise amount σ of the adjacent blocks 2 1, σ 2 It can be seen that the greater the difference between k and σ, the greater the slope k at the block boundary. If the slope k is large, the block boundary on the image will be more noticeable, which may result in a decrease in image quality. Note that "block boundary of the image sensor 100" refers to the boundary between adjacent blocks 131 on the image sensor 100, and "block boundary on the image" refers to the boundary between areas on the image that correspond to adjacent blocks on the image sensor 100.

[0070] The imaging control unit 512 drives the imaging element 100 during imaging, thereby changing the position at which light is received on the imaging element 100. The imaging control unit 512 determines the drive amount of the imaging element 100 in accordance with the difference in exposure time set for adjacent blocks. When determining the drive amount of the imaging element 100, the imaging control unit 512 determines the target drive distance s target Calculate the target driving distance s target is the threshold value of the slope k. threshold Then, it is given by the following formula:

number

[0071] Here, the threshold K of the slope k threshold is the maximum value that the slope k can take, and may be a predetermined value. target is the target value of the distance between the drive position 0 and each drive position, and the slope k is the threshold K threshold The imaging control unit 512 calculates the target driving distance s calculated for each block 131 from the driving position 0. target A plurality of drive positions may be set within the range of the maximum value of .

[0072] The imaging control unit 512 also determines the drive position so that the drive amount for the exposure time of the block 131 having the longest exposure time is equal to or less than the size of one pixel. The drive amount is the distance the imaging element 100 moves when driven from one drive position to another. The pixel size is the length of a side of one pixel in the x-axis direction or y-axis direction on the xy plane on which the imaging chip 113 is located.

[0073] 8B is a conceptual diagram showing the gradient of the noise amount at the block boundary when imaging is performed in drive mode. For example, when the image sensor 100 is driven by a distance s' in the positive and negative directions of the x-axis, the distance of the block boundary region becomes 2s'. In the block boundary region, the noise amount is σ 2 2 or more σ 2 It changes gradually with a slope k within a range of not more than 1. By driving the image sensor 100 in this way, the amount of noise in the block boundary region is leveled, and the block boundaries on the image become less noticeable.

[0074] Target driving distance s target The maximum value of may be a predetermined value. Increasing the distance 2s' reduces the gradient k of the noise amount, which has the effect of making the block boundaries on the image less noticeable. However, when there is a significant difference in brightness between adjacent blocks 131, the following problem occurs.

[0075] For example, if block 1 corresponds to a bright area and block 2 corresponds to a dark area, the exposure time set for block 1 is shorter than the exposure time set for block 2. If, among the pixels of block 1, a pixel located near the boundary with block 2 is driven to a position that was included in block 2 at drive position 0, it receives weak incident light from the dark area with a short exposure time, and there is a risk that a pixel signal will not be generated correctly. Therefore, the target drive distance s target By setting the maximum value of the target driving distance s, it is possible to reduce the slope k while maintaining the appropriateness of the exposure conditions. target may be equal to or less than the length of the block 131 in the x-axis direction or the y-axis direction.

[0076] FIG. 9 is a conceptual diagram showing the relationship between exposure time and drive position. In FIG. 9, the horizontal axis represents the time axis. Block 1, block 2, and block 3 are examples of block 131, and the white arrows represent the exposure time and drive timing of the corresponding block. Block 1 corresponds to the bright area, block 3 corresponds to the dark area, and block 2 corresponds to the area of ​​intermediate brightness between blocks 1 and 3. The exposure times set for blocks 1 to 3 are shorter in this order.

[0077] The number of times each block is imaged is determined based on the exposure time of block 3, which has the longest exposure time. In the example of Fig. 9, block 3 has an exposure time that is half the imaging period, so the number of times imaged is two. Block 1 has an exposure time that is one-fourth that of block 3, so the number of times imaged is four times that of block 3, i.e., eight. Similarly, block 2 has an exposure time that is one-half that of block 3, so the number of times imaged is twice that of block 3, i.e., four.

[0078] The imaging control unit 512 determines the number of times of driving and the driving timing based on the number of times of imaging of block 1. In the example of Fig. 9, the number of times of driving is 7, which is the number of times of imaging of block 1 minus 1, and the driving timing is the same as when the exposure time of block 1 has elapsed. The number of driving positions is 8, the same as the number of times of imaging of block 1, and the imaging element 100 is driven from driving position 0, which is the initial position, to driving position 7 between the start and end of imaging.

[0079] The imaging control unit 512 determines the drive position so that the drive amount during the exposure time of block 3, which has the longest exposure time, is equal to or less than the size of one pixel. In the example of FIG. 9, during the first imaging of block 3, the imaging element 100 is located at drive positions 0 to 3, so the imaging control unit 512 determines the drive position so that the drive amount from drive position 0 to drive position 3 is equal to or less than the size of one pixel. Next, at the start of the second imaging of block 3, the imaging element 100 is driven to drive position 4, but the drive amount from drive position 0 to drive position 4 may exceed the size of one pixel. During the second imaging of block 3, the imaging element 100 is located at drive positions 4 to 7, so the imaging control unit 512 determines the drive position so that the drive amount from drive position 4 to drive position 7 is equal to or less than the size of one pixel.

[0080] In block 1, the drive timing coincides with the elapse of the exposure time, but in blocks 2 and 3, which have longer exposure times set than block 1, the drive timing arrives during exposure. In other words, the imaging control unit 512 drives the image sensor 100 to the next drive position while the pixels in blocks 2 and 3 are being exposed. In this way, even for blocks corresponding to dark regions, exposure is not interrupted when driving the image sensor 100, so a sufficient amount of incident light can be received and accurate pixel signals can be generated.

[0081] The imaging control unit 512 may also move the imaging element 100 more finely by increasing the number of drive positions within one pixel compared to the number of exposures of block 1. The imaging control unit 512 may also create a drive route by connecting multiple drive positions, or may set a drive route without setting drive positions and move the imaging element 100 continuously along the drive route. Even in such cases, the imaging control unit 512 determines the drive positions so that the drive amount during the exposure time of block 3, which has the longest exposure time set, is equal to or less than the size of one pixel.

[0082] Alternatively, the imaging control unit 512 may determine the drive position so that imaging is performed at the same position during the exposure time for block 3, rather than driving the image sensor 100 every time the exposure time for block 1 elapses. In the example of FIG. 9 , if drive positions 1 to 4 are the same and drive positions 5 to 7 are the same, the image sensor 100 is driven from drive position 0 to 1 during the first imaging of block 3, and is driven from drive position 1 to 5 during the first imaging of block 3. In this case, the second to fifth imaging of block 1 is performed at the same drive position 1, and the sixth to eighth imaging is performed at the same drive position 5. The block 2 is driven from drive position 0 to 1 during the first imaging, then performs imaging at drive position 1, then driven to drive position 5 during the third imaging, and then performed at drive position 5 for imaging up to the fourth imaging.

[0083] FIG. 10A shows an example of setting a drive position. FIG. 10A is an example of setting a drive position of the image sensor 100 including blocks 1 to 3 in the example of FIG. 9. The image capture control unit 512 calculates the drive amount of the image sensor 100 from the exposure conditions set for each block 131, and determines the number of times of drive and the drive timing. The image capture control unit 512 calculates the drive amount of the image sensor 100 from the exposure conditions set for each block 131, and determines the number of times of drive and the drive timing. target The drive position corresponding to the number of drives is determined so that the number of drives is within the range.

[0084] 10A, there are 16 drive positions, and the imaging control unit 512 determines drive positions 1 to 15. Here, the time required for drive from drive position 0 to drive position 3 is equal to the exposure time of block 3. The imaging control unit 512 determines drive positions 1 to 3 so that the movement distance from drive position 0 to drive position 3 does not exceed the size of a pixel.

[0085] Similarly, the imaging control unit 512 determines drive positions 4 to 7 so that the movement distance from drive position 4 to drive position 7 does not exceed the size of the pixel. Furthermore, the imaging control unit 512 determines drive positions 8 to 11 and drive positions 12 to 15 in the same manner as drive positions 1 to 3 and drive positions 4 to 7, except that the final drive position 15 is closer to drive position 0.

[0086] In this way, by ensuring that the drive amount in the exposure time of the block for which the longest exposure time is set does not exceed the size of one pixel, it is possible to suppress image blurring due to drive.

[0087] FIG. 10B shows another example of setting the drive positions. In the example of FIG. 10B, the imaging control unit 512 determines drive positions 1 to N so that the x coordinates of the drive positions satisfy the normal distribution expressed by the equation shown in FIG. 10B, and the y coordinates also satisfy the normal distribution. In other words, the imaging control unit 512 determines the drive positions 1 to N so that the target drive distance s target The driving positions 1 to N may be determined randomly within the range.

[0088] FIG. 10C shows another example of setting the drive positions. In the example of FIG. 10C, the imaging control unit 512 determines drive positions 1 to N so as to form a spiral. After being driven from drive position 0 to drive position N in the first imaging, the imaging element 100 may return from drive position n to drive position 0 via the shortest path (passing along the x-axis in FIG. 10C) before the start of the second imaging. Alternatively, in the second imaging, the imaging element 100 may be driven to drive positions N, N-1, N-2, . . . , 1, 0, in the opposite direction to the driving in the first imaging. In FIG. 10C, the distance on the x-axis from drive position 0 to drive position N is the target drive distance s target This becomes:

[0089] FIG. 11A is a diagram showing a specific example of an image captured according to exposure conditions. In FIG. 11A, the diagram on the left shows the exposure conditions set for capturing an image of a certain subject. The subject is a high-rise building at night. The exposure time setting indicates that the brighter (closer to white) the color of the block, the shorter the exposure time (bright area), and the darker (closer to black) the color, the longer the exposure time (dark area). The block pointed to by the arrow corresponds to an area including an aviation obstruction light lit on the top floor of the subject, the high-rise building, and is set as a short-exposure block corresponding to a bright area due to the influence of the brightness of the aviation obstruction light. The adjacent blocks all correspond to the night sky area (dark area) of the subject, and are set as long-exposure blocks.

[0090] 11A, the diagram on the right shows an image obtained as a result of capturing an image without driving the image sensor 100. The arrow indicates the area corresponding to the above-mentioned obstruction light blocks, and the block boundaries on the image stand out unnaturally in the area corresponding to the night sky.

[0091] Fig. 11B is a diagram showing a specific example of an image captured by driving the image sensor 100 under the same exposure conditions as Fig. 11A. As a result of driving, it can be seen that the difference in noise amount between adjacent blocks is smoothed out, resulting in a natural image in which block boundaries are not noticeable.

[0092] 12 is a flow diagram showing an example of an imaging method according to this embodiment. Here, as an example, the imaging method will be described assuming that the image processing unit 511, imaging control unit 512, or mode selection unit 513 of the system control unit 501 is the subject of the operation, but is not limited to this. Other functional blocks included in the imaging device 500 may also be the subject of the operation.

[0093] In step S1310, the imaging control unit 512 sets exposure conditions for each block 131.

[0094] In step S1320, the imaging control unit 512 determines an exposure position on the image sensor 100. Based on the exposure conditions set for adjacent blocks, the imaging control unit 512 calculates the amount of noise or the signal-to-noise ratio (SN ratio) for each block 131. Based on the amount of noise or the SN ratio of each block 131, the imaging control unit 512 also calculates the difference in the amount of noise or the difference in the SN ratio between the blocks 131, and determines the drive amount of the image sensor 100 based on this.

[0095] The imaging control unit 512 may determine the drive amount taking into consideration camera shake correction. The imaging control unit 512 calculates the amount of camera shake and corrects the drive amount by subtracting the amount of camera shake from the calculated drive amount. For example, a measurement unit such as an acceleration sensor may be provided in the imaging device 500 to measure the amount of camera shake. Furthermore, if the imaging device 500 has an image stabilization mechanism, the imaging control unit 512 may determine the drive amount taking into consideration the amount of movement required for camera shake prevention by the image stabilization mechanism. The imaging control unit 512 corrects the drive amount by subtracting the amount of movement required by the image stabilization mechanism from the calculated drive amount. In this way, it is possible to drive within a more appropriate range of drive amounts.

[0096] Before step S1320, the mode selection unit 513 may acquire exposure conditions from the imaging control unit 512 and select an imaging mode. The mode selection unit 513 may select the drive mode when the difference between the values ​​related to the exposure times set for adjacent blocks exceeds a threshold, and may select the standard mode when the difference is equal to or less than the threshold. If the drive mode is selected, processing may proceed to step S1320, and if the standard mode is selected, processing may proceed to step S1330.

[0097] In step S1330, the imaging control unit 512 transmits the exposure conditions and drive information indicating the drive position and drive timing to the drive control unit 420 of the image sensor 100, and performs imaging under the control of the drive control unit 420. Here, if the mode selection unit 513 selects the standard mode, the imaging control unit 512 may transmit only the exposure conditions to the drive control unit 420.

[0098] The image processing unit 511 generates an image by synthesizing and aligning the multiple images generated at the multiple light receiving positions. The multiple images may be synthesized by calculating the sum or average of the pixel values ​​of the pixels captured at the corresponding positions.

[0099] In this way, according to this example, it is possible to make the block boundaries on the image less noticeable.

[0100] Although the present invention has been described above using embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the claims that such modifications and improvements can also be included within the technical scope of the present invention.

[0101] It should be noted that the order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before," "prior to," etc., and that the processes can be performed in any order unless the output of a previous process is used in a subsequent process. Even if the operational flow in the claims, specifications, and drawings is described using "first," "next," etc. for convenience, this does not mean that the processes must be performed in this order. [Explanation of symbols]

[0102] 100 imaging element, 101 microlens, 102 color filter, 103 passivation film, 104 PD, 105 transistor, 106 PD layer, 107 wiring, 108 wiring layer, 109 bump, 110 TSV, 111 signal processing chip, 112 memory chip, 113 imaging chip, 131 block, 302 transfer transistor, 303 reset transistor, 304 amplification transistor, 305 selection transistor, 306 reset wiring, 307 TX wiring, 308 decoder wiring, 309 output wiring, 310 Vdd wiring, 311 load current source, 411 multiplexer, 412 signal processing circuit, 413 demultiplexer, 414 pixel memory, 415 arithmetic circuit, 420 drive control unit, 441 sensor control unit, 443 synchronization control unit, 444 Signal control unit, 500 imaging device, 501 system control unit, 502 drive unit, 503 photometry unit, 504 work memory, 505 recording unit, 506 display unit, 511 image processing unit, 512 imaging control unit, 513 mode selection unit, 520 photographing lens

Claims

1. an image sensor having a plurality of pixels that output pixel signals in response to incident light, the plurality of pixels being divided into a plurality of blocks each including at least two pixels, and exposure conditions including an exposure time being set for each block; an imaging control unit that changes a light receiving position on the imaging element by driving at least one of the imaging element and an optical system that guides the incident light to the plurality of pixels based on a first exposure time set for a first block and a second exposure time set for a second block adjacent to the first block, among the exposure conditions set for each block; an image processing unit that combines a plurality of images generated at the plurality of light receiving positions; Equipped with The imaging control unit determining a range of a driving amount of the image sensor or the optical system based on the first exposure time and the second exposure time; driving the image sensor or the optical system within the range of the driving amount to change the light receiving position on the image sensor; Imaging device.

2. Further comprising a photometry unit for performing photometry, A luminance distribution is detected based on the photometry by the photometry unit, and an exposure condition for each block is set based on the luminance distribution. The imaging device according to claim 1 .

3. 3. The imaging device according to claim 1, wherein the imaging control unit determines the range of the drive amount based on a difference between the amount of noise in the first block estimated from the first exposure time and the amount of noise in the second block estimated from the second exposure time.

4. The imaging device according to claim 3 , wherein the imaging control unit determines the range of the drive amount so that the range of the drive amount becomes wider as the difference between the amount of noise in the first block and the amount of noise in the second block increases.

5. The driving amount of the imaging device or the optical system during the exposure time of the block in which the longest exposure time is set is equal to or less than the size of one pixel. The imaging device according to claim 1 .

6. a longer exposure time is set for the second block than for the first block; The imaging control unit changes the light receiving position every time an exposure time of the first block elapses during exposure of the second block. The imaging device according to claim 1 .

7. The imaging control unit corrects the driving amount by subtracting the amount of camera shake. The imaging device according to claim 1 .

8. An imaging method using an imaging element having a plurality of pixels that output pixel signals in response to incident light, the plurality of pixels being divided into a plurality of blocks each including at least two pixels, and exposure conditions including an exposure time being set for each block, a step of changing a light receiving position on the image sensor by driving at least one of the image sensor and an optical system that guides the incident light to the plurality of pixels based on a first exposure time set for a first block and a second exposure time set for a second block adjacent to the first block, among exposure conditions set for each block; combining a plurality of images generated at a plurality of said light receiving positions; Equipped with The step of changing the light receiving position on the imaging element includes: determining a range of a driving amount of the image sensor or the optical system based on the first exposure time and the second exposure time; and driving the image sensor or the optical system within the range of the driving amount to change the light receiving position on the image sensor. Imaging method.

Images