Imaging element

Abstract

To properly process photoelectric conversion data.SOLUTION: An imaging element includes a plurality of semiconductor substrates. The semiconductor substrates include: first, second, and third photoelectric conversion units; a first conversion unit configured to convert a first signal from the first photoelectric conversion unit into a digital signal; a second conversion unit configured to convert a second signal from the second photoelectric conversion unit into a digital signal; a third conversion unit configured to convert a third signal from the third photoelectric conversion unit into a digital signal; a first adder configured to perform first addition processing using the first signal; a second adder configured to perform second addition processing using the second signal; a third adder configured to perform third addition processing using the third signal; a first storage unit configured to store a signal after the first addition processing; a second storage unit configured to store a signal after the second addition processing; and a third storage unit configured to store a signal after the third addition processing. The first, second, and third photoelectric conversion units are disposed on a first semiconductor substrate. The first, second, and third adders, and the first, second, and third storage units are disposed on a semiconductor substrate different from the first semiconductor substrate.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an imaging device. [Background technology]

[0002] An electronic device has been proposed that includes an imaging element in which a back-illuminated imaging chip and a signal processing chip are stacked (hereinafter, referred to as a stacked imaging element) (see Patent Document 1). In the stacked imaging element, the back-illuminated imaging chip and the signal processing chip are stacked so that they are connected via microbumps in each predetermined area. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2006-49361 A Summary of the Invention [Problem to be solved by the invention]

[0004] In electronic devices equipped with conventional stacked imaging elements, there have been few proposals for dividing an image into blocks having one or more of the above-mentioned regions and acquiring an image for each block, and the usability of electronic devices equipped with stacked imaging elements has not been sufficient. Furthermore, the conventional technology of synthesizing multiple images captured separately under different exposure conditions is not suitable for capturing images of moving subjects. [Means for solving the problem]

[0005] An imaging element according to a first aspect of the invention is an imaging element including a plurality of stacked semiconductor substrates, the plurality of semiconductor substrates including a first photoelectric conversion unit that converts light into electric charges, a second photoelectric conversion unit that converts light into electric charges and that is arranged alongside the first photoelectric conversion unit in the row direction, a third photoelectric conversion unit that converts light into electric charges and that is arranged alongside the first photoelectric conversion unit in the column direction, a first conversion unit that converts a first signal based on the charges converted by the first photoelectric conversion unit into a digital signal, a second conversion unit that converts a second signal based on the charges converted by the second photoelectric conversion unit into a digital signal, a third conversion unit that converts a third signal based on the charges converted by the third photoelectric conversion unit into a digital signal, and a digital signal conversion unit that converts the first signal converted into a digital signal by using the first conversion unit. the first photoelectric conversion unit, the second photoelectric conversion unit, and the third photoelectric conversion unit are disposed on a first semiconductor substrate among the plurality of semiconductor substrates, and the first adder, the second adder, the third adder, the first storage unit, the second storage unit, and the third storage unit are disposed on a semiconductor substrate different from the first semiconductor substrate among the plurality of semiconductor substrates. Effect of the Invention

[0006] According to the present invention, photoelectric conversion data can be appropriately processed. [Brief description of the drawings]

[0007] [Figure 1] FIG. 2 is a cross-sectional view of a stacked imaging element. [Diagram 2] 2 is a diagram illustrating a pixel array and a unit area of ​​an imaging chip. FIG. [Diagram 3] 3 is a circuit diagram corresponding to a unit area of ​​the imaging chip. FIG. [Figure 4] FIG. 2 is a block diagram showing a functional configuration of an imaging element. [Diagram 5] FIG. 2 is a diagram for explaining the flow of a pixel signal per pixel. [Figure 6] FIG. 1 is a block diagram illustrating a configuration of an imaging device having an imaging element. [Figure 7] 2 is a diagram illustrating an example of a region of interest and a peripheral region in an imaging element; [Figure 8] 3A to 3C are diagrams for explaining readout timing, an accumulation signal, and a pixel signal read out from an image sensor via an arithmetic circuit. [Figure 9] 5 is a flowchart illustrating a flow of a photographing operation executed by a control unit according to the first embodiment. [Figure 10] 10 is a flowchart illustrating a flow of a photographing operation executed by a control unit according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. First Embodiment <Explanation of stacked image sensor> First, a stacked imaging element 100 mounted on an electronic device (e.g., imaging device 1) according to a first embodiment of the present invention will be described. This stacked imaging element 100 is described in Japanese Patent Application No. 2012-139026, previously filed by the applicant of the present application. FIG. 1 is a cross-sectional view of the stacked imaging element 100. The imaging element 100 includes a back-illuminated imaging chip 113 that outputs pixel signals corresponding to incident light, a signal processing chip 111 that processes the pixel signals, and a memory chip 112 that stores the pixel signals. The imaging chip 113, the signal processing chip 111, and the memory chip 112 are stacked and electrically connected to each other by conductive bumps 109 made of Cu or the like.

[0009] As shown in the figure, the incident light is mainly incident in the Z-axis positive direction indicated by the white arrow. In this embodiment, the surface of the imaging chip 113 on which the incident light is incident is referred to as the back surface. Also, as shown in the coordinate axes, the left direction on the paper surface perpendicular to the Z-axis is the X-axis positive direction, and the front direction on the paper surface perpendicular to the Z-axis and the X-axis is the Y-axis positive direction. In the following figures, the coordinate axes are displayed so that the orientation of each figure can be understood based on the coordinate axes in FIG. 1.

[0010] An example of the imaging chip 113 is a back-illuminated MOS image sensor. The PD layer 106 is disposed on the back side of the wiring layer 108. The PD layer 106 includes a plurality of PDs (photodiodes) 104 that are two-dimensionally disposed and accumulate electric charges according to incident light, and transistors 105 that correspond to the PDs 104.

[0011] A color filter 102 is provided on the incident side of the PD layer 106 where the incident light is incident, via a passivation film 103. There are a plurality of types of color filters 102 that transmit different wavelength regions, and each has a specific arrangement corresponding to each PD 104. The arrangement of the color filters 102 will be described later. A set of the color filter 102, the PD 104, and the transistor 105 forms one pixel.

[0012] A microlens 101 is provided corresponding to each pixel on the incident light side of the color filter 102. The microlens 101 condenses the incident light toward the corresponding PD 104.

[0013] The wiring layer 108 has wiring 107 that transmits pixel signals from the PD layer 106 to the signal processing chip 111. The wiring 107 may be multi-layered, and may include passive and active elements.

[0014] A plurality of bumps 109 are arranged on the surface of the wiring layer 108. The plurality of bumps 109 are aligned with a plurality of bumps 109 provided on the opposing surface of the signal processing chip 111, and the imaging chip 113 and the signal processing chip 111 are pressed, etc., so that the aligned bumps 109 are joined and electrically connected to each other.

[0015] Similarly, a plurality of bumps 109 are arranged on the opposing surfaces of the signal processing chip 111 and the memory chip 112. These bumps 109 are aligned with each other, and the signal processing chip 111 and the memory chip 112 are pressed together, etc., so that the aligned bumps 109 are bonded to each other and electrically connected.

[0016] The bonding between the bumps 109 is not limited to Cu bump bonding by solid-phase diffusion, and micro-bump bonding by solder melting may be adopted. Also, it is sufficient to provide about one bump 109 for each unit area described later. Therefore, the size of the bump 109 may be larger than the pitch of the PD 104. Also, in a peripheral area other than the pixel area where the pixels are arranged, a bump larger than the bump 109 corresponding to the pixel area may be provided.

[0017] The signal processing chip 111 has TSVs (Through Silicon Vias) 110 that connect the circuits provided on the front and back surfaces of the chip to each other. The TSVs 110 are preferably provided in the peripheral region. The TSVs 110 may also be provided in the peripheral region of the imaging chip 113 and the memory chip 112.

[0018] 2 is a diagram for explaining the pixel array of the imaging chip 113 and the unit area 131. In particular, the imaging chip 113 is shown as observed from the back side. For example, more than 20 million pixels are arranged in a matrix in the pixel area. In this embodiment, for example, adjacent 4 pixels x 4 pixels, or 16 pixels, form one unit area 131. The grid lines in the figure show the concept of forming the unit area 131 by grouping adjacent pixels. The number of pixels forming the unit area 131 is not limited to this, and may be about 1000, for example, 32 pixels x 64 pixels, or may be more or less than that.

[0019] As shown in the partially enlarged view of the pixel region, the unit region 131 contains four so-called Bayer arrays, each consisting of four pixels: green pixels Gb, Gr, blue pixel B, and red pixel R, arranged vertically and horizontally. The green pixels are pixels having a green filter as the color filter 102 and receive light in the green wavelength band of the incident light. Similarly, the blue pixels are pixels having a blue filter as the color filter 102 and receive light in the blue wavelength band, and the red pixels are pixels having a red filter as the color filter 102 and receive light in the red wavelength band.

[0020] In this embodiment, a plurality of blocks are defined so that each block includes at least one unit region 131, and each block can control the pixels included in the block with different control parameters. In other words, imaging signals with different imaging conditions can be acquired for a pixel group included in one block and a pixel group included in another block. Examples of the control parameters include a frame rate, a gain, a thinning rate, the number of summation rows or columns for adding pixel signals, the accumulation time or number of accumulations of electric charge, the number of bits for digitization, etc. Furthermore, the control parameters may be parameters for image processing after acquiring image signals from pixels.

[0021] Fig. 3 is a circuit diagram corresponding to a unit area 131 of the imaging chip 113. In Fig. 3, a rectangle surrounded by a dotted line representatively represents a circuit corresponding to one pixel. Note that at least a part of each transistor described below corresponds to the transistor 105 in Fig. 1.

[0022] As described above, the unit area 131 is formed of 16 pixels. The 16 PDs 104 corresponding to the respective pixels are connected to the transfer transistors 302, and each gate of the transfer transistors 302 is connected to the TX wiring 307 to which a transfer pulse is supplied. In this embodiment, the TX wiring 307 is commonly connected to the 16 transfer transistors 302.

[0023] The drain of each transfer transistor 302 is connected to the source of the corresponding reset transistor 303, and a so-called floating diffusion FD between the drain of the transfer transistor 302 and the source of the reset transistor 303 is connected to the gate of the amplification transistor 304. The drain of the reset transistor 303 is connected to a Vdd wiring 310 to which a power supply voltage is supplied, and the gate of the reset transistor 303 is connected to a reset wiring 306 to which a reset pulse is supplied. In this embodiment, the reset wiring 306 is commonly connected to the 16 reset transistors 303.

[0024] The drain of each of the amplification transistors 304 is connected to a Vdd wiring 310 to which a power supply voltage is supplied. In addition, the source of each of the amplification transistors 304 is connected to the drain of each of the corresponding selection transistors 305. Each gate of the selection transistor 305 is connected to a decoder wiring 308 to which a selection pulse is supplied. In this embodiment, the decoder wiring 308 is provided independently for each of the 16 selection transistors 305. Then, the source of each selection transistor 305 is connected to a common output wiring 309. The load current source 311 supplies a current to the output wiring 309. That is, the output wiring 309 for the selection transistor 305 is formed by a source follower. The load current source 311 may be provided on the imaging chip 113 side or on the signal processing chip 111 side.

[0025] Here, the flow from the start of charge accumulation to pixel output after accumulation is completed will be described. When a reset pulse is applied to the reset transistor 303 via the reset wiring 306 and at the same time a transfer pulse is applied to the transfer transistor 302 via the TX wiring 307, the potentials of the PD 104 and the floating diffusion FD are reset.

[0026] When the application of the transfer pulse is released, the PD 104 converts the received incident light into electric charges and accumulates them. After that, when the transfer pulse is applied again without the reset pulse being applied, the accumulated electric charges are transferred to the floating diffusion FD, and the potential of the floating diffusion FD changes from the reset potential to the signal potential after the electric charges are accumulated. Then, when a selection pulse is applied to the selection transistor 305 through the decoder wiring 308, the fluctuation of the signal potential of the floating diffusion FD is transmitted to the output wiring 309 through the amplification transistor 304 and the selection transistor 305. As a result, a pixel signal corresponding to the reset potential and the signal potential is output from the unit pixel to the output wiring 309.

[0027] As shown in FIG. 3, in this embodiment, the reset wiring 306 and the TX wiring 307 are common to the 16 pixels forming the unit area 131. That is, the reset pulse and the transfer pulse are each applied simultaneously to all 16 pixels. Therefore, all pixels forming the unit area 131 start charge accumulation at the same timing, and end charge accumulation at the same timing. However, pixel signals corresponding to the accumulated charges are selectively output from the output wiring 309 by sequentially applying a selection pulse to each selection transistor 305. In addition, the reset wiring 306, the TX wiring 307, and the output wiring 309 are provided separately for each unit area 131.

[0028] In this way, by configuring the circuit based on the unit area 131, it is possible to control the charge accumulation time for each unit area 131. In other words, it is possible to output pixel signals at different frame rates between the unit areas 131. Furthermore, while one unit area 131 is performing a single charge accumulation, the other unit area 131 is caused to repeatedly accumulate charge and output a pixel signal each time, so that each frame for a moving image can be output at a different frame rate between these unit areas 131.

[0029] 4 is a block diagram showing a functional configuration of the image sensor 100. An analog multiplexer 411 sequentially selects the 16 PDs 104 forming a unit area 131 and outputs the pixel signals of each of the PDs 104 to the output wiring 309 provided corresponding to the unit area 131. The multiplexer 411 is formed in the image sensor chip 113 together with the PDs 104.

[0030] The pixel signals output via the multiplexer 411 undergo CDS and A / D conversion by a signal processing circuit 412 that performs correlated double sampling (CDS) and analog / digital (A / D) conversion and is formed in the signal processing chip 111. The A / D converted pixel signals are passed to a demultiplexer 413. The pixel signals output from the demultiplexer 413 are input to adders 416 corresponding to the respective pixels. The adder 416 adds the pixel signal output from the demultiplexer 413 and the pixel signal read out from the pixel memory 414 corresponding to the respective pixels, and outputs the pixel signal after the addition back to the pixel memory 414.

[0031] The pixel memories 414 store the pixel signals from the adders 416. Each of the pixel memories 414 has a capacity capable of storing the pixel signals after addition. The demultiplexer 413, the adders 416, and the pixel memories 414 are formed in the memory chip 112.

[0032] Fig. 5 is a diagram explaining the flow of pixel signals per pixel. In Fig. 5, a pixel signal S output from the demultiplexer 413 is input to a corresponding adder n of the adders 416. At this time, a pixel signal P stored in the corresponding memory n of the pixel memory 414 is read out from the memory n and input to the adder n.

[0033] The adder n adds the input pixel signals S and P, and outputs the pixel signal S+P after the addition to the pixel memory n. The pixel memory n stores the input pixel signal S+P and waits for it to be read out to the arithmetic circuit 415. Here, by controlling the pixel memory 414 so as not to read out the pixel signal P stored in the pixel memory n when the adder n performs the addition, it is possible to output only the pixel signal S input to the adder n from the adder n to the pixel memory n as it is. In other words, it is also possible to read out the pixel signal S from the imaging chip 113 from the memory n to the arithmetic circuit 415 as it is, without addition by the adder n.

[0034] The arithmetic circuit 415 processes the pixel signals stored in the pixel memory 414 and passes them to a downstream image processing unit. The arithmetic circuit 415 may be provided in the signal processing chip 111 or in the memory chip 112.

[0035] The drive control unit 417 generates a timing control signal to synchronize the timing at which pixel signals are sent from the imaging chip 113 to the signal processing chip 111 and the memory chip 112, the timing at which the pixel signals are read out and stored in the pixel memory 414, the timing at which the pixel signals are added in the adder 416, and the timing at which the pixel signals are passed to the arithmetic circuit 415.

[0036] 4 shows the connections for one unit area 131, but in reality, these exist for each unit area 131 and operate in parallel. However, a calculation circuit 415 does not need to exist for each unit area 131. For example, one calculation circuit 415 may process sequentially while referring to the values ​​of the pixel memories 414 corresponding to each unit area 131 in order.

[0037] As described above, output wiring 309 is provided corresponding to each unit area 131. Since image pickup element 100 has imaging chip 113, signal processing chip 111, and memory chip 112 stacked on top of each other, by using bumps 109 for electrical connection between the chips for output wiring 309, it is possible to route wiring without increasing the size of each chip in the planar direction.

[0038] <Explanation of imaging device> Fig. 6 is a block diagram illustrating a configuration of an imaging device 1 having the above-mentioned imaging element 100. In Fig. 6, the imaging device 1 has an imaging optical system 10, an imaging section 20, an image processing section 30, a work memory 40, a display section 50, a recording section 60, and a control section 70.

[0039] The imaging optical system 10 is composed of a plurality of lenses, and guides a light beam from a subject field to the imaging section 20. The imaging optical system 10 may be configured integrally with the imaging device 1, or may be configured to be replaceable with respect to the imaging device 1. The imaging optical system 10 may also include a built-in focus lens or a built-in zoom lens.

[0040] The imaging section 20 has the imaging element 100 described above and a driving section 21 that drives the imaging element 100. The imaging element 100 is capable of independent accumulation control in block units described above by being driven and controlled by a control signal output from the driving section 21. The control section 70 issues instructions to the driving section 21 regarding the positions, shapes, ranges, etc. of the blocks.

[0041] The image processing unit 30 cooperates with the work memory 40 to perform image processing on the image data captured by the imaging unit 20. In this embodiment, the image processing unit 30 performs detection processing of a main subject included in an image in addition to normal image processing (color signal processing, gamma correction, etc.). The detection of the main subject by the image processing unit 30 can be performed using a known face detection function. In addition to face detection, a human body included in an image may be detected as a main subject, as described in, for example, JP 2010-16621 A (US 2010 / 0002940 A).

[0042] The work memory 40 temporarily stores image data before and after JPEG compression or before and after MPEG compression. The display unit 50 is configured with, for example, a liquid crystal display panel 51, and displays images (still images and videos) captured by the imaging unit 20, various information, and an operation input screen. The display unit 50 has a configuration in which a touch panel 52 is layered on the display surface of the liquid crystal display panel 51. The touch panel 52 outputs a signal indicating the position on the liquid crystal display panel 51 touched by the user.

[0043] The recording unit 60 stores various data such as image data in a storage medium such as a memory card. The control unit 70 has a CPU and controls the overall operation of the imaging device 1. In this embodiment, the control unit 70 divides the imaging surface of the imaging element 100 (imaging chip 113) into multiple blocks and captures images at different frame rates (accumulation times) and gains between the blocks. To this end, the control unit 70 instructs the driving unit 21 on the position, shape, and range of the blocks, as well as control parameters for each block.

[0044] The control unit 70 also calculates the focus adjustment state of the imaging optical system 10 using an AF calculation unit 71. The control unit 70 further performs exposure calculations using an AE / AWB calculation unit 72 so as to obtain appropriate exposure.

[0045] <Attention area and surrounding area> In this embodiment, the concepts of an attention area and a peripheral area are introduced into the screen to correspond to the above-mentioned multiple blocks. Fig. 7 is a diagram illustrating an attention area 80 and a peripheral area 90 in the image sensor 100 (imaging chip 113). The control unit 70 determines the positions of the attention area 80 and the peripheral area 90 in the image sensor 100 (imaging chip 113) through scene recognition based on a live view image.

[0046] Here, the live view image is also called a preview image before actual imaging is performed, and refers to an image for a monitor acquired by the image sensor 100 at a predetermined frame rate (e.g., 30 fps). In Fig. 7, a person is included on the right side of the screen of the live view image, and a tree is included on the left side of the screen of the live view image. The control unit 70 sends an instruction to the image processing unit 30 to perform a known scene recognition process on the live view image data. The image processing unit 30 performs the scene recognition process to analyze the live view image and extract the main subject area.

[0047] The image processing unit 30 determines the area including the detected human body as the main subject area as described above. Note that it is also possible to detect animals such as pets and determine the area including the animals as the main subject area, not limited to people. The control unit 70 then determines the main subject area as the attention area 80, and the area outside the attention area 80 as the peripheral area 90.

[0048] In addition, when a live view image is displayed on the liquid crystal display panel 51 of the display unit 50, the area of ​​the main subject (which is displayed) corresponding to the position where the user viewing the live view image touches the touch panel 52 may be defined as the attention area 80.

[0049] For example, when acquiring a live view image, the control unit 70 causes the AE / AWB calculation unit 72 to perform automatic exposure calculation and determine a white balance adjustment value based on pixel signals output from the attention area 80 and the surrounding area 90 after accumulation for a first accumulation time. The AE / AWB calculation unit 72 calculates exposure (exposure time, gain, etc.) so as to bring the average level of the pixel signals closer to a predetermined level. The AE / AWB calculation unit 72 also determines a white balance adjustment value for expressing a white color as white.

[0050] Furthermore, the control unit 70 generates an image for monitoring based on pixel signals output from the attention area 80 and the peripheral area 90 after accumulation for the first accumulation time, and causes the display unit 50 to display the image as the live view image.

[0051] The control unit 70 further causes the AF (autofocus) calculation unit 71 to calculate the focus adjustment state of the imaging optical system 10 based on pixel signals output from the attention area 80 after accumulation for the second accumulation time, for example, when acquiring a live view image. In this embodiment, the second accumulation time is controlled to be longer than the first accumulation time.

[0052] The AF calculation unit 71 detects the focus adjustment state by, for example, a contrast detection method. Specifically, while moving the position of the focus lens of the imaging optical system 10, the position of the focus lens of the imaging optical system 10 is adjusted so as to increase the contrast of an image formed by pixel signals output from the attention area 80.

[0053] The focus detection process may be performed by a phase difference detection method. In this case, focus detection pixels are provided in advance in the image sensor 100 (image capture chip 113). Then, a phase difference detection calculation is performed using output signals from the focus detection pixels included in the attention area 80, thereby detecting the focus adjustment state (specifically, the defocus amount) of the image capture optical system 10. The focus detection pixels and phase difference detection calculation are publicly known, as described in, for example, JP 2009-94881 A, and therefore detailed description thereof will be omitted.

[0054] When acquiring a live view image, the control unit 70 sends an instruction to the driving unit 21 to read out pixel signals having different accumulation times (i.e., pixel signals after the first accumulation time has elapsed and pixel signals after the second accumulation time has elapsed) in multiple separate readouts from the attention area 80 of the image sensor 100 (imaging chip 113) as described above. Here, accumulation control is performed separately for the attention area 80 and the peripheral area 90 before an instruction to perform actual imaging (still image recording or video recording) is given by operating a release switch (not shown).

[0055] In other words, until an actual imaging instruction is given, exposure calculations, white balance adjustment values ​​are determined, and a live view image is displayed based on the images acquired in the attention area 80 and the surrounding area 90, and focus detection processing is performed based on the image acquired in the attention area 80.

[0056] <Reading out pixel signals with different accumulation times> Reading of pixel signals with different accumulation times will be described with reference to FIG. 8 which illustrates the readout timing of pixel signals, accumulation signals in the imaging chip 113, and pixel signals read out from the imaging element 100 via the arithmetic circuit 415.

[0057] The driving unit 21 controls the image sensor 100 as follows. That is, in each frame of a live view image, the period from the accumulation start time t0 to time t1 is set as the first accumulation time, and the period from time t0 to time t2 is set as the second accumulation time. At time t0, the driving unit 21 starts charge accumulation for pixels included in the attention area 80 and the peripheral area 90. Then, at time t1, the driving unit 21 outputs pixel signals from the attention area 80 and the peripheral area 90 while controlling the pixel memory 414 so as not to read out pixel signals stored in the pixel memory n illustrated in FIG. 5. As a result, the pixel signal a accumulated during the first accumulation time (from time t0 to time t1) is output from the demultiplexer 413 and output as it is as the signal A via the arithmetic circuit 415. This pixel signal A (=a) is also stored in the pixel memory n.

[0058] Furthermore, when the driving unit 21 reads out the pixel signal at time t1, it immediately starts charge accumulation for the pixels included in the region of interest 80. Then, at time t2, it controls the pixel memory 414 to read out the pixel signal a stored in the pixel memory n illustrated in FIG. 5, while outputting the pixel signal from the region of interest 80. As a result, the pixel signal b accumulated between time t1 and time t2 is output from the demultiplexer 413, and the pixel signal b and the pixel signal a read out from the pixel memory n are added by the adder n. The pixel signal a+b after the addition is output as the signal B via the arithmetic circuit 415. The pixel signal B (=a+b) is the sum of the pixel signals accumulated from time t0 to time t1 and from time t1 to time t2, and therefore corresponds to the pixel signal accumulated during the second accumulation time (from time t0 to time t2).

[0059] As described above, in the first embodiment, the pixel signal B with a high signal level accumulated over the second accumulation time longer than the first accumulation time is used for focus detection processing, so that higher focus detection accuracy can be obtained compared to the case where the pixel signal A with a low signal level is used for focus detection processing. Also, the pixel signal A accumulated over the first accumulation time shorter than the second accumulation time is used for exposure calculation, determination of a white balance adjustment value, and display of a live view image, so that calculations can be performed with high accuracy while avoiding the influence of noise, and a live view image that is not too bright and easy to see can be obtained.

[0060] <Explanation of the flow chart> Fig. 9 is a flowchart for explaining the flow of the photographing operation executed by the control unit 70 of the imaging device 1 in the first embodiment. When an ON-OFF switch (not shown) is operated to turn on the power and electricity is being supplied to each unit of the imaging device 1, the control unit 70 repeatedly starts the process shown in Fig. 9. In step S101 of Fig. 9, the control unit 70 determines control parameters such as frame rate and gain for the attention area 80 and the surrounding area 90, respectively, and proceeds to step S102. For example, values ​​to be applied in steps S102 and S110 described later are read out from program data and prepared.

[0061] In step S102, the control unit 70 sends an instruction to the driving unit 21 to start live view imaging by the imaging unit 20. Acquisition of the live view image started in step S102 is performed by setting control parameters for the surrounding area 90 for, for example, substantially the entire imaging surface of the image sensor 100.

[0062] In step S103, the control unit 70 sends an instruction to the drive unit 21 to transfer first data from the imaging unit 20. The first data corresponds to the pixel signal A (=a) read out at the time t1. In step S104, the control unit 70 causes the image processing unit 30 to perform image processing on live view image data based on the pixel signal A output from the imaging unit 20, and then causes the display unit 50 to display the image.

[0063] In step S105, the control unit 70 sends an instruction to the drive unit 21 to transfer second data from the imaging unit 20. The second data corresponds to the pixel signal B (=a+b) read out at the above-mentioned time t2. In step S106, the control unit 70 causes the AF calculation unit 71 to perform AF calculation (detection of the focus adjustment state) based on the pixel signal B for the AF calculation area. This allows focus adjustment of the imaging optical system 10. Note that since the AF calculation area is not yet determined in the first frame after the power-on operation, the AF calculation is performed for substantially the entire imaging surface of the image sensor 100.

[0064] In step S107, the control unit 70 sends an instruction to the image processing unit 30 to extract the main subject region from the live view image. In step S108, the control unit 70 sets the region including the main subject as the attention region 80, determines this attention region 80 as the AF calculation region, and proceeds to step S109. The AF calculation region is applied to the live view image acquired in the next frame.

[0065] In step S109, the control unit 70 determines whether or not a release operation has been performed. The release operation is used as a main imaging instruction to the imaging device 1. If a release button (not shown) has been pressed, the control unit 70 makes an affirmative decision in step S109 and proceeds to step S110, and if a release button (not shown) has not been pressed, the control unit 70 makes a negative decision in step S109 and proceeds to step S112.

[0066] When a tap operation on the release icon displayed on the liquid crystal display panel 51 is detected to determine whether or not an actual image capture instruction has been issued, a positive judgment can be made in step S109 if the release icon being displayed is tapped.

[0067] In step S110, the control unit 70 sends an instruction to the driving unit 21 to set control parameters (exposure time, gain, etc.) required for the exposure conditions for shooting determined based on the first data (pixel signal A (=a)), and proceeds to step S111.

[0068] In step S111, the control unit 70 executes a photographing process, and stores the acquired image data in a memory card or the like by the recording unit 60, and the process returns to step S102. In the photographing process, photographing control parameters set to be common (the same) across all areas of the image sensor 100 are applied to acquire and record one still image (main image capture), and also acquire multiple frames of images after acquiring the still image. Then, based on the multiple frames of images acquired within a predetermined time before and after the release operation, slow motion playback video data is generated and recorded. The slow motion playback video data refers to moving image data that is played back at a frame rate (e.g., 15 fps) slower than the frame rate (e.g., 30 fps) acquired by the image sensor 100.

[0069] The control unit 70 generates slow-motion video data as follows. That is, the slow-motion video data is generated based on a plurality of frame images (for example, 18 frames for 0.6 seconds when the frame rate is 30 fps) based on the first data (pixel signal A (=a)) temporarily stored in the work memory 40 for displaying the above-mentioned live view image from the time before the release operation time (tx) (for example, 0.6 seconds before) to the time tx, and a plurality of frame images (for example, 12 frames for 0.4 seconds when the frame rate is 30 fps) based on the first data (pixel signal A (=a)) stored in the work memory 40 from the time tx to the time after the time tx (for example, 0.4 seconds after). As a result, slow-motion video data with a playback time of about 2 seconds is generated based on a plurality of frame images (30 images in total) stored in the work memory 40 during one second (from 0.6 seconds before the time tx to 0.4 seconds after the time tx) that sandwiches the time tx. In this way, slow motion playback video data based on the frame images acquired before and after the release operation is obtained. The slow motion playback video data is generated by the image processor 30 as MPEG data or JPEG data.

[0070] In step S112, which is reached after a negative decision is made in step S109, the control unit 70 decides whether or not the power has been turned off. If an ON-OFF switch (not shown) has been turned off, the control unit 70 makes a positive decision in step S112, performs a predetermined power-off process, and ends the process in Fig. 9. If an ON-OFF switch (not shown) has not been turned off, the control unit 70 makes a negative decision in step S112 and returns to step S102. If returning to step S102, the above-mentioned process is repeated.

[0071] According to the first embodiment described above, the following advantageous effects can be obtained. (1) The image sensor 100 includes a demultiplexer 413 that inputs data from a plurality of pixels that perform photoelectric conversion, a pixel memory 414 that stores the input data for each pixel, an adder 416 that adds the data stored in the pixel memory 414 and the input data for each pixel, and an arithmetic circuit 415 that outputs the input data or the data added by the adder 416. This allows the photoelectric conversion data to be appropriately processed, improving the ease of use of the image sensor 100.

[0072] (2) The pixel memory 414 stores the data added by the adder 416 for each pixel, so that it is possible to accumulate the input data. This makes it possible to accumulate the photoelectric conversion data and output it, or to output it without accumulating it.

[0073] (3) The adder 416 adds data with different photoelectric conversion times for each pixel, so that data equivalent to the case where the photoelectric conversion time is changed to a longer time can be obtained.

[0074] (4) The drive control unit 417 is provided to read out the data stored in the pixel memory 414 and to cause the adder 416 to add the data when data is input from the pixels, thereby enabling control at appropriate timing.

[0075] (5) When the data output from the arithmetic circuit 415 is the above-mentioned input data, exposure calculation, determination of a white balance adjustment value, and display of a live view image are performed. When the data output from the arithmetic circuit 415 is added data, focus detection processing is performed. This makes it possible to appropriately combine photoelectric conversion data and processing using the same.

[0076] (6) Whether to output the input data or the sum data from the arithmetic circuit 415 is determined by analyzing the image formed by the input data, so that appropriate photoelectric conversion data can be output according to the image.

[0077] (7) In the above determination, additive data is output for pixels corresponding to the main subject region of the image, and input data is output for pixels corresponding to regions other than the main subject region of the image, so that appropriate photoelectric conversion data can be output.

[0078] (8) Since imaging chip 113 having a plurality of pixels and memory chip 112 for processing photoelectric conversion data are stacked, imaging element 100 can be configured compactly without increasing the size of each chip in the planar direction.

[0079] (9) The imaging device 1, which is an example of an electronic device, includes a memory chip 112 that processes photoelectric conversion data and an imaging chip 113 having a plurality of pixels. This makes it possible to provide an imaging device 1 that is easy to use and that appropriately processes the photoelectric conversion data.

[0080] Second Embodiment In the first embodiment, an example was described in which the screen is divided into an attention area 80 and a peripheral area 90, and images with different accumulation times are obtained in the attention area 80 and the peripheral area 90 before an instruction to perform actual imaging is given. In the second embodiment, the screen is divided into a bright area 85 and a dark area 95, and images with different accumulation times are obtained in the bright area 85 and the dark area 95 after an instruction to perform actual imaging is given. Here, the bright area 85 is defined as an area formed of pixel signals having a value higher than a predetermined pixel signal value (higher than a predetermined luminance value) by analyzing a live view image, for example. The dark area 95 is defined as an area other than the bright area 85.

[0081] The imaging device 1 according to the second embodiment acquires an image based on pixel signals accumulated in the bright region 85 for a first accumulation time and an image based on pixel signals accumulated in the dark region 95 for a second accumulation time, and synthesizes both images to obtain an image with a wide dynamic range. For this reason, the control unit 70 sends an instruction to the driving unit 21 when acquiring an image for recording, and reads out pixel signals after the first accumulation time has elapsed and pixel signals after the second accumulation time has elapsed from the dark region 95 of the imaging element 100 (imaging chip 113) in multiple batches.

[0082] <Reading out pixel signals with different accumulation times> The timing of reading out pixel signals, the accumulated signals in the imaging chip 113, and the pixel signals read out from the imaging element 100 via the arithmetic circuit 415 will be described with reference to FIG.

[0083] The driving unit 21 controls the image sensor 100 as follows. That is, the first accumulation time is from the accumulation start time t0 to time t1 of the image for recording after the main imaging instruction, and the second accumulation time is from time t0 to time t2. The driving unit 21 starts charge accumulation in the pixels included in the bright area 85 and the dark area 95 at time t0. Then, at time t1, the driving unit 21 outputs pixel signals from the bright area 85 and the dark area 95 while controlling the pixel memory 414 not to read out the pixel signals stored in the pixel memory n illustrated in FIG. 5. As a result, the pixel signal a accumulated during the first accumulation time (from time t0 to time t1) is output from the demultiplexer 413 and output as it is as the signal A via the arithmetic circuit 415. This pixel signal A (=a) is also stored in the pixel memory n.

[0084] Furthermore, when the driving unit 21 reads out the pixel signal at time t1, it immediately starts charge accumulation in the pixels included in the dark area 95. Then, at time t2, it controls the pixel memory 414 to read out the pixel signal a stored in the pixel memory n illustrated in FIG. 5, while outputting the pixel signal from the dark area 95. As a result, the pixel signal b accumulated between time t1 and time t2 is output from the demultiplexer 413, and the pixel signal b and the pixel signal a read out from the pixel memory n are added by the adder n. The pixel signal a+b after the addition is output as the signal B via the arithmetic circuit 415. The pixel signal B (=a+b) is the sum of the pixel signals accumulated from time t0 to time t1 and from time t1 to time t2, and therefore corresponds to the pixel signal accumulated during the second accumulation time (from time t0 to time t2).

[0085] As described above, in the second embodiment, the image of the dark area 95 is formed by the pixel signal B accumulated in the second accumulation time longer than the first accumulation time, so the image is less likely to be crushed into black compared to when the image of the dark area 95 is formed by the pixel signal A with a short accumulation time and a low signal level. Also, the image of the bright area 85 is formed by the pixel signal A accumulated in the first accumulation time shorter than the second accumulation time, so the image is less likely to be blown out compared to when the image of the bright area 85 is formed by the pixel signal B with a long accumulation time and a high signal level. As a result, an image with a wide dynamic range from bright areas to dark areas can be obtained by performing the accumulation process from time t0 to time t2 only once. Also, since the accumulation start time is the same for the first accumulation time and the second accumulation time (both are time t0), this is also suitable for shooting a moving subject, unlike the conventional technology in which multiple images with different accumulation start times are synthesized.

[0086] <Explanation of the flow chart> Fig. 10 is a flowchart for explaining the flow of the photographing operation executed by the control unit 70 of the imaging device 1 in the second embodiment. When an ON-OFF switch (not shown) is operated to turn on the power and electricity is being applied to each unit of the imaging device 1, the control unit 70 repeatedly starts the process shown in Fig. 10. In step S201 of Fig. 10, the control unit 70 determines control parameters such as frame rate and gain for the bright area 85 and the dark area 95, respectively, and proceeds to step S202. For example, values ​​to be applied in steps S202, S205, and S208 described later are read out from program data and prepared.

[0087] In step S202, the control unit 70 sends an instruction to the driving unit 21 to start live view imaging by the imaging unit 20. Acquisition of the live view image started in step S202 is performed by setting control parameters for the dark area 95 for, for example, substantially the entire imaging surface of the image sensor 100.

[0088] In step S203, the control unit 70 determines the bright area 85 and the dark area 95 based on the live view image, and proceeds to step S204. In step S204, the control unit 70 determines whether or not a release operation has been performed. The release operation is used as a main image capture instruction to the imaging device 1. If the release button (not shown) has been pressed, the control unit 70 makes an affirmative decision in step S204 and proceeds to step S205, and if the release button has not been pressed, the control unit 70 makes a negative decision in step S204 and proceeds to step S211.

[0089] When a tap operation on the release icon displayed on the liquid crystal display panel 51 is detected to determine whether or not an actual image capture instruction has been issued, a positive judgment can be made in step S204 if the release icon being displayed is tapped.

[0090] In step S205, the control unit 70 sends an instruction to the driving unit 21 to start capturing an image for recording by the imaging unit 20. The acquisition of the image for recording that starts in step S205 is performed by setting control parameters for the bright region 85 for, for example, substantially the entire imaging surface of the imaging element 100.

[0091] In step S206, the control unit 70 sends an instruction to the driving unit 21 to transfer the first data from the imaging unit 20. The first data corresponds to the pixel signal A (=a) read out at the time t1. In step S207, the control unit 70 sends an instruction to the image processing unit 30 to temporarily store the data corresponding to the bright area 85 out of the first data in the work memory 40, and the process proceeds to step S208.

[0092] In step S208, the control unit 70 sends an instruction to the driving unit 21 to transfer second data from the imaging unit 20. The second data corresponds to the pixel signal B (=a+b) read out at the time t2. In step S209, the control unit 70 sends an instruction to the image processing unit 30 to temporarily store the second data (i.e., data corresponding to the dark area 95) in the work memory 40, and the process proceeds to step S210.

[0093] In step S210, the control unit 70 sends an instruction to the image processing unit 30 to perform image synthesis of the first data and the second data temporarily stored in the work memory 40. Specifically, in the image based on the first data, a region corresponding to the dark area 95 is replaced with an image based on the second data. This results in a synthetic image in which the dark area 95 is composed of pixel signal B (=a+b) and the bright area is composed of pixel signal A (=a). When the image synthesis is completed, the control unit 70 returns to step S202. When returning to step S202, the above-mentioned processing is repeated.

[0094] In step S211, which is reached after making a negative decision in step S204, the control unit 70 decides whether or not the power has been turned off. If an ON-OFF switch (not shown) has been turned off, the control unit 70 makes a positive decision in step S211, performs a predetermined power-off process, and ends the process in Fig. 10. If an ON-OFF switch (not shown) has not been turned off, the control unit 70 makes a negative decision in step S211 and returns to step S202.

[0095] According to the second embodiment described above, the following advantageous effects can be obtained. (1) The image sensor 100 forms bright areas 85 of the image when the data output from the arithmetic circuit 415 is the data input to the demultiplexer 413, and forms dark areas 95 of the image when the data output from the arithmetic circuit 415 is the data added by the adder 416. Therefore, it is possible to obtain an image with a wide dynamic range by appropriately combining the photoelectric conversion data and the processing using the same.

[0096] (2) Whether to output the input data or the sum data from the arithmetic circuit 415 is determined by analyzing the image formed by the input data, so that appropriate photoelectric conversion data can be output according to the image.

[0097] (3) In the above determination, the input data is output for pixels corresponding to areas where the image brightness is higher than a predetermined value, and the added data is output for pixels corresponding to areas where the image brightness is lower than a predetermined value, so that appropriate photoelectric conversion data can be output.

[0098] (Variation 1) The imaging device 1 according to the first and second embodiments described above may be configured as a high-performance mobile phone or a tablet terminal. In this case, the camera unit mounted on the high-performance mobile phone (or tablet terminal) is configured using the stacked imaging element 100 described above.

[0099] (Variation 2) In the above-described second embodiment, an example has been described in which the image processing unit 30 synthesizes images using the first data and the second data temporarily stored in the work memory 40. Instead of this, the image synthesis may be performed using the pixel memory 414 and the adder 416 illustrated in Figs. 4 and 5. For example, when reading out pixel signals at time t1, control is performed so that only pixel signals A corresponding to the bright area 85 are selectively read out from the image sensor 100. Also, when reading out pixel signals at time t2, control is performed so that only pixel signals B corresponding to the dark area 95 are selectively read out from the image sensor 100.

[0100] (Variation 3) In the above-described first and second embodiments, an example has been described in which pixel signals after a first accumulation time has elapsed and pixel signals after a second accumulation time have elapsed are read out twice from a predetermined region of the image sensor 100 (image sensor chip 113). The number of readouts is not limited to the above-described two times, and may be three, four, or more.

[0101] Furthermore, images read out in multiple steps before an instruction to perform actual imaging may be divided into images for use in AF calculation, images for use in exposure calculation and determining a white balance adjustment value, images for use in extracting a main subject, images for use in live view display, etc. Furthermore, images read out in multiple steps after an instruction to perform actual imaging may be divided into images for use according to the brightness of the images, such as extremely bright areas, bright areas, dark areas, extremely dark areas, etc.

[0102] The above description is merely an example, and the present invention is not limited to the configurations of the above-described embodiments. The configurations of the above-described embodiments and modifications may be combined as appropriate. [Explanation of symbols]

[0103] 1...Imaging device 10...Imaging optical system 20...Imaging unit 30...Image processing unit 40…Work memory 50...Display section 51...Liquid crystal display panel 52...Touch panel 60…Recording section 70...Control unit 71...AF calculation section 72...AE, AWB calculation section 100...Image sensor 109…Bump 111...Signal processing chip 112…Memory chip 113...Imaging chip 131...Unit area 413…Demultiplexer 414…Pixel memory 415...Arithmetic circuit 416…Adder 417...Drive control unit

Claims

[Claim 1] An imaging element including a plurality of stacked semiconductor substrates, The plurality of semiconductor substrates include A first photoelectric conversion unit that converts light into an electric charge; a second photoelectric conversion unit that converts light into electric charges and is arranged alongside the first photoelectric conversion unit in the row direction; a third photoelectric conversion unit that converts light into electric charges and is arranged alongside the first photoelectric conversion unit in the column direction; a first conversion unit that converts a first signal based on the charge converted by the first photoelectric conversion unit into a digital signal; a second conversion unit that converts a second signal based on the charge converted by the second photoelectric conversion unit into a digital signal; a third conversion unit that converts a third signal based on the charge converted by the third photoelectric conversion unit into a digital signal; a first adder that performs a first addition process using the first signal that has been converted into a digital signal by the first conversion unit; a second adder that performs a second addition process using the second signal that has been converted into a digital signal by the second conversion unit; a third adder that performs a third addition process using the third signal that has been converted into a digital signal by the third conversion unit; a first storage unit that stores the signal that has been subjected to the first addition process; a second storage unit that stores the signal that has been subjected to the second addition process; a third storage unit for storing the signal subjected to the third addition process; having the first photoelectric conversion unit, the second photoelectric conversion unit, and the third photoelectric conversion unit are disposed on a first semiconductor substrate among the plurality of semiconductor substrates; the first adder, the second adder, the third adder, the first storage unit, the second storage unit, and the third storage unit are disposed on a semiconductor substrate different from the first semiconductor substrate among the plurality of semiconductor substrates, Image sensor.

Images