Imaging element

The image sensor addresses the limitation of conventional stacked image sensors by allowing for multiple moving images with varied imaging conditions, enhancing device usability through differential charge accumulation and processing.

JP2024159886AInactive Publication Date: 2024-11-08NIKON CORP
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2024139612
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2013-08-12
Filing Date
2024-08-21
Publication Date
2024-11-08
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Conventional stacked image sensors lack the ability to capture images in units of multiple blocks with different imaging conditions, limiting their usability.

Method used

An image sensor with a first and second photoelectric conversion section on semiconductor substrates, allowing for different charge accumulation timings and imaging conditions across multiple regions, enabling generation of multiple moving images with varied settings.

Benefits of technology

Enables the generation of multiple moving images with distinct imaging conditions, improving the usability of electronic devices by allowing for varied imaging parameters and maintaining resolution across different regions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024159886000001_ABST
    Figure 2024159886000001_ABST
Patent Text Reader

Abstract

To divide an imaging region into a plurality of regions to which different imaging conditions are set and generate a plurality of motion pictures corresponding to the plurality of regions.SOLUTION: A plurality of semiconductor substrates comprise a first semiconductor substrate in which a plurality of photoelectric conversion parts that convert light into electric charges are arranged in a row direction. The plurality of semiconductor substrates comprise a second semiconductor substrate having: a drive part that controls a first storage time during which electric charges respectively converted by a first photoelectric conversion part and a second photoelectric conversion part among the plurality of photoelectric conversion parts are stored and a second storage time during which electric charges respectively converted by a third photoelectric conversion part and a fourth photoelectric conversion part among the plurality of photoelectric conversion parts are stored so that the storage times are different from each other; a first conversion part that converts a first signal based on electric charges of the first photoelectric conversion part and a second signal based on electric charges of the second photoelectric conversion part into a digital signal; and a second conversion part that converts a third signal based on electric charges of the third photoelectric conversion part and a fourth signal based on electric charges of the fourth photoelectric conversion part into a digital signal.SELECTED DRAWING: Figure 3
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an imaging device. [Background technology]

[0002] An electronic device has been proposed that includes an imaging element in which a back-illuminated imaging chip and a signal processing chip are stacked (hereinafter, this imaging element is referred to as a stacked imaging element) (see, for example, Patent Document 1). In the stacked imaging element, the back-illuminated imaging chip and the signal processing chip are stacked so that they are connected via microbumps for each block unit that groups together multiple pixels. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2006-49361 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the case of electronic devices equipped with a conventional stack-type imaging element, there have not been many proposals for capturing images in units of multiple blocks to obtain images, and the usability of electronic devices equipped with a stack-type imaging element has been insufficient.

[0005] An object of one aspect of the present invention is to divide an imaging area into a plurality of areas for which different imaging conditions are set, and to generate a plurality of moving images corresponding to the plurality of areas. [Means for solving the problem]

[0006] One aspect of the present invention is an imaging device. The imaging device includes a first semiconductor substrate having a first photoelectric conversion unit that converts light into electric charges, and a second photoelectric conversion unit that converts light into electric charges. The imaging device includes a second semiconductor substrate that is a semiconductor substrate stacked together with the first semiconductor substrate and has a drive unit that controls the timing at which accumulation of electric charges converted by the first photoelectric conversion unit is ended to be different from the timing at which accumulation of electric charges converted by the second photoelectric conversion unit is ended. The drive unit executes a first mode for generating first image data based on a first signal output by the electric charges converted by the first photoelectric conversion unit and a second signal output by the electric charges converted by the second photoelectric conversion unit, and a second mode for generating second image data based on the first signal and third image data based on the second signal. Effect of the Invention

[0007] According to an aspect of the present invention, it is possible to generate a plurality of moving images corresponding to a plurality of areas for which different imaging conditions are set. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 is a cross-sectional view of a stacked imaging element. [Diagram 2] 1 is a diagram illustrating a pixel array and unit groups of an imaging chip. [Diagram 3] FIG. 2 is a circuit diagram corresponding to a unit group of the imaging chip. [Figure 4] FIG. 2 is a block diagram showing a functional configuration of an imaging element. [Diagram 5] 1 is a block diagram showing a configuration of an electronic device according to a first embodiment. [Figure 6] FIG. 4 is a diagram showing an example of a display screen of a display unit. [Figure 7] FIG. 2 is a functional block diagram of an image processing unit and a system control unit. [Figure 8] FIG. 4 is a diagram showing an arrangement pattern in each block. [Figure 9] 4 is a flowchart illustrating a photographing operation executed by a system control unit. [Figure 10] 13 is a flowchart illustrating an arrangement pattern setting process. [Figure 11] 13 is a diagram showing an example of how a second arrangement pattern is set when a second moving image mode is executed. FIG. [Figure 12] 11A and 11B are diagrams illustrating an example of a live view image displayed in an image display area. [Figure 13] FIG. 13 is a diagram showing the range and position of electronic zoom in an image display area. [Figure 14] 11A and 11B are diagrams showing the range and position of a sub-screen in an image display area. [Figure 15] 13 is a flowchart illustrating an operation during execution of the second moving image mode. [Figure 16] 13 is a timing chart showing the timing of charge accumulation in the second moving image mode. [Figure 17] FIG. 11 is a block diagram showing the configurations of an imaging device and an electronic device according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited thereto. In addition, in the drawings, in order to explain the embodiment, some parts are enlarged or emphasized, and the scale is appropriately changed.

[0010] First Embodiment 1 is a cross-sectional view of a stacked imaging element. This stacked imaging element 100 is described in Japanese Patent Application No. 2012-139026, previously filed by the applicant of the present application. The imaging element 100 includes an imaging chip 113 that outputs pixel signals corresponding to incident light, a signal processing chip 111 that processes the pixel signals, and a memory chip 112 that stores the pixel signals. The imaging chip 113, the signal processing chip 111, and the memory chip 112 are stacked, and are electrically connected to each other by conductive bumps 109 made of Cu or the like.

[0011] As shown in the figure, the incident light is mainly incident in the Z-axis positive direction indicated by the white arrow. In this embodiment, the surface of the imaging chip 113 on which the incident light is incident is referred to as the back surface. As shown in the coordinate axes, the left direction on the paper surface perpendicular to the Z-axis is the X-axis positive direction, and the front direction on the paper surface perpendicular to the Z-axis and the X-axis is the Y-axis positive direction. In the following figures, the coordinate axes are displayed so that the orientation of each figure can be understood based on the coordinate axes in FIG. 1.

[0012] An example of the imaging chip 113 is a back-illuminated MOS image sensor. The PD layer 106 is disposed on the back side of the wiring layer 108. The PD layer 106 includes a plurality of photodiodes (hereinafter, referred to as PDs) 104 that are two-dimensionally disposed and accumulate electric charges according to incident light, and transistors 105 provided corresponding to the PDs 104.

[0013] A color filter 102 is provided on the incident light side of the PD layer 106 via a passivation film 103. The color filter 102 is a filter that transmits a specific wavelength range of visible light. There are multiple types of color filters 102 that transmit different wavelength ranges, and each has a specific arrangement corresponding to each PD 104. The arrangement of the color filters 102 will be described later. A set of the color filter 102, the PD 104, and the transistor 105 forms one pixel.

[0014] A microlens 101 is provided corresponding to each pixel on the incident light side of the color filter 102. The microlens 101 condenses the incident light toward the corresponding PD 104.

[0015] The wiring layer 108 has wiring 107 that transmits pixel signals from the PD layer 106 to the signal processing chip 111. The wiring 107 may be multi-layered, and may include passive elements and active elements. A plurality of bumps 109 are arranged on the surface of the wiring layer 108. These plurality of bumps 109 are aligned with a plurality of bumps 109 provided on the opposing surface of the signal processing chip 111. Then, the imaging chip 113 and the signal processing chip 111 are pressed, etc., so that the aligned bumps 109 are joined together and electrically connected.

[0016] Similarly, a plurality of bumps 109 are arranged on the opposing surfaces of the signal processing chip 111 and the memory chip 112. These bumps 109 are aligned with each other. Then, the signal processing chip 111 and the memory chip 112 are pressed together, etc., so that the aligned bumps 109 are bonded to each other and electrically connected.

[0017] The bonding between the bumps 109 is not limited to Cu bump bonding by solid-phase diffusion, and micro-bump bonding by solder melting may be adopted. Also, it is sufficient to provide about one bump 109 for each unit group described later. Therefore, the size of the bump 109 may be larger than the pitch of the PD 104. Also, in a peripheral area other than the pixel area (pixel area 113A shown in FIG. 2) where the pixels are arranged, a bump larger than the bump 109 corresponding to the pixel area may be provided.

[0018] The signal processing chip 111 has TSVs (Through-Silicon Vias) 110 that connect the circuits provided on the front and back surfaces of the chip to each other. The TSVs 110 are provided in the peripheral region. The TSVs 110 may also be provided in the peripheral region of the imaging chip 113 or in the memory chip 112.

[0019] FIG. 2 is a diagram for explaining the pixel arrangement and unit groups of an imaging chip. FIG. 2 particularly shows the imaging chip 113 observed from the back side. The region in which pixels are arranged in the imaging chip 113 is called pixel region 113A. More than 20 million pixels are arranged in a matrix in pixel region 113A. In the example shown in FIG. 2, adjacent 4 pixels x 4 pixels, or 16 pixels, form one unit group 131. The grid lines in FIG. 2 show the concept of grouping adjacent pixels to form unit group 131. The number of pixels forming unit group 131 is not limited to this, and may be about 1000, for example, 32 pixels x 64 pixels, or may be more or less than that.

[0020] As shown in a partially enlarged view of pixel region 113A, unit group 131 includes four so-called Bayer arrays, each consisting of four pixels: green pixels Gb, Gr, a blue pixel B, and a red pixel R, arranged vertically and horizontally. The green pixels are pixels having a green filter as color filter 102 and receive light in the green wavelength band of incident light. Similarly, the blue pixels are pixels having a blue filter as color filter 102 and receive light in the blue wavelength band. The red pixels are pixels having a red filter as color filter 102 and receive light in the red wavelength band.

[0021] 3 is a circuit diagram corresponding to a unit group of an imaging chip. In FIG. 3, a rectangle surrounded by a dotted line typically represents a circuit corresponding to one pixel. Note that at least a part of each transistor described below corresponds to the transistor 105 in FIG. 1.

[0022] As described above, the unit group 131 is formed of 16 pixels. The 16 PDs 104 corresponding to the respective pixels are connected to the transfer transistors 302. The gates of the transfer transistors 302 are connected to the TX wiring 307 to which a transfer pulse is supplied. In this embodiment, the TX wiring 307 is commonly connected to the 16 transfer transistors 302.

[0023] The drain of each transfer transistor 302 is connected to the source of the corresponding reset transistor 303, and a so-called floating diffusion FD (charge detection unit) between the drain of the transfer transistor 302 and the source of each reset transistor 303 is connected to the gate of the amplification transistor 304. The drain of each reset transistor 303 is connected to a Vdd wiring 310 to which a power supply voltage is supplied. The gate of each reset transistor 303 is connected to a reset wiring 306 to which a reset pulse is supplied. In this embodiment, the reset wiring 306 is commonly connected to the 16 reset transistors 303.

[0024] The drain of each of the amplification transistors 304 is connected to a Vdd wiring 310 to which a power supply voltage is supplied. In addition, the source of each of the amplification transistors 304 is connected to the drain of each of the corresponding selection transistors 305. The gate of each of the selection transistors 305 is connected to a decoder wiring 308 to which a selection pulse is supplied. In this embodiment, the decoder wiring 308 is provided independently for each of the 16 selection transistors 305. Then, the sources of each of the selection transistors 305 are connected to a common output wiring 309. The load current source 311 supplies a current to the output wiring 309. That is, the output wiring 309 for the selection transistor 305 is formed by a source follower. The load current source 311 may be provided on the imaging chip 113 side or on the signal processing chip 111 side.

[0025] Here, the flow from the start of charge accumulation to pixel output after accumulation is completed will be described. A reset pulse is applied to the reset transistor 303 through the reset wiring 306. At the same time, a transfer pulse is applied to the transfer transistor 302 through the TX wiring 307. This resets the potentials of the PD 104 and the floating diffusion FD.

[0026] When the application of the transfer pulse is released, the PD 104 converts the received incident light into electric charges and accumulates them. Thereafter, when the transfer pulse is applied again in a state where the reset pulse is not applied, the electric charges accumulated in the PD 104 are transferred to the floating diffusion FD. As a result, the potential of the floating diffusion FD changes from the reset potential to the signal potential after the electric charges are accumulated. Then, when a selection pulse is applied to the selection transistor 305 through the decoder wiring 308, the fluctuation of the signal potential of the floating diffusion FD is transmitted to the output wiring 309 through the amplification transistor 304 and the selection transistor 305. By the operation of such a circuit, a pixel signal corresponding to the reset potential and the signal potential is output from the unit pixel to the output wiring 309.

[0027] As shown in Fig. 3, in this embodiment, the reset wiring 306 and the TX wiring 307 are common to the 16 pixels forming the unit group 131. That is, the reset pulse and the transfer pulse are each applied simultaneously to all 16 pixels. Therefore, all pixels forming the unit group 131 start charge accumulation at the same timing, and end charge accumulation at the same timing. However, pixel signals corresponding to the accumulated charges are selectively output to the output wiring 309 by sequentially applying a selection pulse to each selection transistor 305. In addition, the reset wiring 306, the TX wiring 307, and the output wiring 309 are provided separately for each unit group 131.

[0028] In this way, by configuring the circuit based on the unit group 131, it is possible to control the charge accumulation time for each unit group 131. In other words, it is possible to output pixel signals with different charge accumulation times between the unit groups 131. Furthermore, while one unit group 131 is performing one charge accumulation, the other unit group 131 is caused to repeat charge accumulation many times and output a pixel signal each time, so that each frame for a moving image can be output at a different frame rate between these unit groups 131.

[0029] 4 is a block diagram showing the functional configuration of the image sensor. An analog multiplexer 411 sequentially selects the 16 PDs 104 forming the unit group 131. The multiplexer 411 then outputs pixel signals of the 16 PDs 104 to the output wiring 309 provided corresponding to the unit group 131. The multiplexer 411 is formed in the image sensor chip 113 together with the PDs 104.

[0030] The pixel signals, which are analog signals output via the multiplexer 411, are amplified by an amplifier 412 formed in the signal processing chip 111. The pixel signals amplified by the amplifier 412 are subjected to correlated double sampling signal processing and A / D conversion (conversion from an analog signal to a digital signal) by a signal processing circuit 413 formed in the signal processing chip 111, which performs correlated double sampling (CDS) and analog / digital conversion. The pixel signals are subjected to correlated double sampling signal processing in the signal processing circuit 413, thereby reducing noise in the pixel signals. The A / D converted pixel signals are passed to a demultiplexer 414 and stored in pixel memories 415 corresponding to the respective pixels. The demultiplexer 414 and pixel memories 415 are formed in the memory chip 112.

[0031] The arithmetic circuit 416 processes the pixel signals stored in the pixel memory 415 and passes them to a downstream image processing unit. The arithmetic circuit 416 may be provided in the signal processing chip 111 or in the memory chip 112. Note that while Fig. 4 shows connections for one unit group 131, in reality, these exist for each unit group 131 and operate in parallel. However, it is not necessary for there to be a arithmetic circuit 416 for each unit group 131. For example, one arithmetic circuit 416 may perform sequential processing while referring to values ​​of the pixel memories 415 corresponding to each unit group 131 in order.

[0032] As described above, output wiring 309 is provided corresponding to each unit group 131. The imaging element 100 has the imaging chip 113, the signal processing chip 111, and the memory chip 112 stacked on top of each other. For this reason, by using electrical connection between the chips using bumps 109 for these output wiring 309, it is possible to route the wiring without increasing the size of each chip in the planar direction.

[0033] Next, blocks set in the pixel region 113A (see FIG. 2) of the image sensor 100 will be described. In this embodiment, the pixel region 113A of the image sensor 100 is divided into a plurality of blocks. The plurality of blocks are defined so as to include at least one unit group 131 per block. The pixels included in each block are controlled with different control parameters. That is, pixel signals having different control parameters are acquired for a pixel group included in one block and a pixel group included in another block. Examples of the control parameters include the charge accumulation time or number of accumulations, frame rate, gain, thinning rate, number of rows or columns for adding pixel signals, and number of bits for digitization. Furthermore, the control parameters may be parameters for image processing after image signals are acquired from pixels.

[0034] Here, the charge accumulation time refers to the time from when the PD 104 starts accumulating charge to when it ends. Moreover, the number of times the charge is accumulated refers to the number of times the PD 104 accumulates charge per unit time. Moreover, the frame rate refers to a value that indicates the number of frames that are processed (displayed or recorded) in a moving image per unit time. The unit of the frame rate is expressed in fps (Frames Per Second). The higher the frame rate, the smoother the movement of the subject (i.e., the object being imaged) in the moving image.

[0035] Moreover, the gain refers to the gain rate (amplification rate) of the amplifier 412. By changing this gain, the ISO sensitivity can be changed. This ISO sensitivity is a standard for photographic film established by ISO, and indicates how weak light a photographic film can record. However, generally, the ISO sensitivity is also used to express the sensitivity of the image sensor 100. In this case, the ISO sensitivity is a value that indicates the ability of the image sensor 100 to capture light. Increasing the gain also improves the ISO sensitivity. For example, doubling the gain also doubles the electrical signal (pixel signal), and the appropriate brightness is achieved even with half the amount of incident light. However, increasing the gain also amplifies the noise contained in the electrical signal, resulting in more noise.

[0036] The thinning rate refers to the ratio of the number of pixels whose pixel signals are not read out to the total number of pixels in a predetermined region. For example, when the thinning rate of a predetermined region is 0, it means that pixel signals are read out from all pixels in the predetermined region. When the thinning rate of a predetermined region is 0.5, it means that pixel signals are read out from half of the pixels in the predetermined region. Specifically, when the unit group 131 is a Bayer array, pixels whose pixel signals are read out and pixels whose pixel signals are not read out are set alternately every other unit of the Bayer array in the vertical direction, that is, every two pixels (every two rows) of pixel units. Note that thinning out the readout of pixel signals reduces the resolution of the image. However, since the image sensor 100 has more than 20 million pixels arranged therein, for example, even if thinning out is performed at a thinning rate of 0.5, an image can be displayed with more than 10 million pixels. For this reason, it is considered that the user (photographer) does not notice the reduction in resolution.

[0037] The number of added rows refers to the number of pixels (rows) in the vertical direction to be added when pixel signals of pixels adjacent in the vertical direction are added. The number of added columns refers to the number of pixels (columns) in the horizontal direction to be added when pixel signals of pixels adjacent in the horizontal direction are added. Such an addition process is performed, for example, in the arithmetic circuit 416. The arithmetic circuit 416 performs a process of adding pixel signals of a predetermined number of pixels adjacent in the vertical direction or horizontal direction, thereby achieving the same effect as a process of thinning out pixel signals at a predetermined thinning rate and reading them out. Note that in the above-mentioned addition process, the arithmetic circuit 416 may calculate an average value by dividing the sum by the number of rows or columns added.

[0038] The number of digitization bits refers to the number of bits when an analog signal is converted into a digital signal in A / D conversion by the signal processing circuit 413. The more bits of the digital signal, the more detailed the luminance, color changes, and the like are expressed.

[0039] In this embodiment, the accumulation conditions refer to conditions related to the accumulation of electric charge in the image sensor 100. Specifically, the accumulation conditions refer to the electric charge accumulation time or number of accumulations, the frame rate, and the gain among the above-mentioned control parameters. Since the frame rate can change depending on the electric charge accumulation time or number of accumulations, the frame rate is included in the accumulation conditions. In addition, the light amount of the proper exposure changes depending on the gain, and the electric charge accumulation time or number of accumulations can also change depending on the light amount of the proper exposure. For this reason, the gain is included in the accumulation conditions.

[0040] Moreover, the imaging conditions refer to conditions related to imaging of a subject. Specifically, the imaging conditions refer to control parameters including the above-mentioned accumulation conditions. The imaging conditions include control parameters for controlling the imaging element 100 (e.g., charge accumulation time or accumulation count, frame rate, gain), as well as control parameters for controlling reading of signals from the imaging element 100 (e.g., thinning rate) and control parameters for processing signals from the imaging element 100 (e.g., number of summation rows or number of summation columns for summing pixel signals, number of digitization bits, and control parameters for image processing by the image processing unit 30 described later).

[0041] FIG. 5 is a block diagram showing the configuration of an electronic device according to the first embodiment. The electronic device 1 shown in FIG. 5 is composed of a device such as a digital camera, a smartphone, a mobile phone, or a personal computer equipped with an imaging function. As shown in FIG. 5, the electronic device 1 includes a lens unit 10, an imaging unit 20, an image processing unit 30, a work memory 40, a display unit 50, an operation unit 55, a recording unit 60, and a system control unit 70. The lens unit 10 is an imaging optical system composed of a plurality of lens groups. The lens unit 10 guides a light beam from a subject to the imaging unit 20. The lens unit 10 may be integral with the electronic device 1, or may be an interchangeable lens that is detachable from the electronic device 1. The lens unit 10 may also include a built-in focus lens or a built-in zoom lens.

[0042] The imaging unit 20 includes an imaging element 100 and a driving unit 21. The driving unit 21 is a control circuit that controls the driving of the imaging element 100 according to instructions from the system control unit 70. Here, the driving unit 21 controls the accumulation time or the number of accumulations of electric charges, which are control parameters, by controlling the timing (or the timing period) of applying a reset pulse and a transfer pulse to the reset transistor 303 and the transfer transistor 302, respectively. The driving unit 21 also controls the frame rate by controlling the timing (or the timing period) of applying a reset pulse, a transfer pulse, and a selection pulse to the reset transistor 303, the transfer transistor 302, and the selection transistor 305, respectively. The driving unit 21 also controls the thinning rate by setting the pixels to which the reset pulse, the transfer pulse, and the selection pulse are applied.

[0043] The driver 21 also controls the ISO sensitivity of the image sensor 100 by controlling the gain (also called gain rate or amplification rate) of the amplifier 412. The driver 21 also sets the number of rows or columns to be added together for adding pixel signals by sending an instruction to the arithmetic circuit 416. The driver 21 also sets the number of bits for digitization by sending an instruction to the signal processing circuit 413. The driver 21 also sets blocks in the pixel region (imaging region) 113A of the image sensor 100. In this way, the driver 21 functions as an image sensor control unit that causes the image sensor 100 to capture images under different imaging conditions for each of a plurality of blocks and output pixel signals. The system control unit 70 gives instructions to the driver 21 regarding the position, shape, range, etc. of the blocks.

[0044] The imaging element 100 passes pixel signals from the imaging element 100 to the image processing unit 30. The image processing unit 30 uses the work memory 40 as a workspace to perform various image processes on RAW data consisting of pixel signals of each pixel, and generates image data. The image processing unit 30 is provided with a first image processing unit 30A and a second image processing unit 30B. When the load of image processing is high, processing is assigned to the first image processing unit 30A and the second image processing unit 30B, respectively. Then, the first image processing unit 30A and the second image processing unit 30B each execute the assigned processing in parallel.

[0045] In this embodiment, as described later, the system control unit 70 (specifically, the division unit 71 shown in FIG. 7) divides the pixel region (imaging region) 113A of the imaging element 100 into at least a first region and a second region. In addition, the system control unit 70 (specifically, the imaging control unit 72 shown in FIG. 7) drives and controls the imaging element 100 so that imaging is performed under different imaging conditions for the first region and the second region. In this case, for example, the first image processing unit 30A performs image processing of the signal from the first region. In addition, the second image processing unit 30B performs image processing of the signal from the second region. Note that the pixel region (imaging region) 113A of the imaging element 100 is not limited to being divided into two regions, the first region and the second region, but may be divided into a plurality of regions such as the first region, the second region, the third region, etc. Then, image processing for the plurality of regions is appropriately assigned to the first image processing unit 30A and the second image processing unit 30B. The allocation of image processing may be determined in advance based on the number and range of the divided regions, or may be determined by the system control unit 70 based on the number and range of the divided regions.

[0046] The image processing unit 30 executes various types of image processing. For example, the image processing unit 30 generates an RGB image signal by performing color signal processing (color correction) on the signal obtained by the Bayer array. The image processing unit 30 also performs image processing such as white balance adjustment, sharpness adjustment, gamma correction, and gradation adjustment on the RGB image signal. The image processing unit 30 also performs compression processing in a predetermined compression format (JPEG format, MPEG format, etc.) as necessary. The image processing unit 30 outputs the generated image data to the recording unit 60. The image processing unit 30 also outputs the generated image data to the display unit 50.

[0047] In this embodiment, in addition to the above processes, the image processing unit 30 performs a process of detecting a main subject from image data. Here, the "main subject" refers to a subject that is the object to be imaged and that is noticed by a user (photographer) or is assumed to be noticed by a user. There is no limit to the case where only one main subject exists in image data, and there may be multiple main subjects (see, for example, FIG. 11).

[0048] The control parameters (imaging conditions) also include parameters referenced when the image processing unit 30 performs image processing. For example, the control parameters include parameters such as color signal processing (color correction), white balance adjustment, gradation adjustment, and compression rate. The signal read from the imaging element 100 changes depending on the charge accumulation time, and the parameters referenced when performing image processing change depending on the change in the signal. The image processing unit 30 sets different control parameters for each block, and performs image processing such as color signal processing based on these control parameters.

[0049] The image processing unit 30 extracts frames at a predetermined timing from among a plurality of frames obtained in a time series from the imaging unit 20. Alternatively, the image processing unit 30 discards frames at a predetermined timing from among a plurality of frames obtained in a time series from the imaging unit 20. This reduces the amount of data, thereby reducing the load of subsequent processing. The image processing unit 30 also calculates one or more frames to be interpolated between each frame based on the plurality of frames obtained in a time series from the imaging unit 20. Then, the image processing unit 30 adds the calculated one or more frames between each frame. This allows a video with smoother movement to be played back when playing the video. In addition, the driving unit 21 is configured to control the thinning rate, but this is not limited to the configuration. For example, the driving unit 21 may read pixel signals from all pixels, but the image processing unit 30 or the arithmetic circuit 416 may control the thinning rate by discarding a predetermined pixel signal from the read pixel signals.

[0050] The work memory 40 temporarily stores image data and the like when image processing is performed by the image processing unit 30. The display unit 50 is configured, for example, with a liquid crystal display panel. As shown in FIG. 5, the display unit 50 has a display panel 51 and a touch panel 52. The display panel 51 displays images (still images, videos, live view images) captured by the imaging unit 20 and various information. The touch panel 52 is formed on the display screen of the display panel 51. The touch panel 52 outputs a signal indicating a position touched by a user to the system control unit 70 when the user selects an area in an image. In addition, the touch panel 52 outputs a signal indicating a position touched by a user to the system control unit 70 when the user sets a shooting mode, shooting conditions, and screen switching.

[0051] The operation unit 55 includes a release switch, a video switch, various operation switches, etc., which are operated by the user. This operation unit 55 outputs a signal according to the operation by the user to the system control unit 70. Note that when the release switch is half-pressed, preparations for shooting such as AF (Automatic Focusing) and AE (Automatic Exposure) are performed.

[0052] The recording unit 60 has two card slots into which two storage media such as memory cards (a first recording medium 61 and a second recording medium 62) can be attached. The recording unit 60 stores image data and various data generated in the image processing unit 30 in the recording media (the first recording medium 61 and the second recording medium 62) attached to the card slots. In this embodiment, as described above, the first image processing unit 30A and the second image processing unit 30B each execute image processing based on a signal from the first area and image processing based on a signal from the second area in parallel. In this case, the first recording medium 61 stores image data based on a signal from the first area in response to the operation of a release switch or a video switch. Also, the second recording medium 62 stores image data based on a signal from the second area in response to the operation of a release switch or a video switch. Also, the recording unit 60 has an internal memory. The recording unit 60 can also record image data and various data generated in the image processing unit 30 in the internal memory.

[0053] The system control unit 70 controls the overall processing and operation of the electronic device 1. The system control unit 70 has a CPU (Central Processing Unit) 70A. In this embodiment, the system control unit 70 divides the imaging surface (pixel area 113A) of the imaging element 100 (imaging chip 113) into a plurality of blocks, and causes the blocks to acquire images with different charge accumulation times (or charge accumulation times), frame rates, and gains. For this reason, the system control unit 70 instructs the driving unit 21 on the positions, shapes, and ranges of the blocks, and the accumulation conditions for each block. The system control unit 70 also causes the blocks to acquire images with different thinning rates, numbers of added rows or columns for adding pixel signals, and numbers of bits for digitization, for each block. For this reason, the system control unit 70 instructs the driving unit 21 on the imaging conditions for each block (thinning rate, numbers of added rows or columns for adding pixel signals, and numbers of bits for digitization). The image processing unit 30 also executes image processing with different imaging conditions (control parameters such as color signal processing, white balance adjustment, gradation adjustment, and compression rate) for each block. For this reason, the system control unit 70 instructs the image processing unit 30 on the imaging conditions for each block (control parameters such as color signal processing, white balance adjustment, gradation adjustment, and compression rate).

[0054] The system control unit 70 also records the image data generated in the image processing unit 30 in the recording unit 60. The system control unit 70 also causes the display unit 50 to display an image by outputting the image data generated in the image processing unit 30 to the display unit 50. The system control unit 70 also causes the display unit 50 to display an image by reading out the image data recorded in the recording unit 60 and outputting it to the display unit 50. Images displayed on the display unit 50 include still images, videos, and live view images. Here, the live view image is an image displayed on the display unit 50 by sequentially outputting the image data generated in the image processing unit 30 to the display unit 50. The live view image is used for the user to check the image of the subject being captured by the imaging unit 20. The live view image is also called a through image or a preview image.

[0055] Fig. 6 is a diagram showing an example of a display screen in the display unit. As shown in Fig. 6, the display panel 51 of the display unit 50 is provided with an image display area 510 and an operation button display area 520. The image display area 510 is an area for displaying images captured by the imaging unit 20, i.e., still images, videos, and live view images. The operation button display area 520 is an area for displaying a menu for the user to set imaging conditions and the like.

[0056] The operation button display area 520 is provided near the image display area 510. In the operation button display area 520, a menu (menu image) for the user to set the shooting mode, the image capturing conditions, and the display screen is displayed. The menu for the user to set the shooting mode is the shooting mode button 521. The menu for the user to set the image capturing conditions is the ISO sensitivity button 522 and the frame rate button 523. The menu for the user to set the screen switching is the screen switching button 524. Hereinafter, the shooting mode button 521, the ISO sensitivity button 522, the frame rate button 523, and the screen switching button 524 are simply referred to as the "shooting mode 521", the "ISO sensitivity 522", the "frame rate 523", and the "screen switching 524", respectively.

[0057] Shooting mode 521 is a button that the user presses when setting (selecting) a shooting mode. ISO sensitivity 522 is a button that the user presses when setting the ISO sensitivity (i.e., gain). Frame rate 523 is a button that the user presses when setting the frame rate of a moving image. Screen switching 524 is a button that the user presses when enlarging an image of a partial area to fill the entire display screen by electronic zoom (digital zoom) (see Figs. 13 and 14) or when displaying an image on a sub-screen (see Fig. 14).

[0058] Here, the shooting modes include a still image mode for shooting a still image and a moving image mode for shooting a moving image. The moving image modes include a first moving image mode and a second moving image mode. The still image mode refers to a shooting mode in which the dividing unit 71 does not divide the pixel area (imaging area) 113A of the imaging element 100, and the imaging element 100 takes a still image of a subject with the pixel area 113A as one area. This still image mode is a normal still image shooting mode that is generally used.

[0059] The first moving image mode refers to a shooting mode in which the dividing unit 71 does not divide the pixel region (imaging region) 113A of the image sensor 100, and the image sensor 100 shoots a moving image of the subject with the pixel region 113A as one region. This first moving image mode is a normal moving image shooting mode that is generally performed. The second moving image mode refers to a shooting mode in which the dividing unit 71 divides the pixel region 113A into a plurality of regions including at least a first region and a second region, and the image sensor 100 shoots a moving image of the same subject in each of the plurality of regions.

[0060] Touch panel 52 is provided on image display area 510 and operation button display area 520. In touch panel 52, touch area 510a is formed on image display area 510. When touch area 510a detects that it has been pressed (touched) by the user, it outputs a detection signal indicating the pressed position to system control unit 70.

[0061] Moreover, touch area 521a is formed so as to overlap with shooting mode 521. Moreover, touch area 522a is formed so as to overlap with ISO sensitivity 522. Moreover, touch area 523a is formed so as to overlap with frame rate 523. Moreover, touch area 524a is formed so as to overlap with screen switching 524. When each of touch areas 521a to 524a detects that it has been pressed (touched) by the user, it outputs a detection signal indicating the pressed position (which touch area it is) to the system control unit 70.

[0062] When the user selects a shooting mode, image capturing conditions, and switching of a display screen, the user may operate the operation unit 55 instead of operating the touch panel 52.

[0063] 7 is a functional block diagram of the image processing unit and the system control unit shown in FIG. 5. As shown in FIG. 7, the first image processing unit 30A includes an image generating unit (moving image generating unit) 31A and a detection unit 32A. The image generating unit 31A performs various image processing on RAW data consisting of pixel signals of each pixel of the first region output from the imaging unit 20 to generate image data. The detection unit 32A detects a main subject from the image data generated by the image generating unit 31A. In this embodiment, the detection unit 32A compares a plurality of image data obtained in time series from the live view images generated by the image generating unit 31A to detect a moving subject (moving subject) as the main subject. The detection unit 32A also detects the main subject using a face detection function as described in, for example, JP2010-16621A (US2010 / 0002940A). In addition to face detection, the detection unit 32A detects a human body included in image data as a main subject, as described in, for example, Japanese Patent Application Laid-Open No. 2010-16621 (US2010 / 0002940).

[0064] The second image processing unit 30B also includes an image generating unit (moving image generating unit) 31B. The image generating unit 31B generates image data by performing various image processes on RAW data consisting of pixel signals of each pixel in the second region output from the imaging unit 20. The second image processing unit 30B does not include a detection unit, but may include a detection unit. Also, the first image processing unit 30A may not include a detection unit 32A, and the second image processing unit 30B may include a detection unit. In this embodiment, the image generating unit 31A and the image generating unit 31B may be collectively referred to as the image generating unit 31.

[0065] The system control unit 70 includes a division unit 71, an imaging control unit 72, a recording control unit 73, and a display control unit 74. The division unit 71 divides the pixel region (imaging region) 113A of the image sensor 100 into a plurality of regions in units of blocks. The division unit 71 divides the pixel region 113A into a plurality of regions based on a predetermined arrangement pattern in each block of the pixel region 113A (see Figs. 8A to 8D). The imaging control unit 72 sets different imaging conditions in the plurality of regions divided by the division unit 71 in response to a touch operation by the user on a menu (ISO sensitivity 522, frame rate 523) for setting imaging conditions. The imaging control unit 72 also executes drive control of the image sensor 100 in response to an operation of a release switch or a video switch by the user. The imaging control unit 72 also executes drive control of the image sensor 100 when capturing a live view image (i.e., after the start of a shooting operation after power-on).

[0066] The recording control unit 73 controls the recording of the video data in the multiple regions generated by the image generation unit 31 in the recording unit 60. For example, the recording control unit 73 records the image data of the first region generated by the image generation unit 31A in the first recording medium 61, and records the image data of the second region generated by the image generation unit 31B in the second recording medium 62. The display control unit 74 controls the display unit 50 to output the image data generated by the image generation unit 31, and displays an image (still image, video, live view image) in the image display area 510 of the display unit 50. In addition, the display control unit 74 switches the display screen in response to the operation of the screen switching 524 by the user. For example, the display control unit 74 enlarges an image of a part of the region to the entire display screen by electronic zoom in response to the operation of the screen switching 524 by the user, and displays the image on the sub screen.

[0067] In the system control unit 70, the division unit 71, the imaging control unit 72, the recording control unit 73, and the display control unit 74 are realized by the CPU 70A executing processes based on a control program.

[0068] Next, the arrangement pattern in each block set by the division unit 71 will be described. Fig. 8 is a diagram showing the arrangement pattern in each block. Here, Fig. 8(A) shows a first arrangement pattern in each block. Fig. 8(B) shows a second arrangement pattern in each block. Fig. 8(C) shows a third arrangement pattern in each block. Fig. 8(D) shows a fourth arrangement pattern in each block.

[0069] 8A is an arrangement pattern in which pixel region 113A is divided into a first region and a second region. In this first arrangement pattern, in pixel region 113A, the first region is made up of blocks of odd-numbered columns (2m-1), and the second region is made up of blocks of even-numbered columns (2m). That is, each block in pixel region 113A is divided into odd-numbered columns and even-numbered columns. m is a positive integer (m=1, 2, 3,...).

[0070] The second arrangement pattern shown in Fig. 8(B) is also an arrangement pattern in which the pixel region 113A is divided into a first region and a second region. In this second arrangement pattern, in the pixel region 113A, the first region is composed of blocks of odd-numbered rows (2n-1), and the second region is composed of blocks of even-numbered rows (2n). That is, each block in the pixel region 113A is divided into odd-numbered rows and even-numbered rows. n is a positive integer (n=1, 2, 3,...).

[0071] The third arrangement pattern shown in FIG. 8(C) is also an arrangement pattern in which the pixel region 113A is divided into a first region and a second region. In this third arrangement pattern, in the pixel region 113A, the first region is composed of blocks of odd-numbered rows (2n-1) in odd-numbered columns (2m-1) and blocks of even-numbered rows (2n) in even-numbered columns (2m). The second region is composed of blocks of odd-numbered rows (2n-1) in even-numbered columns (2m) and blocks of even-numbered rows (2n) in odd-numbered columns (2m-1). That is, each block in the pixel region 113A is divided to form a checkerboard pattern. m and n are positive integers (m=1, 2, 3, . . ., n=1, 2, 3, . . .).

[0072] 8(D), pixel region 113A is divided into three regions, a first region, a second region, and a third region. In this fourth array pattern, in pixel region 113A, the first region is made up of blocks in the column two columns before the column that is a multiple of 3 (3m-2), the second region is made up of blocks in the column one column before the column that is a multiple of 3 (3m-1), and the third region is made up of blocks in the column that is a multiple of 3 (3m), where m is a positive integer (m=1, 2, 3,...).

[0073] In FIG. 8, a small number of blocks are set in pixel region 113A in order to make the arrangement of blocks in each region easier to see, but a larger number of blocks than the number of blocks shown in FIG. 8 may be set.

[0074] Next, the photographing operation according to the first embodiment will be described. FIG. 9 is a flowchart for explaining the photographing operation executed by the system control unit. FIG. 10 is a flowchart for explaining the setting process. In the process shown in FIG. 9, when the power supply to the electronic device 1 is turned on, the system control unit 70 starts the photographing operation. Although not shown in FIG. 9, when photographing starts, the display control unit 73 displays a live view image captured by the imaging unit 20 in the image display area 510 of the display panel 51, and displays various menus 521 to 524 in the operation button display area 520 of the display panel 51. At this time, since the live view image does not need to be an image of a smooth movement of the subject, the imaging control unit 72 drives and controls the imaging element 100 so as to capture images at a low frame rate.

[0075] The user touches the shooting mode 521 to select the shooting mode. The division unit 71 checks the shooting mode selected by the user's touch operation of the shooting mode 521 (step S1). The division unit 71 determines whether the shooting mode selected by the user is the still image mode (step S2). If the division unit 71 determines that the shooting mode is the still image mode, it performs a process of setting the shooting mode to the still image mode (step S3). After that, the imaging control unit 72 determines whether the user has operated the release switch (a half-press operation followed by a full press operation) (step S4).

[0076] When the imaging control unit 72 determines that the release switch has been operated, it causes the imaging unit 20 to capture an image in still image mode (step S5). Also, in step S5, the recording control unit 73 controls the imaging unit 20 to record the still image captured by the imaging unit 20 in the recording unit 60 (first recording medium 61 or the first recording medium 61). Furthermore, in step S5, the display control unit 74 controls the imaging unit 20 to display the still image captured by the imaging unit 20 in the image display area 510 of the display panel 51. Note that imaging in the still image mode is similar to normal still image shooting, and therefore a detailed description thereof will be omitted.

[0077] Furthermore, when the division unit 71 determines in step S2 that the shooting mode is not the still image mode, i.e., when the division unit 71 determines that the shooting mode is the moving image mode, it determines whether or not the moving image mode is the first moving image mode (step S10). When the division unit 71 determines that the moving image mode is the first moving image mode, it performs a process of setting the shooting mode to the first moving image mode (step S11). On the other hand, when the division unit 71 determines that the moving image mode is not the first moving image mode, i.e., when the division unit 71 determines that the moving image mode is the second moving image mode, it performs a process of setting the shooting mode to the second moving image mode (step S12).

[0078] In the process of step S11 or step S12, the division unit 71 executes the setting process shown in FIG. 10. In the process shown in FIG. 10, the division unit 71 instructs the image processing unit 30 (first image processing unit 30A) to detect a main subject (step S21). The detection unit 32A compares a plurality of image data obtained in time series from a live view image to detect a moving subject and a non-moving subject. In addition, in this embodiment, the detection unit 32A recognizes a face from the image data using the eyes, mouth, skin color, etc., and detects the face as the main subject. Furthermore, in this embodiment, in addition to face detection, the detection unit 32A detects a human body (person) included in the image data as the main subject. Then, the detection unit 32A outputs the detection result together with the image data to the system control unit 70. The division unit 71 checks whether or not there is a main subject based on the detection result of the detection unit 32A. Then, the division unit 71 sets an area in the pixel area 113A according to the main subject and the shooting mode (step S22).

[0079] Specifically, when the shooting mode is the first moving image mode, the division unit 71 does not divide the pixel region 113A into a plurality of regions. That is, the division unit 71 sets all the pixel regions 113A as one region. At this time, the division unit 71 outputs an instruction signal to the driving unit 21 to instruct the driving unit 21 to set all the pixel regions 113A as one region.

[0080] On the other hand, when the shooting mode is the second moving image mode, the division unit 71 selects one of the arrangement patterns shown in Fig. 8(A) to (D). Then, the division unit 71 checks whether the main subject is a moving subject or not based on the detection result of the detection unit 32A. When the main subject is not a moving subject but a non-moving subject, the division unit 71 sets the first region and the second region according to the third arrangement pattern shown in Fig. 8(C). When the main subject is a moving subject, the division unit 71 checks the moving direction of the moving subject. When the moving direction of the moving subject is mainly in the up-down direction, the division unit 71 sets the first region and the second region according to the first arrangement pattern shown in Fig. 8(A). When the moving direction of the moving subject is mainly in the left-right direction, the division unit 71 sets the first region and the second region according to the second arrangement pattern shown in Fig. 8(B). Furthermore, when the moving subject moves quickly in the vertical direction, the division unit 71 sets the first region, the second region, and the third region according to the fourth arrangement pattern shown in Fig. 8(D). In step S22, the division unit 71 outputs an instruction signal to the driving unit 21 to instruct the position of the blocks of each region (the first region, the second region, and the first region to the third region).

[0081] FIG. 11 is a diagram showing an example of setting the second arrangement pattern when the second moving image mode is executed. In FIG. 11, each block is shown enlarged to make the arrangement of the blocks easier to see. However, in reality, blocks smaller than the size of the blocks shown in FIG. 11 are set in the pixel region 113A. In the example shown in FIG. 11, the detection unit 32A detects people O1 and O2 playing soccer and a soccer ball O3 as main subjects (moving subjects). Based on the detection result of the detection unit 32A, the division unit 71 determines that the main subjects O1 to O3 are moving subjects and that the moving direction of the moving subjects is mainly the left-right direction. As a result, the division unit 71 sets the first region and the second region according to the second arrangement pattern shown in FIG. 8(B). At this time, the division unit divides the pixel region 113A so that the first region and the second region include the main subjects O1 to O3.

[0082] Returning to the explanation of Fig. 10, the imaging control unit 72 sets imaging conditions for each area (first area and second area in the example shown in Fig. 11) set in step S22 in response to the touch operation of ISO sensitivity 522 or frame rate 523 by the user (step S23). Specifically, the division unit 71 outputs an instruction signal to the drive unit 21 instructing the imaging conditions (gain, frame rate) in response to the touch operation of ISO sensitivity 522 or frame rate 523 by the user.

[0083] The imaging control unit 72 may automatically set a frame rate value that provides optimal exposure in response to a change in gain. The imaging control unit 72 may automatically set a gain value that provides optimal exposure in response to a change in frame rate. The imaging control unit 72 may also output an instruction signal to the image processing unit 30 instructing parameters (imaging conditions) such as color signal processing, white balance adjustment, gradation adjustment, and compression rate so as to perform optimal image correction in response to changes in gain and frame rate.

[0084] In step S23, if the user sets different frame rates for the first area and the second area, the number of frames recorded per unit time will be different between the first recording medium 61 and the second recording medium 62. Therefore, if the first recording medium 61 and the second recording medium 62 have the same recording capacity, the recording time of the first moving image on the first recording medium 61 and the recording time of the second moving image on the second recording medium 62 will also be different. In other words, if the frame rate of the first area is higher than the frame rate of the second area, the recording time of the first moving image on the first recording medium 61 will be shorter than the recording time of the second moving image on the second recording medium 62.

[0085] Thereafter, the display control unit 74 sets (switches) the display screen in response to the user's touch operation of screen switching 524 (step S24). Fig. 12 is a diagram showing a display example of a live view image in the image display area. Fig. 13 is a diagram showing the range and position of electronic zoom in the image display area. Fig. 14 is a diagram showing the range and position of a sub-screen in the image display area.

[0086] In step S24, when the user is not touching screen switching 524, a live view image (an image of multiple people playing soccer) as shown in Fig. 12 is displayed in image display area 510. Specifically, in the live view image shown in Fig. 12, multiple people O1, O2, O4 to O7, a soccer ball O3, and a goal O8 are displayed as main subjects.

[0087] In step S24, when the user touches the screen switch 524 to select electronic zoom, the display control unit 74 detects the main subjects O1 to O8 based on the detection result of the detection unit 32A. The display control unit 74 also detects the main subjects O1, O2, O4 to O7 of people from the main subjects O1 to O8. Next, the display control unit 74 extracts a partial area 511 in which many of the main subjects O1, O2, O4 to O7 of people are gathered from the image display area 510 as shown in FIG. 13. Then, the display control unit 74 performs control to enlarge and display a live view image of the extracted partial area 511 over the entire image display area 510 as shown in FIG. 14. At this time, the imaging control unit 72 controls the drive unit 21 not to perform imaging for areas of the image display area 510 other than the partial area 511 (areas where a live view image is not displayed). This makes it possible to reduce power consumption when an enlarged display of a live view image is being performed by electronic zoom.

[0088] In step S24, when the user touches the screen switching 524 to select display of the sub-screen (specific area) 512, the display control unit 74 executes control to display the live view image in the first area or the second area on the sub-screen 512 shown in FIG. 14. Specifically, when the live view image of the first area is displayed in the image display area 510, the display control unit 74 instructs the image processing unit 30 to combine the live view images of the first area and the second area so that the live view image of the second area is displayed on the sub-screen 512. In addition, when the live view image of the second area is displayed in the image display area 510, the display control unit 74 instructs the image processing unit 30 to combine the live view images of the first area and the second area so that the live view image of the first area is displayed on the sub-screen 512. Then, the display control unit 74 causes the display unit 50 to output the live view image combined by the image processing unit 30 and causes the live view image to be displayed on the image display area 510. In the example shown in FIG. 14, the sub-screen 512 is provided in the lower right corner of the image display area 510.

[0089] Returning to the explanation of Fig. 9, the imaging control unit 72 judges whether or not the user has operated the video switch (step S13). If the imaging control unit 72 judges that the video switch has been operated, it causes the imaging unit 20 to perform imaging in a video mode (first video mode or second video mode) (step S14). Note that imaging in the first video mode is the same as normal video shooting, and therefore a detailed explanation will be omitted.

[0090] Fig. 15 is a flow chart for explaining the operation during execution of the second video mode (step S14). In the process shown in Fig. 15, the division unit 71 determines whether or not the user has performed a setting operation of the block arrangement (i.e., the area) (step S31). Also, the imaging control unit 72 determines whether or not the user has performed a setting operation of the imaging conditions (step S31). In this embodiment, the user can set the block arrangement and the imaging conditions after the start of video shooting, in addition to setting the block arrangement and the imaging conditions before the start of video shooting (see steps S22 and S23 in Fig. 10).

[0091] For example, the user operates the operation unit 55 to set a block arrangement by selecting one of the arrangement patterns shown in Fig. 8(A) to (D) displayed on the display panel 51. Also, for example, the user touches the ISO sensitivity 522 or the frame rate 523 to set the imaging conditions (gain and frame rate) in the same manner as in step S23 of Fig. 10.

[0092] When the dividing unit 71 determines that the user has performed a setting operation of the block arrangement, it resets the block arrangement (step S32). That is, the imaging control unit 72 outputs an instruction signal to the driving unit 21 to instruct the position of the blocks in each area (first area, second area, first area to third area), etc. Furthermore, when the imaging control unit 72 determines that the user has performed a setting operation of the imaging conditions, it resets the imaging conditions (step S32). That is, the imaging control unit 72 outputs an instruction signal to the driving unit 21 to instruct the imaging conditions (gain and frame rate).

[0093] Next, the display control unit 74 judges whether or not the user has performed an operation to select electronic zoom (i.e., a touch operation of the screen switching 524) (step S33). When the display control unit 74 judges that an operation to select electronic zoom has been performed, it performs control to enlarge and display the partial area 511 over the entire image display area 510 as described in step S24 of FIG. 10 (step S34). In addition, the display control unit 74 judges whether or not the user has performed an operation to select display of the sub-screen 512 (i.e., a touch operation of the screen switching 524) (step S35). When the display control unit 74 judges that an operation to select display of the sub-screen 512 has been performed, it performs control to display the sub-screen 512 in the lower right corner of the image display area 510 as described in step S24 of FIG. 10 (step S36).

[0094] Next, the imaging control unit 72 drives and controls the imaging element 100 so as to capture an image of the subject in the block arrangement set in step S22 or step S32, in the range of electronic zoom (i.e., partial area 511) if enlarged display is performed by electronic zoom in step S34, and under the imaging conditions set in step S23 or step S32 (step S37). Then, the recording control unit 73 records image data of a moving image captured by the imaging element 100 and processed by the image processing unit 30 in the recording unit 60 (step S38). Specifically, the recording control unit 73 records image data of the first moving image in the first area in the first recording medium 61. Moreover, the recording control unit 73 records image data of the second moving image in the second area in the second recording medium 62.

[0095] In step S38, the recording control unit 73 outputs an instruction signal to the image processing unit 30 to interpolate image data of the moving image of the second region for image data of the first moving image in the first region. The recording control unit 73 also outputs an instruction signal to the image processing unit 30 to interpolate image data of the moving image of the first region for image data of the second moving image in the second region. When the pixel region (imaging region) 113A is divided into two regions (first region, second region), the resolution is reduced by half. However, by performing the above-mentioned processing, image data of defective pixels resulting from the division of the region is interpolated, so that the resolution is not reduced.

[0096] Furthermore, in step S38, when the display control unit 74 is enlarging and displaying a moving image captured in the partial area 511 of the first area in the image display area 510 by electronic zoom (step S34), the recording control unit 73 records the second moving image in the second area in the second recording medium 62, and records the moving image captured in the partial area 511 of the first area in the first recording medium 61. With this configuration, not only the moving image of the partial area 511 enlarged and displayed in the image display area 510, but also moving images that are not displayed in the image display area 510 can be recorded.

[0097] Furthermore, in step S38, when the display control unit 74 displays either the moving image captured in the partial area 511 of the first area or the second moving image in the image display area 510 and displays the other moving image in the sub-screen 512 (step S37), the recording control unit 73 records a moving image (see FIG. 14) obtained by combining the one moving image and the other moving image in the recording unit 60. With this configuration, moving images with different angles of view (a moving image in the electronic zoom range and a moving image covering the entire image display area 510) can be recorded as one moving image. At this time, if the frame rates of the one moving image and the other moving image are different, the image processing unit 30 thins out frames or adds frames so that the frame rates become the same. This allows the moving images with different angles of view to be recorded as one moving image in the first recording medium 61 or the second recording medium 62.

[0098] FIG. 16 is a timing chart showing the timing of charge accumulation in the second moving image mode (imaging in step S37). In the second moving image mode shown in FIG. 16, for example, a first region and a second region are set. In the second moving image mode, the imaging control unit 72 outputs an instruction signal to the driving unit 21 to perform imaging of a moving image in the first region and to perform imaging of a moving image in the second region while the moving image switch is being operated. In FIG. 16, the driving unit 21 causes each pixel in the first region to image a moving image for a charge accumulation time T1 while the moving image switch is being operated. Also, the driving unit 21 causes each pixel in the second region to image a moving image for a charge accumulation time T2 longer than the charge accumulation time T1 while the moving image switch is being operated. The frame rate changes depending on the charge accumulation time. Therefore, the frame rate of the moving image differs between the case where imaging is performed in the charge accumulation time T1 and the case where imaging is performed in the charge accumulation time T2. For example, the frame rate corresponding to the charge accumulation time T1 in the first region is 60 fps, and the frame rate corresponding to the charge accumulation time T2 in the second region is 30 fps. The charge accumulation times and the frame rates are set in the imaging condition setting process in step S23 or step S32.

[0099] Furthermore, the pixel signals of each pixel read out from the first region of the image sensor 100 are amplified by the amplifier 412 with a gain instructed by the imaging control unit 72, and then output to the image processing unit 30. The image generating unit 31A checks parameters used in image processing such as color signal processing, based on an instruction signal instructing the imaging conditions of the first region output from the imaging control unit 72. Then, the image generating unit 31A generates image data of the first region by performing various image processing based on the parameters on RAW data consisting of pixel signals of each pixel of the first region.

[0100] Furthermore, the pixel signals of each pixel read out from the second region of the image sensor 100 are amplified by the amplifier 412 with a gain instructed by the imaging control unit 72, and then output to the image processing unit 30. The image generating unit 31B checks parameters used in image processing such as color signal processing, based on an instruction signal instructing the imaging conditions of the second region output from the imaging control unit 72. Then, the image generating unit 31B generates image data for the second region by performing various image processing based on the parameters on RAW data consisting of pixel signals of each pixel in the second region.

[0101] In this way, because the frame rate of the first region is higher than the frame rate of the second region, the movement of the main subject of a person in the first moving image is smoother than the movement of the main subject of a person in the second moving image.

[0102] As described above, the first embodiment includes the imaging unit 20 having the imaging element 100, the dividing unit 71 that divides the imaging area 113A of the imaging element 100 into at least a first area and a second area, the imaging control unit 72 that sets different imaging conditions for the first area and the second area, and the moving image generating unit 31 that generates a first moving image by imaging the first area and a second moving image by imaging the second area. With this configuration, it is possible to generate multiple types of moving images corresponding to multiple areas for which different imaging conditions are set. Therefore, it is possible to generate multiple types of moving images according to the subject and the shooting situation, and the usability of the electronic device 1 equipped with the imaging element 100 is improved.

[0103] Moreover, since the moving image generating unit generates a first moving image by imaging the first region and a second moving image by imaging the second region for the same subject, it is possible to generate a plurality of types of moving images for the same subject. Furthermore, since the dividing unit 71 makes the number of pixels of the first region and the number of pixels of the second region the same, that is, since the blocks of the first region and the blocks of the second region are evenly arranged in the image display region 510, there is no variation in resolution in the image display region 510. Since the dividing unit 71 forms the first region from a plurality of discrete regions (a plurality of discrete blocks), there is no possibility of partial reduction in resolution in the moving image. Furthermore, since the dividing unit 71 variably divides the first region and the second region, it is possible to divide the image into regions according to various situations such as the type of subject.

[0104] The imaging control unit 72 at least differs in frame rate as an imaging condition between the first area and the second area of ​​the imaging element 100, so that the user can obtain a plurality of types of moving images captured under different imaging conditions. Also, the moving image generating unit 31 performs correction on the first image and the second image by making at least one of white balance, gradation, and color tone correction different, so that the user can obtain a plurality of types of moving images that have been image-processed based on different parameters.

[0105] Moreover, the moving image generating unit 31 interpolates the moving image of the second region to the first moving image, and interpolates the moving image of the first region to the second moving image. With this configuration, image data of defective pixels caused by dividing the imaging area 113A is interpolated, and the resolution does not decrease. In addition, since the recording control unit 73 is provided to record the first moving image and the second moving image generated by the moving image generating unit 31 in the recording unit 60, two types of moving images generated at the same time can be recorded.

[0106] Also, since a display control unit 74 that causes the video generated by the video generating unit 31 to be displayed on the display unit 50 is provided, the user can check the image displayed on the display unit 50. Furthermore, the display control unit 74 enlarges and displays a video obtained by capturing an image of a partial area 511 of the first area on the display unit 50, and the recording control unit 73 records the second video and the video obtained by capturing an image of the partial area 511 in the recording unit 60. With this configuration, not only the video of the partial area 511 that is enlarged and displayed on the display unit 50, but also the video of the entire area of ​​the display unit 50 that is not displayed on the display unit 50 (the entire image display area 510) can be recorded.

[0107] Furthermore, the display control unit 74 displays either the video captured in the partial area 511 of the first area or the second video on the display unit 50, and displays the other video on a specific area (sub-screen 512) of the display unit 50, and the recording control unit 73 records a video (see FIG. 14) in which one video and the other video are combined in the recording unit 60. With this configuration, videos with different angles of view can be recorded as one video.

[0108] Also, a detection unit 32A is provided that detects a main subject from a moving image generated by the moving image generating unit 31, and the division unit 71 performs division so that the first region and the second region include the main subject, so that a first moving image of the first region and a second moving image of the second region can be generated for the main subject. Also, since the imaging element 100 has a structure in which a back-illuminated imaging chip and a signal processing chip are stacked, the volume for accommodating the imaging element 100 can be reduced. Also, the driving control of the imaging element 100 can be performed based on an instruction from the system control unit 70, which reduces the burden on the system control unit 70 and makes it easy to mount the imaging element 100 in the electronic device 1.

[0109] In the electronic device 1 according to the first embodiment shown in FIG. 5, the display unit 50 may be provided outside the electronic device. In this case, the system control unit 70 and the display unit 50 are each provided with a communication unit that transmits and receives signals (image data, control signals, etc.) by wire or wirelessly. The image processing unit 30 and the system control unit 70 may be integrated. In this case, the system control unit having one or more CPUs performs processing based on a control program to assume the functions of the image processing unit 30 and the system control unit 70. In addition, although the image processing unit 30 includes two image processing units 30A and 30B, it may be configured to include only one image processing unit.

[0110] <Second embodiment> In the second embodiment, the electronic device 1 in the first embodiment described above is configured to be separated into an imaging device 1A and an electronic device 1B.

[0111] Fig. 17 is a block diagram showing the configuration of an imaging device and an electronic device according to the third embodiment. In the configuration shown in Fig. 17, the imaging device 1A is a device that captures an image of a subject. This imaging device 1A includes a lens unit 10, an imaging unit 20, an image processing unit 30, a work memory 40, an operation unit 55, a recording unit 60, and a first system control unit 75. Note that, among the imaging device 1A, the configurations of 10, the imaging unit 20, the image processing unit 30, the work memory 40, the operation unit 55, and the recording unit 60 are the same as those shown in Fig. 5. Therefore, the same components are denoted by the same reference numerals, and duplicated explanations will be omitted.

[0112] Moreover, the electronic device 1B is a device that displays images (still images, videos, and live view images). The electronic device 1B includes a display unit 50 and a second system control unit (control unit) 70B. The configuration of the display unit 50 in the electronic device 1B is the same as the configuration shown in FIG. 5. Therefore, the same components are denoted by the same reference numerals, and duplicated explanations will be omitted.

[0113] The first system control unit 75 has a first communication unit 75A. The second system control unit 76 has a second communication unit 76B. The first communication unit 75A and the second communication unit 76B transmit and receive signals to each other by wire or wirelessly. The first system control unit 75 has components corresponding to, for example, the splitter 71, the imaging control unit 72, and the recording control unit 73 of the components shown in FIG. 7. The second system control unit 76 has only a component corresponding to, for example, the display control unit 74 of the components shown in FIG. 7.

[0114] 7 (the division unit 71, the imaging control unit 72, the recording control unit 73, and the display control unit 74) may be provided in either the first system control unit 75 or the second system control unit 76. All of the components shown in Fig. 7 may be provided in the first system control unit 75 or the second system control unit 76, or a part of the components shown in Fig. 7 may be provided in the first system control unit 75, and the components other than the part of the components shown in Fig. 7 may be provided in the second system control unit 76.

[0115] The imaging device 1A may be, for example, a digital camera, smartphone, mobile phone, or personal computer equipped with imaging and communication functions, and the electronic device 1B may be, for example, a mobile terminal equipped with communication functions, such as a smartphone, mobile phone, or portable personal computer.

[0116] The first system control unit 75 shown in Fig. 17 is realized by a CPU (not shown) executing processes based on a control program. Also, the second system control unit 76 shown in Fig. 17 is realized by a CPU (not shown) executing processes based on a control program.

[0117] 17, the image processing unit 30 and the first system control unit 75 may be integrated together. In this case, a system control unit having one or more CPUs performs processing based on a control program to thereby assume the functions of the image processing unit 30 and the first system control unit 75.

[0118] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. Various modifications or improvements can be made to the above embodiments without departing from the spirit of the present invention. In addition, one or more of the requirements described in the above embodiments may be omitted. Such modifications or improvements, and omitted forms, are also included in the technical scope of the present invention. In addition, it is also possible to apply a suitable combination of the configurations of the above embodiments and modified examples.

[0119] In Fig. 8(A) to (C), the arrangement pattern of each block is set so that the area of ​​the first region is the same as the area of ​​the second region. That is, the arrangement pattern of each block is set so that the number of pixels of the first region is the same as the number of pixels of the second region. Also, in Fig. 8(D), the arrangement pattern of each block is set so that the area of ​​the first region is the same as the area of ​​the second region. That is, the arrangement pattern of each block is set so that the number of pixels of the first region is the same as the number of pixels of the second region and the number of pixels of the third region. However, the arrangement pattern of each block may be set so that the area (number of pixels) of each region is different.

[0120] In addition, in each of the above-described embodiments, the color filters 102 are arranged in a Bayer array, but other arrays may be used. The number of pixels forming the unit group 131 may be at least one pixel. A block may also be at least one pixel. Therefore, it is possible to perform imaging under different imaging conditions for each pixel.

[0121] In each of the above-described embodiments, the drive unit 21 may have a part or all of its configuration mounted on the imaging chip 113, or a part or all of its configuration mounted on the signal processing chip 111. A part of the configuration of the image processing unit 30 may be mounted on the imaging chip 113 or the signal processing chip 111. A part of the configuration of the system control unit 70 may be mounted on the imaging chip 113 or the signal processing chip 111.

[0122] In the first embodiment described above, the gain and the frame rate are configured to be changeable as the imaging conditions, but it is sufficient if at least one of them is changeable. Also, although only the case where the imaging conditions are set by the user's operation has been described, the imaging conditions may be automatically set according to the main subject, etc.

[0123] 8(A) to 8(D) are shown as examples of block arrangement patterns in the above-mentioned embodiments, but the arrangement patterns are not limited to these. Also, in the above-mentioned embodiments, the size of the block area is set in advance, but the size of the block area may be set by the user.

[0124] In the first embodiment described above, the dividing unit 71 recognizes the subject based on a live view image and sets the area. However, the dividing unit 71 may recognize the subject based on the image at the time when the release switch 55a or the movie switch 55c is half-pressed and set the area. [Explanation of symbols]

[0125] Reference Signs List 1, 1B...electronic device, 1A...imaging device, 20...imaging section, 30...image processing section, 31...image generating section (video generating section), 32A...detection section, 50...display section, 51...display panel, 52...touch panel, 70...system control section, 70A...first system control section, 70B...second system control section, 71...division section, 72...imaging control section, 73...recording control section, 74...display control section, 100...imaging element

Claims

[Claim 1] A first semiconductor substrate having a first photoelectric conversion unit that converts light into electric charges and a second photoelectric conversion unit that converts light into electric charges; a second semiconductor substrate which is stacked together with the first semiconductor substrate and has a drive unit which controls a timing at which accumulation of the electric charge converted by the first photoelectric conversion unit is ended to be different from a timing at which accumulation of the electric charge converted by the second photoelectric conversion unit is ended; An imaging element, wherein the driving unit executes a first mode for generating first image data based on a first signal output by the charge converted by the first photoelectric conversion unit and a second signal output by the charge converted by the second photoelectric conversion unit, and a second mode for generating second image data based on the first signal and third image data based on the second signal.

Citation Information

Patent Citations

  • Image pickup element and image pickup device

    JP7544116B2

  • Semiconductor module and MOS solid-state imaging device

    JP2006049361A