Imaging element and imaging apparatus

Abstract

To solve such a problem that an imaging unit has a control line for each cell, but a charge storage time and reading of a pixel signal between the cells are not finely controlled.SOLUTION: An imaging element including a plurality of laminated semiconductor substrates includes: a first pixel having a first photoelectric conversion part for converting light into a charge; a second pixel having a second photoelectric conversion part for converting the light into the charge; and an arithmetic part for performing an arithmetic operation of a first evaluation value for controlling driving of the first pixel and an arithmetic operation of a second evaluation value for controlling driving of the second pixel. The first photoelectric conversion part and the second photoelectric conversion part are arranged on a first semiconductor substrate among the plurality of semiconductor substrates. The arithmetic part is arranged on a second semiconductor substrate among the plurality of semiconductor substrates.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an imaging element and an imaging device. [Background technology]

[0002] An imaging unit is known in which a back-illuminated imaging chip and a signal processing chip are connected via microbumps for each cell unit that groups together multiple pixels. [Prior art document] [Patent documents] [Patent Document 1] JP 2006-49361 A Summary of the Invention [Problem to be solved by the invention]

[0003] In the imaging unit, each cell has a control line, but the charge accumulation time between cells and the readout of pixel signals are not precisely controlled. [Means for solving the problem]

[0004] In a first aspect of the present invention, there is provided an imaging element having a plurality of stacked semiconductor substrates, the imaging element comprising: a first pixel having a first photoelectric conversion unit that converts light into an electric charge; a second pixel having a second photoelectric conversion unit that converts light into an electric charge; and a calculation unit that calculates a first evaluation value for controlling the driving of the first pixel and a second evaluation value for controlling the driving of the second pixel, wherein the first photoelectric conversion unit and the second photoelectric conversion unit are arranged on a first semiconductor substrate of the plurality of semiconductor substrates, and the calculation unit is arranged on a second semiconductor substrate of the plurality of semiconductor substrates.

[0005] A second aspect of the present invention is an imaging device comprising the imaging element described above.

[0006] The above summary of the invention does not list all of the necessary features of the present invention, and subcombinations of these features may also constitute inventions. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a cross-sectional view of a back-illuminated MOS imaging element according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating a pixel array and unit groups of an imaging chip. [Figure 3] FIG. 2 is a circuit diagram corresponding to a unit group of the imaging chip. [Figure 4] FIG. 2 is a block diagram showing the functional configuration of an imaging element. [Figure 5] 1 is a block diagram showing a configuration of an imaging apparatus according to an embodiment of the present invention. [Figure 6] FIG. 2 is a functional block diagram of an image processing unit. [Figure 7] 10 is a flowchart showing the operation of the imaging device to generate and record a moving image. [Figure 8] 2 shows an example of an image captured by an imaging element. [Figure 9] 2 shows an example of an image captured by an imaging element. [Figure 10] The relationship between each frame rate and the output timing of the image signal is shown. [Figure 11] 3A and 3B schematically illustrate an attention area moving image and a peripheral area moving image generated by a moving image generating unit. [Figure 12] 10 shows an example of header information added by the video generating unit. [Figure 13] 10 is a flowchart showing an operation of the imaging device to play back and display a moving image. [Figure 14] 10 is a flowchart showing another example of the operation of the imaging device to generate and record a moving image. [Figure 15] An example of pixels read out for a unit group at a thinning rate of 0.5 is shown below. [Figure 16] 10 is a flowchart showing an operation of the imaging device to play back and display a moving image. [Figure 17] 1A and 1B are diagrams illustrating an example of a scene and region division. [Figure 18] 18 is a diagram illustrating charge accumulation control for each divided region in the example of FIG. 17. FIG. [Figure 19] FIG. 10 is a diagram showing the relationship between the number of integrations and the dynamic range. [Figure 20] FIG. 10 is a flowchart showing processing of a photographing operation. [Figure 21] FIG. 2 is a block diagram showing a specific configuration as an example of a signal processing chip. [Figure 22] FIG. 10 is a cross-sectional view of another back-illuminated MOS imaging element according to the present embodiment. [Figure 23] FIG. 2 is a diagram illustrating a pixel array and unit groups of an imaging chip. [Figure 24] FIG. 2 is a circuit diagram corresponding to a unit group of the imaging chip. [Figure 25] 1 is a block diagram showing a configuration of an imaging apparatus according to an embodiment of the present invention. [Figure 26] FIG. 2 is a block diagram showing a specific configuration as an example of a signal processing chip. [Figure 27] 14 is an example of a functional block of an arithmetic circuit 1415. [Figure 28] An example of the correspondence between the difference d between frames and the frame rate f is shown below. [Figure 29] 2 shows an example of an image captured by an imaging element. [Figure 30] 2 shows an example of an image captured by an imaging element. [Figure 31] 10 is an example of a functional block of another arithmetic circuit. [Figure 32] An example of a pixel 1188 read out at a thinning rate of 0.5 for one unit group is shown. [Figure 33] 10 shows an example of a functional block of yet another arithmetic circuit. [Figure 34] 10 is a diagram illustrating the relationship between gain and pixel signal. [Figure 35] 1 is a cross-sectional view of a back-illuminated MOS imaging element according to an embodiment of the present invention. [Figure 36] 2A and 2B are diagrams illustrating a pixel array and pixel blocks of an imaging chip. [Figure 37] FIG. 2 is a circuit diagram corresponding to a pixel block of the imaging chip. [Figure 38] FIG. 2 is a diagram illustrating a part of the configuration of an imaging element and an example of its operation. [Figure 39] 1 is a block diagram showing a configuration of an imaging apparatus according to an embodiment of the present invention. [Figure 40] FIG. 2 is a functional block diagram of an image processing unit. [Figure 41] 10 is a flowchart showing the operation of the imaging device to generate and record a moving image. [Figure 42] 2 shows an example of an image captured by an imaging element. [Figure 43] 2 shows an example of an image captured by an imaging element. [Figure 44] The relationship between each frame rate and the output timing of the image signal is shown. [Figure 45] 3A and 3B schematically illustrate an attention area moving image and a peripheral area moving image generated by a moving image generating unit. [Figure 46] 10 shows an example of header information added by the video generating unit. [Figure 47] 10 is a flowchart showing an operation of the imaging device to play back and display a moving image. [Figure 48] 10 is a flowchart showing another example of the operation of the imaging device to generate and record a moving image. [Figure 49] An example of pixels read out at a thinning rate of 0.5 for one pixel block is shown below. [Figure 50] 10 is a flowchart showing an operation of the imaging device to play back and display a moving image. [Figure 51A] FIG. 1 is a diagram illustrating an example of a scene. [Figure 51B] FIG. 10 is a diagram illustrating region division. [Figure 52] FIG. 51C is a diagram illustrating charge accumulation control for each divided region in the example of FIG. 51B. [Figure 53] FIG. 10 is a diagram showing the relationship between the number of integrations and the dynamic range. [Figure 54] FIG. 10 is a flowchart showing processing of a photographing operation. [Figure 55] FIG. 2 is a block diagram showing a specific configuration as an example of a signal processing chip. [Figure 56] FIG. 2 is a block diagram showing a configuration of a surrounding pixel data processing unit. [Figure 57] FIG. 2 is a block diagram showing an example of the configuration of an arithmetic circuit. [Figure 58] 10 is a flowchart illustrating an example of the operation of an arithmetic circuit. [Figure 59] 1 shows the structure of a data array generated by an output circuit. [Figure 60] The contents of the data array shown in Figure 59 are shown below. [Figure 61] 1 is a cross-sectional view of a back-illuminated MOS imaging element according to an embodiment of the present invention. [Figure 62] 2A and 2B are diagrams illustrating a pixel array and pixel blocks of an imaging chip. [Figure 63] FIG. 2 is a circuit diagram corresponding to a pixel block of the imaging chip. [Figure 64A] FIG. 2 is a diagram illustrating a part of the configuration of an imaging element and an example of its operation. [Figure 64B] FIG. 10 is a diagram illustrating another example of the operation of the imaging element. [Figure 64C] FIG. 10 is a diagram illustrating another example of the operation of the imaging element. [Figure 65] 1 is a block diagram showing a configuration of an imaging apparatus according to an embodiment of the present invention. [Figure 66] FIG. 2 is a functional block diagram of an image processing unit. [Figure 67] 10 is a flowchart showing the operation of the imaging device to generate and record a moving image. [Figure 68] 2 shows an example of an image captured by an imaging element. [Figure 69] 2 shows an example of an image captured by an imaging element. [Figure 70] The relationship between each frame rate and the output timing of the image signal is shown. [Figure 71] 3A and 3B schematically illustrate an attention area moving image and a peripheral area moving image generated by a moving image generating unit. [Figure 72] 10 shows an example of header information added by the video generating unit. [Figure 73] 10 is a flowchart showing an operation of the imaging device to play back and display a moving image. [Figure 74] 1 is a plan view of a pixel area of ​​an imaging element and an example of its operation. [Figure 75] 10A and 10B are plan views of another configuration of a pixel region of an imaging element and an example of its operation. [Figure 76] 10A and 10B are plan views of another configuration of a pixel region of an imaging element and an example of its operation. [Figure 77] 10A and 10B are plan views of another configuration of a pixel region of an imaging element and an example of its operation. [Figure 78] 10A and 10B are diagrams illustrating other configurations of pixel regions of an imaging element and examples of their operations. [Figure 79] 10 is a flowchart showing another example of the operation of the imaging device to generate and record a moving image. [Figure 80] An example of pixels read out at a thinning rate of 0.5 is shown below. [Figure 81] 10 is a flowchart showing an operation of the imaging device to play back and display a moving image. [Figure 82A] FIG. 1 is a diagram illustrating an example of a scene. [Figure 82B] FIG. 10 is a diagram illustrating region division. [Figure 83] FIG. 82C is a diagram illustrating charge accumulation control for each divided region in the example of FIG. 82B. [Figure 84] FIG. 10 is a diagram showing the relationship between the number of integrations and the dynamic range. [Figure 85] FIG. 10 is a flowchart showing processing of a photographing operation. [Figure 86] FIG. 2 is a block diagram showing a specific configuration as an example of a signal processing chip. DETAILED DESCRIPTION OF THE INVENTION

[0008] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0009] 1 is a cross-sectional view of a back-illuminated image sensor 100 according to this embodiment. The image sensor 100 includes an image sensor chip 113 that outputs pixel signals corresponding to incident light, a signal processing chip 111 that processes the pixel signals, and a memory chip 112 that stores the pixel signals. The image sensor chip 113, signal processing chip 111, and memory chip 112 are stacked and electrically connected to each other by conductive bumps 109 made of Cu or the like.

[0010] As shown in the figure, incident light is mainly incident in the positive direction of the Z axis, as indicated by the white arrow. In this embodiment, the surface of the imaging chip 113 on which the incident light is incident is referred to as the back surface. As shown by the coordinate axes, the left direction on the paper, perpendicular to the Z axis, is the positive X axis, and the front direction on the paper, perpendicular to the Z axis and the X axis, is the positive Y axis. In the following figures, the coordinate axes are displayed so that the orientation of each figure can be understood, based on the coordinate axes in Figure 1.

[0011] An example of the imaging chip 113 is a back-illuminated MOS image sensor. The PD layer 106 is arranged on the back side of the wiring layer 108. The PD layer 106 has a plurality of PDs (photodiodes) 104 arranged two-dimensionally and accumulating charges according to incident light, and transistors 105 provided corresponding to the PDs 104.

[0012] A color filter 102 is provided on the incident light side of the PD layer 106 via a passivation film 103. There are multiple types of color filters 102 that transmit different wavelength ranges, and each has a specific arrangement corresponding to each PD 104. The arrangement of the color filters 102 will be described later. A set of a color filter 102, a PD 104, and a transistor 105 forms one pixel.

[0013] A microlens 101 is provided corresponding to each pixel on the incident light side of the color filter 102. The microlens 101 condenses the incident light toward the corresponding PD 104.

[0014] The wiring layer 108 has wiring 107 that transmits pixel signals from the PD layer 106 to the signal processing chip 111. The wiring 107 may be multi-layered, and may be provided with passive elements and active elements.

[0015] A plurality of bumps 109 are arranged on the surface of the wiring layer 108. The plurality of bumps 109 are aligned with a plurality of bumps 109 provided on the opposing surface of the signal processing chip 111, and the imaging chip 113 and the signal processing chip 111 are pressed together, whereby the aligned bumps 109 are bonded together and electrically connected.

[0016] Similarly, a plurality of bumps 109 are arranged on the opposing surfaces of the signal processing chip 111 and the memory chip 112. These bumps 109 are aligned with each other, and the signal processing chip 111 and the memory chip 112 are pressed together, whereby the aligned bumps 109 are bonded together and electrically connected.

[0017] The bonding between the bumps 109 is not limited to Cu bump bonding by solid-phase diffusion, but may also employ micro-bump bonding by solder melting. For example, it is sufficient to provide one bump 109 for each unit group described below. Therefore, the size of the bumps 109 may be larger than the pitch of the PDs 104. Furthermore, in a peripheral region other than the pixel region where the pixels are arranged, bumps larger than the bumps 109 corresponding to the pixel region may also be provided.

[0018] The signal processing chip 111 has through-silicon vias (TSVs) 110 that connect circuits provided on the front and back surfaces of the chip to each other. The TSVs 110 are preferably provided in the peripheral region. The TSVs 110 may also be provided in the peripheral region of the imaging chip 113 and the memory chip 112.

[0019] FIG. 2 is a diagram illustrating the pixel array of the imaging chip 113 and the unit groups 131. In particular, the image capturing chip 113 is shown as viewed from the back side. More than 20 million pixels are arranged in a matrix in the pixel region. In this embodiment, 16 pixels, 4 pixels by 4 pixels adjacent to each other, form one unit group 131. The grid lines in the figure show the concept of forming the unit group 131 by grouping adjacent pixels. The number of pixels forming the unit group 131 is not limited to this, and may be around 1000, for example, 32 pixels by 64 pixels, or may be more or less than that.

[0020] As shown in the partially enlarged view of the pixel region, unit group 131 includes four so-called Bayer arrays, arranged vertically and horizontally, each consisting of four pixels: green pixels Gb, Gr, blue pixel B, and red pixel R. The green pixels are pixels that have a green filter as their color filter 102 and receive light in the green wavelength band of incident light. Similarly, the blue pixels are pixels that have a blue filter as their color filter 102 and receive light in the blue wavelength band, and the red pixels are pixels that have a red filter as their color filter 102 and receive light in the red wavelength band.

[0021] In this embodiment, at least one unit group is selected from the plurality of unit groups 131, and the pixels included in each unit group are controlled by control parameters different from those of the other unit groups. Examples of the control parameters include a frame rate, a thinning rate, the number of rows or columns for adding pixel signals, the charge accumulation time or number of accumulations, the number of digitization bits, etc. Furthermore, the control parameters may be parameters for image processing after image signals are acquired from the pixels.

[0022] 3 is a circuit diagram corresponding to a unit group 131 of the imaging chip 113. In the figure, a rectangle surrounded by a dotted line typically represents a circuit corresponding to one pixel. Note that at least some of the transistors described below correspond to the transistor 105 in FIG.

[0023] As described above, the unit group 131 is formed of 16 pixels. The 16 PDs 104 corresponding to the respective pixels are connected to transfer transistors 302, and each gate of each transfer transistor 302 is connected to a TX wiring 307 through which a transfer pulse is supplied. In this embodiment, the TX wiring 307 is commonly connected to the 16 transfer transistors 302.

[0024] The drain of each transfer transistor 302 is connected to the source of the corresponding reset transistor 303, and a so-called floating diffusion FD between the drain of the transfer transistor 302 and the source of the reset transistor 303 is connected to the gate of the amplification transistor 304. The drain of the reset transistor 303 is connected to a Vdd wiring 310 to which a power supply voltage is supplied, and the gate of the reset transistor 303 is connected to a reset wiring 306 to which a reset pulse is supplied. In this embodiment, the reset wiring 306 is commonly connected to the 16 reset transistors 303.

[0025] The drain of each amplification transistor 304 is connected to a Vdd wiring 310 to which a power supply voltage is supplied. The source of each amplification transistor 304 is connected to the drain of a corresponding selection transistor 305. The gate of each selection transistor is connected to a decoder wiring 308 to which a selection pulse is supplied. In this embodiment, the decoder wiring 308 is provided independently for each of the 16 selection transistors 305. The sources of each selection transistor 305 are connected to a common output wiring 309. A load current source 311 supplies a current to the output wiring 309. In other words, the output wiring 309 for the selection transistor 305 is formed by a source follower. The load current source 311 may be provided on the imaging chip 113 side or on the signal processing chip 111 side.

[0026] Here, we will explain the flow from the start of charge accumulation to pixel output after accumulation ends. When a reset pulse is applied to the reset transistor 303 via the reset wiring 306 and at the same time a transfer pulse is applied to the transfer transistor 302 via the TX wiring 307, the potentials of the PD 104 and the floating diffusion FD are reset.

[0027] When the application of the transfer pulse is stopped, the PD 104 converts the incident light it receives into electric charges and accumulates them. Then, when the transfer pulse is applied again without the reset pulse being applied, the accumulated electric charges are transferred to the floating diffusion FD, and the potential of the floating diffusion FD changes from the reset potential to the signal potential after charge accumulation. When a selection pulse is applied to the selection transistor 305 via the decoder wiring 308, the fluctuation in the signal potential of the floating diffusion FD is transmitted to the output wiring 309 via the amplification transistor 304 and the selection transistor 305. As a result, a pixel signal corresponding to the reset potential and the signal potential is output from the unit pixel to the output wiring 309.

[0028] As shown in the figure, in this embodiment, the reset wiring 306 and the TX wiring 307 are common to the 16 pixels that form the unit group 131. That is, the reset pulse and the transfer pulse are each applied simultaneously to all 16 pixels. Therefore, all pixels that form the unit group 131 start and end charge accumulation at the same timing. However, pixel signals corresponding to the accumulated charges are sequentially applied to the respective selection transistors 305 by selection pulses, and are selectively output to the output wiring 309. Furthermore, the reset wiring 306, the TX wiring 307, and the output wiring 309 are provided separately for each unit group 131.

[0029] By configuring the circuit in this way based on the unit group 131, it is possible to control the charge accumulation time for each unit group 131. In other words, it is possible to cause adjacent unit groups 131 to output pixel signals with different charge accumulation times. Furthermore, while one unit group 131 is performing a single charge accumulation, the other unit group 131 is caused to repeatedly accumulate charge and output a pixel signal each time, so that these unit groups 131 can output frames for a moving image at different frame rates.

[0030] 4 is a block diagram showing the functional configuration of the image sensor 100. An analog multiplexer 411 sequentially selects the 16 PDs 104 that form a unit group 131 and outputs the pixel signals from each of the 16 PDs 104 to the output wiring 309 provided corresponding to the unit group 131. The multiplexer 411 is formed on the image sensor chip 113 together with the PDs 104.

[0031] The pixel signals output via the multiplexer 411 undergo correlated double sampling (CDS) and analog-to-digital (A / D) conversion by a signal processing circuit 412 formed in the signal processing chip 111, which performs CDS and A / D conversion. The A / D converted pixel signals are passed to a demultiplexer 413 and stored in pixel memories 414 corresponding to the respective pixels. Each pixel memory 414 has a capacity capable of storing pixel signals corresponding to the maximum number of integrations described below. The demultiplexer 413 and pixel memories 414 are formed in the memory chip 112.

[0032] The arithmetic circuit 415 processes the pixel signals stored in the pixel memory 414 and passes them to a downstream image processing unit. The arithmetic circuit 415 may be provided in the signal processing chip 111 or in the memory chip 112. Note that although the diagram shows connections for one unit group 131, in reality, there is one for each unit group 131 and they operate in parallel. However, there does not need to be a arithmetic circuit 415 for each unit group 131; for example, one arithmetic circuit 415 may perform sequential processing while referring to values ​​in the pixel memories 414 corresponding to each unit group 131 in order.

[0033] As described above, output wiring 309 is provided corresponding to each unit group 131. Since the imaging element 100 has the imaging chip 113, the signal processing chip 111, and the memory chip 112 stacked on top of each other, by using the bumps 109 for electrical connection between the chips for these output wiring 309, it is possible to route the wiring without increasing the size of each chip in the planar direction.

[0034] 5 is a block diagram showing the configuration of an imaging device according to this embodiment. The imaging device 500 includes a photographing lens 520 as an imaging optical system, which guides a subject light beam incident along an optical axis OA to the image sensor 100. The photographing lens 520 may be an interchangeable lens that can be attached to and detached from the imaging device 500. The imaging device 500 mainly includes the image sensor 100, a system control unit 501, a drive unit 502, a photometry unit 503, a work memory 504, a recording unit 505, and a display unit 506.

[0035] The photographing lens 520 is composed of a group of multiple optical lenses and focuses a subject light beam from a scene near its focal plane. Note that in FIG. 1, the photographing lens 520 is represented by a virtual single lens placed near the pupil. The driver 502 is a control circuit that performs charge accumulation control such as timing control and area control of the image sensor 100 in accordance with instructions from the system controller 501. In this sense, the driver 502 can be said to function as an image sensor controller that causes the image sensor 100 to accumulate charges and output pixel signals.

[0036] The image sensor 100 passes the pixel signals to an image processing unit 511 in the system control unit 501. The image processing unit 511 performs various image processes using the work memory 504 as a workspace to generate image data. For example, when generating image data in JPEG file format, a color video signal is generated from a signal obtained using the Bayer array, and then compression processing is performed. The generated image data is recorded in a recording unit 505 and converted into a display signal, which is then displayed on a display unit 506 for a preset time.

[0037] The photometry unit 503 detects the luminance distribution of a scene prior to a series of shooting sequences for generating image data. The photometry unit 503 includes, for example, an AE sensor with approximately one million pixels. The calculation unit 512 of the system control unit 501 receives the output of the photometry unit 503 and calculates the luminance of each area of ​​the scene. The calculation unit 512 determines the shutter speed, aperture value, and ISO sensitivity according to the calculated luminance distribution. The image sensor 100 may also function as the photometry unit 503. The calculation unit 512 also performs various calculations for operating the image capture device 500.

[0038] The driving unit 502 may be partly or entirely mounted on the imaging chip 113, or partly or entirely mounted on the signal processing chip 111. A part of the system control unit 501 may be mounted on the imaging chip 113 or the signal processing chip 111.

[0039] 6 is a functional block diagram of the image processing unit. In addition to the functions described above, image processing unit 511 has subject estimation unit 150, group selection unit 152, video generation unit 154, and video synthesis unit 156. Each of these functions will be described later.

[0040] Fig. 7 is a flowchart showing the operation of the imaging device to generate and record a moving image. Fig. 8 and Fig. 9 show examples of images captured by the imaging element. Fig. 10 shows the relationship between each frame rate and the output timing of the image signal.

[0041] 7 starts when the user instructs the image capturing device 500 to generate a video by pressing a record button, etc. First, the subject estimation unit 150 drives the drive unit 502 to acquire image data based on an image signal from the image capturing element 100, and estimates a main subject included in an image represented by the image data (S100).

[0042] In this case, it is preferable that the driving unit 502 outputs image signals from the unit groups 131 included in the entire imaging area, for example, from all the unit groups 131. The driving unit 502 may also output image signals from all the pixels included in each unit group 131, or may output image signals from pixels thinned out at a predetermined thinning rate. The subject estimation unit 150 compares multiple images obtained in time series from the image sensor 100 and identifies a moving subject as a main subject. Note that other methods may be used to estimate the main subject.

[0043] For example, when image 170 in Fig. 8 and image 178 in Fig. 9 are acquired from image sensor 100 as images successive in time, subject estimation section 150 identifies a child as main subject 171 from the difference between these images. Note that, although the grid lines in images 170 and 178 indicate boundaries of unit groups 131, the number of unit groups 131 is merely an example and is not limited to the number shown in these figures.

[0044] The group selection unit 152 selects at least one unit group 131 into which image light of the main subject 171 estimated by the subject estimation unit 150 is incident (S102). For example, in the image 170, a unit group 131 that includes at least a portion of the main subject 171 is selected. Furthermore, taking into consideration that the main subject 171 moves within the imaging area, it is preferable that the group selection unit 152 also selects unit groups 131 that are further surrounding the unit group 131 that includes at least a portion of the main subject 171.

[0045] The group selection unit 152 sets a collection of these selected unit groups 131 as a region of interest 172. Furthermore, the group selection unit 152 sets a collection of unit groups 131 that are not included in the region of interest 172 in the entire imaging area as a peripheral area 176. The group selection unit 152 specifies area information 174 that indicates the range of the region of interest 172 relative to the entire imaging area.

[0046] 8, the region of interest 172 is a rectangular region made up of a total of 28 unit groups 131, 7 horizontally and 4 vertically. In contrast, the peripheral region 176 is made up of 98 unit groups 131, which is the total of 126 units in the imaging region, 21 horizontally and 6 vertically, excluding the region of interest 172. Furthermore, as region information 174, the position (9, 2) counted from the left and top of the unit group 131 at the top left corner of the region of interest 172 in the imaging region in the drawing is specified. Furthermore, as size information, the number of vertical and horizontal directions of the region of interest 172 is specified as 7×4.

[0047] The group selection unit 152 transmits information specifying the unit groups 131 included in the region of interest 172 and information specifying the peripheral region 176 to the drive unit 502. In this case, information on the frame rates to be applied to the region of interest 172 and the peripheral region 176 is also transmitted. Here, it is preferable that the frame rate applied to the region of interest 172 is higher than the frame rate to be applied to the peripheral region 176. For example, if the frame rate to be applied to the peripheral region 176 is 60 fps, the frame rate to be applied to the region of interest 172 is set to 180 fps. It is preferable that these frame rate values ​​are set in advance and stored so that the group selection unit 152 can refer to them, but the values ​​may also be changeable by the user later.

[0048] The driving unit 502 drives the image sensor 100 to capture images at each frame rate (S104). That is, the driving unit 502 causes the unit groups 131 included in the region of interest 172 to accumulate charges and output image signals at a high frame rate, and causes the unit groups 131 included in the peripheral region 176 to accumulate charges and output image signals at a low frame rate. In other words, the driving unit 502 obtains image signals corresponding to a plurality of frames arranged in time series for the unit groups 131 included in the region of interest 172, while obtaining an image signal corresponding to one frame for the unit groups 131 included in the peripheral region 176.

[0049] 10 , when the frame rate of the peripheral region 176 is set to 60 fps and the frame rate of the attention region 172 is set to 180 fps, the driving unit 502 obtains image signals of three frames A1, A2, and A3 from the attention region 172 (1 / 60 s = 3 × 1 / 180 s) during the 1 / 60 s it takes to obtain an image signal of one frame B1 from the peripheral region 176. In this case, the driving unit 502 separately drives the set of the reset transistor 303, the transfer transistor 302, and the selection transistor 305 of the unit group 131 included in the peripheral region 176 and the set of the reset transistor 303, the transfer transistor 302, and the selection transistor 305 of the unit group 131 included in the attention region 172, thereby obtaining image signals at different frame rates.

[0050] 10 shows the timing of outputting the image signal, but does not show the length of the exposure time. The driving unit 502 drives the above-mentioned transistor pairs for the peripheral region 176 and the region of interest 172 so as to achieve the exposure time calculated in advance by the calculation unit 512.

[0051] In addition, the length of the exposure time may be changed according to the frame rate. For example, in the example shown in Fig. 10, the exposure time of one frame of the peripheral region 176 may be set to 1 / 3, so that it is substantially the same exposure time as that of the region of interest 172. Furthermore, after outputting the image signal, the image signal may be corrected based on the frame rate ratio. Furthermore, the timing of outputting the image signal between the peripheral region 176 and the region of interest 172 may not be synchronized as shown in Fig. 10, but may be asynchronous.

[0052] The image processing unit 511 stores the image signals from the region of interest 172 in a predetermined storage area of ​​the work memory 504, one frame at a time (S106). Similarly, the image processing unit 511 stores the image signals from the peripheral region 176 in a predetermined storage area of ​​the work memory 504, one frame at a time (same step).

[0053] The video generation unit 154 reads out the image signal of the attention area 172 stored in the work memory 504 (S108), and generates data of the attention area video including multiple frames of the attention area 172 (S110). Similarly, the video generation unit 154 reads out the image signal of the surrounding area 176 stored in the work memory 504, and generates data of the surrounding area video including multiple frames of the surrounding area 176 (same step). Here, the attention area video and the surrounding area video may each be generated in a general-purpose format such as MPEG so that they can be played separately, or may be generated in a dedicated format that cannot be played without undergoing a synthesis process described below.

[0054] 11 schematically shows the attention area moving image and the surrounding area moving image generated by the moving image generating unit. The moving image generating unit 154 generates the attention area moving image at a frame rate corresponding to the frame rate at which the driving unit 502 drives the attention area 172. In the example shown in FIG. 11, the attention area moving image is generated at a frame rate of 1 / 180 fps, which is the same as the frame rate at which the driving unit 502 drives the attention area 172, 1 / 180 fps.

[0055] Similarly, the video generation unit 154 generates the surrounding area video at a frame rate corresponding to the frame rate at which the driving unit 502 drives the surrounding area 176. In the example shown in Fig. 11, the surrounding area video is generated at a frame rate of 1 / 60 fps, which is the same as the frame rate of 1 / 60 fps at which the driving unit 502 drives the surrounding area 176. Note that there are no valid values ​​in the area of ​​the surrounding area video that corresponds to the attention area 172, and this is indicated by diagonal lines in the figure.

[0056] Furthermore, the video generation unit 154 adds header information to the attention area video and the peripheral area video, and records this data in the recording unit 505 (S112). The header information includes area information indicating the position of the attention area 172 relative to the entire imaging area, size information indicating the size of the attention area 172, and timing information indicating the relationship between the output timing of the image signal of the attention area 172 and the output timing of the image signal of the peripheral area 176.

[0057] The system control unit 501 determines whether or not to capture the next unit of time (S114). Whether or not to capture the next unit of time is determined based on whether or not the user has pressed the video record button at that point. If to capture the next unit of time (S114: Yes), the process returns to step S102, and if not to capture the next unit of time (S114: No), the process ends.

[0058] Here, the "unit time" is a time of about several seconds that is set in advance in the system control unit 501. The memory capacity used for storage in step S106 is determined by this unit time, the frame rate and number of unit groups of the attention area 172, and the frame rate and number of unit groups of the peripheral area 176. Furthermore, based on this information, an area in the memory capacity for storing data of the attention area 172 and an area for storing data of the peripheral area 176 are determined.

[0059] As a result, it is possible to obtain image signals at a high frame rate from the region of interest 172 that includes the main subject 171, while reducing the amount of data by keeping the frame rate low for the peripheral region 176. Therefore, compared to high-speed readout from all pixels, it is possible to reduce the load on driving and image processing, and suppress power consumption and heat generation.

[0060] 7, when the next unit time starts, a new unit group 131 is selected in step S102, and the region information and size information are updated. This allows the region of interest 172 to be successively updated to track the main subject 171. In the example shown in FIG. 11, in the first frame A7 of the unit time in the region of interest moving image, a region of interest 182 made up of a different unit group 131 from the last frame A6 of the previous unit time is selected, and the region information 184 and surrounding region 186 are also updated accordingly.

[0061] Fig. 12 shows an example of header information added by the video generation unit. The header information in Fig. 12 includes an attention area video ID that identifies the attention area video, the frame rate of the attention area video, a surrounding area video ID that identifies the surrounding area video corresponding to the attention area video, the frame rate of the surrounding area video, timing information, area information, and size information. This header information may be added as header information to either the attention area video or the surrounding area video, or to both.

[0062] 13 is a flowchart showing the operation of the imaging device to play and display a moving image. This operation starts when the user selects one of the area-of-interest moving images displayed as thumbnails on the display unit 506 and presses the play button.

[0063] The moving image synthesizing unit 156 reads data of the attention area moving image specified by the user from the recording unit 505 (S150). The moving image synthesizing unit 156 reads data of the surrounding area moving image corresponding to the attention area moving image from the recording unit 505 (S152).

[0064] In this case, the video composition unit 156 identifies the surrounding area video by the surrounding area video ID indicated in the header information of the attention area video read in step S150. Alternatively, a surrounding area image including, as header information, the same timing information as the timing information indicated in the header information may be searched for and identified.

[0065] In the above example, it is assumed that the attention area video contains header information. On the other hand, if the attention area video does not contain header information but the surrounding area video does, the surrounding area video may be first read by having the user specify it in step S150, and the attention area video may then be specified and read from the header information in step S152.

[0066] The moving image synthesizing unit 156 synthesizes frames of the display moving image using frames of the attention area moving image and frames of the surrounding area moving image (S154). In this case, first, the first frame A1 of the attention area moving image is inserted into the position indicated by the area information 174 in the first frame B1 of the surrounding area moving image, thereby synthesizing the first frame C1 of the display moving image. As shown in FIG. 11, the moving image synthesizing unit 156 displays the first frame C1 of the display moving image on the display unit 506 (S156).

[0067] The moving image composition unit 156 determines whether or not the next frame of the attention area moving image is present before the next frame B2 of the surrounding area moving image (S158). If the next frame of the attention area moving image is present (S158: Yes), the moving image composition unit 156 updates the attention area 172 with the next frames A2 and A3 and holds the surrounding area 176 in the previous frame B1 (S162), thereby combining the next frames C2 and C3 of the displayed moving image (S162) and displaying them sequentially (S156).

[0068] On the other hand, if there is no next frame of the attention area video until the next frame B2 in the surrounding area video in step S158 (S158), the video synthesis unit 156 updates the attention area 172 with the next frame A4 and also updates the surrounding area 176 with the next frame B2 (S164), thereby synthesizing (S162) and displaying (S156) the next frame C4 of the displayed video.

[0069] As long as there is a next frame of peripheral area 176 in the peripheral area video (S160: Yes), steps S154 to S160 are repeated. If there is no next frame of peripheral area 176 in the peripheral area video (S160: No), video composition unit 156 searches for whether there is a pair of attention area video and peripheral area video in the unit time next to the unit time of the pair of attention area video and peripheral area video (S166). For example, video composition unit 156 searches for whether there is an attention area video in the same folder in recording unit 505, the header information of which includes timing information indicating a timing immediately after the timing indicated by the timing information of the attention area video.

[0070] As long as there is a pair of attention area moving image and peripheral area moving image in the next unit time (S166: Yes), steps S150 to S166 are repeated. If there is no pair of attention area moving image and peripheral area moving image in the next unit time (S166: No), the operation ends.

[0071] As described above, it is possible to display a smooth moving image of the region of interest 172 containing the main subject 171 while reducing the overall amount of data. In step S162, the region of interest 172 is updated with the next frame and the frames of the displayed image are synthesized. However, this synthesis method is not limited to this. As another example, the boundary of the main subject 171 in the region of interest 172 may be identified by image processing, and the main subject 171 enclosed by the boundary may be updated to the next frame, while the previous frame may be maintained for the area outside the boundary of the main subject 171 even within the region of interest 172, and synthesized with a frame of the peripheral region 176. In other words, the frame rate for the area outside the boundary of the region of interest 172 may be reduced to that of the peripheral region 176. This prevents the boundary between smoothness in the displayed moving image from appearing unnatural. Furthermore, the frame rate for playback does not need to be the same as the frame rate used during shooting (180 fps for the region of interest and 60 fps for the peripheral region). For example, the region of interest may be 60 fps and the peripheral region may be 20 fps. In this case, slow-motion playback is performed.

[0072] 14 is a flowchart showing another example of the operation of the imaging device to generate and record a moving image. In FIG. 14, the same operations as those in FIG. 7 are denoted by the same reference numerals, and the description thereof will be omitted.

[0073] 14, instead of or in addition to the frame rate of Fig. 7, different thinning rates are used between the region of interest 172 and the peripheral region 176. More specifically, in step S120, the driving unit 502 causes pixels thinned out at a low thinning rate for the unit group 131 included in the region of interest 172 to accumulate charge and output image signals, and causes pixels thinned out at a high thinning rate for the unit group 131 included in the peripheral region 176 to accumulate charge and output image signals. For example, a thinning rate of 0 is used for the unit group 131 included in the region of interest 172, i.e., all pixels are read out, and a thinning rate of 0.5 is used for the unit group 131 included in the peripheral region 176, i.e., half the pixels are read out.

[0074] In this case, the driving unit 502 separately drives the set of reset transistor 303, transfer transistor 302 and selection transistor 305 of the unit group 131 included in the peripheral region 176 and the set of reset transistor 303, transfer transistor 302 and selection transistor 305 of the unit group 131 included in the target region 172, thereby obtaining image signals at different thinning rates.

[0075] In step S110, based on the image signal of attention area 172 output at a low thinning rate, moving image generation unit 154 generates an attention area moving image corresponding to attention area 172. Similarly, based on the image signal of surrounding area 176 output at a high thinning rate, moving image generation unit 154 generates a surrounding area moving image corresponding to surrounding area 176. Furthermore, in step S112, moving image generation unit 154 records the attention area moving image and the surrounding area moving image in recording unit 505, adding information on the respective thinning rates.

[0076] Fig. 15 shows an example of pixels 188 that are read out at a thinning rate of 0.5 for one unit group. In the example shown in Fig. 15, when the unit groups 132 in the peripheral region 176 are in a Bayer array, pixels 188 that are read out and pixels 188 that are not read out are set every other unit of the Bayer array in the vertical direction, that is, every two rows when viewed in pixel units. This allows thinning readout to be performed without disrupting the color balance.

[0077] Fig. 16 is a flowchart showing the operation of the imaging device for playing back and displaying a moving image, corresponding to Fig. 14. In Fig. 16, the same operations as those in Fig. 13 are given the same reference numerals, and the description thereof will be omitted.

[0078] 16, the moving image synthesis unit 156 complements the pixels of the frames of the peripheral area moving image to match the resolution of the frames of the attention area moving image, and then synthesizes the frames of the displayed image by fitting the frames of the attention area moving image into the frames of the peripheral area moving image. This makes it possible to obtain a high-resolution image signal from the attention area 172 containing the main subject 171, while reducing the amount of data by keeping the peripheral area 176 at a lower resolution. Therefore, compared to high-speed readout from all pixels, it is possible to reduce the load on driving and image processing, and suppress power consumption and heat generation.

[0079] 1 to 16, the region of interest 172 is rectangular, but the shape of the region of interest 172 is not limited to this. The region of interest 172 may be a convex polygon, a concave polygon, or a doughnut shape with the surrounding region 176 embedded therein, as long as it is along the boundary line of the unit group 131. Furthermore, multiple regions of interest 172 may be set at intervals. In this case, different frame rates may be set for the regions of interest 172.

[0080] The frame rates of the region of interest 172 and the peripheral region 176 may also be variable. For example, the amount of movement of the main subject 171 may be detected each time a unit time elapses, and a higher frame rate may be set for the region of interest 172 as the amount of movement of the main subject 171 increases. Furthermore, the selection of unit groups 131 to be included in the region of interest 172 may be updated as needed within the unit time to follow the main subject 171.

[0081] Although the generation of the moving images in Figures 7 and 14 is initiated by the user pressing a record button, and the playback of the moving images in Figures 13 and 16 is initiated by the user pressing a play button, the start points are not limited to these. As another example, the operation of generating the moving images and the operation of playing the moving images may be executed consecutively by the user pressing a single button, and a through image (also referred to as a live view display) may be displayed on the display unit 506. In this case, a display that allows the user to recognize the attention area 172 may be superimposed. For example, a frame may be displayed on the display unit 506 around the boundary of the attention area 172, or the brightness of the surrounding area 176 may be reduced or the brightness of the attention area 172 may be increased.

[0082] In the operation of FIG. 14 , the thinning rate is made different between the region of interest 172 and the peripheral region 176. Instead of making the thinning rate different, the number of rows used when adding pixel signals from pixels in adjacent rows may be made different. For example, the region of interest 172 may have one row, i.e., pixel signals are output without adding adjacent rows, while the peripheral region 176 may have a larger number of rows than the region of interest 172, e.g., two rows, and output pixel signals from pixels in the same column of two adjacent rows. This allows the overall signal amount to be reduced while maintaining a higher resolution in the region of interest 172 than in the peripheral region 176, as in FIG. 14 . Furthermore, instead of adding pixel signals from adjacent rows, pixel signals from adjacent columns may be added. In this case, the number of columns used when adding pixel signals from pixels in adjacent columns is made different between the region of interest 172 and the peripheral region 176. Furthermore, the addition may include a process of calculating an average by dividing the added value by the number of rows or columns added.

[0083] Note that the moving image synthesizing unit 156 may be provided in an external display device, for example, a PC, instead of being provided in the image processing unit 511 of the imaging device 500. Also, the above embodiment is not limited to being applied to the case of generating moving images, but may also be applied to the case of generating still images.

[0084] Furthermore, in all of the above embodiments, the plurality of unit groups 131 are divided into two regions, the region of interest 172 and the peripheral region 176, but the present invention is not limited to this and the unit groups 131 may be divided into three or more regions. In this case, the unit group 131 corresponding to the boundary between the region of interest 172 and the peripheral region 176 may be set as a boundary region, and the boundary region may be controlled using an intermediate value between the control parameter value used for the region of interest 172 and the control parameter value used for the peripheral region 176. This makes it possible to prevent the boundary between the region of interest 172 and the peripheral region 176 from looking unnatural.

[0085] The charge accumulation time, accumulation count, etc. may be different between the region of interest 172 and the peripheral region 176. In this case, the region of interest 172 and the peripheral region 176 may be divided based on brightness, and an intermediate region may also be provided.

[0086] FIG. 17 illustrates an example of a scene and explains region division. FIG. 17(a) shows a scene captured by the pixel region of the imaging chip 113. Specifically, the scene simultaneously captures a shadow object 601 and a middle object 602 in an indoor environment and a highlight object 603 in an outdoor environment, observed inside a window frame 604. When capturing a scene with a large contrast between highlight and shadow areas, a conventional imaging device would experience crushed shadows in the shadow areas if charge accumulation was performed based on the highlight areas, while experiencing blown-out highlights in the highlight areas if charge accumulation was performed based on the shadow areas. In other words, the dynamic range of the photodiode is insufficient for a scene with a large contrast between highlight and shadow areas to output an image signal by uniformly accumulating charge in both the highlight and shadow areas. Therefore, in this embodiment, the scene is divided into partial regions, such as highlight and shadow regions, and the photodiodes corresponding to each region are allowed to accumulate charge at different times, thereby substantially expanding the dynamic range.

[0087] FIG. 17(b) shows the division of the pixel region of the imaging chip 113. The calculation unit 512 analyzes the scene of FIG. 17(a) captured by the photometry unit 503 and divides the pixel region based on brightness. For example, the system control unit 501 causes the photometry unit 503 to capture the scene multiple times while changing the exposure time, and the calculation unit 512 determines the dividing line of the pixel region by referring to the changes in the distribution of blown-out highlight regions and crushed shadow regions. In the example of FIG. 17(b), the calculation unit 512 divides the image into three regions: a shadow region 611, a middle region 612, and a highlight region 613.

[0088] The division lines are defined along the boundaries of the unit groups 131. That is, each divided region includes an integer number of groups. The pixels of each group included in the same region perform charge accumulation and pixel signal output the same number of times within a period corresponding to the shutter speed determined by the calculation unit 512. If the pixels belong to different regions, they perform charge accumulation and pixel signal output different times.

[0089] FIG. 18 is a diagram illustrating charge accumulation control for each divided region in the example of FIG. 17. When the calculation unit 512 receives a shooting preparation instruction from the user, it determines the shutter speed T0 from the output of the photometry unit 503. Furthermore, as described above, it divides the region into a shadow region 611, an intermediate region 612, and a highlight region 613, and determines the number of charge accumulations from the brightness information for each region. The number of charge accumulations is determined so that pixels are not saturated by charge accumulation per one time. For example, the number of charge accumulations is determined based on the criterion that 80 to 90 percent of the charge that can be accumulated in one charge accumulation operation is accumulated.

[0090] Here, the shadow region 611 is set to one charge accumulation. That is, the charge accumulation time is made to match the determined shutter speed T0. The number of charge accumulations in the intermediate region 612 is set to two. That is, one charge accumulation time is set to T0 / 2, and charge accumulation is repeated two times during the shutter speed T0. The number of charge accumulations in the highlight region 613 is set to four. That is, one charge accumulation time is set to T0 / 4, and charge accumulation is repeated four times during the shutter speed T0.

[0091] When a shooting instruction is received from the user at time t = 0, the driver 502 applies a reset pulse and a transfer pulse to the pixels of the groups belonging to any area. This application serves as a trigger for all pixels to start accumulating charges.

[0092] At time t=T0 / 4, the driving unit 502 applies a transfer pulse to the pixels in the group that belongs to the highlighted region 613. Then, it sequentially applies a selection pulse to the pixels in each group, causing each pixel signal to be output to the output wiring 309. After the pixel signals of all pixels in the group have been output, the driving unit 502 again applies a reset pulse and a transfer pulse to the pixels in the group that belongs to the highlighted region 613, causing a second charge accumulation to begin.

[0093] Note that because it takes time to selectively output pixel signals, there is a time lag between the end of the first charge accumulation and the start of the second charge accumulation. If this time lag can be practically ignored, then, as described above, the time obtained by dividing the shutter speed T0 by the number of charge accumulations can be set as the charge accumulation time for one cycle. On the other hand, if it cannot be ignored, then the shutter speed T0 can be adjusted to take this time into consideration, or the charge accumulation time for one cycle can be set shorter than the time obtained by dividing the shutter speed T0 by the number of charge accumulations.

[0094] At time t=T0 / 2, the driving unit 502 applies transfer pulses to the pixels in the groups belonging to the intermediate region 612 and the highlight region 613. Then, it sequentially applies selection pulses to the pixels in each group, causing each pixel signal to be output to the output wiring 309. After outputting pixel signals from all pixels in the group, the driving unit 502 again applies reset pulses and transfer pulses to the pixels in the groups belonging to the intermediate region 612 and the highlight region 613, causing a second charge accumulation to begin in the intermediate region 612 and a third charge accumulation to begin in the highlight region 613.

[0095] At time t=3T0 / 4, the driving unit 502 applies a transfer pulse to the pixels in the group that belongs to the highlighted region 613. Then, the driving unit 502 sequentially applies a selection pulse to the pixels in each group, causing each pixel signal to be output to the output wiring 309. After the pixel signals of all pixels in the group have been output, the driving unit 502 again applies a reset pulse and a transfer pulse to the pixels in the group that belongs to the highlighted region 613, causing the fourth charge accumulation to begin.

[0096] At time t=T0, the driver 502 applies transfer pulses to the pixels in all regions. Then, it sequentially applies selection pulses to the pixels in each group, causing each pixel signal to be output to the output wiring 309. Through the above control, one pixel signal is stored in each pixel memory 414 corresponding to the shadow region 611, two pixel signals are stored in each pixel memory 414 corresponding to the intermediate region 612, and four pixel signals are stored in each pixel memory 414 corresponding to the highlight region 613.

[0097] These pixel signals are sequentially transferred to the image processing unit 511. The image processing unit 511 generates high dynamic range image data from these pixel signals. Specific processing will be described later.

[0098] 19 is a diagram showing the relationship between the number of integrations and the dynamic range. Multiple pixel signals corresponding to repeatedly executed charge accumulations are integrated by the image processing unit 511 to form part of image data with a high dynamic range.

[0099] If the dynamic range of an area where the number of integrations is one, i.e., charge accumulation is performed once, is taken as the standard, the expansion of the dynamic range of an area where the number of integrations is two, i.e., charge accumulation is performed twice and the output signal is accumulated, is one step. Similarly, if the number of integrations is increased to four, it becomes two steps, and if it is increased to 128, it becomes seven steps. In other words, to expand the dynamic range by n steps, n It is sufficient to integrate the output signals.

[0100] Here, a 3-bit exponent digit indicating the number of accumulations is added to the image signal so that the image processing unit 511 can identify which divided region has accumulated charge and how many times. As shown in the figure, the exponent digits are assigned in order, such as 000 for 1 accumulation, 001 for 2 accumulations, ..., 111 for 128 accumulations.

[0101] The image processing unit 511 references the exponent digits of each pixel signal received from the arithmetic circuit 415, and if the reference result indicates two or more integrations, it performs integration processing on the pixel signals. For example, if the number of integrations is two (one stage), the upper 11 bits of the 12-bit pixel signals corresponding to the charge accumulation for two pixel signals are added together to generate one 12-bit pixel signal. Similarly, if the number of integrations is 128 (seven stages), the upper 5 bits of the 12-bit pixel signals corresponding to the charge accumulation for 128 pixel signals are added together to generate one 12-bit pixel signal. In other words, the upper bits obtained by subtracting the number of stages corresponding to the number of integrations from 12 are added together to generate one 12-bit pixel signal. Note that the lower bits that are not subject to addition are discarded.

[0102] By processing in this way, the brightness range to which gradation is applied can be shifted to the high brightness side in accordance with the number of integrations. In other words, 12 bits are allocated to a limited range on the high brightness side. This means that gradation can be applied to image areas that would previously have been blown out.

[0103] However, since 12 bits are assigned to different brightness ranges for the other divided regions, image data cannot be generated by simply combining each region. Therefore, the image processing unit 511 performs re-quantization processing based on the maximum and minimum brightness pixels to convert all regions into 12-bit image data while maintaining the obtained gradation as much as possible. Specifically, quantization is performed by applying gamma conversion so that gradation is maintained more smoothly. By processing in this manner, image data with a high dynamic range can be obtained.

[0104] The number of accumulations is not limited to the case where a 3-bit exponent digit is added to the pixel signal as described above, but may be described as additional information separate from the pixel signal. Furthermore, the exponent digit may be omitted from the pixel signal, and instead the number of pixel signals stored in the pixel memory 414 may be counted to obtain the number of accumulations during the addition process.

[0105] In the image processing described above, requantization processing was performed to fit the entire region into 12-bit image data, but the number of output bits can be increased to match the upper limit of the number of integrations for the number of bits of the pixel signal. For example, if the upper limit of the number of integrations is set to 16 times (4 stages), then the entire region can be made into 16-bit image data for a 12-bit pixel signal. Processing in this way makes it possible to generate image data without digit cancellation.

[0106] Next, a series of photographing operation processes will be described. Fig. 20 is a flow diagram showing the photographing operation processes. The flow starts when the power of the imaging device 500 is turned on.

[0107] In step S201, the system control unit 501 waits until the switch SW1 is pressed, which is an instruction to prepare for shooting. If the system control unit 501 detects that the switch SW1 has been pressed, the process proceeds to step S202.

[0108] In step S202, the system control unit 501 executes photometry processing. Specifically, the calculation unit 512 calculates the luminance distribution of the scene based on the output of the photometry unit 503. Then, the process proceeds to step S203, where the shutter speed, area division, number of integrations, etc. are determined as described above.

[0109] When the photographing preparation operation is completed, the process proceeds to step S204 and waits until the switch SW2 is pressed, which is an instruction to photograph. At this time, if the elapsed time exceeds a predetermined time Tw (YES in step S205), the process returns to step S201. If the pressing of the switch SW2 is detected before Tw is exceeded (NO in step S205), the process proceeds to step S206.

[0110] In step S206, the driving unit 502 receives an instruction from the system control unit 501 and executes the charge accumulation process and signal readout process described with reference to Fig. 18. Then, when all signal readout is complete, the process proceeds to step S207, where the image processing described with reference to Fig. 19 is executed, and a recording process is executed to record the generated image data in the recording unit.

[0111] When the recording process is completed, the process proceeds to step S208, where it is determined whether or not the power supply to the image capture device 500 has been turned off. If the power supply has not been turned off, the process returns to step S201, and if the power supply has been turned off, the series of photographing operation processes ends.

[0112] Fig. 21 is a block diagram showing a specific configuration as an example of the signal processing chip 111. In the explanation using Fig. 4 above, an example was shown in which the demultiplexer 413 and the pixel memory 414 were formed on the memory chip 112, but here an example in which they are formed on the signal processing chip 111 will be explained.

[0113] The signal processing chip 111 performs the functions of the drive unit 502. The signal processing chip 111 includes a sensor control unit 441, a block control unit 442, a synchronization control unit 443, and a signal control unit 444 as distributed control functions, and a drive control unit 420 that controls these control units in an integrated manner. The drive control unit 420 converts instructions from the system control unit 501 into control signals that can be executed by each control unit and passes them on to each unit.

[0114] The sensor control unit 441 is responsible for controlling the transmission of control pulses related to charge accumulation and charge readout of each pixel, which are sent to the imaging chip 113. Specifically, the sensor control unit 441 controls the start and end of charge accumulation by sending a reset pulse and a transfer pulse to the target pixel, and outputs a pixel signal to the output wiring 309 by sending a selection pulse to the readout pixel.

[0115] The block control unit 442 transmits specific pulses to the imaging chip 113 to identify the unit groups 131 to be controlled. As described with reference to FIG. 17 and other figures, a divided region may include multiple adjacent unit groups 131. These unit groups 131 belonging to the same region form a block. Pixels included in the same block start and end charge accumulation at the same timing. Therefore, the block control unit 442 transmits specific pulses to the target unit groups 131 based on instructions from the drive control unit 420, thereby dividing the unit groups 131 into blocks. The transfer pulses and reset pulses received by each pixel via the TX wiring 307 and reset wiring 306 are the logical product of the pulses transmitted by the sensor control unit 441 and the specific pulse transmitted by the block control unit 442. In this way, the charge accumulation control described with reference to FIG. 18 is realized by controlling each region as a mutually independent block. The block division instructions from the drive control unit 420 will be described in detail later.

[0116] The synchronization control unit 443 sends a synchronization signal to the imaging chip 113. Each pulse becomes active in the imaging chip 113 in synchronization with the synchronization signal. For example, by adjusting the synchronization signal, random control, thinning control, etc., which control only specific pixels of the pixels belonging to the same unit group 131, can be realized.

[0117] The signal control unit 444 is mainly responsible for timing control of the A / D converter 412b. The pixel signals output via the output wiring 309 are input to the A / D converter 412b via the CDS circuit 412a and multiplexer 411. The A / D converter 412b is controlled by the signal control unit 444 and converts the input pixel signals into digital signals. The converted pixel signals are passed to the demultiplexer 413 and stored as digital data pixel values ​​in the pixel memories 414 corresponding to each pixel.

[0118] The signal processing chip 111 has a timing memory 430 as an accumulation control memory that stores block division information about which unit groups 131 are to be combined to form a block, and accumulation count information about how many times each formed block will repeat charge accumulation. The timing memory 430 is configured, for example, by a flash RAM.

[0119] As described above, which unit groups are to be combined to form blocks is determined by the system control unit 501 based on the detection results of scene luminance distribution detection executed prior to a series of shooting sequences. The determined blocks are divided into, for example, a first block, a second block, and so on, and each block is defined by which unit group 131 it includes. The drive control unit 420 receives this block division information from the system control unit 501 and stores it in the timing memory 430.

[0120] Furthermore, the system control unit 501 determines how many times charge accumulation should be repeated for each block based on the detection result of the luminance distribution. The drive control unit 420 receives this accumulation count information from the system control unit 501 and stores it in the timing memory 430 as a pair with the corresponding block division information. By storing the block division information and accumulation count information in the timing memory 430 in this way, the drive control unit 420 can independently execute a series of charge accumulation controls by sequentially referring to the timing memory 430. In other words, once the drive control unit 420 receives an image capture instruction signal from the system control unit 501 in the control of acquiring one image, it can complete the accumulation control without subsequently receiving instructions from the system control unit 501 for the control of each pixel.

[0121] The drive control unit 420 receives from the system control unit 501 block division information and accumulation count information that are updated based on photometry results (detection results of luminance distribution) executed in synchronization with a shooting preparation instruction, and appropriately updates the contents stored in the timing memory 430. For example, the drive control unit 420 updates the timing memory 430 in synchronization with a shooting preparation instruction or a shooting instruction. This configuration realizes faster charge accumulation control, and the system control unit 501 can execute other processes in parallel while the drive control unit 420 is executing charge accumulation control.

[0122] The drive control unit 420 not only controls charge accumulation for the imaging chip 113, but also refers to the timing memory 430 when executing readout control. For example, the drive control unit 420 refers to accumulation count information for each block and stores the pixel signal output from the demultiplexer 413 at a corresponding address in the pixel memory 414.

[0123] In response to a transfer request from the system control unit 501, the drive control unit 420 reads the target pixel signal from the pixel memory 414 and transfers it to the image processing unit 511. As described above, the pixel memory 414 has a memory space capable of storing pixel signals corresponding to the maximum number of integrations for each pixel, and stores each pixel signal corresponding to the number of accumulations performed as a pixel value. For example, if charge accumulation is repeated four times in a certain block, the pixels included in that block output pixel signals for four times, and four pixel values ​​are stored in the memory space for each pixel in the pixel memory 414. When the drive control unit 420 receives a transfer request from the system control unit 501 requesting the pixel signal of a specific pixel, it specifies the address of the specific pixel in the pixel memory 414, reads all stored pixel signals, and transfers them to the image processing unit 511. For example, if four pixel values ​​are stored, all four pixel values ​​are transferred sequentially; if only one pixel value is stored, that pixel value is transferred.

[0124] The drive control unit 420 can read the pixel signals stored in the pixel memory 414 to the arithmetic circuit 415 and cause the arithmetic circuit 415 to execute the above-mentioned integration process. The pixel signals that have undergone the integration process are stored at a target pixel address in the pixel memory 414. The target pixel address may be provided adjacent to the address space before the integration process, or may be the same address so that the pixel signals before the integration process are overwritten. Alternatively, a dedicated space may be provided to collectively store the pixel values ​​of each pixel after the integration process. When the drive control unit 420 receives a delivery request from the system control unit 501 requesting the pixel signal of a specific pixel, the drive control unit 420 can deliver the pixel signal after the integration process to the image processing unit 511 depending on the nature of the delivery request. Of course, the pixel signals before and after the integration process can also be delivered together.

[0125] The pixel memory 414 is provided with a data transfer interface that transmits pixel signals in accordance with a transfer request. The data transfer interface is connected to a data transfer line that is connected to the image processing unit 511. The data transfer line is configured, for example, by a data bus among bus lines. In this case, a transfer request from the system control unit 501 to the drive control unit 420 is executed by address specification using the address bus.

[0126] The transmission of pixel signals via the data transfer interface is not limited to the addressing method, and various other methods can be used. For example, a double data rate method can be used, which uses both the rising and falling edges of the clock signal used to synchronize each circuit when transferring data. A burst transfer method can also be used, which transfers data all at once by omitting some steps such as addressing, thereby increasing speed. It is also possible to use a combination of a bus method using lines connecting the control unit, memory unit, and input / output unit in parallel, and a serial method that transfers data one bit at a time in series.

[0127] With this configuration, the image processing unit 511 can receive only the necessary pixel signals, and can complete image processing at high speed, especially when forming a low-resolution image. Also, when the arithmetic circuit 415 is made to perform the accumulation process, the image processing unit 511 does not need to perform the accumulation process, and therefore, the image processing can be speeded up by function sharing and parallel processing.

[0128] 17 to 21, the number of times charge is accumulated in the region of interest 172 and the peripheral region 176 is made different from each other, so that the number of bits when the pixel signals of the region of interest 172 are digitized is larger than that of the peripheral region 176. The number of bits when digitizing the region of interest 172 and the peripheral region 176 may be made different by other methods. For example, the A / D circuit of the signal processing circuit 412 may, in response to an instruction from the drive unit 502, digitize the region of interest 172 at a higher number of bits than the peripheral region 176 for the same number of accumulations.

[0129] Using the signal processing chip 111 of FIG. 21, image processing may be performed after pixel signals are acquired using different control parameters for the region of interest 172 and the peripheral region 176. For example, in FIGS. 7 to 10, a video is generated from images acquired at different frame rates for the region of interest 172 and the peripheral region 176. Alternatively, image processing may be performed to average images acquired at a higher frame rate to improve the S / N ratio. In this case, for example, while the drive control unit 420 acquires one pixel signal from the peripheral region 176, it acquires multiple image signals (e.g., four) from the region of interest 142 and stores them in the pixel memory 414. The arithmetic circuit 415 reads out the multiple pixel signals acquired for each pixel in the region of interest 142 from the pixel memory 414 and averages them for each pixel. This reduces random noise in each pixel of the region of interest 172, improving the S / N ratio of the region of interest 172.

[0130] 7 to 10, a video is generated from images acquired at different frame rates in the attention area 172 and the peripheral area 176. However, different frame rates may be used based on the speed of the subject's movement. In this case, the subject estimation unit 150 estimates the vertical and horizontal speed of the subject from changes in the position of the subject between frames. The subject estimation unit 150 also estimates the forward and backward speed of the subject from changes in the size of the subject between frames. Based on this estimation, the group selection unit 152 identifies unit groups 131 that receive light from slow-moving or stationary subjects, unit groups 131 that receive light from medium-moving subjects, and unit groups 131 that receive light from fast-moving subjects.

[0131] The driving unit 502 drives the image sensor 100 to capture images at a low frame rate for the unit groups 131 that receive light of a slow-moving or stationary subject, at a medium frame rate for the unit groups 131 that receive light of a medium-moving subject, and at a high frame rate for the unit groups 131 that receive light of a high-speed subject. Examples of the respective frame rates are 60 fps, 120 fps, and 240 fps.

[0132] 22 is a cross-sectional view of another back-illuminated image sensor 1100 according to this embodiment. The image sensor 1100 includes an image sensor chip 1113 that outputs pixel signals corresponding to incident light, a signal processing chip 1111 that processes the pixel signals, and a memory chip 1112 that stores the pixel signals. The image sensor chip 1113, signal processing chip 1111, and memory chip 1112 are stacked and electrically connected to each other by conductive bumps 1109 made of Cu or the like.

[0133] As shown in the figure, incident light is mainly incident in the positive direction of the Z axis, as indicated by the white arrow. In this embodiment, the surface of the imaging chip 1113 on which incident light is incident is referred to as the back surface. Also, as shown by the coordinate axes, the left direction on the paper, perpendicular to the Z axis, is the positive X axis, and the front direction on the paper, perpendicular to the Z axis and the X axis, is the positive Y axis. In the following figures, the coordinate axes are displayed so that the orientation of each figure can be understood, based on the coordinate axes in FIG. 22.

[0134] An example of the imaging chip 1113 is a back-illuminated MOS image sensor. The PD layer 1106 is arranged on the back side of the wiring layer 1108. The PD layer 1106 has a plurality of PDs (photodiodes) 104 arranged two-dimensionally and accumulating charges according to incident light, and transistors 1105 provided corresponding to the PDs 1104.

[0135] A color filter 1102 is provided on the incident light side of the PD layer 1106 via a passivation film 1103. There are multiple types of color filters 1102 that transmit different wavelength ranges, and each has a specific arrangement corresponding to the PD 1104. The arrangement of the color filters 1102 will be described later. A set of the color filter 1102, the PD 1104, and the transistor 1105 forms one pixel.

[0136] A microlens 1101 is provided for each pixel on the incident light side of the color filter 1102. The microlens 1101 condenses the incident light toward the corresponding PD 1104.

[0137] The wiring layer 1108 has wiring 1107 that transmits pixel signals from the PD layer 1106 to the signal processing chip 1111. The wiring 1107 may be multi-layered and may be provided with passive elements and active elements.

[0138] A plurality of bumps 1109 are arranged on the surface of the wiring layer 1108. The plurality of bumps 1109 are aligned with a plurality of bumps 1109 provided on the opposing surface of the signal processing chip 1111, and the imaging chip 1113 and the signal processing chip 1111 are pressed together, whereby the aligned bumps 1109 are bonded together and electrically connected.

[0139] Similarly, a plurality of bumps 1109 are arranged on the opposing surfaces of the signal processing chip 1111 and the memory chip 1112. These bumps 1109 are aligned with each other, and the signal processing chip 1111 and the memory chip 1112 are pressed together, etc., so that the aligned bumps 1109 are bonded and electrically connected to each other.

[0140] The bonding between the bumps 1109 is not limited to Cu bump bonding by solid-phase diffusion, but may also employ micro-bump bonding by solder melting. For example, it is sufficient to provide one bump 1109 for each unit group described below. Therefore, the size of the bumps 1109 may be larger than the pitch of the PDs 1104. Furthermore, in a peripheral region other than the pixel region where pixels are arranged, bumps larger than the bumps 1109 corresponding to the pixel region may also be provided.

[0141] The signal processing chip 1111 has TSVs (through silicon vias) 110 that connect circuits provided on the front and back surfaces of the chip. The TSVs 1110 are preferably provided in the peripheral region. The TSVs 1110 may also be provided in the peripheral region of the imaging chip 1113 and the memory chip 1112.

[0142] FIG. 23 is a diagram illustrating the pixel arrangement of the imaging chip 1113 and the unit groups 1131. In particular, it shows the imaging chip 1113 observed from the back side. More than 20 million pixels are arranged in a matrix in the pixel region. In the example of FIG. 23, 16 pixels, 4 pixels by 4 pixels adjacent to each other, form one unit group 1131. The grid lines in the figure show the concept of forming the unit group 1131 by grouping adjacent pixels. The number of pixels forming the unit group 1131 is not limited to this, and may be around 1000, for example, 32 pixels by 64 pixels, or may be more or less than that.

[0143] As shown in the partially enlarged view of the pixel region, the unit group 1131 includes four so-called Bayer arrays, arranged vertically and horizontally, each consisting of four pixels: green pixels Gb, Gr, blue pixel B, and red pixel R. The green pixels are pixels that have a green filter as the color filter 1102 and receive light in the green wavelength band of incident light. Similarly, the blue pixels are pixels that have a blue filter as the color filter 1102 and receive light in the blue wavelength band, and the red pixels are pixels that have a red filter as the color filter 1102 and receive light in the red wavelength band.

[0144] In this embodiment, an evaluation value is calculated for each of the multiple unit groups 1131, and exposure or readout of pixels included in the unit group is controlled by control parameters based on the evaluation value. Examples of the evaluation value include an average of pixel signals within the unit group 1131, a weighted average of pixel signals inside and outside the unit group 1131, contrast within the unit group 1131, a weighted average of contrasts inside and outside the unit group 1131, luminance within the unit block 1131, and a weighted average of luminance inside and outside the unit group 1131. Examples of the control parameters include a frame rate, a thinning rate, the number of rows or columns for adding pixel signals, the accumulation time or number of accumulations of charge, and the number of bits for digitization. Furthermore, the control parameters may be parameters for image processing after image signals are acquired from the pixels.

[0145] 24 is a circuit diagram corresponding to a unit group 1131 of the imaging chip 1113. In the figure, a rectangle surrounded by a dotted line typically represents a circuit corresponding to one pixel. Note that at least a part of the transistors described below corresponds to the transistor 1105 in FIG. 22.

[0146] As described above, the unit group 1131 is formed of 16 pixels. The 16 PDs 1104 corresponding to the respective pixels are connected to transfer transistors 1302, and each gate of each transfer transistor 1302 is connected to a TX wiring 1307 through which a transfer pulse is supplied. In this embodiment, the TX wiring 1307 is commonly connected to the 16 transfer transistors 1302.

[0147] The drain of each transfer transistor 1302 is connected to the source of the corresponding reset transistor 1303, and a so-called floating diffusion FD between the drain of the transfer transistor 1302 and the source of the reset transistor 1303 is connected to the gate of the amplification transistor 1304. The drain of the reset transistor 1303 is connected to a Vdd wiring 1310 to which a power supply voltage is supplied, and the gate of the reset transistor 1303 is connected to a reset wiring 1306 to which a reset pulse is supplied. In this embodiment, the reset wiring 1306 is commonly connected to the 16 reset transistors 1303.

[0148] The drain of each amplification transistor 1304 is connected to a Vdd wiring 1310 to which a power supply voltage is supplied. The source of each amplification transistor 1304 is connected to the drain of a corresponding selection transistor 1305. The gate of each selection transistor is connected to a decoder wiring 1308 to which a selection pulse is supplied. In this embodiment, the decoder wiring 1308 is provided independently for each of the 16 selection transistors 1305. The sources of each selection transistor 1305 are connected to a common output wiring 1309. A load current source 1311 supplies a current to the output wiring 1309. In other words, the output wiring 1309 for the selection transistor 1305 is formed by a source follower. The load current source 1311 may be provided on the imaging chip 1113 side or on the signal processing chip 1111 side.

[0149] Here, we will explain the flow from the start of charge accumulation to pixel output after accumulation ends. When a reset pulse is applied to the reset transistor 1303 via the reset wiring 1306 and at the same time a transfer pulse is applied to the transfer transistor 1302 via the TX wiring 1307, the potentials of the PD 1104 and floating diffusion FD are reset.

[0150] When the application of the transfer pulse is stopped, the PD 1104 converts the incident light it receives into electric charges and accumulates them. Then, when the transfer pulse is applied again without the reset pulse being applied, the accumulated electric charges are transferred to the floating diffusion FD, and the potential of the floating diffusion FD changes from the reset potential to the signal potential after charge accumulation. When a selection pulse is applied to the selection transistor 1305 via the decoder wiring 1308, the fluctuation in the signal potential of the floating diffusion FD is transmitted to the output wiring 1309 via the amplification transistor 1304 and the selection transistor 1305. As a result, a pixel signal corresponding to the reset potential and the signal potential is output from the unit pixel to the output wiring 1309.

[0151] As shown in the figure, in this embodiment, the reset wiring 1306 and the TX wiring 1307 are common to the 16 pixels that form the unit group 1131. That is, the reset pulse and the transfer pulse are each applied to all 16 pixels simultaneously. Therefore, all pixels that form the unit group 1131 start and end charge accumulation at the same timing. However, pixel signals corresponding to the accumulated charges are sequentially applied to the respective selection transistors 1305 by selection pulses, and are selectively output to the output wiring 1309. Furthermore, the reset wiring 1306, the TX wiring 1307, and the output wiring 1309 are provided separately for each unit group 1131.

[0152] In this way, by configuring the circuit based on the unit group 1131, it is possible to control the charge accumulation time for each unit group 1131. In other words, it is possible to cause adjacent unit groups 1131 to output pixel signals with different charge accumulation times. Furthermore, while one unit group 1131 is performing a single charge accumulation, the other unit group 1131 is caused to repeatedly accumulate charge and output a pixel signal each time, so that these unit groups 1131 can output frames for a moving image at different frame rates.

[0153] 25 is a block diagram showing the configuration of an imaging device according to this embodiment. The imaging device 1500 includes a photographing lens 1520 as an imaging optical system, which guides a subject light beam incident along an optical axis OA to the image sensor 1100. The photographing lens 1520 may be an interchangeable lens that can be attached to and detached from the imaging device 1500. The imaging device 1500 mainly includes the image sensor 1100, a system control unit 1501, a drive unit 1502, a photometry unit 1503, a work memory 1504, a recording unit 1505, and a display unit 1506.

[0154] The photographing lens 1520 is composed of a group of multiple optical lenses and focuses a subject light beam from a scene near its focal plane. Note that in Figure 25, the photographing lens 1520 is represented by a single virtual lens placed near the pupil. The driver 1502 is a control circuit that performs charge accumulation control such as timing control and area control of the image sensor 1100 in accordance with instructions from the system controller 1501.

[0155] The image sensor 1100 passes pixel signals to an image processing unit 1511 in the system control unit 1501. The image processing unit 1511 performs various image processes using a work memory 1504 as a workspace to generate image data. For example, when generating image data in JPEG file format, a color video signal is generated from a signal obtained using the Bayer array, and then compression processing is performed. The generated image data is recorded in a recording unit 1505 and converted into a display signal, which is displayed on a display unit 1506 for a preset time.

[0156] The photometry unit 1503 detects the luminance distribution of a scene prior to a series of shooting sequences for generating image data. The photometry unit 1503 includes, for example, an AE sensor with approximately one million pixels. The calculation unit 1512 of the system control unit 1501 receives the output of the photometry unit 1503 and calculates the luminance of each area of ​​the scene. The calculation unit 1512 determines the shutter speed, aperture value, and ISO sensitivity according to the calculated luminance distribution. The image sensor 1100 may also function as the photometry unit 1503. The calculation unit 1512 also performs various calculations for operating the imaging device 1500.

[0157] The driving unit 1502 may be partly or entirely mounted on the imaging chip 1113, or partly or entirely mounted on the signal processing chip 1111. A part of the system control unit 1501 may be mounted on the imaging chip 1113 or the signal processing chip 1111.

[0158] 26 is a block diagram showing a specific configuration as an example of the signal processing chip 1111. The signal processing chip 1111 performs the function of the driving unit 1502.

[0159] The signal processing chip 1111 includes a sensor control unit 1441, a block control unit 1442, a synchronization control unit 1443, a signal control unit 1444, an individual circuit unit 1450A, etc., which serve as distributed control functions, and a drive control unit 1420 that performs overall control of these control units. The signal processing chip 1111 further includes an I / F circuit 1418 between the drive control unit 1420 and a system control unit 1501 of the main body of the imaging device 1500. Each of the sensor control unit 1441, block control unit 1442, synchronization control unit 1443, signal control unit 1444, and drive control unit 1420 is provided for the signal processing chip 1111.

[0160] Meanwhile, individual circuit units 1450A, 450B, 450C, 450D, and 450E are provided for each of the unit groups 1131A, 131B, 131C, 131D, and 131E. Since the individual circuit units 1450A, 450B, 450C, 450D, and 450E have the same configuration, only the individual circuit unit 1450A will be described below. The individual circuit unit 1450A includes a CDS circuit 1410, a multiplexer 1411, an A / D conversion circuit 1412, a demultiplexer 1413, a pixel memory 1414, and an arithmetic circuit 1415. The arithmetic circuit 1415 transmits and receives signals to and receives signals from the system control unit 1501 via an I / F circuit 1418.

[0161] The individual circuit section 1450A is preferably arranged in an area overlapping an area in which the pixels of the corresponding unit group 1131A are arranged, so that the individual circuit section 1450A can be provided for each of the plurality of unit groups 1131A without increasing the size of each chip in the planar direction.

[0162] The drive control unit 1420 refers to the timing memory 1430, converts instructions from the system control unit 1501 into control signals that can be executed by each control unit, and passes them on to each unit. In particular, when the drive control unit 1420 controls each of the unit groups 1131A, etc. using separate control parameters, it passes on the control parameters to each control unit along with information that identifies the unit group 1131A. Once the drive control unit 1420 receives an imaging instruction signal from the system control unit 1501 in control of acquiring one image, it can thereafter complete accumulation control without having to receive instructions from the system control unit 1501 each time regarding control of each pixel.

[0163] The sensor control unit 1441 is responsible for controlling the transmission of control pulses related to charge accumulation and charge readout of each pixel, which are sent to the imaging chip 1113. Specifically, the sensor control unit 1441 controls the start and end of charge accumulation by sending reset pulses and transfer pulses to the target pixel, and outputs a pixel signal to the output wiring 1309 by sending a selection pulse to the readout pixel.

[0164] The block control unit 1442 executes transmission of specific pulses that specify the unit group 1131 to be controlled, to be transmitted to the imaging chip 1113. The transfer pulses and reset pulses that each pixel receives via the TX wiring 1307 and reset wiring 1306 are the logical product of the pulses transmitted by the sensor control unit 1441 and the specific pulses transmitted by the block control unit 1442. In this way, each region can be controlled as a mutually independent block.

[0165] The synchronization control unit 1443 sends a synchronization signal to the imaging chip 1113. Each pulse synchronizes with the synchronization signal and becomes active in the imaging chip 1113. For example, by adjusting the synchronization signal, random control, thinning control, etc., can be realized, in which only specific pixels belonging to the same unit group 1131A, etc., are targeted for control.

[0166] The signal control unit 1444 is mainly responsible for timing control of the A / D conversion circuit 1412. The pixel signal output via the output wiring 1309 is input to the A / D conversion circuit 1412 via the CDS circuit 1410 and multiplexer 1411. The CDS circuit 1410 removes noise from the pixel signal.

[0167] The A / D conversion circuit 1412 converts the input pixel signals into digital signals under the control of a signal control unit 1444. The converted digital pixel signals are passed to a demultiplexer 1413 and stored as digital pixel values ​​in pixel memories 1414 corresponding to the respective pixels.

[0168] The pixel memory 1414 is provided with a data transfer interface that transmits pixel signals in accordance with a transfer request. The data transfer interface is connected to a data transfer line that is connected to the image processing unit 1511. The data transfer line is configured, for example, by a data bus among bus lines. In this case, a transfer request from the system control unit 1501 to the drive control unit 1420 is executed by address specification using the address bus.

[0169] The transmission of pixel signals via the data transfer interface is not limited to the addressing method, and various other methods can be used. For example, a double data rate method can be used, which uses both the rising and falling edges of the clock signal used to synchronize each circuit when transferring data. A burst transfer method can also be used, which transfers data all at once by omitting some steps such as addressing, thereby increasing speed. It is also possible to use a combination of a bus method using lines connecting the control unit, memory unit, and input / output unit in parallel, and a serial method that transfers data one bit at a time in series.

[0170] With this configuration, the image processing unit 1511 can receive only the necessary pixel signals, and can complete image processing at high speed, especially when forming a low-resolution image. Also, when the arithmetic circuit 1415 is made to perform accumulation processing, the image processing unit 1511 does not need to perform accumulation processing, and therefore, function sharing and parallel processing can speed up image processing.

[0171] The signal processing chip 1111 has a timing memory 1430 formed of a flash RAM or the like. The timing memory 1430 stores control parameters such as accumulation count information about how many times charge accumulation is to be repeated for each unit group 1131A, etc., in association with information specifying the unit group 1131A, etc. Any of the control parameters is calculated by the arithmetic circuit 1415 of the individual circuit section 1450A, etc., and stored in the timing memory 1430.

[0172] The drive control unit 1420 not only controls charge accumulation for the imaging chip 1113, but also refers to the timing memory 1430 when executing readout control. For example, the drive control unit 1420 refers to accumulation count information for each unit group 1131, and stores the pixel signal output from the demultiplexer 1413 at a corresponding address in the pixel memory 1414.

[0173] In response to a transfer request from the system control unit 1501, the drive control unit 1420 reads out a target pixel signal from the pixel memory 1414 and transfers it to the image processing unit 1511. The pixel memory 1414 has a memory space capable of storing pixel signals corresponding to the maximum number of integrations for each pixel, and stores each pixel signal corresponding to the number of integrations performed as a pixel value. For example, if charge accumulation is repeated four times in a certain block, the pixels included in that block output pixel signals for four times, and four pixel values ​​are stored in the memory space of each pixel in the pixel memory 1414. When the drive control unit 1420 receives a transfer request from the system control unit 1501 requesting the pixel signal of a specific pixel, it specifies the address of the specific pixel in the pixel memory 1414, reads out all stored pixel signals, and transfers them to the image processing unit 1511. For example, if four pixel values ​​are stored, all four pixel values ​​are transferred sequentially; if only one pixel value is stored, that pixel value is transferred.

[0174] The drive control unit 1420 can read the pixel signals stored in the pixel memory 1414 to the arithmetic circuit 1415 and cause the arithmetic circuit 1415 to execute the above-mentioned integration process. The pixel signals that have undergone the integration process are stored at a target pixel address in the pixel memory 1414. The target pixel address may be provided adjacent to the address space before the integration process, or may be the same address so as to overwrite the pixel signal before the integration process. Alternatively, a dedicated space may be provided to collectively store the pixel values ​​of each pixel after the integration process. When the drive control unit 1420 receives a delivery request from the system control unit 1501 requesting the pixel signal of a specific pixel, the drive control unit 1420 can deliver the pixel signal after the integration process to the image processing unit 1511 depending on the nature of the delivery request. Of course, the pixel signals before and after the integration process can also be delivered together.

[0175] As described above, output wiring 1309 is provided corresponding to each unit group 1131. Since the imaging element 1100 has the imaging chip 1113, the signal processing chip 1111, and the memory chip 1112 stacked on top of each other, by using electrical connection between the chips using bumps 1109 for these output wiring 1309, it is possible to route the wiring without increasing the size of each chip in the planar direction. Similarly, by using electrical connection between the chips using bumps 1109 for signal lines from each control unit to the unit groups, it is possible to route the wiring without increasing the size of each chip in the planar direction.

[0176] Fig. 27 shows an example of functional blocks of the arithmetic circuit 1415. The arithmetic circuit 1415 calculates an evaluation value using pixel signals stored in the pixel memory 1414 of the individual circuit section 1450A, and outputs control parameters for controlling exposure or readout of the corresponding unit group 1131A based on the evaluation value. In the example shown in Fig. 27, the arithmetic circuit 1415 calculates a frame rate to be applied to the pixel unit group 1131A based on the difference in the time series of the average pixel signals of the unit group 1131A.

[0177] 27 includes an average calculation unit 1452, an average storage unit 1454, a difference calculation unit 1456, and a frame rate calculation unit 1458. The average calculation unit 1452 calculates the average value Ag by simply averaging the G pixel signals of each pixel of the unit group 1131A stored in the pixel memory 1414. In this case, the average calculation unit 1452 calculates the average value Ag for the frame at that time at a time interval corresponding to a predetermined frame rate.

[0178] In the above example, one average value Ag is calculated for each unit group 1131A and stored in the average storage unit 1454. Since the difference between the average values ​​Ag of the previous and next frames is calculated, the average storage unit 1454 is provided with memory space for storing at least two values.

[0179] The difference calculation unit 1456 calculates the difference d between the average value Ag in the latest frame stored in the average storage unit 1454 and the average value Ag in the immediately previous frame. This difference may be output as an absolute value.

[0180] The frame rate calculation unit 1458 calculates the frame rate f by comparing the difference d calculated by the difference calculation unit 1456 with a predetermined reference value d0, etc. Here, for example, a correspondence relationship such that the larger the difference d between frames, the higher the frame rate f is is stored in the frame rate calculation unit 1458 as a table.

[0181] The frame rate calculation unit 1458 outputs the calculated frame rate f to the drive control unit 1420. Alternatively or in addition to this, the frame rate calculation unit 1458 may write the frame rate f directly into the timing memory 1430.

[0182] Fig. 28 shows an example of the correspondence between the difference d between frames and the frame rate f. In Fig. 28, the difference between frames has two reference values ​​d0 and d1, and correspondingly, three frame rates f0, f1, and f2 are provided.

[0183] If the difference d between frames is equal to or less than the lower reference value d0, the frame rate calculation unit 1458 outputs the lowest frame rate f0 as the frame rate f to be applied to the unit group 1131A. If the difference d between frames is between the reference value d0 and the higher reference value d1, the frame rate calculation unit 1458 outputs the intermediate frame rate f1. If the difference d between frames is greater than the reference value d1, the frame rate calculation unit 1458 outputs the highest frame rate f2.

[0184] Here, the time interval at which the calculation circuit 1415 performs the above series of calculations is preferably set to (1 / f0) seconds, which corresponds to the interval between frames at the lowest frame rate f0. This allows the next frame rate to be calculated at the same timing for multiple unit groups 1131A, 1131B, etc., regardless of the frame rate being driven at that time. Furthermore, even when driven at the lowest frame rate f0, a new frame rate f can be calculated based on frames different from those used in the previous calculation.

[0185] 29 and 30 show examples of images captured by an imaging element. The grid lines in image 1170 and image 1178 indicate the boundaries of unit groups 1131, but the number of unit groups 1131 is merely an example and is not limited to the number shown in these figures. Also, unit groups 1131A and the like are simply indicated by "A" and the like. The unit group including the main subject 1171 is indicated by a thick line.

[0186] For example, suppose the image sensor 1100 acquires an image 1170 in Fig. 29 and an image 1178 in Fig. 30 as images acquired before and after in time. Focusing on a unit group 1131A in the figure, the unit group 1131A does not include the main subject 1171 in the image 1170 of the previous frame, but does include the main subject 1171 in the image 1178 of the subsequent frame. Therefore, the difference d between the average values ​​Ag of the unit group 1131A in the images 1170 and 1178 calculated by the average calculation unit 1452 appears large.

[0187] As a result, the frame rate calculation unit 1458 calculates a high frame rate f for the unit groups 1131A after the image 1178 based on the correspondence relationship in Fig. 28. Therefore, the drive control unit 1420 drives each pixel of the unit groups 1131A after the image 1178 at a high frame rate f2, etc. Therefore, the drive control unit 1420 can acquire pixel signals at a high frame rate f2, etc. for a subject that moves significantly between the previous and next frames in terms of time.

[0188] The unit group 1131 driven at the high frame rate f2 can perform electrical accumulation multiple times while the unit group 1131 driven at the low frame rate f0 performs electrical accumulation once. Therefore, the number of bits when the pixel signal of the unit group 1131 driven at the high frame rate f2 or the like is digitized can be made larger than that of the unit group 1131 driven at the low frame rate f0. This makes it possible to generate an image with high gradation from the unit group 1131 driven at the high frame rate f2 or the like.

[0189] Instead of increasing the number of digitization bits, the S / N ratio may be improved by performing image processing to average images acquired at a high frame rate, such as f2. In this case, while the unit group 1131 driven at a low frame rate, f0, performs one electrical accumulation, multiple image signals, for example, four, are obtained from the unit group 1131 driven at a high frame rate, f2, and stored in the pixel memory 1414. The arithmetic circuit 1415 reads out from the pixel memory 1414 multiple pixel signals obtained for each pixel of the unit group 1131 controlled at the high frame rate, f2, and averages them for each pixel. This reduces random noise in each pixel of the unit group 1131, thereby improving the S / N ratio.

[0190] As a result, the frame rate f can be calculated quickly and with less power consumption compared to when the downstream image processing unit 1511 acquires all pixel signals of the image 1170, etc., estimates the main subject, and then calculates the frame rate f of each unit group 1131A, etc. Furthermore, even if there is a problem with the pixels themselves, wiring, processing circuit, etc. of any of the unit groups 1131, the frame rate f can be calculated quickly and with less power consumption for the other unit groups 1131.

[0191] Note that the average calculation unit 1452 in FIG. 27 averages the pixel signals of the G pixels of the corresponding unit group 1131A. Alternatively, the average calculation unit 1452 may calculate an average including pixel signals of the R and B pixels. Furthermore, the average calculation unit 1452 may calculate the average of the G pixels, the average of the R pixels, and the average of the B pixels. In this case, the frame rate calculation unit 1458 may calculate the frame rate f based on conditions such as whether any one of the difference in the average of the G pixels, the difference in the average of the R pixels, and the difference in the B pixels exceeds a threshold. Furthermore, the frame rate calculation unit 1458 may make a determination based on the result of adding the average of the G pixels, the average of the R pixels, and the average of the B pixels at a predetermined ratio. Furthermore, the average value may be calculated as the average value of a partial region arranged in a unit group.

[0192] 29 and the like, from the arithmetic circuits 1415 of other individual circuit units 1450, and may add the average value Ag of the unit groups 1131B, 131C, 131D, 131E, etc., surrounding the unit group 1131A in question to the average value Ag of the unit group 1131A in question. For example, these average values ​​may be weighted. Instead of acquiring the average value Ag of the unit groups 1131B, 131C, 131D, 131E, etc., surrounding the unit group 1131A in question from the arithmetic circuits 1415 and the like, the average calculation unit 1452 may read pixel signals from the pixel memories 1414 of other individual circuit units 1450B and calculate the average value Ag by itself.

[0193] In addition, in the example of FIG. 28, there are two reference values ​​for the difference and three levels of the frame rate, but the number of reference values ​​for the difference and the number of levels of the frame rate are not limited to these.

[0194] Fig. 31 shows an example of functional blocks of another arithmetic circuit 1416. In the example shown in Fig. 31, the arithmetic circuit 1416 calculates a thinning rate to be applied to the pixel unit group 1131A based on the contrast of the pixel signals of the pixel unit group 1131A.

[0195] 31 includes a high-frequency component calculation unit 1460, a sum calculation unit 1462, and a thinning rate calculation unit 1464. The high-frequency component calculation unit 1460 reads out the G pixel signals of each pixel in the unit group 1131A stored in the pixel memory 1414, and extracts spatial high-frequency components Gh by performing high-pass filter processing based on the two-dimensional array. Similarly, the high-frequency component calculation unit 1460 calculates high-frequency components Rh of R pixels and high-frequency components Bh of B pixels.

[0196] The sum calculation unit 1462 calculates the sum of the absolute values ​​of the high-frequency components Gh, Rh, and Bh. The thinning rate calculation unit 1464 calculates the thinning rate for thinning out and reading out the pixels included in the unit group 1131A based on the sum. In this case, it is preferable that a table indicating a correspondence relationship in which the larger the sum, the lower the thinning rate is stored in advance in the thinning rate calculation unit 1464. For example, instead of the correspondence relationship in FIG. 28, a reference value of the sum and a thinning rate are associated with each other.

[0197] For example, one reference value for the sum is set, and if the sum is higher than the reference value, no thinning is performed, i.e., all pixels are read, and if the sum is lower than the reference value, a thinning rate of 0.5 is calculated. The thinning rate calculation unit 1464 outputs the calculated thinning rate to the drive control unit 1420. Alternatively or in addition to this, the thinning rate calculation unit 1464 may write the calculated thinning rate directly to the timing memory 1430.

[0198] The drive control unit 1420 outputs image signals by thinning out the pixels included in the corresponding unit group 1131 at the thinning rate calculated by the thinning rate calculation unit 1464. In this case, the drive control unit 1420 separately drives the set of the reset transistor 1303, the transfer transistor 1302, and the selection transistor 1305 of the unit group 1131 with a thinning rate of 0.5 and the set of the reset transistor 1303, the transfer transistor 1302, and the selection transistor 1305 of the unit group 1131 with a thinning rate of 0, thereby obtaining pixel signals at different thinning rates.

[0199] This makes it possible to reduce the signal amount of the unit group 1131 for a low-contrast area while maintaining a high resolution of the unit group 1131 corresponding to a high-contrast area. Furthermore, in this case, the thinning rate can be calculated quickly and with less power consumption than when the thinning rate is calculated by the downstream image processing unit 1511. Furthermore, even if there is a problem with the pixels themselves, wiring, processing circuit, or the like of one of the unit groups 1131, the thinning rates for the other unit groups 1131 can be calculated quickly and with less power consumption.

[0200] Fig. 32 shows an example of pixels 1188 that are read out at a thinning rate of 0.5 for one unit group. In the example shown in Fig. 32, when the unit group 1132 is a Bayer array, pixels 1188 that are read out and pixels that are not read out are set every other unit of the Bayer array in the vertical direction, that is, every two rows when viewed in pixel units. This allows thinning readout to be performed without disrupting the color balance.

[0201] 32, thinning readout is performed row by row, but thinning readout may alternatively be performed column by column. Furthermore, the high-frequency component calculation unit 1460 may extract high-wavelength components in both the column direction and the row direction, and the thinning rate calculation unit 1464 may calculate the thinning rate in the column direction and the thinning rate in the row direction, respectively.

[0202] 31 and 32, the thinning rate calculation unit 1464 calculates the thinning rate for the corresponding pixel group. Alternatively, the number of pixels to be used when adding pixel signals from adjacent pixels of the same color may be calculated. For example, if the sum calculated by the sum calculation unit 1462 is equal to or greater than a reference value, the number of rows is set to 1, i.e., a pixel signal is output without adding pixels from adjacent rows of the same color. If the sum is smaller than the reference value, a larger number of rows, for example, 2, is set, and a pixel signal is output in which pixels from the same column in two adjacent rows of the same color are added.

[0203] As a result, similar to Figure 32, the overall signal amount can be reduced while maintaining high resolution in high-contrast areas. Also, instead of adding pixel signals of adjacent rows of the same color, pixel signals of adjacent columns of the same color may be added. Furthermore, the addition may include a process of calculating an average by dividing the added value by the number of rows or columns added. Also, pixel signals of adjacent rows and columns of the same color may be added.

[0204] Note that the high-frequency component calculation unit 1460 and the like use the high-wavelength components Rh, Gh, and Bh for each R pixel, G pixel, and B pixel. Alternatively, the high-frequency components may be calculated using luminance components calculated from the R pixel, G pixel, and B pixel. In this case, the high-frequency components may be calculated after adjusting the gain between the luminance components of the R pixel, G pixel, and B pixel.

[0205] Furthermore, the sum calculation unit 1462 may acquire high-frequency components of unit groups 1131B, 131C, 131D, 131E, etc., surrounding the unit group 1131A in question, as shown in FIG. 29 etc., from the arithmetic circuits 1416 of other individual circuit units 1450B etc., and add them to the high-frequency components of the unit group 1131A in question. For example, a weighted average of these may be calculated. Instead of acquiring the average values ​​Ag of unit groups 1131B, 131C, 131D, 131E, etc., surrounding the unit group 1131A in question, from other arithmetic circuits 1416 etc., the sum calculation unit 1462 may read pixel signals from the pixel memories 1414 of other individual circuit units 1450B etc., and calculate the high-frequency components itself.

[0206] Furthermore, the number of bits for digitizing pixel signals may be set to be larger for unit groups whose pixel signals exceed the thresholds in the frame rate calculation unit 1458 and the thinning rate calculation unit 1464 than for unit groups whose pixel signals are equal to or smaller than the thresholds. For example, the A / D conversion circuit 1412 may digitize pixel signals at a higher number of bits for the same number of accumulations in response to an instruction from the drive unit 1502.

[0207] 33 shows an example of functional blocks of yet another arithmetic circuit 1417. The arithmetic circuit 1417 has a self-average calculation section 1472, an adjacent average calculation section 1470, a gain calculation section 1474, and a correction section 1476.

[0208] The self-average calculation unit 1472 calculates the average value Ag by simply averaging the G pixel signals of each pixel of the unit group 1131A stored in the pixel memory 1414. Similarly, the self-average calculation unit 1472 calculates the average values ​​Ar and Ab by simply averaging the R pixel signals and B pixel signals of each pixel of the unit group 1131A stored in the pixel memory 1414. Furthermore, the self-average calculation unit 1472 outputs the average values ​​Ag, Ar, and Ab of the unit group 1131A to the adjacent average calculation unit 1470 of the surrounding unit groups 1131B, etc.

[0209] The adjacent average calculation unit 1470 obtains average values ​​Ag, Ar, and Ab from the self-average calculation units 1472 corresponding to the other unit groups 1131B, 131C, 131D, and 131E adjacent to the unit group 1131A, and calculates a weighted average of them. The gain calculation unit 1474 performs a weighted average for each of RGB the average values ​​Ag, Ar, and Ab calculated by the self-average calculation unit 1472 and the average values ​​Ag, Ar, and Ab calculated by the adjacent average calculation unit 1470, and calculates the gains of the R pixel signal and the B pixel signal relative to the G pixel signal based on the ratio between them. In this case, for example, a weighted average is used in which the weighting of the average value of the unit group 1131A is 4 / 8 and the weighting of the average value of the adjacent unit group 1131B, etc. is 1 / 8.

[0210] The gains of the R pixel signal and the B pixel signal are transmitted as additional information to the system control unit 1501 via the I / F circuit 1418. Note that the adjacent average calculation unit 1470 may read pixel signals from the pixel memories 1414 of the other individual circuit units 1450B, etc., and calculate the average values ​​Ag, etc. of the unit groups 1131B, 131C, 131D, 131E, etc. surrounding the unit group 1131A in question, instead of acquiring them from the arithmetic circuits 1417, etc. of the other individual circuit units 1450B, etc.

[0211] The correction unit 1476 corrects the R pixel signals and B pixel signals using the gain calculated by the gain calculation unit 1474, and writes the corrected results to the pixel memory 1414. In this case, the correction unit 1476 multiplies each R pixel signal by the gain for the R pixel signal, and multiplies each B pixel signal by the gain for the B pixel signal. Note that the correction unit 1476 may obtain feedback information from the system control unit 1501 and further correct the gains.

[0212] FIG. 34 schematically shows the relationship between gain and pixel signal. The operation of calculating the gain and correcting the pixel signal is preferably performed, for example, for each frame at a frame rate of f0, that is, every (1 / f0) seconds. As shown in FIG. 34, the gain of the R pixel signal and the gain of the B pixel signal are calculated every (1 / f0) seconds, and the output values ​​of the R pixel signal and the B pixel signal are corrected. As a result, the gain can be calculated quickly and with less power consumption, and the pixel signal can be corrected, compared to calculating the gain and correcting the pixel signal in the image processing unit 1511 at a subsequent stage.

[0213] In the above embodiment, one sensor control unit 1441, one block control unit 1442, one synchronization control unit 1443, one signal control unit 1444, and one drive control unit 1420 are provided for each signal processing chip 1111, and one individual circuit unit 1450A, 450B, 450C, 450D, and 450E are provided for each unit group 1131A, 131B, 131C, 131D, and 131E. Alternatively, a plurality of sensor control units 1441, block control units 1442, synchronization control units 1443, signal control units 1444, and drive control units 1420 may be provided for each signal processing chip 1111, and each may share the responsibility of controlling a plurality of unit groups 1131.

[0214] Alternatively, one individual circuit unit 1450A etc. may be provided for each of a plurality of unit groups 1131 and may be shared by the plurality of unit groups 1131. The individual circuit unit 1450A etc. may be provided for each pixel. That is, in the above embodiment, the unit group 1131 may be composed of a single pixel.

[0215] 35 is a cross-sectional view of another image sensor 2100 according to this embodiment. The image sensor 2100 includes an image sensor chip 2113 that outputs pixel signals corresponding to incident light, a signal processing chip 2111 that processes the pixel signals, and a memory chip 2112 that stores the pixel signals. The image sensor chip 2113, signal processing chip 2111, and memory chip 2112 are stacked and electrically connected to each other by conductive bumps 2109 made of Cu or the like.

[0216] As shown in the figure, incident light is mainly incident in the positive direction of the Z axis, as indicated by the white arrow. In this specification, the surface of the imaging chip 2113 on which incident light is incident is referred to as the back surface. Also, as shown by the coordinate axes, the left direction on the paper, perpendicular to the Z axis, is the positive X axis, and the front direction on the paper, perpendicular to the Z axis and the X axis, is the positive Y axis. In the following figures, the coordinate axes are displayed based on the coordinate axes in Figure 35 so that the orientation of each figure can be understood.

[0217] An example of the imaging chip 2113 is a back-illuminated MOS image sensor. The PD layer 2106 is arranged on the back side of the wiring layer 2108. The PD layer 2106 has a plurality of PDs (photodiodes) 2104 arranged two-dimensionally, which accumulate charges according to incident light and generate pixel signals according to the accumulated charges, and transistors 2105 provided corresponding to the PDs 2104.

[0218] A color filter 2102 is provided on the incident light side of the PD layer 2106 via a passivation film 2103. There are multiple types of color filters 2102 that transmit different wavelength ranges, and each has a specific arrangement corresponding to each PD 2104. The arrangement of the color filters 2102 will be described later. A set of the color filter 2102, the PD 2104, and the transistor 2105 forms one pixel.

[0219] A microlens 2101 is provided corresponding to each pixel on the incident light side of the color filter 2102. The microlens 2101 condenses the incident light toward the corresponding PD 2104.

[0220] The wiring layer 2108 has wiring 2107 that transmits pixel signals from the PD layer 2106 to the signal processing chip 2111. The wiring 2107 may be multi-layered, and may be provided with passive elements and active elements.

[0221] A plurality of bumps 2109 are arranged on the surface of the wiring layer 2108. The plurality of bumps 2109 are aligned with a plurality of bumps 2109 provided on the opposing surface of the signal processing chip 2111, and the imaging chip 2113 and the signal processing chip 2111 are pressed together, whereby the aligned bumps 2109 are bonded together and electrically connected.

[0222] Similarly, a plurality of bumps 2109 are arranged on the opposing surfaces of the signal processing chip 2111 and the memory chip 2112. These bumps 2109 are aligned with each other, and the signal processing chip 2111 and the memory chip 2112 are pressed together, etc., so that the aligned bumps 2109 are bonded and electrically connected to each other.

[0223] The bonding between the bumps 2109 is not limited to Cu bump bonding by solid-phase diffusion, and micro-bump bonding by solder melting may also be used. For example, it is sufficient to provide one bump 2109 for each pixel block described below. Therefore, the size of the bumps 2109 may be larger than the pitch of the PDs 2104. Furthermore, in a peripheral region other than the imaging region where the pixels are arranged, bumps larger than the bumps 2109 corresponding to the imaging region may also be provided.

[0224] The signal processing chip 2111 has TSVs (through silicon vias) 2110 that connect circuits provided on the front and back surfaces of the chip to each other. The TSVs 2110 are preferably provided in the peripheral region. The TSVs 2110 may also be provided in the peripheral region of the imaging chip 2113 and the memory chip 2112.

[0225] FIG. 36 is a diagram illustrating the pixel array and pixel blocks 2131 of the imaging chip 2113. FIG. 36 shows the imaging chip 2113 as viewed from the back side. Multiple pixels are arranged in a matrix in the imaging region 2700. The imaging region 2700 has multiple pixel blocks 2131 formed by dividing the multiple pixels in the row and column directions. Each pixel block 2131 has m×n pixels in the row and column directions. Here, m and n are integers of 2 or greater. The row and column directions refer to two different directions within the plane of the imaging region 2700 and do not necessarily need to be perpendicular to each other. In FIG. 36, adjacent 4×4 pixels, or 16 pixels, form one pixel block 2131. The grid lines in the figure indicate the concept of grouping adjacent pixels to form the pixel block 2131. The number of pixels forming the pixel block 2131 is not limited to 1,000, and may be, for example, 32×64 pixels, or more or less.

[0226] As shown in the partially enlarged view of the imaging region 2700, the pixel block 2131 contains four pixels arranged vertically and horizontally in a so-called Bayer array, each consisting of green pixels Gb, Gr, blue pixels B, and red pixels R. The green pixels are pixels that have a green filter as the color filter 2102 and receive light in the green wavelength band of incident light. Similarly, the blue pixels are pixels that have a blue filter as the color filter 2102 and receive light in the blue wavelength band, and the red pixels are pixels that have a red filter as the color filter 2102 and receive light in the red wavelength band.

[0227] In this embodiment, at least one pixel block is selected from the multiple pixel blocks 2131, and the pixels included in each pixel block are controlled using control parameters different from those of the other pixel blocks. Examples of the control parameters include the frame rate, the thinning rate, the number of rows for adding pixel signals, the charge accumulation time or number of accumulations, and the number of digitization bits. Furthermore, the control parameters may be parameters for image processing after image signals are acquired from the pixels. The frame rate refers to the cycle at which pixel signals are generated. Note that in this specification, the frame rate may refer to the frame rate for each pixel block 2131. For example, the reference frame rate and the high-speed frame rate refer to the frame rates for each pixel block 2131.

[0228] 37 is a circuit diagram corresponding to a pixel block 2131 of an imaging chip 2113. In the figure, a rectangle surrounded by a dotted line typically represents a circuit corresponding to one pixel. Note that at least a portion of the transistors described below correspond to the transistor 2105 in FIG.

[0229] 37 shows a pixel block 2131 formed of 16 pixels, but the number of pixels in the pixel block 2131 is not limited to this. The 16 PDs 104 corresponding to the respective pixels are connected to transfer transistors 2302, and each gate of each transfer transistor 2302 is connected to a TX wiring 2307 that supplies a transfer pulse. In the example shown in FIG. 37, the TX wiring 2307 is commonly connected to the 16 transfer transistors 2302.

[0230] The drain of each transfer transistor 2302 is connected to the source of the corresponding reset transistor 2303, and a so-called floating diffusion FD between the drain of the transfer transistor 2302 and the source of the reset transistor 2303 is connected to the gate of the amplification transistor 2304. The drain of the reset transistor 2303 is connected to a Vdd wiring 2310 to which a power supply voltage is supplied, and the gate thereof is connected to a reset wiring 2306 to which a reset pulse is supplied. In the example shown in FIG. 37 , the reset wiring 2306 is commonly connected to 16 reset transistors 2303.

[0231] The drain of each amplification transistor 2304 is connected to a Vdd wiring 2310 to which a power supply voltage is supplied. The source of each amplification transistor 2304 is connected to the drain of a corresponding selection transistor 2305. The gate of each selection transistor is connected to a decoder wiring 2308 to which a selection pulse is supplied. In the example shown in FIG. 37 , the decoder wiring 2308 is provided independently for each of the 16 selection transistors 2305. The sources of each selection transistor 2305 are connected to a common output wiring 2309. A load current source 2311 supplies a current to the output wiring 2309. In other words, the output wiring 2309 for the selection transistor 2305 is formed by a source follower. The load current source 2311 may be provided on the imaging chip 2113 side or on the signal processing chip 2111 side.

[0232] Here, the flow from the start of charge accumulation to pixel output after accumulation is completed will be described. When a reset pulse is applied to the reset transistor 2303 via the reset wiring 2306 and at the same time a transfer pulse is applied to the transfer transistor 2302 via the TX wiring 2307, the potentials of the PD 2104 and the floating diffusion FD are reset.

[0233] When the application of the transfer pulse is stopped, the PD 2104 converts the incident light it receives into electric charges and accumulates them. Then, when the transfer pulse is applied again without the reset pulse being applied, the accumulated electric charges are transferred to the floating diffusion FD, and the potential of the floating diffusion FD changes from the reset potential to the signal potential after charge accumulation. When a selection pulse is applied to the selection transistor 2305 via the decoder wiring 2308, the fluctuation in the signal potential of the floating diffusion FD is transmitted to the output wiring 2309 via the amplification transistor 2304 and the selection transistor 2305. As a result, a pixel signal corresponding to the reset potential and the signal potential is output from the unit pixel to the output wiring 2309.

[0234] In the example shown in FIG. 37, the reset wiring 2306 and TX wiring 2307 are common to the 16 pixels that form the pixel block 2131. That is, the reset pulse and transfer pulse are each applied simultaneously to all 16 pixels. Therefore, all pixels that form the pixel block 2131 start and end charge accumulation at the same timing. However, pixel signals corresponding to the accumulated charges are sequentially applied to the respective selection transistors 2305 by selection pulses, and are selectively output to the output wiring 2309. Furthermore, the reset wiring 2306, TX wiring 2307, and output wiring 2309 are provided separately for each pixel block 2131.

[0235] By configuring the circuit based on the pixel block 2131 in this way, the charge accumulation time can be controlled for each pixel block 2131. In other words, adjacent pixel blocks 2131 can output pixel signals with different charge accumulation times. Furthermore, by causing one pixel block 2131 to perform a single charge accumulation while the other pixel block 2131 repeatedly accumulates charge and outputs a pixel signal each time, these pixel blocks 2131 can output frames of a moving image at different frame rates. Note that at least some of the transistors and wiring shown in FIG. 37 function as readout circuits that read out pixel signals output from each pixel. A readout circuit is provided for each pixel. In each readout circuit, some of the wiring and other components may be shared between pixels.

[0236] 38 shows part of the configuration of the image sensor 2100 and an example of its operation. The image sensor 2100 of this example further includes a storage unit 2114 in addition to the configuration shown in FIG. 35. The storage unit 2114 may be provided in the signal processing chip 2111. In this case, the image sensor 2100 does not need to include the memory chip 2112. The storage unit 2114 may also be provided in the memory chip 2112.

[0237] The imaging chip 2113 has an imaging region 2700 in which a plurality of pixels are arranged, each of which generates a pixel signal in response to incident light. For ease of explanation, FIG. 38 shows three pixel blocks 2131 in each row and column direction. It is preferable that the number of pixels included in each pixel block 2131 is the same. It is also preferable that the number of pixels included in each pixel block 2131 in the imaging region 2700 is fixed.

[0238] The signal processing chip 111 of this example has a multiplexer 2411, an A / D converter 2412, a demultiplexer 2413, a control unit 2740, and an arithmetic circuit 2415 for each pixel block 2131. The multiplexer 2411 sequentially selects pixels included in the corresponding pixel block 2131 and inputs pixel signals corresponding to the selected pixels to the A / D converter 2412. The A / D converter 2412 converts the analog pixel signals into digital pixel data and inputs it to the demultiplexer 2413. The demultiplexer 2413 stores the pixel data in a memory area corresponding to the pixel in the corresponding memory block 2730. Each memory block 2730 passes the stored pixel data to the subsequent arithmetic circuit 2415.

[0239] The memory unit 2114 has a plurality of memory blocks 2730 provided corresponding to a plurality of pixel blocks 2131, each capable of storing pixel data of the corresponding pixel block 2131. The memory blocks 2730 correspond one-to-one to the pixel blocks 2131. The memory blocks 2730 may be connected to the corresponding pixel blocks 2131 via a bus 2720. The memory blocks 2730 may be buffer memories.

[0240] Furthermore, at least some of the storage blocks 2730 can also store pixel data of pixel blocks other than the corresponding pixel block 2131. That is, one storage block 2730 may be shared by multiple pixel blocks 2131. In other words, the control unit 2740 can store pixel data of one pixel block 2131 in multiple storage blocks 2730. By sharing the storage blocks 2730, the multiple storage blocks 2730 can be used efficiently, as will be described later, and the overall memory capacity of the storage unit 2114 can be reduced.

[0241] It is preferable that, for all pixel blocks 2131, pixel data can be read from and written to at least one other storage block 2730 in addition to the corresponding storage block 2730. The other storage block 2730 may be predetermined for each pixel block 2131, or may be dynamically changeable. It is also preferable that, for all storage blocks 2730, pixel data can be read from and written to at least one other pixel block 2131 in addition to the corresponding pixel block 2131. The other pixel block 2131 may be predetermined for each storage block 2730, or may be dynamically changeable.

[0242] Each memory block 2730 may be a memory provided for each pixel block 2131 in an area overlapping the corresponding pixel block 2131 in the signal processing chip 2111. That is, the memory block 2730 may be provided in an area directly below the corresponding pixel block 2131 in the signal processing chip 2111. In this case, the pixel block 2131 and the memory block 2730 may be electrically connected by a TSV. Also, the corresponding memory block 2730, A / D converter 2412, arithmetic circuit 2415, etc. are provided in the area overlapping each pixel block 2131 in the signal processing chip 2111. Also, each memory block 2730 may be a memory provided outside the area overlapping with the imaging area 2700 in the signal processing chip 2111.

[0243] Furthermore, when each memory block 2730, A / D converter 2412, and arithmetic circuit 2415 is provided in an area overlapping with the corresponding pixel block 2131, and when each memory block 2730 stores pixel data of a pixel block 2131 other than the corresponding pixel block 2131, it may transmit an analog pixel signal or digital pixel data to the area in which the memory block 2730 is provided. In the former case, the A / D converter 2412 corresponding to the memory block 2730 converts the pixel signal into pixel data and inputs it to the memory block 2730. In the latter case, the A / D converter 2412 in the area overlapping with the pixel block 2131 converts the pixel signal into pixel data, and then transmits the pixel data to the memory block 2730 in which it is to be stored. The signal processing chip 2111 is provided with wiring for transmitting these pixel signals or pixel data.

[0244] An arithmetic circuit 2415, which will be described later, processes the pixel data stored in the memory block 2730 and passes it on to the downstream image processing unit 2511. The arithmetic circuit 2415 may be provided in the signal processing chip 2111. Note that while the diagram shows connections for one pixel block 2131, in reality, there is a arithmetic circuit for each pixel block 2131, and they operate in parallel. It is preferable to provide a arithmetic circuit 2415 for each pixel block 2131.

[0245] As described above, output wiring 2309 is provided corresponding to each pixel block 2131. Since the imaging element 2100 has an imaging chip 2113, a signal processing chip 2111, and a memory unit 2114 stacked on top of each other, by using electrical connections between the chips using bumps 2109 for these output wiring 2309, it is possible to route the wiring without increasing the size of each chip in the planar direction.

[0246] The control unit 2740 is provided with rate information regarding the frame rate of each pixel block 2131. Based on the rate information, the control unit 2740 selects a storage block 2730 in which to store pixel data of the pixel block 2131 having the high frame rate. For example, the control unit 2740 selects the storage block 2730 corresponding to the pixel block 2131 having the reference frame rate as the storage block 2730 in which to store the pixel data.

[0247] In the examples shown in the figures, an arithmetic circuit 2415 is provided for each pixel block 2131 including multiple pixels. However, an arithmetic circuit 2415 may be provided for each pixel. An arithmetic circuit 2415 does not have to be provided for every pixel. That is, at least a first pixel and a second pixel are arranged in the imaging region 2700, and the imaging element 2100 has at least a first arithmetic circuit 2415 corresponding to the first pixel and a second arithmetic circuit 2415 corresponding to the second pixel.

[0248] The first pixel signal output by the first pixel is read out by the first readout circuit, and the second pixel signal output by the second pixel is read out by the second readout circuit. The first arithmetic circuit 2415 calculates a first evaluation value based on the first pixel signal output from the first pixel and transmits it to the downstream image processing unit 2511. The second arithmetic circuit 2415 calculates a second evaluation value based on the second pixel signal output from the second pixel and transmits it to the downstream image processing unit 2511. Here, the evaluation value is a value obtained by performing a predetermined calculation using the values ​​of the pixel signals. For example, the evaluation value may be the difference or average between the value of a pixel signal output by a given pixel and the value of an adjacent pixel signal output by an adjacent pixel adjacent to that pixel. Alternatively, the evaluation value may be the difference or average between the values ​​of multiple pixel signals output by a given pixel in different frames. Various parameters may be used in this calculation.

[0249] 39 is a block diagram showing the configuration of an imaging device according to this embodiment. The imaging device 2500 includes a photographing lens 2520 as an imaging optical system, and the photographing lens 2520 guides a subject light beam incident along an optical axis OA to the image sensor 2100. The photographing lens 2520 may be an interchangeable lens that can be attached to and detached from the imaging device 2500. The imaging device 2500 mainly includes the image sensor 2100, a system control unit 2501, a drive unit 2502, a photometry unit 2503, a work memory 2504, a recording unit 2505, and a display unit 2506.

[0250] The photographing lens 2520 is composed of a group of multiple optical lenses and focuses a subject light beam from a scene near its focal plane. Note that in FIG. 35, the photographing lens 2520 is represented by a virtual single lens placed near the pupil. The driver 2502 is a control circuit that performs charge accumulation control such as timing control and area control of the image sensor 100 in accordance with instructions from the system controller 2501. In this sense, the driver 2502 can be said to function as an image sensor controller that causes the image sensor 2100 to accumulate charges and output pixel signals.

[0251] The image sensor 2100 passes the pixel signals to the image processing unit 2511 of the system control unit 2501. The image processing unit 2511 performs various image processing using the work memory 2504 as a workspace to generate image data. The image processing unit 2511, located downstream of the first and second arithmetic circuits 2415, performs image processing on first pixel data of an image corresponding to the first pixel signal based on the first evaluation value received from the first arithmetic circuit 2415, and performs image processing on second pixel data of an image corresponding to the second pixel signal based on the second evaluation value received from the second arithmetic circuit 2415. For example, when generating image data in JPEG file format, a color video signal is generated from a signal obtained using the Bayer array and then compressed. The generated image data is recorded in the recording unit 2505 and converted into a display signal, which is displayed on the display unit 2506 for a predetermined time. The image processing unit 2511 may be provided in the image sensor 2100 or in the system control unit 2501 external to the image sensor 2100. Furthermore, the image processing unit 2511 may be provided for each pixel, or may be provided for each pixel block 2131 that includes multiple pixels.

[0252] The photometry unit 2503 detects the luminance distribution of a scene prior to a series of shooting sequences for generating image data. The photometry unit 2503 includes, for example, an AE sensor with approximately one million pixels. The calculation unit 2512 of the system control unit 2501 receives the output of the photometry unit 2503 and calculates the luminance of each area of ​​the scene. The calculation unit 2512 determines the shutter speed, aperture value, and ISO sensitivity according to the calculated luminance distribution. The image sensor 2100 may also function as the photometry unit 2503. The calculation unit 2512 also performs various calculations for operating the imaging device 2500.

[0253] The drive unit 2502 may be partly or entirely mounted on the imaging chip 2113, or partly or entirely mounted on the signal processing chip 2111. A part of the system control unit 2501 may be mounted on the imaging chip 2113 or the signal processing chip 2111. In the imaging device 2500 of this example, at least a part of the image processing function of the image processing unit 2511 is provided in the imaging element 2100.

[0254] 40 is a functional block diagram of the image processing unit. The image processing unit 2511 in this example extracts pixel blocks 2131 (peripheral regions 2176 described below) that operate at the reference frame rate and pixel blocks 2131 (regions of interest 2172 described below) that operate at a high frame rate. In addition to the above functions, the image processing unit 2511 also has a subject estimation unit 2150, a group selection unit 2152, a video generation unit 2154, and a video synthesis unit 2156. Each of these functions will be described later.

[0255] Fig. 41 is a flowchart showing the operation of an imaging device to generate and record a moving image. Fig. 42 and Fig. 43 show examples of images captured by an imaging element. Fig. 44 shows the relationship between each frame rate and the output timing of an image signal.

[0256] 41 starts when a user instructs the image capturing device 2500 to generate a video by pressing a record button, etc. First, the subject estimation unit 2150 drives the drive unit 2502 to acquire image data based on an image signal from the image capturing element 2100, and estimates a main subject included in an image represented by the image data (S100).

[0257] In this case, it is preferable that the driving unit 2502 output image signals from pixel blocks 2131 included in the entire imaging area, for example, from all pixel blocks 2131. The driving unit 2502 may also output image signals from all pixels included in each pixel block 2131, or may output image signals from pixels thinned out at a predetermined thinning rate. The subject estimation unit 2150 compares multiple images obtained in time series from the image sensor 2100 and identifies a moving subject as the main subject. Note that other methods may be used to estimate the main subject.

[0258] For example, when image 2170 in Fig. 42 and image 2178 in Fig. 43 are acquired from the image sensor 2100 as images taken before and after in time, the subject estimation unit 2150 uses the difference between these images to identify a child as the main subject 2171. Note that the grid lines in images 2170 and 2178 indicate boundaries of pixel blocks 2131, but the number of pixel blocks 2131 is merely an example and is not limited to the number shown in these figures.

[0259] The group selection unit 2152 selects at least one pixel block 2131 onto which image light of the main subject 2171 estimated by the subject estimation unit 2150 is incident (S2102). For example, in the image 2170, a pixel block 2131 that includes at least a portion of the main subject 2171 is selected. Furthermore, taking into consideration that the main subject 2171 moves within the imaging area, it is preferable that the group selection unit 2152 also selects pixel blocks 2131 further surrounding the pixel block 2131 that includes at least a portion of the main subject 2171.

[0260] The group selection unit 152 designates a collection of these selected pixel blocks 2131 as a region of interest 2172. Furthermore, the group selection unit 2152 designates a collection of pixel blocks 2131 in the entire imaging area that are not included in the region of interest 2172 as a peripheral region 2176. The group selection unit 2152 specifies region information 2174 that indicates the range of the region of interest 2172 relative to the entire imaging area.

[0261] 42, the region of interest 2172 is a rectangular region made up of 28 pixel blocks 2131, 7 horizontally and 4 vertically. In contrast, the peripheral region 2176 is made up of 98 pixel blocks 2131, excluding the region of interest 2172, from the 126 pixel blocks of the imaging region, 21 horizontally and 6 vertically. Furthermore, as region information 2174, the position (9, 2) counted from the left and top of the pixel block 2131 at the top left corner of the region of interest 2172 in the imaging region in the figure is specified. Furthermore, as size information, the number of vertical and horizontal dimensions of the region of interest 2172 is specified as 7×4.

[0262] The group selection unit 2152 transmits information specifying pixel blocks 2131 included in the region of interest 2172 and information specifying the peripheral region 2176 to the drive unit 2502. In this case, information on the frame rates to be applied to the region of interest 2172 and the peripheral region 2176 is also transmitted. Here, it is preferable that the frame rate applied to the region of interest 2172 is higher than the frame rate to be applied to the peripheral region 2176. For example, if the frame rate to be applied to the peripheral region 2176 is 60 fps, the frame rate to be applied to the region of interest 2172 is set to 180 fps. It is preferable that these frame rate values ​​are set in advance and stored so that the group selection unit 2152 can refer to them, but the user may be able to change the values ​​later.

[0263] The driver 2502 drives the image sensor 2100 to capture images at each frame rate (S2104). That is, the driver 2502 causes the pixel blocks 2131 included in the region of interest 2172 to accumulate charge and output image signals at a high frame rate, and causes the pixel blocks 2131 included in the peripheral region 2176 to accumulate charge and output image signals at a low frame rate. In other words, the driver 2502 obtains image signals corresponding to multiple frames arranged in time series for the pixel blocks 2131 included in the region of interest 2172 while obtaining an image signal corresponding to one frame for the pixel blocks 2131 included in the peripheral region 2176.

[0264] 44, when the frame rate of the peripheral region 2176 is set to 60 fps and the frame rate of the region of interest 2172 is set to 180 fps, the driving unit 2502 obtains image signals of three frames A1, A2, and A3 from the region of interest 2172 (1 / 60 s = 3 × 1 / 180 s) during the 1 / 60 s it takes to obtain an image signal of one frame B1 from the peripheral region 2176. In this case, the driving unit 2502 separately drives the set of the reset transistor 2303, transfer transistor 2302, and selection transistor 2305 of the pixel block 2131 included in the peripheral region 2176 and the set of the reset transistor 2303, transfer transistor 2302, and selection transistor 2305 of the pixel block 2131 included in the region of interest 2172, thereby obtaining image signals at different frame rates.

[0265] 44 shows the timing of outputting the image signal, but does not show the length of the exposure time. The driving unit 2502 drives the above-mentioned sets of transistors for the peripheral region 2176 and the region of interest 2172 so as to achieve the exposure time calculated in advance by the calculation unit 2512.

[0266] In addition, the length of the exposure time may be changed according to the frame rate. For example, in the example shown in Fig. 44, the exposure time of one frame of the peripheral region 2176 may be set to 1 / 3, so that it is substantially the same exposure time as that of the region of interest 2172. Furthermore, after outputting the image signal, the image signal may be corrected based on the frame rate ratio. Furthermore, the timing of outputting the image signal between the peripheral region 2176 and the region of interest 2172 may not be synchronized as shown in Fig. 44, but may be asynchronous.

[0267] The image processing unit 2511 stores the image signals from the region of interest 2172 sequentially for each frame in a predetermined storage area of ​​the work memory 2504 (S2106). Similarly, the image processing unit 2511 stores the image signals from the peripheral region 2176 sequentially for each frame in a predetermined storage area of ​​the work memory 2504 (same step). The work memory 2504 has a plurality of storage blocks 2730, as described in FIG. 38 . The work memory 2504 may be a memory made up of memory groups corresponding to the respective pixel blocks 2131.

[0268] The video generation unit 2154 reads out the image signal of the attention area 2172 stored in the work memory 2504 (S2108), and generates data of the attention area video including multiple frames of the attention area 2172 (S2110). Similarly, the video generation unit 2154 reads out the image signal of the peripheral area 2176 stored in the work memory 2504, and generates data of the peripheral area video including multiple frames of the peripheral area 2176 (same step). Here, the attention area video and the peripheral area video may each be generated in a general-purpose format such as MPEG so that they can be played separately, or may be generated in a dedicated format that cannot be played without undergoing a synthesis process described later.

[0269] Fig. 45 schematically shows the attention area moving image and the surrounding area moving image generated by the moving image generation unit. The moving image generation unit 2154 generates the attention area moving image at a frame rate corresponding to the frame rate at which the driving unit 2502 drives the attention area 2172. In the example shown in Fig. 45, the attention area moving image is generated at a frame rate of 1 / 180 fps, which is the same as the frame rate of 1 / 180 fps at which the driving unit 2502 drives the attention area 2172.

[0270] Similarly, the video generation unit 2154 generates the surrounding area video at a frame rate corresponding to the frame rate at which the drive unit 2502 drives the surrounding area 2176. In the example shown in Fig. 45, the surrounding area video is generated at a frame rate of 1 / 60 fps, which is the same as the frame rate of 1 / 60 fps at which the drive unit 2502 drives the surrounding area 2176. Note that there are no valid values ​​in the area of ​​the surrounding area video that corresponds to the attention area 2172, and this is indicated by diagonal lines in the figure.

[0271] Furthermore, the video generation unit 2154 adds header information to the attention area video and the peripheral area video, and records this data in the recording unit 2505 (S2112). The header information includes area information indicating the position of the attention area 2172 relative to the entire imaging area, size information indicating the size of the attention area 172, and timing information indicating the relationship between the output timing of the image signal of the attention area 2172 and the output timing of the image signal of the peripheral area 2176.

[0272] The system control unit 2501 determines whether or not to capture the next unit of time (S2114). Whether or not to capture the next unit of time is determined based on whether or not the user has pressed the video record button at that point. If to capture the next unit of time (S2114: Yes), the process returns to step S2102, and if not to capture the next unit of time (S2114: No), the process ends.

[0273] Here, the "unit time" is a time of about several seconds that is set in advance in the system control unit 2501. The memory capacity used for storage in step S2106 is determined by this unit time, the frame rate and number of pixel blocks of the region of interest 2172, and the frame rate and number of pixel blocks of the peripheral region 2176. Furthermore, based on this information, an area in the memory capacity for storing data of the region of interest 2172 and an area for storing data of the peripheral region 2176 are determined.

[0274] As a result, it is possible to obtain image signals at a high frame rate from the region of interest 2172 that includes the main subject 2171, while reducing the amount of data by keeping the frame rate low for the peripheral region 2176. Therefore, compared to high-speed readout from all pixels, it is possible to reduce the load on driving and image processing, and suppress power consumption and heat generation.

[0275] 41, when the next unit time starts, pixel block 2131 is selected anew in step S2102, and the region information and size information are updated. This allows region of interest 2172 to be successively updated to track main subject 2171. In the example shown in FIG. 45, in the first frame A7 of the unit time in the region of interest moving image, region of interest 2182 consisting of pixel block 2131 different from that in the last frame A6 of the previous unit time is selected, and region information 2184 and surrounding region 2186 are also updated accordingly.

[0276] Fig. 46 shows an example of header information added by the video generation unit. The header information in Fig. 46 includes an attention area video ID that identifies the attention area video, the frame rate of the attention area video, a surrounding area video ID that identifies the surrounding area video corresponding to the attention area video, the frame rate of the surrounding area video, timing information, area information, and size information. This header information may be added as header information to either the attention area video or the surrounding area video, or to both.

[0277] 47 is a flowchart showing the operation of the imaging device to play and display a moving image. This operation starts when the user selects one of the area-of-interest moving images displayed as thumbnails on the display unit 2506 and presses the play button.

[0278] The video composition unit 2156 reads data of the attention area video specified by the user (S2150) from the storage unit 2505. The video composition unit 2156 reads data of the surrounding area video corresponding to the attention area video from the storage unit 2505 (S2152).

[0279] In this case, the video composition unit 2156 identifies the surrounding area video by the surrounding area video ID indicated in the header information of the attention area video read in step S2150. Alternatively, a surrounding area image including, as header information, the same timing information as the timing information indicated in the header information may be searched for and identified.

[0280] In the above example, it is assumed that the attention area video contains header information. On the other hand, if the attention area video does not contain header information but the surrounding area video does, the surrounding area video may be first read by having the user specify it in step S2150, and the attention area video may then be specified and read from the header information in step S2152.

[0281] The moving image synthesizing unit 2156 synthesizes frames of the display moving image using frames of the attention area moving image and frames of the surrounding area moving image (S154). In this case, first, the first frame A1 of the attention area moving image is inserted into the position indicated by the area information 2174 in the first frame B1 of the surrounding area moving image, thereby synthesizing the first frame C1 of the display moving image. As shown in FIG. 45, the moving image synthesizing unit 2156 displays the first frame C1 of the display moving image on the display unit 2506 (S2156).

[0282] The video composition unit 2156 determines whether the next frame of the attention area video is present before the next frame B2 of the surrounding area video (S2158). If the next frame of the attention area video is present (S2158: Yes), the video composition unit 2156 updates the attention area 2172 with the next frames A2 and A3 and maintains the surrounding area 2176 in the previous frame B1 (S2162), thereby combining the next frames C2 and C3 of the displayed video (S2162) and displaying them sequentially (S2156).

[0283] On the other hand, if in step S2158 there is no next frame of the attention area video until the next frame B2 in the surrounding area video (S2158), the video synthesis unit 2156 updates the attention area 2172 with the next frame A4 and also updates the surrounding area 2176 with the next frame B2 (S2164), thereby synthesizing (S2162) and displaying (S2156) the next frame C4 of the displayed video.

[0284] As long as there is a next frame of the surrounding area 2176 in the surrounding area video (S2160: Yes), steps S2154 to S2160 are repeated. If there is no next frame of the surrounding area 2176 in the surrounding area video (S2160: No), the video composition unit 2156 searches whether there is a pair of the attention area video and the surrounding area video in the unit time next to the unit time of the pair of the attention area video and the surrounding area video (S2166). For example, the video composition unit 2156 searches whether there is an attention area video in the same folder of the recording unit 2505, the header information of which includes timing information indicating a timing immediately after the timing indicated by the timing information of the attention area video.

[0285] As long as there is a pair of attention area video and peripheral area video for the next unit time (S2166: Yes), steps S2150 to S2166 are repeated. If there is no pair of attention area video and peripheral area video for the next unit time (S2166: No), the operation ends.

[0286] As described above, it is possible to display a smooth moving image of the region of interest 2172 containing the main subject 2171 while reducing the overall amount of data. In step S2162, the region of interest 2172 is updated with the next frame as is, and the frames of the displayed image are synthesized. However, this synthesis method is not limited to this. As another example, the boundary line of the main subject 2171 in the region of interest 2172 may be identified by image processing, and the main subject 2171 enclosed by this boundary line may be updated to the next frame, while the previous frame may be maintained for the area outside the boundary line of the main subject 2171 even within the region of interest 2172, and synthesized with a frame of the peripheral region 2176. In other words, the frame rate for the area outside the boundary line of the region of interest 2172 may be reduced to that of the peripheral region 2176. This prevents the boundary line of smoothness in the displayed moving image from appearing unnatural. Furthermore, the frame rate for playback does not need to be the same as the frame rate used during shooting (180 fps for the region of interest and 60 fps for the peripheral region). For example, the region of interest may be 60 fps, and the peripheral region may be 20 fps. In this case, the video will be played in slow motion.

[0287] Fig. 48 is a flowchart showing another example of the operation of the imaging device to generate and record a moving image. In Fig. 48, the same operations as those in Fig. 41 are assigned the same reference numerals, and the description thereof will be omitted.

[0288] In the operation of Fig. 48, instead of or in addition to the frame rate of Fig. 41, different thinning rates are used between the region of interest 2172 and the peripheral region 2176. More specifically, in step S2120, the driving unit 2502 causes pixels thinned out at a low thinning rate in the pixel block 2131 included in the region of interest 2172 to accumulate charge and output image signals, and causes pixels thinned out at a high thinning rate in the pixel block 2131 included in the peripheral region 2176 to accumulate charge and output image signals. For example, a thinning rate of 0 is used in the pixel block 2131 included in the region of interest 2172, meaning that all pixels are read out, and a thinning rate of 0.5 is used in the pixel block 2131 included in the peripheral region 2176, meaning that half the pixels are read out.

[0289] In this case, the driving unit 2502 separately drives the set of reset transistor 2303, transfer transistor 2302 and selection transistor 2305 of pixel block 2131 included in the peripheral region 2176 and the set of reset transistor 2303, transfer transistor 2302 and selection transistor 2305 of pixel block 2131 included in the target region 2172, thereby obtaining image signals at different thinning rates.

[0290] In step S2110, the video generation unit 2154 generates an attention area video corresponding to the attention area 2172, based on the image signal of the attention area 2172 output at a low thinning rate. Similarly, the video generation unit 2154 generates a surrounding area video corresponding to the surrounding area 2176, based on the image signal of the surrounding area 2176 output at a high thinning rate. Furthermore, in step S2112, the video generation unit 2154 records the attention area video and the surrounding area video in the recording unit 2505, adding information about the respective thinning rates.

[0291] Figure 49 shows an example of pixels 2188 that are read out at a thinning rate of 0.5 for one pixel block. In the example shown in Figure 49, when the pixel block 2132 in the peripheral region 2176 is in a Bayer array, pixels 2188 that are read out and pixels that are not read out are set every other Bayer array unit in the vertical direction, that is, every two rows when viewed pixel by pixel. This allows thinning readout to be performed without disrupting the color balance.

[0292] Fig. 50 is a flowchart showing the operation of the imaging device for playing back and displaying a moving image, corresponding to Fig. 48. In Fig. 50, the same operations as those in Fig. 47 are given the same reference numerals, and the description thereof will be omitted.

[0293] 50, the video composition unit 2156 complements the pixels of the frames of the peripheral area video to match the resolution of the frames of the attention area video, and then combines the frames of the display image by fitting the frames of the attention area video into the frames of the peripheral area video. This makes it possible to obtain a high-resolution image signal from the attention area 2172 containing the main subject 2171, while reducing the amount of data by limiting the peripheral area 2176 to a lower resolution. Therefore, compared to high-speed readout from all pixels, the load on driving and image processing can be reduced, and power consumption and heat generation can be suppressed.

[0294] 35 to 50, the region of interest 2172 is rectangular, but the shape of the region of interest 2172 is not limited to this. The region of interest 2172 may be a convex polygon, a concave polygon, or a doughnut shape with the surrounding region 2176 embedded therein, as long as it follows the boundary line of the pixel block 2131. Furthermore, multiple regions of interest 2172 may be set at intervals. In this case, different frame rates may be set for the regions of interest 2172.

[0295] The frame rates of the region of interest 2172 and the peripheral region 2176 may also be variable. For example, the amount of movement of the main subject 2171 may be detected each time a unit of time elapses, and a higher frame rate may be set for the region of interest 2172 the greater the amount of movement of the main subject 2171. Furthermore, the selection of pixel blocks 2131 to be included in the region of interest 2172 may be updated as needed within the unit of time to follow the main subject 2171.

[0296] Although generation of the moving images in FIGS. 41 and 48 is initiated by the user pressing a record button, and playback of the moving images in FIGS. 47 and 50 is initiated by the user pressing a play button, the start points are not limited to these. As another example, a single button operation by the user may cause the operation of generating a moving image and the operation of playing the moving image to be executed consecutively, and a through image (also referred to as a live view display) may be displayed on the display unit 2506. In this case, a display that allows the user to recognize the region of interest 2172 may be superimposed. For example, a frame may be displayed on the display unit 2506 around the boundary of the region of interest 2172, or the brightness of the peripheral region 2176 may be reduced or the brightness of the region of interest 2172 may be increased.

[0297] In the operation of FIG. 48 , the thinning rate is made different between the region of interest 172 and the peripheral region 176. Instead of making the thinning rate different, the number of rows when adding pixel signals of pixels in adjacent rows may be made different. For example, the number of rows in the region of interest 172 is 1, i.e., pixel signals are output without adding adjacent rows, and the number of rows in the peripheral region 176 is made larger than that of the region of interest 172, for example, 2, and pixel signals of pixels in the same column of two adjacent rows are output. This makes it possible to reduce the overall signal amount while maintaining a higher resolution in the region of interest 172 than in the peripheral region 176, as in FIG. 48 .

[0298] The moving image synthesizing unit 2156 may be provided in an external display device, for example, a PC, instead of being provided in the image processing unit 2511 of the imaging device 2500. Also, the above embodiment is not limited to being applied to the case of generating moving images, but may also be applied to the case of generating still images.

[0299] Furthermore, in all of the above embodiments, the multiple pixel blocks 2131 are divided into two regions, the region of interest 2172 and the surrounding region 2176, but the present invention is not limited to this and the pixel blocks 2131 may be divided into three or more regions. In this case, the pixel block 2131 corresponding to the boundary between the region of interest 2172 and the surrounding region 2176 may be set as the boundary region, and the boundary region may be controlled using an intermediate value between the control parameter value used for the region of interest 2172 and the control parameter value used for the surrounding region 2176. This makes it possible to prevent the boundary between the region of interest 2172 and the surrounding region 2176 from looking unnatural.

[0300] The charge accumulation time, accumulation count, etc. may be different between the region of interest 2172 and the peripheral region 2176. In this case, the region of interest 2172 and the peripheral region 2176 may be divided based on brightness, and an intermediate region may also be provided.

[0301] 51A and 51B are diagrams illustrating an example of a scene and its region division. FIG. 51A shows a scene captured by the imaging region of the imaging chip 2113. Specifically, the scene simultaneously captures a shadow object 2601 and a middle object 2602 in an indoor environment and a highlight object 2603 in an outdoor environment, observed inside a window frame 2604. When capturing a scene with a large contrast between highlight and shadow, a conventional imaging device would experience crushed shadows in the shadows if charge accumulation was performed based on the highlights, while performing charge accumulation based on the shadows would result in blown-out highlights in the highlights. In other words, the dynamic range of the photodiode is insufficient for a scene with a large contrast between highlights and shadows to output an image signal by uniformly accumulating charge across both the highlights and shadows. Therefore, in this embodiment, the scene is divided into partial regions, such as highlights and shadows, and the photodiodes corresponding to each region are allowed to accumulate charge at different times, thereby substantially expanding the dynamic range.

[0302] FIG. 51B shows the division of the imaging region of the imaging chip 2113. The calculation unit 2512 analyzes the scene of FIG. 51A captured by the photometry unit 2503 and divides the imaging region based on brightness. For example, the system control unit 2501 causes the photometry unit 2503 to capture the scene multiple times while changing the exposure time, and the calculation unit 2512 determines the division line of the imaging region by referring to the changes in the distribution of blown-out highlight regions and crushed shadow regions. In the example of FIG. 51B, the calculation unit 2512 divides the imaging region into three regions: a shadow region 2611, a middle region 2612, and a highlight region 2613.

[0303] The division lines are defined along the boundaries of the pixel block 2131. That is, each divided region includes an integer number of groups. Pixels in each group included in the same region accumulate charge and output pixel signals the same number of times within a period corresponding to the shutter speed determined by the calculation unit 2512. Pixels belonging to different regions accumulate charge and output pixel signals differently.

[0304] FIG. 52 is a diagram illustrating charge accumulation control for each divided region in the examples of FIGS. 51A and 51B. When the calculation unit 2512 receives a shooting preparation instruction from the user, it determines the shutter speed T0 from the output of the photometry unit 2503. Furthermore, as described above, it divides the region into a shadow region 2611, an intermediate region 2612, and a highlight region 2613, and determines the number of charge accumulations from the brightness information for each region. The number of charge accumulations is determined so that pixels are not saturated by charge accumulation per one time. For example, the number of charge accumulations is determined based on the criterion that 80 to 90 percent of the charge that can be accumulated in one charge accumulation operation is accumulated.

[0305] Here, the shadow region 2611 is set to one charge accumulation. That is, the charge accumulation time is made to match the determined shutter speed T0. The number of charge accumulations in the intermediate region 2612 is set to two. That is, one charge accumulation time is set to T0 / 2, and charge accumulation is repeated two times during the shutter speed T0. The number of charge accumulations in the highlight region 2613 is set to four. That is, one charge accumulation time is set to T0 / 4, and charge accumulation is repeated four times during the shutter speed T0.

[0306] When a shooting instruction is received from the user at time t = 0, the driving unit 2502 applies a reset pulse and a transfer pulse to the pixels of the groups belonging to any area. This application serves as a trigger to start charge accumulation in all pixels.

[0307] At time t=T0 / 4, the driving unit 2502 applies a transfer pulse to the pixels of the group belonging to the highlighted region 2613. Then, it sequentially applies a selection pulse to the pixels in each group, causing each pixel signal to be output to the output wiring 2309. After the pixel signals of all pixels in the group have been output, the driving unit 2502 again applies a reset pulse and a transfer pulse to the pixels of the group belonging to the highlighted region 2613, causing a second charge accumulation to begin.

[0308] Note that because it takes time to selectively output pixel signals, there is a time lag between the end of the first charge accumulation and the start of the second charge accumulation. If this time lag can be practically ignored, then, as described above, the time obtained by dividing the shutter speed T0 by the number of charge accumulations can be set as the charge accumulation time for one cycle. On the other hand, if it cannot be ignored, then the shutter speed T0 can be adjusted to take this time into consideration, or the charge accumulation time for one cycle can be set shorter than the time obtained by dividing the shutter speed T0 by the number of charge accumulations.

[0309] At time t=T0 / 2, the driving unit 2502 applies transfer pulses to the pixels in the groups belonging to the intermediate region 2612 and the highlight region 2613. Then, it sequentially applies selection pulses to the pixels in each group, causing each pixel signal to be output to the output wiring 2309. After outputting pixel signals from all pixels in the group, the driving unit 2502 again applies reset pulses and transfer pulses to the pixels in the groups belonging to the intermediate region 2612 and the highlight region 2613, causing a second charge accumulation to begin in the intermediate region 2612 and a third charge accumulation to begin in the highlight region 2613.

[0310] At time t=3T0 / 4, the driving unit 2502 applies a transfer pulse to the pixels of the group belonging to the highlighted region 2613. Then, the driving unit 2502 sequentially applies a selection pulse to the pixels in each group, causing each pixel signal to be output to the output wiring 2309. After the pixel signals of all pixels in the group have been output, the driving unit 2502 again applies a reset pulse and a transfer pulse to the pixels of the group belonging to the highlighted region 2613, causing the fourth charge accumulation to begin.

[0311] At time t=T0, the drive unit 2502 applies transfer pulses to the pixels in all regions. Then, it sequentially applies selection pulses to the pixels in each group, causing each pixel signal to be output to the output wiring 2309. Through the above control, one pixel signal is stored in each pixel memory 2414 corresponding to the shadow region 2611, two pixel signals are stored in each pixel memory 2414 corresponding to the intermediate region 2612, and four pixel signals are stored in each pixel memory 2414 corresponding to the highlight region 2613.

[0312] The driving unit 2502 may sequentially apply a reset pulse to the pixels of the groups belonging to any of the regions, thereby sequentially resetting the pixels of the groups belonging to each region. The driving unit 2502 may also sequentially apply a transfer pulse to the pixels of the reset groups. This application may trigger the pixels of each group to sequentially start charge accumulation. After charge accumulation has been completed for the pixels of the groups belonging to all regions, the driving unit 2502 may apply a transfer pulse to the pixels of all regions. Then, a selection pulse may be sequentially applied to the pixels in each group, causing each pixel signal to be output to the output wiring 2309.

[0313] These pixel signals are sequentially transferred to the image processing unit 2511. The image processing unit 2511 generates high dynamic range image data from these pixel signals. Specific processing will be described later.

[0314] 53 is a diagram showing the relationship between the number of integrations and the dynamic range. Multiple pixel data corresponding to repeatedly executed charge accumulations are integrated by the image processing unit 2511 to form part of image data with a high dynamic range.

[0315] If the dynamic range of an area where the number of integrations is one, i.e., charge accumulation is performed once, is used as the reference, the expansion of the dynamic range of an area where the number of integrations is two, i.e., charge accumulation is performed twice and the output signal is accumulated, is one step. Similarly, if the number of integrations is increased to four, it becomes two steps, and if it is increased to 128, it becomes seven steps. In other words, to expand the dynamic range by n steps, the output signal needs to be accumulated 2n times.

[0316] Here, a 3-bit exponent digit indicating the number of accumulations is added to the image signal so that the image processing unit 2511 can identify which divided region has accumulated charge and how many times. As shown in the figure, the exponent digits are assigned in order, such as 000 for 1 accumulation, 001 for 2 accumulations, ..., 111 for 128 accumulations.

[0317] The image processing unit 2511 references the exponent digit of each pixel data received from the arithmetic circuit 2415, and if the reference result indicates two or more integrations, it performs integration processing on the pixel data. For example, if the number of integrations is two (one stage), the upper 11 bits of the 12-bit pixel data corresponding to the charge accumulation for two pixel data are added together to generate one 12-bit pixel data. Similarly, if the number of integrations is 128 (seven stages), the upper 5 bits of the 12-bit pixel data corresponding to the charge accumulation for 128 pixel data are added together to generate one 12-bit pixel data. In other words, the upper bits obtained by subtracting the number of stages corresponding to the number of integrations from 12 are added together to generate one 12-bit pixel data. Note that the lower bits that are not subject to addition are discarded.

[0318] By processing in this way, the brightness range to which gradation is applied can be shifted to the high brightness side in accordance with the number of integrations. In other words, 12 bits are allocated to a limited range on the high brightness side. This means that gradation can be applied to image areas that would previously have been blown out.

[0319] However, since 12 bits are assigned to different brightness ranges for the other divided regions, image data cannot be generated by simply combining each region. Therefore, the image processing unit 2511 performs re-quantization processing based on the maximum and minimum brightness pixels to convert all regions into 12-bit image data while maintaining the obtained gradation as much as possible. Specifically, quantization is performed by applying gamma conversion so that gradation is maintained more smoothly. By processing in this manner, image data with a high dynamic range can be obtained.

[0320] The number of accumulations is not limited to the case where a 3-bit exponent is added to the pixel data as described above, but may be described as additional information separate from the pixel data. Furthermore, the exponent may be omitted from the pixel data, and instead the number of pixel data stored in pixel memory 2414 may be counted to obtain the number of accumulations during addition processing.

[0321] In the image processing described above, requantization processing was performed to fit the entire region into 12-bit image data, but the number of output bits can be increased to match the upper limit of the number of integrations for the number of bits of pixel data. For example, if the upper limit of the number of integrations is set to 16 times (4 stages), then the entire region can be made into 16-bit image data for 12-bit pixel data. Processing in this way makes it possible to generate image data without digit cancellation.

[0322] Next, a series of photographing operation processes will be described. Fig. 54 is a flow diagram showing the photographing operation processes. The flow starts when the power of the imaging device 500 is turned on.

[0323] In step S2201, the system control unit 2501 waits until the switch SW1 is pressed, which is an instruction to prepare for imaging. If the system control unit 2501 detects that the switch SW1 has been pressed, the process proceeds to step S2202.

[0324] In step S2202, the system control unit 2501 executes photometry processing. Specifically, the output of the photometry unit 2503 is obtained, and the calculation unit 2512 calculates the luminance distribution of the scene. Then, the process proceeds to step S2203, where the shutter speed, area division, number of integrations, etc. are determined as described above.

[0325] When the shooting preparation operation is complete, the process proceeds to step S2204 and waits until the switch SW2 is pressed, which is an instruction to shoot. At this time, if the elapsed time exceeds a predetermined time Tw (YES in step S2205), the process returns to step S2201. If the pressing of switch SW2 is detected before Tw is exceeded (NO in step S2205), the process proceeds to step S2206.

[0326] In step S2206, the driving unit 2502 receives an instruction from the system control unit 2501 and executes the charge accumulation process and signal readout process described with reference to Fig. 52. Then, when all signal readout is complete, the process proceeds to step S2207, where the image processing described with reference to Fig. 53 is executed, and a recording process is executed to record the generated image data in the recording unit.

[0327] When the recording process is completed, the process proceeds to step S2208, where it is determined whether or not the power supply to the image capture device 2500 has been turned off. If the power supply has not been turned off, the process returns to step S2201, and if the power supply has been turned off, the series of image capture operation processes ends.

[0328] Fig. 55 is a block diagram showing a specific configuration as an example of the signal processing chip 2111. The pixel data processing unit 2910 shown in Fig. 55 is provided for each pixel block 2131. However, similar to the arithmetic circuit 2415 described in relation to Fig. 38, the pixel data processing unit 2910 may be provided for each pixel, for two or more pixels. Furthermore, of the components of the pixel data processing unit 2910, the components other than the arithmetic circuit 2415 may be provided for each pixel block 2131.

[0329] The control unit 2740 in the signal processing chip 2111 of this example performs some or all of the functions of the drive unit 2502. The control unit 2740 includes a sensor control unit 2441, a block control unit 2442, a synchronization control unit 2443, and a signal control unit 2444 as distributed control functions, and a drive control unit 2420 that controls these control units in an integrated manner. The drive control unit 2420 converts instructions from the system control unit 2501 into control signals that can be executed by each control unit and passes them on to each unit.

[0330] The sensor control unit 2441 is responsible for controlling the transmission of control pulses related to charge accumulation and charge readout of each pixel, which are sent to the imaging chip 2113. Specifically, the sensor control unit 2441 controls the start and end of charge accumulation by sending a reset pulse and a transfer pulse to the target pixel, and outputs a pixel signal to the output wiring 2309 by sending a selection pulse to the readout pixel.

[0331] The block control unit 2442 transmits specific pulses to the imaging chip 2113 to identify the pixel blocks 2131 to be controlled. As described with reference to FIG. 51B and other figures, each of the regions divided into the region of interest 2172 and the peripheral region 2176 may include multiple adjacent pixel blocks 2131. These pixel blocks 2131 belonging to the same region form a block group. Pixels in the same block group start and end charge accumulation at the same timing. Therefore, the block control unit 2442 sends specific pulses to the target pixel blocks 2131 based on instructions from the drive control unit 2420, thereby dividing the pixel blocks 2131 into blocks. The transfer pulses and reset pulses that each pixel receives via the TX wiring 2307 and reset wiring 2306 are the logical product of the pulses sent by the sensor control unit 2441 and the specific pulses sent by the block control unit 2442.

[0332] In this way, by controlling each region as a mutually independent block group, the charge accumulation control described with reference to Figure 52 is realized. The drive control unit 2420 may apply reset pulses and transfer pulses at different timings to pixels included in the same block group. Furthermore, the drive control unit 2420 may complete charge accumulation of pixels included in the same block group at the same timing, and then apply selection pulses to the pixels in the block group sequentially to sequentially read out each pixel signal.

[0333] The synchronization control unit 2443 sends a synchronization signal to the imaging chip 2113. Each pulse becomes active in synchronization with the synchronization signal in the imaging chip 2113. For example, by adjusting the synchronization signal, random control, thinning control, and the like can be realized, in which only specific pixels belonging to the same pixel block 2131 are controlled.

[0334] The signal control unit 2444 is mainly responsible for timing control of the A / D converter 2412. The pixel signals output via the output wiring 2309 are input to the A / D converter 2412 via the CDS circuit 2410 and multiplexer 2411. The A / D converter 2412 is controlled by the signal control unit 2444 to convert the input pixel signals into digital pixel data. The pixel data converted into digital signals is passed to a demultiplexer 2413 and stored as digital pixel values ​​in pixel memories 2414 corresponding to each pixel. The pixel memories 2414 are an example of a memory block 2730.

[0335] The signal processing chip 2111 has a timing memory 2430 as an accumulation control memory that stores block division information about which pixel blocks 2131 are to be combined to form the block groups of the target area 2172 and the peripheral area 2176, and accumulation count information about how many times charge accumulation is to be repeated for each of the formed block groups. The timing memory 2430 is configured, for example, by a flash RAM.

[0336] As described above, which pixel blocks 2131 are to be combined to form a block group is determined by the system control unit 2501 based on the detection results of scene luminance distribution detection performed prior to a series of shooting sequences. The determined block groups are divided into, for example, a first block group, a second block group, and so on, and each block group is defined by which pixel blocks 2131 it includes. The drive control unit 2420 receives this block division information from the system control unit 2501 and stores it in the timing memory 2430.

[0337] Furthermore, the system control unit 2501 determines how many times charge accumulation should be repeated for each block group based on the detection result of the luminance distribution. The drive control unit 2420 receives this accumulation count information from the system control unit 2501 and stores it in the timing memory 2430 as a pair with the corresponding block division information. By storing the block division information and accumulation count information in the timing memory 2430 in this way, the drive control unit 2420 can independently execute a series of charge accumulation controls by sequentially referring to the timing memory 2430. In other words, once the drive control unit 2420 receives an image capture instruction signal from the system control unit 2501 in control of acquiring one image, it can complete accumulation control thereafter without receiving instructions from the system control unit 2501 each time regarding control of each pixel.

[0338] The drive control unit 2420 receives from the system control unit 2501 block division information and accumulation count information that are updated based on photometry results (detection results of luminance distribution) executed in synchronization with a shooting preparation instruction, and appropriately updates the contents stored in the timing memory 2430. For example, the drive control unit 2420 updates the timing memory 2430 in synchronization with a shooting preparation instruction or a shooting instruction. This configuration realizes faster charge accumulation control, and the system control unit 2501 can execute other processes in parallel while the drive control unit 2420 is executing charge accumulation control.

[0339] The drive control unit 2420 not only controls charge accumulation for the imaging chip 2113, but also refers to the timing memory 2430 when executing readout control. For example, the drive control unit 2420 refers to accumulation count information for each block group and stores pixel data output from the demultiplexer 2413 at a corresponding address in the pixel memory 2414.

[0340] In response to a transfer request from the system control unit 2501, the drive control unit 2420 reads out the target pixel data for each pixel block from the pixel memory 2414 and transfers it to the image processing unit 2511. At this time, the drive control unit 2420 also transfers the target pixel data to the image processing unit 2511 together with additional data corresponding to each target pixel data.

[0341] The arithmetic circuit 2415 performs a predetermined calculation for each pixel block 2131 on pixel data corresponding to a pixel signal generated by the corresponding pixel block 2131. That is, the arithmetic circuit 2415 is provided corresponding to each pixel block 2131, and performs calculation processing for each pixel block 2131. The arithmetic circuit 2415 is provided in a one-to-one relationship with the pixel block 2131. That is, the arithmetic circuit 2415 is a circuit provided in the signal processing chip 2111 directly below the pixel block 2131. The drive control unit 2420 reads out the pixel data stored in the pixel memory 2414 to the arithmetic circuit 2415, and causes the arithmetic circuit 2415 to perform predetermined calculation processing.

[0342] The pixel memory 2414 is provided with a data transfer interface that transmits pixel data or differential data (described later) in accordance with a transfer request. The data transfer interface is connected to a data transfer line 2920 that is connected to the image processing unit 2511. The data transfer line 2920 is configured, for example, by a serial bus. In this case, a transfer request from the system control unit 2501 to the drive control unit 2420 is executed by address specification using the address bus.

[0343] Using the signal processing chip 2111 of FIG. 55, a predetermined calculation may be performed after pixel data is acquired using different control parameters for the region of interest 2172 and the peripheral region 2176. For example, in FIGS. 41 to 44, a video is generated from images acquired at different frame rates for the region of interest 2172 and the peripheral region 2176. Alternatively, image processing may be performed to average images acquired at a higher frame rate to improve the S / N ratio. In this case, for example, while the drive control unit 2420 acquires one pixel signal from the peripheral region 2176, it acquires multiple pixel signals (e.g., four pixel signals) from the region of interest 2172 and stores the pixel data in the pixel memory 2414. The calculation circuit 2415 reads the multiple pixel data acquired for each pixel in the region of interest 2172 from the pixel memory 2414 and averages them for each pixel. This reduces random noise in each pixel of the region of interest 2172, improving the S / N ratio of the region of interest 2172.

[0344] A memory 2940 is connected to the data transfer line 2920. The memory 2940 may be a volatile memory that sequentially stores pixel data from the pixel memory 2414 at designated addresses. For example, the memory 2940 is a DRAM. The memory 2940 stores one frame's worth of RGB data using the pixel data of each pixel block 2131 that it receives.

[0345] The control unit 2740 causes the arithmetic circuit 2415 corresponding to the pixel block 2131 to exchange data with the arithmetic circuits 2415 corresponding to the surrounding pixel blocks 2131. In the example of FIG. 55, the drive control unit 2420 causes data to be transmitted between multiple arithmetic circuits 2415. Each arithmetic circuit 2415 receives at least a portion of the other arithmetic results from other arithmetic circuits 2415 corresponding to other pixel blocks 2131. Each arithmetic circuit 2415 may generate its own arithmetic result further based on the other arithmetic results it has received.

[0346] The arithmetic circuit 2415 also inputs the arithmetic results for each pixel block 2131 that has been subjected to the arithmetic processing to the output circuit 2922. The output circuit 2922 outputs the arithmetic results of the arithmetic circuit 2415 in association with the pixel data to the system control unit 2501. Here, outputting in association with pixel data means outputting the arithmetic results performed by the arithmetic circuit 2415 on the pixel data of the pixel block 2131 in question in association with information indicating which pixel block the pixel data subjected to the arithmetic processing belongs to.

[0347] The data transferred to the system control unit 2501 via the output circuit 2922 is the calculation result for each pixel block 2131. However, the system control unit 2501 cannot use the received data unless it knows what calculation was performed in each pixel block 2131 and what result was obtained. In this example, the output circuit 2922 outputs the calculation result with a data code indicating the calculation performed in each calculation circuit 2415. The data code may be predetermined for each calculation circuit 2415. Furthermore, if the calculation circuit 2415 can perform multiple types of calculations, it is preferable that the calculation circuit 2415 notify the output circuit 2922 of information indicating what calculation was performed. In other words, the output circuit 2922 generates and outputs the calculation results, calculation results, and control information performed for each pixel block 2131 as a single data array. Specific examples of data arrays output by the output circuit 2922 will be described later.

[0348] FIG. 56 shows multiple arithmetic circuits 2415 exchanging calculation results. For example, the first arithmetic circuit 2415 receives the second evaluation value from the second arithmetic circuit 2415 or the calculation result obtained when the second arithmetic circuit 2415 calculates the second evaluation value. In this case, the first arithmetic circuit 2415 calculates a first evaluation value based on the second evaluation value or the calculation result. Alternatively, each arithmetic circuit 2415 may read a pixel signal corresponding to another arithmetic circuit 2415 from the pixel memory 2414 corresponding to the arithmetic circuit 2415 and perform calculation on that pixel signal itself. For example, the first arithmetic circuit 2415 reads a second pixel signal corresponding to the second arithmetic circuit 2415. In this case, the first arithmetic circuit 2415 calculates a first evaluation value based on the read second pixel signal.

[0349] In this example, the pixel blocks 2131 corresponding to the arithmetic circuits 2415-1, 2415-2, and 2415-4 are adjacent in the column direction, and the pixel blocks 2131 corresponding to the arithmetic circuits 2415-1, 2415-3, and 2415-5 are adjacent in the row direction. Each arithmetic circuit 2415 receives at least a portion of the results of calculations performed by other arithmetic circuits 2415 corresponding to its corresponding pixel block 2131 and the adjacent pixel blocks 2131. Here, "adjacent" does not necessarily mean adjacent in the row and column directions. It may also include cases where the pixel blocks 2131 are adjacent in the diagonal direction. In this example, the case where the pixel blocks 2131 are adjacent in the row and column directions will be described.

[0350] Adjacent arithmetic circuits 2415 are connected via an output bus that outputs the arithmetic results to the arithmetic circuit 2415 corresponding to the adjacent pixel block 2131, and an input bus that inputs the arithmetic results to the arithmetic circuit 2415 corresponding to the adjacent pixel block 2131. The control unit 2740 causes the arithmetic circuit 2415 corresponding to the pixel block 2131 to generate its own arithmetic result based on the arithmetic results from the arithmetic circuits 2415 corresponding to the other adjacent pixel blocks 2131.

[0351] 57 is a block diagram showing an example of the configuration of the arithmetic circuits 2415. Each arithmetic circuit 2415 includes a current block calculation unit 2912, an average calculation unit 2913, an average-average calculation unit 2914, a surrounding block calculation unit 2911, and a pixel-average calculation unit 2915. The input of the current block calculation unit 2912 is connected to the output of the pixel memory 2414 corresponding to the current pixel block 2131, and the output of the current block calculation unit 2912 is connected to the input of the average calculation unit 2913, the input of the average-average calculation unit 2914, the input of the output circuit 2922, and each of the arithmetic circuits 2415 corresponding to the adjacent pixel blocks 2131. For example, the current block calculation unit 2912 outputs the average of the pixel values ​​of each color in the corresponding pixel block 2131.

[0352] The surrounding block calculation unit 2911 has multiple inputs, each connected to the output of the arithmetic circuit 2415 corresponding to multiple pixel blocks 2131 adjacent to the pixel block 2131 in question. The output of the surrounding block calculation unit 2911 is connected to the input of the average calculation unit 913. For example, the surrounding block calculation unit 2911 may calculate the average of the pixel values ​​of each color based on the average of these values ​​received from other arithmetic circuits 2415. Alternatively, the surrounding block calculation unit 2911 may output the average of the pixel values ​​of each color received from other arithmetic circuits 2415 as is.

[0353] The average calculation unit 2913 has two input units, one input connected to the output of the block calculation unit 2912, and the other input connected to the output of the surrounding block calculation unit 2911. For example, the average calculation unit 2913 outputs the average of the pixel values ​​of each color in the corresponding pixel block 2131 and the adjacent pixel blocks 2131, based on the average value output by the block calculation unit 2912 and the average value output by the surrounding block calculation unit 2911.

[0354] The mean-mean calculation unit 2914 has two inputs, one input connected to the output of the mean calculation unit 2913 and the other input connected to the output of the block calculation unit 2912. The output of the mean-mean calculation unit 2914 is connected to the input of the output circuit 2922. For example, the mean-mean calculation unit 2914 calculates the difference between the average of the pixel values ​​of each color calculated by the mean calculation unit 2913 and the average of the pixel values ​​of each color calculated by the block calculation unit 2912.

[0355] The pixel-average calculation unit 2915 has two inputs, one input connected to the output of the average calculation unit 2913, and the other input connected to the output of the pixel memory 2414 corresponding to the pixel block 2131. The output of the pixel-average calculation unit 2915 is connected to the input of the pixel memory 2414 corresponding to the pixel block 2131. For example, the pixel-average calculation unit 2915 outputs the difference between each pixel value in the pixel block 2131 and the average value of the corresponding color among the averages of the pixel values ​​of each color calculated by the average calculation unit 2913.

[0356] The control unit 2740 transmits the calculation result in the block calculation unit 2912 to the other calculation circuits 2415 and the output circuit 2922. The control unit 2740 also transmits the calculation result in the average-average calculation unit 2914 to the output circuit 2922. Furthermore, the control unit 2740 feeds back the calculation result in the pixel-average calculation unit 2915 to the pixel memory 2414 of the pixel block 2131.

[0357] Each calculation unit of the arithmetic circuit 2415 can be configured with an adder circuit, a subtractor circuit, and a divider circuit. By simplifying the circuit configuration of the arithmetic circuit 2415 in this way, it is possible to implement the arithmetic circuit 2415 for each pixel block 2131.

[0358] 58 is a flowchart illustrating an example of the operation of the arithmetic circuit 2415. After the arithmetic circuit 2415 starts operating, in step S2300, the control unit 2740 reads out RGB pixel data of the pixel block 2131 that was captured at the frame rate of the pixel block 2131 from the pixel memory 2414 corresponding to the pixel block 2131, and inputs the data to the block calculation unit 2912. In step S2310, in synchronization with step S2300, the control unit 2740 inputs at least a portion of the calculation results for the neighboring pixel block 2131 from the neighboring arithmetic circuit 2415 to the surrounding block calculation unit 2911. In this example, each arithmetic circuit 2415 calculates the average pixel values ​​for each RGB pixel, and the surrounding block calculation unit 2911 receives the average pixel values ​​for each RGB pixel calculated by the neighboring arithmetic circuit 2415.

[0359] In step S2320, the control unit 2740 causes the block calculation unit 2912 to perform a predetermined calculation on the pixel data of the corresponding pixel block 2131. For example, the block calculation unit 2912 calculates the average value (Ar, Ag, Ab) for each RGB pixel of the pixel block 2131. The average value is calculated by Ai = Σ(i pixel in pixel block) / (i number of pixels in pixel block) (i = r, g, b). In step S2322, the control unit 2740 causes the block calculation unit 2912 to input the average values ​​(Ar, Ag, Ab) to the input of the output circuit 2922 and to the input of each of the corresponding calculation circuits 2415 of the four adjacent pixel blocks 2131.

[0360] In step S2340, the control unit 2740 causes the surrounding block calculation unit 2911 to calculate the average (Br, Bg, Bb) of the plurality of adjacent pixel blocks 2131 based on the arithmetic mean values ​​of each of the RGB pixels of the adjacent pixel blocks 2131 (this is referred to as the adjacent pixel block average). For example, the adjacent pixel block average is calculated by Bi=ΣAi / 4 (i=r, g, b) (where the number of adjacent pixel blocks 2131 is 4). In step S2350, the control unit 2740 causes the average calculation unit 2913 to perform a predetermined calculation on the other calculation result received from the other calculation circuit 2415 and the calculation result in the block calculation unit 2912. For example, the average calculation unit 2913 calculates the overall average (Cr, Cg, Cb) of the four adjacent pixel block average values ​​(Br, Bg, Bb) calculated in step S2340 and the additive average values ​​(Ar, Ag, Ab) of the pixel block 2131 calculated in step S2320. The overall average is calculated by Ci = (Bi + Ai) / 2 (i = r, g, b).

[0361] In step S2360, the control unit 2740 causes the average-average calculation unit 2914 to calculate the difference values ​​(ΔAr, ΔAg, ΔAb) between the arithmetic average values ​​(Ar, Ag, Ab) of the block calculated by the block calculation unit 2912 in step S2320 and the overall average values ​​(Cr, Cg, Cb) calculated by the average calculation unit 2913 in step S2350. The difference values ​​are calculated by ΔAi = (Ai - Ci) (i = r, g, b). In step S2370, the control unit 2740 causes the average-average calculation unit 2914 to input the difference values ​​(ΔAr, ΔAg, ΔAb) to the output circuit 2922. The calculation circuit 2415 may not have the average-average calculation section 2914 , and may input the calculation result in the average calculation section 2913 to the output circuit 2922 instead of the calculation result in the average-average calculation section 2914 .

[0362] In step S2380, the control unit 2740 causes the pixel-average calculation unit 2915 to calculate the difference values ​​(ΔCr, ΔCg, ΔCb) between the RGB pixel data of the pixel block acquired in step S2310 and the overall average values ​​(Cr, Cg, Cb) calculated by the average calculation unit 2913 in step S2350. The difference values ​​are calculated using ΔCi = (Ci - i pixel in the pixel block) (i = r, g, b). This makes it possible to preserve the information of the original pixel data using the small difference value and the average value. In other words, the pixel data of the corresponding pixel block 2131 can be compressed based on the calculation result of the average calculation unit 2913.

[0363] In step S2390, the control unit 2740 feeds back (ΔCr, ΔCg, ΔCb) to the pixel memory 2414 of the pixel block 2131. In step S2392, the control unit 2740 determines whether to continue the calculation, and in the former case, returns to step S2300, and in the latter case, ends the calculation process.

[0364] The control unit 2740 executes the above operation of the arithmetic circuit 2415 for each pixel block 2131. The arithmetic circuit 2415 may perform a predetermined operation on pixel data in the current frame using pixel data from a previous frame. In this case, the control unit 2740 may instruct the arithmetic circuit 2415 to use the average values ​​(Dr, Dg, Db) of each RGB pixel of the pixel block 2131 in question, for example, in the previous frame, instead of the average values ​​of each RGB pixel of the adjacent pixel block 2131. The average value of the previous frame is calculated by Di = Σ (i pixel in the pixel block of the previous frame) / (i number of pixels in the pixel block of the previous frame) (i = r, g, b). The control unit 2740 reads the RGB pixel data of the previous frame from the memory 2940 and causes the fourth arithmetic unit to calculate the average values ​​(Dr, Dg, Db). Other operations are the same as those in FIG. 58, and therefore will not be described here.

[0365] As described above, according to this example, the calculation results and calculation contents for each pixel block 2131, as well as control information for each pixel block 2131 from the control unit 2740, can be sent from the pixel block 2131 to the system control unit 2501 via the output circuit 2922. As a result, the image processing load on the system control unit 2501 can be significantly reduced. Furthermore, because the calculation circuit 2415 only needs to output the correlation value with pixel data of surrounding pixel blocks 2131 as the evaluation value of the pixel block 2131, the amount of data to be transmitted to the system control unit 2501 can be reduced. Furthermore, because the calculation circuit 2415 of this example feeds back the difference values ​​(ΔCr, ΔCg, ΔCb) to the pixel memory 2414 corresponding to the pixel block 2131, the amount of data transmitted to the system control unit 2501 can be reduced accordingly. Furthermore, the image processing unit 2511 included in the system control unit 2501 can generate one image data based on the calculation results received from each output circuit 2922, thereby improving the image processing speed compared to when RGB pixel data of all pixel blocks 2131 is stored in the memory 2940 and then read out to reconstruct one image. The signal processing chip 2111 in this example has at least a portion of the image processing functions of the image processing unit 2511. For example, the calculation circuit 2415 further functions as an image processing unit that performs image processing on image data of an image corresponding to a corresponding pixel signal based on each evaluation value. As an example, the image processing function may be a function that feeds back difference values ​​(ΔCr, ΔCg, ΔCb) to the pixel memory 2414. Examples of the evaluation value include the average of pixel signals within the pixel block 2131, a weighted average of pixel signals within and outside the pixel block 2131, the contrast within the pixel block 2131, a weighted average of the contrast within and outside the pixel block 2131, the luminance within the pixel block 2131, and a weighted average of the luminance within and outside the pixel block 2131. Furthermore, the evaluation value may be a value obtained by adding together the average of G pixels, the average of R pixels, and the average of B pixels at a predetermined ratio. Furthermore, the average value may be calculated as the average value of partial regions arranged within the unit group.

[0366] 59 shows an example of a data array 2950 generated by the output circuit 2922 based on input from the arithmetic circuit 2415. The data array 2950 has a data code area 2952 and a data area 2954. The data code area 2952 may be allocated 4 bits for data codes. In this example, D12 to D15 are allocated to the data codes. The data area 2954 may be allocated 12 bits for additional data corresponding to each data code. In this example, D0 to D11 are allocated to the data codes. The number of bits of the data array 2950 is not limited to 16 bits, and the number of bits to be allocated to the data codes and additional data can be set arbitrarily.

[0367] The control unit 2740 may output the calculation result data from the calculation circuit 2415 via a route different from that for the pixel data from the pixel memory 2414. For example, the control unit 2740 may transmit the calculation result of the calculation circuit 2415 to the system control unit 2501 via the output circuit 2922. The control unit 2740 may also store the pixel data of the pixel memory 2414 in the memory 2940 via the data transfer line 2920. In another example, the control unit 2740 may attach the calculation result of the pixel data of the pixel block 2131 to the pixel data of the pixel block 2131 and transmit them together from the output circuit 2922 to the system control unit 2501.

[0368] Note that, although an example of calculating the average of pixel values ​​has been described above, the calculation content in the calculation circuit 2415 is not limited to this. The parameters used in the calculation circuit 2415 may include information other than pixel values. For example, the calculation circuit 2415 may perform a predetermined calculation using parameters such as the position of the pixel on the XY plane, information about the distance to the subject, the aperture value, the charge accumulation time in the PD 2104, the charge-voltage conversion gain in the pixel block 2131, and the drive frame frequency (frame rate) in the pixel block 2131.

[0369] FIG. 60 shows an example of the contents of the data array 2950 shown in FIG. 59. The data code area 2952 stores 16 types of data codes (0 to 9, a to f). Data code 0 is assigned the R pixel average value (Ar) of the pixel block 2131 in question, and is output as 12-bit additional data. Data code 1 is assigned the G pixel average value (Ag) of the pixel block 2131 in question, and is output as 12-bit additional data. Data code 2 is assigned the B pixel average value (Ar) of the pixel block 2131 in question, and is output as 12-bit additional data. Data code 3 is assigned the difference ΔAr between the total average values ​​Cr and Ar, and is output as 12-bit additional data. Data code 4 is assigned the difference ΔAg between the total average values ​​Cg and Ag, and is output as 12-bit additional data. Data code 5 is assigned the difference ΔAb between the total average values ​​Cb and Ab, and is output as 12-bit additional data. The above is an example of the calculation content and calculation result data output by the calculation circuit 2415.

[0370] The data array 2950 also includes control information of the control unit 2740. In this example, the charge-voltage conversion gain of the pixel block 2131 is assigned to the data code d, and is output as 12-bit additional data. The drive frame frequency of the pixel block 2131 is assigned to the data code e, and is output as 12-bit additional data. The accumulation time of the pixel block 2131 is assigned to the data code f, and is output as 12-bit additional data. By adding the control information (control log) of the control unit 2740 to the data array 2950, ​​it is possible to send control information indicating how the control unit 2740 controlled each pixel block 2131 from the pixel block side to the system control unit 2501.

[0371] In other words, the system control unit 2501 can receive the data array 2950 illustrated in FIG. 59 for each pixel block 2131. Therefore, by accessing the differential data for each RGB pixel of the pixel block 2131 stored in the memory 2940 and processing the read data based on the data codes in the data array 2950, ​​image processing for each pixel block 2131 can be easily performed. In other words, since part of the processing in the system control unit 2501 is performed by the arithmetic circuit 2415, the pixel data processing load on the system control unit 2501 for generating moving images can be significantly reduced. Furthermore, the system control unit 2501 can effectively utilize the contents of the data array 2950 output by the output circuit 2922 while reducing the load on the system control unit 2501 itself. For example, the system control unit 2501 can generate moving images by changing the compression rate for each pixel block 2131 based on the contents of the data array 2950.

[0372] 61 is a cross-sectional view of another image sensor 3100 according to this embodiment. The image sensor 3100 includes an image sensor chip 3113 that outputs pixel signals corresponding to incident light, a signal processing chip 3111 that processes the pixel signals, and a memory chip 3112 that stores the pixel signals. The image sensor chip 3113, signal processing chip 3111, and memory chip 3112 are stacked and electrically connected to each other by conductive bumps 3109 made of Cu or the like.

[0373] As shown in the figure, incident light is mainly incident in the positive direction of the Z axis, as indicated by the white arrow. In this specification, the surface of the imaging chip 3113 on which incident light is incident is referred to as the back surface. Also, as shown by the coordinate axes, the left direction on the paper, perpendicular to the Z axis, is the positive X axis, and the front direction on the paper, perpendicular to the Z axis and the X axis, is the positive Y axis. In the following figures, the coordinate axes are displayed based on the coordinate axes in Figure 61 so that the orientation of each figure can be understood.

[0374] An example of the imaging chip 3113 is a back-illuminated MOS image sensor. The PD layer 3106 is arranged on the back side of the wiring layer 3108. The PD layer 3106 has a plurality of PDs (photodiodes) 104 arranged two-dimensionally, which accumulate charges according to incident light and generate pixel signals according to the accumulated charges, and transistors 3105 provided corresponding to the PDs 3104.

[0375] A color filter 3102 is provided on the incident light side of the PD layer 3106 via a passivation film 3103. There are multiple types of color filters 3102 that transmit different wavelength ranges, and each has a specific arrangement corresponding to the PD 3104. The arrangement of the color filters 3102 will be described later. A set of the color filter 3102, the PD 3104, and the transistor 3105 forms one pixel.

[0376] A microlens 3101 is provided for each pixel on the incident light side of the color filter 3102. The microlens 3101 condenses the incident light toward the corresponding PD 3104.

[0377] The wiring layer 3108 has wiring 3107 that transmits pixel signals from the PD layer 3106 to the signal processing chip 3111. The wiring 3107 may be multi-layered, and may be provided with passive elements and active elements.

[0378] A plurality of bumps 3109 are arranged on the surface of the wiring layer 3108. The plurality of bumps 3109 are aligned with a plurality of bumps 3109 provided on the opposing surface of the signal processing chip 3111, and the imaging chip 3113 and the signal processing chip 3111 are pressed together, whereby the aligned bumps 3109 are bonded together and electrically connected.

[0379] Similarly, a plurality of bumps 3109 are arranged on the opposing surfaces of the signal processing chip 3111 and the memory chip 3112. These bumps 3109 are aligned with each other, and the signal processing chip 3111 and the memory chip 3112 are pressed together, etc., so that the aligned bumps 3109 are bonded and electrically connected to each other.

[0380] The bonding between the bumps 3109 is not limited to Cu bump bonding by solid-phase diffusion, and micro-bump bonding by solder melting may also be used. Furthermore, it is sufficient to provide approximately one bump 3109 for each pixel block, as described below. Therefore, the size of the bumps 3109 may be larger than the pitch of the PDs 3104. Furthermore, in a peripheral region other than the pixel region where the pixels are arranged, bumps larger than the bumps 3109 corresponding to the pixel region may also be provided.

[0381] The signal processing chip 3111 has TSVs (through silicon vias) 110 that connect the circuits provided on the front and back surfaces to each other. The TSVs 3110 are preferably provided in the peripheral region. The TSVs 3110 may also be provided in the peripheral region of the imaging chip 3113 and the memory chip 3112.

[0382] FIG. 62 is a diagram illustrating the pixel arrangement of the imaging chip 3113 and the pixel block 3131. FIG. 62 shows the imaging chip 3113 as viewed from the back side. A plurality of pixels are arranged in a matrix in the pixel region 3700. In FIG. 62, adjacent 4 pixels x 4 pixels, or 16 pixels, form one pixel block 3131. The grid lines in the figure show the concept of adjacent pixels being grouped to form the pixel block 3131. The number of pixels forming the pixel block 3131 is not limited to this and may be around 1000, for example, 32 pixels x 64 pixels, or it may be more or less than that.

[0383] As shown in the partially enlarged view of pixel region 3700, pixel block 3131 contains four pixels arranged vertically and horizontally in a so-called Bayer array, each consisting of green pixels Gb, Gr, blue pixels B, and red pixels R. The green pixels are pixels that have a green filter as their color filter 3102 and receive light in the green wavelength band of incident light. Similarly, the blue pixels are pixels that have a blue filter as their color filter 3102 and receive light in the blue wavelength band, and the red pixels are pixels that have a red filter as their color filter 3102 and receive light in the red wavelength band.

[0384] In this embodiment, at least one pixel block is selected from the multiple pixel blocks 3131, and the pixels included in each pixel block are controlled using control parameters different from those of the other pixel blocks. Examples of the control parameters include the frame rate, the thinning rate, the number of rows for adding pixel signals, the charge accumulation time or number of accumulations, and the number of digitization bits. Furthermore, the control parameters may be parameters for image processing after image signals are acquired from the pixels. The frame rate refers to the cycle at which pixel signals are generated. Note that in this specification, the frame rate may refer to the frame rate for each pixel block 3131. For example, the reference frame rate and the high-speed frame rate refer to the frame rates for each pixel block 3131.

[0385] 63 is a circuit diagram corresponding to a pixel block 3131 of an imaging chip 3113. In the figure, a rectangle surrounded by a dotted line typically represents a circuit corresponding to one pixel. Note that at least a portion of the transistors described below correspond to the transistor 3105 in FIG.

[0386] 63 shows a pixel block 3131 formed of 16 pixels, but the number of pixels in the pixel block 3131 is not limited to this. The 16 PDs 3104 corresponding to the respective pixels are connected to transfer transistors 3302, and each gate of each transfer transistor 3302 is connected to a TX wiring 3307 that supplies a transfer pulse. In the example shown in FIG. 63, the TX wiring 3307 is commonly connected to the 16 transfer transistors 3302.

[0387] The drain of each transfer transistor 3302 is connected to the source of the corresponding reset transistor 3303, and a so-called floating diffusion FD between the drain of the transfer transistor 3302 and the source of the reset transistor 3303 is connected to the gate of the amplification transistor 3304. The drain of the reset transistor 3303 is connected to a Vdd wiring 3310 to which a power supply voltage is supplied, and the gate of the reset transistor 3303 is connected to a reset wiring 3306 to which a reset pulse is supplied. In the example shown in FIG. 63, the reset wiring 3306 is commonly connected to 16 reset transistors 3303.

[0388] The drain of each amplification transistor 3304 is connected to a Vdd wiring 3310 to which a power supply voltage is supplied. The source of each amplification transistor 3304 is connected to the drain of a corresponding selection transistor 3305. The gate of each selection transistor is connected to a decoder wiring 3308 to which a selection pulse is supplied. In the example shown in FIG. 63, the decoder wiring 3308 is provided independently for each of the 16 selection transistors 3305. The sources of each selection transistor 3305 are connected to a common output wiring 3309. A load current source 3311 supplies a current to the output wiring 3309. In other words, the output wiring 3309 for the selection transistor 3305 is formed by a source follower. The load current source 3311 may be provided on the imaging chip 3113 side or on the signal processing chip 3111 side.

[0389] Here, the flow from the start of charge accumulation to pixel output after accumulation is completed will be described. When a reset pulse is applied to the reset transistor 3303 via the reset wiring 3306 and at the same time a transfer pulse is applied to the transfer transistor 3302 via the TX wiring 3307, the potentials of the PD 3104 and the floating diffusion FD are reset.

[0390] When the application of the transfer pulse is stopped, the PD 3104 converts the incident light it receives into electric charges and accumulates them. Then, when the transfer pulse is applied again without the reset pulse being applied, the accumulated electric charges are transferred to the floating diffusion FD, and the potential of the floating diffusion FD changes from the reset potential to the signal potential after charge accumulation. Then, when a selection pulse is applied to the selection transistor 3305 via the decoder wiring 3308, the fluctuation in the signal potential of the floating diffusion FD is transmitted to the output wiring 3309 via the amplification transistor 3304 and the selection transistor 3305. As a result, a pixel signal corresponding to the reset potential and the signal potential is output from the unit pixel to the output wiring 3309.

[0391] In the example shown in FIG. 63, the reset wiring 3306 and TX wiring 3307 are common to the 16 pixels that make up the pixel block 3131. That is, the reset pulse and transfer pulse are each applied simultaneously to all 16 pixels. Therefore, all of the pixels that make up the pixel block 3131 start and end charge accumulation at the same timing. However, pixel signals corresponding to the accumulated charges are sequentially applied to the respective selection transistors 3305 by selection pulses, and are selectively output to the output wiring 3309. Furthermore, the reset wiring 3306, TX wiring 3307, and output wiring 3309 are provided separately for each pixel block 3131.

[0392] In this way, by configuring the circuit based on the pixel block 3131, it is possible to control the charge accumulation time for each pixel block 3131. In other words, it is possible to cause adjacent pixel blocks 3131 to output pixel signals with different charge accumulation times. Furthermore, by causing one pixel block 3131 to perform a single charge accumulation while the other pixel block 3131 is caused to repeatedly accumulate charge and output a pixel signal each time, it is also possible for these pixel blocks 3131 to output frames for a moving image at different frame rates.

[0393] 64A shows part of the configuration of the image sensor 3100 and an example of its operation. The image sensor 3100 of this example further includes a storage unit 3114 in addition to the configuration shown in FIG. 61. The storage unit 3114 may be provided in the signal processing chip 3111. In this case, the image sensor 3100 does not need to include the memory chip 3112. The storage unit 3114 may also be provided in the memory chip 3112.

[0394] The imaging chip 3113 has a pixel region 3700 in which a plurality of pixels are arranged, each of which generates a pixel signal in response to incident light. Here, the pixel region 3700 may be configured by arranging a plurality of pixels two-dimensionally. Each pixel block 3131 has m×n pixels in the row and column directions, where m and n are integers of 2 or greater. The pixel region 3700 has a plurality of pixel blocks 3131 divided in the row and column directions. As shown in FIG. 62, the pixel block 3131 refers to a pixel group in which a plurality of pixels are arranged in a matrix. Furthermore, the row direction and the column direction refer to two different directions within the plane of the pixel region 3700 and do not necessarily need to be perpendicular to each other.

[0395] For ease of explanation, Figures 64A to 64C show three pixel blocks 3131 in each row and column direction, but the number of pixel blocks 3131 included in pixel region 3700 may be greater. It is preferable that the number of pixels included in each pixel block 3131 is equal. It is also preferable that the number of pixels included in each pixel block 3131 in pixel region 3700 is fixed. The pixel block 3131 is composed of, for example, 32 x 64 pixels.

[0396] The signal processing chip 3111 of this example has a multiplexer 3411, an A / D converter 3412, a demultiplexer 3413, and a control unit 3740 for each pixel block 3131. The multiplexer 3411 sequentially selects pixels included in the corresponding pixel block 3131 and inputs pixel signals corresponding to the selected pixels to the A / D converter 3412. The A / D converter 3412 converts the analog pixel signals into digital pixel data and inputs it to the demultiplexer 3413. The demultiplexer 3413 stores the pixel data in a memory area corresponding to the pixel in the corresponding memory block 3730. Each memory block 3730 passes the stored pixel data to a subsequent calculation circuit.

[0397] The memory unit 3114 has a plurality of memory blocks 3730 provided corresponding to a plurality of pixel blocks 3131, each capable of storing pixel data of the corresponding pixel block 3131. The memory blocks 3730 correspond one-to-one to the pixel blocks 3131. The memory blocks 3730 may be connected to the corresponding pixel blocks 3131 via a bus 3720. The memory blocks 3730 may be buffer memories.

[0398] Furthermore, at least some of the memory blocks 3730 can also store pixel data of pixel blocks other than the corresponding pixel block 3131. That is, one memory block 3730 may be shared by multiple pixel blocks 3131. In other words, the control unit 3740 can store pixel data of one pixel block 3131 in multiple memory blocks 3730. By sharing the memory blocks 3730, the multiple memory blocks 3730 can be used efficiently, as will be described later, and the overall memory capacity of the memory unit 3114 can be reduced.

[0399] It is preferable that, for all pixel blocks 3131, pixel data can be read from and written to at least one other memory block 3730 in addition to the corresponding memory block 3730. The other memory block 3730 may be predetermined for each pixel block 3131, or may be dynamically changeable. It is also preferable that, for all memory blocks 3730, pixel data can be read from and written to at least one other pixel block 3131 in addition to the corresponding pixel block 3131. The other pixel block 3131 may be predetermined for each memory block 3730, or may be dynamically changeable.

[0400] Each memory block 3730 may be a memory provided for each pixel block 3131 in an area overlapping the corresponding pixel block 3131 in the signal processing chip 3111. That is, the memory block 3730 may be provided in an area directly below the corresponding pixel block 3131 in the signal processing chip 3111. In this case, the pixel block 3131 and the memory block 3730 may be electrically connected by a TSV. Also, the corresponding memory block 3730 and A / D converter 3412, etc. are provided in an area overlapping with each pixel block 3131 in the signal processing chip 3111. Also, each memory block 3730 may be a memory provided outside the area overlapping with the pixel area 3700 in the signal processing chip 3111.

[0401] Furthermore, when each memory block 3730 and A / D converter 3412 is provided in an area overlapping with the corresponding pixel block 3131, and when each memory block 3730 stores pixel data of a pixel block 3131 other than the corresponding pixel block 3131, it may transmit an analog pixel signal or digital pixel data to the area where the memory block 3730 is provided. In the former case, the A / D converter 3412 corresponding to the memory block 3730 converts the pixel signal into pixel data and inputs it to the memory block 3730. In the latter case, the A / D converter 3412 in the area overlapping with the pixel block 3131 converts the pixel signal into pixel data, and then transmits the pixel data to the memory block 3730 where it is to be stored. The signal processing chip 3111 is provided with wiring for transmitting these pixel signals or pixel data.

[0402] FIG. 64B shows another example of the operation of the image sensor 3100. Note that the configuration of the signal processing chip 3111 shown in FIG. 64A is omitted from FIG. 64B. In this example, pixel data of pixel block 3712, among multiple pixel blocks 3131, is stored in one of storage blocks 3731, 732, or 733 other than the corresponding storage block 3734. In this example, the analog pixel signal generated by pixel block 3712 is converted into digital pixel data by A / D converters 3412 corresponding to the other storage blocks 3731 to 733. In this way, by making it possible to store pixel data of any one of pixel block 3712 in multiple storage blocks 3731 to 734, memory usage efficiency can be improved.

[0403] For example, the multiple pixel blocks 3131 may generate pixel signals of an object captured at a different frame rate for each pixel block 3131, at timing corresponding to the frame rate. As will be described later, the control unit 3740 selects a corresponding storage block 3730 for each pixel block 3131 from at least two frame rates: a base frame rate and a high-speed frame rate having a shorter period than the base frame rate. The period of the high-speed frame rate may be an integer multiple of the period of the base frame rate. Each pixel block 3131 may output one block of pixel signals for each frame rate period.

[0404] In this example, the frame rate of pixel block 3712 is five times the reference frame rate. At approximately the same time that pixel block 3131 with the reference frame rate outputs a pixel signal, pixel block 3712 with the high frame rate also outputs a pixel signal. In this case, pixel block 3712 outputs a pixel signal four times before pixel block 3131 outputs the next pixel signal.

[0405] When the pixel block 3131 operating at the reference frame rate is not outputting a pixel signal, the control unit 3740 causes the multiple storage blocks 3731-3734 to respectively store pixel data corresponding to the pixel signals output four times by the pixel block 3712 operating at the high frame rate. Note that one frame's worth of pixel data corresponding to the pixel signals output by each pixel block in synchronization with the reference frame rate may be stored in a memory different from the multiple storage blocks 3730, or may be transferred to a memory or circuit downstream of the storage block 3730 after being temporarily stored in the multiple storage blocks 3730, before the next pixel data of the pixel block 3712 operating at the high frame rate is input to the multiple storage blocks 3730. This allows the multiple storage blocks to be used efficiently.

[0406] When pixel data is already stored in storage block 3734 corresponding to pixel block 3712 with a high frame rate, control unit 3740 stores the pixel data corresponding to pixel block 3712 in one of storage blocks 3731, 732, or 733 where no pixel data is stored. In other words, control unit 3740 allocates and stores the pixel data of pixel block 3712 with a high frame rate to storage blocks 3731, 732, and 733 other than corresponding storage block 3734 where no pixel data is stored. At this time, the allocated pixel data may include, as additional data, position data of corresponding pixel block 3712 in pixel region 3700 and frame data indicating the frame to which the pixel data belongs. The position of storage block 3730 to which pixel data is allocated may be fixed for each pixel block 3712 or may change dynamically. When the position of storage block 3730 to which pixel data is allocated is fixed for each pixel block 3712, the position data can be omitted from the additional data.

[0407] FIG. 64C shows another example of the operation of the image sensor 3100. Note that the configuration of the signal processing chip 3111 shown in FIG. 64A is omitted in FIG. 64C. In this example, as in the example of FIG. 64B, pixel data of pixel block 3712 is stored in one of memory blocks 3735-3737 other than the corresponding memory block 3734. However, in this example, the pixel signal is converted into pixel data in the A / D converter 3412 in the area overlapping with pixel block 3712, and then the pixel data is transmitted to the memory block in which it should be stored. In this example, the pixel data moves between memory blocks.

[0408] 64B in that, when pixel data is already stored in memory block 3734 corresponding to pixel block 3712 with a high frame rate, control unit 3740 of this example moves the pixel data from memory block 3734 to memory blocks 3735, 3736, 3737, and 3738 that do not store pixel data, and stores the data in the respective memory blocks. That is, in this example, memory unit 3114 has memory blocks connected to each other by wiring so that data can be sent and received between the memory blocks.

[0409] The control unit 3740 moves and stores the pixel data in storage block 3734 in one of storage blocks 3735, 3736, 3737, and 3738, which do not store pixel data. Preferably, the control unit 3740 moves and stores the pixel data in storage block 3734 toward the storage block corresponding to pixel block 3131 on the outermost periphery of pixel region 3700. Since the frame rate of pixel block 3131 often decreases as one moves away from pixel block 3712, which has a high frame rate, toward the periphery of pixel region 3700, it is preferable for the control unit 3740 to two-dimensionally distribute the pixel data toward the periphery. By doing so, the multiple storage blocks 3730 can be used evenly, without increasing the capacity of the buffer memory, thereby reducing the overall memory capacity of storage unit 3114. Note that the control unit 3740 may select a storage block 3730 corresponding to a pixel block 3131 other than the outermost pixel block 3131 based on the frame rate information of each pixel block 3131, and write the pixel data therein.

[0410] In this example, too, the locations of the storage blocks into which pixel data is to be distributed may be fixed or may change dynamically. If the locations of the storage blocks into which pixel data is to be distributed are fixed, the location data may be omitted from the additional data to be added to the pixel data being moved. In this case, it is more preferable that the storage blocks into which pixel data is to be distributed are storage blocks corresponding to the pixel blocks 3131 on the outermost periphery of the pixel region 3700. Furthermore, the pixel data stored in each storage block 3730 may be moved sequentially in synchronization with a high-speed frame rate. This allows the pixel data to be transmitted between spaced-apart storage blocks 3730. By repeating this movement of pixel data, the pixel data can be moved to any storage block 3730.

[0411] The arithmetic circuit 3415, which will be described later, processes the pixel data stored in the memory block 3730 and passes it on to a downstream image processing unit. The arithmetic circuit 3415 may be provided in the signal processing chip 3111 or in the memory unit 3114. Note that while the diagram shows connections for one pixel block 3131, in reality, these exist for each pixel block 3131 and operate in parallel. However, a arithmetic circuit 3415 does not have to exist for each pixel block 3131; for example, one arithmetic circuit may perform sequential processing while referring to the values ​​of the memory blocks 3730 corresponding to each pixel block 3131 in order.

[0412] As described above, output wiring 3309 is provided corresponding to each pixel block 3131. Since the imaging element 3100 has an imaging chip 3113, a signal processing chip 3111, and a memory unit 3114 stacked on top of each other, by using bumps 3109 for electrical connection between the chips for these output wiring 3309, it is possible to route the wiring without increasing the size of each chip in the planar direction.

[0413] The control unit 3740 is provided with rate information regarding the frame rate of each pixel block 3131. Based on the rate information, the control unit 3740 selects a storage block 3730 in which to store pixel data of the pixel block 3131 having a high frame rate. For example, the control unit 3740 may select the storage block 3730 corresponding to the pixel block 3131 having the reference frame rate as the storage block 3730 in which to store the pixel data. The control unit 3740 may also determine a route for moving pixel data in the form shown in FIG. 64C based on the rate information. For example, when moving pixel data of each storage block 3730, the control unit 3740 selects a storage block 3730 that is adjacent to the storage block 3730 and that corresponds to the reference frame rate, and that is located at an increasing distance from the storage block 3730 corresponding to the high frame rate.

[0414] 65 is a block diagram showing the configuration of an imaging device according to this embodiment. The imaging device 3500 includes a photographing lens 3520 as an imaging optical system, and the photographing lens 3520 guides a subject light beam incident along an optical axis OA to the image sensor 3100. The photographing lens 3520 may be an interchangeable lens that can be attached to and detached from the imaging device 3500. The imaging device 3500 mainly includes the image sensor 3100, a system control unit 3501, a drive unit 3502, a photometry unit 3503, a work memory 3504, a recording unit 3505, and a display unit 3506.

[0415] The photographing lens 3520 is composed of a group of multiple optical lenses and forms an image of a subject light beam from a scene near its focal plane. Note that in FIG. 61, the photographing lens 3520 is represented by a virtual single lens placed near the pupil. The driver 3502 is a control circuit that performs charge accumulation control such as timing control and area control of the image sensor 3100 in accordance with instructions from the system controller 3501. In this sense, the driver 3502 can be said to function as an image sensor controller that causes the image sensor 3100 to perform charge accumulation and output pixel signals.

[0416] The image sensor 3100 passes pixel signals to an image processing unit 3511 in the system control unit 3501. The image processing unit 3511 performs various image processes using a work memory 3504 as a workspace to generate image data. For example, when generating image data in JPEG file format, a color video signal is generated from a signal obtained using the Bayer array, and then compression processing is performed. The generated image data is recorded in a recording unit 3505 and converted into a display signal, which is displayed on a display unit 3506 for a preset time.

[0417] The photometry unit 3503 detects the luminance distribution of a scene prior to a series of shooting sequences for generating image data. The photometry unit 3503 includes, for example, an AE sensor with approximately one million pixels. The calculation unit 3512 of the system control unit 3501 receives the output of the photometry unit 3503 and calculates the luminance of each area of ​​the scene. The calculation unit 3512 determines the shutter speed, aperture value, and ISO sensitivity according to the calculated luminance distribution. The image sensor 3100 may also function as the photometry unit 3503. The calculation unit 3512 also performs various calculations for operating the image capture device 3500.

[0418] The drive unit 3502 may be partly or entirely mounted on the imaging chip 3113, or partly or entirely mounted on the signal processing chip 3111. A part of the system control unit 3501 may be mounted on the imaging chip 3113 or the signal processing chip 3111.

[0419] 66 is a functional block diagram of the image processing unit. Image processing unit 3511 in this example extracts pixel blocks 3131 (peripheral regions 3176 described below) that operate at the reference frame rate and pixel blocks 3131 (regions of interest 3172 described below) that operate at a high frame rate. In addition to the above functions, image processing unit 3511 also has a subject estimation unit 3150, a group selection unit 3152, a video generation unit 3154, and a video synthesis unit 3156. Each of these functions will be described later.

[0420] Fig. 67 is a flowchart showing the operation of an imaging device to generate and record a moving image. Fig. 68 and Fig. 69 show examples of images captured by an imaging element. Fig. 70 shows the relationship between each frame rate and the output timing of an image signal.

[0421] 67 starts when the user instructs the image capturing device 3500 to generate a video by pressing the record button, etc. First, the subject estimation unit 3150 drives the drive unit 3502 to acquire image data based on the image signal from the image capturing element 3100, and estimates the main subject included in the image represented by the image data (S3100).

[0422] In this case, it is preferable that the driving unit 3502 output image signals from pixel blocks 3131 included in the entire imaging area, for example, from all pixel blocks 3131. The driving unit 3502 may also output image signals from all pixels included in each pixel block 3131, or may output image signals from pixels thinned out at a predetermined thinning rate. The subject estimation unit 3150 compares multiple images obtained in time series from the image sensor 3100 and identifies a moving subject as the main subject. Note that other methods may be used to estimate the main subject.

[0423] For example, when image 3170 in Fig. 68 and image 3178 in Fig. 69 are acquired from image sensor 3100 as images taken before and after in time, subject estimation section 3150 uses the difference between these images to identify a child as main subject 3171. Note that, although grid lines in images 3170 and 3178 indicate boundaries of pixel blocks 3131, the number of pixel blocks 3131 is merely an example and is not limited to the number shown in these figures.

[0424] The group selection unit 3152 selects at least one pixel block 3131 onto which image light of the main subject 3171 estimated by the subject estimation unit 3150 is incident (S3102). For example, in the image 3170, a pixel block 3131 that includes at least a portion of the main subject 3171 is selected. Furthermore, taking into consideration that the main subject 3171 moves within the imaging area, it is preferable that the group selection unit 3152 also selects pixel blocks 3131 further surrounding the pixel block 3131 that includes at least a portion of the main subject 3171.

[0425] The group selection unit 3152 designates a collection of these selected pixel blocks 3131 as a region of interest 3172. Furthermore, the group selection unit 3152 designates a collection of pixel blocks 3131 in the entire imaging area that are not included in the region of interest 3172 as a peripheral region 3176. The group selection unit 3152 specifies region information 3174 that indicates the range of the region of interest 3172 relative to the entire imaging area.

[0426] In the example shown in Fig. 68, the region of interest 3172 is a rectangular region made up of 28 pixel blocks 3131, 7 horizontally and 4 vertically. In contrast, the peripheral region 3176 is made up of 98 pixel blocks 3131, excluding the region of interest 3172, from the 126 pixel blocks of the imaging region, 21 horizontally and 6 vertically. Furthermore, as region information 3174, the position (9,2) counted from the left and top of the pixel block 3131 at the top left corner of the region of interest 3172 in the imaging region in the figure is specified. Furthermore, as size information, the number of vertical and horizontal dimensions of the region of interest 3172 is specified as 7 x 4.

[0427] The group selection unit 3152 transmits information specifying pixel blocks 3131 included in the region of interest 3172 and information specifying the peripheral region 3176 to the drive unit 3502. In this case, information on the frame rates to be applied to the region of interest 3172 and the peripheral region 3176 is also transmitted. Here, it is preferable that the frame rate applied to the region of interest 3172 is higher than the frame rate to be applied to the peripheral region 3176. For example, if the frame rate to be applied to the peripheral region 3176 is 60 fps, the frame rate to be applied to the region of interest 3172 is set to 180 fps. It is preferable that these frame rate values ​​are set in advance and stored so that the group selection unit 3152 can refer to them, but the values ​​may also be changeable by the user later.

[0428] The driver 3502 drives the image sensor 3100 to capture images at each frame rate (S3104). That is, the driver 3502 causes the pixel blocks 3131 included in the region of interest 3172 to accumulate charge and output image signals at a high frame rate, and causes the pixel blocks 3131 included in the peripheral region 3176 to accumulate charge and output image signals at a low frame rate. In other words, the driver 3502 obtains image signals corresponding to multiple frames arranged in time series for the pixel blocks 3131 included in the region of interest 3172, while obtaining an image signal corresponding to one frame for the pixel blocks 3131 included in the peripheral region 3176.

[0429] 70, when the frame rate of the peripheral region 3176 is set to 60 fps and the frame rate of the region of interest 3172 is set to 180 fps, the driver 3502 obtains image signals of three frames A1, A2, and A3 from the region of interest 3172 (1 / 60 s = 3 × 1 / 180 s) during the 1 / 60 s it takes to obtain an image signal of one frame B1 from the peripheral region 3176. In this case, the driver 3502 separately drives the set of the reset transistor 3303, transfer transistor 3302, and selection transistor 3305 of the pixel block 3131 included in the peripheral region 3176 and the set of the reset transistor 3303, transfer transistor 3302, and selection transistor 3305 of the pixel block 3131 included in the region of interest 3172, thereby obtaining image signals at different frame rates.

[0430] 70 shows the timing of outputting the image signal, but does not show the length of the exposure time. The driving unit 3502 drives the above-mentioned sets of transistors for the peripheral region 3176 and the region of interest 3172 so as to achieve the exposure time calculated in advance by the calculation unit 3512.

[0431] In addition, the length of the exposure time may be changed according to the frame rate. For example, in the example shown in FIG. 70, the exposure time of one frame of the peripheral region 3176 may be set to 1 / 3, so that it is substantially the same exposure time as that of the region of interest 3172. Furthermore, after outputting the image signal, the image signal may be corrected based on the frame rate ratio. Furthermore, the timing of outputting the image signal between the peripheral region 3176 and the region of interest 3172 may not be synchronized as shown in FIG. 70, but may be asynchronous.

[0432] The image processing unit 3511 stores the image signals from the region of interest 3172 sequentially for each frame in a predetermined storage area of ​​the work memory 3504 (S3106). Similarly, the image processing unit 3511 stores the image signals from the peripheral region 3176 sequentially for each frame in a predetermined storage area of ​​the work memory 3504 (same step). The work memory 3504 has a plurality of storage blocks 3730, as described with reference to FIGS. 64A to 64C. The work memory 3504 may be a memory made up of memory groups corresponding to the respective pixel blocks 3131.

[0433] The video generation unit 3154 reads out the image signal of the attention area 3172 stored in the work memory 3504 (S3108), and generates data of the attention area video including multiple frames of the attention area 3172 (S3110). Similarly, the video generation unit 3154 reads out the image signal of the surrounding area 3176 stored in the work memory 3504, and generates data of the surrounding area video including multiple frames of the surrounding area 3176 (same step). Here, the attention area video and the surrounding area video may each be generated in a general-purpose format such as MPEG so that they can be played separately, or may be generated in a dedicated format that cannot be played without undergoing a synthesis process described below.

[0434] 71 schematically shows the attention area moving image and the surrounding area moving image generated by the moving image generation unit. The moving image generation unit 3154 generates the attention area moving image at a frame rate corresponding to the frame rate at which the driving unit 3502 drives the attention area 3172. In the example shown in FIG. 71, the attention area moving image is generated at a frame rate of 1 / 180 fps, which is the same as the frame rate of 1 / 180 fps at which the driving unit 3502 drives the attention area 3172.

[0435] Similarly, the video generation unit 3154 generates the surrounding area video at a frame rate corresponding to the frame rate at which the drive unit 3502 drives the surrounding area 3176. In the example shown in Fig. 71, the surrounding area video is generated at a frame rate of 1 / 60 fps, which is the same as the frame rate of 1 / 60 fps at which the drive unit 3502 drives the surrounding area 3176. Note that the area in the surrounding area video that corresponds to the attention area 3172 does not have a valid value, and is indicated by diagonal lines in the figure.

[0436] Furthermore, the video generation unit 3154 adds header information to the attention area video and the peripheral area video, and records this data in the recording unit 3505 (S3112). The header information includes area information indicating the position of the attention area 3172 relative to the entire imaging area, size information indicating the size of the attention area 3172, and timing information indicating the relationship between the output timing of the image signal of the attention area 3172 and the output timing of the image signal of the peripheral area 3176.

[0437] The system control unit 3501 determines whether or not to capture the next unit of time (S3114). Whether or not to capture the next unit of time is determined based on whether or not the user has pressed the video record button at that point. If to capture the next unit of time (S3114: Yes), the process returns to step S3102, and if not to capture the next unit of time (S3114: No), the process ends.

[0438] Here, the "unit time" is a time of about several seconds that is set in advance in the system control unit 3501. The memory capacity used for storage in step S3106 is determined by this unit time, the frame rate and number of pixel blocks of the region of interest 3172, and the frame rate and number of pixel blocks of the peripheral region 3176. Furthermore, based on this information, an area in the memory capacity for storing data of the region of interest 3172 and an area for storing data of the peripheral region 3176 are determined.

[0439] As a result, it is possible to obtain image signals at a high frame rate from the region of interest 3172 that includes the main subject 3171, while reducing the amount of data by keeping the frame rate low for the peripheral region 3176. Therefore, compared to high-speed readout from all pixels, it is possible to reduce the load on driving and image processing, and suppress power consumption and heat generation.

[0440] In the example shown in Fig. 67, when the next unit time starts, pixel block 3131 is selected anew in step S3102, and the region information and size information are updated. This makes it possible to sequentially update region of interest 3172 in accordance with main subject 3171. In the example shown in Fig. 71, in the first frame A7 of the unit time in the region of interest moving image, region of interest 3182 consisting of pixel block 3131 different from that in the last frame A6 of the previous unit time is selected, and region information 3184 and surrounding region 3186 are also updated accordingly.

[0441] Fig. 72 shows an example of header information added by the video generation unit. The header information in Fig. 72 includes an attention area video ID that identifies the attention area video, the frame rate of the attention area video, a surrounding area video ID that identifies the surrounding area video corresponding to the attention area video, the frame rate of the surrounding area video, timing information, area information, and size information. This header information may be added as header information to either the attention area video or the surrounding area video, or to both.

[0442] 73 is a flowchart showing the operation of the imaging device to play and display a moving image. This operation starts when the user selects one of the area-of-interest moving images displayed as thumbnails on the display unit 3506 and presses the play button.

[0443] The video composition unit 3156 reads data of the attention area video specified by the user (S3150) from the storage unit 3505. The video composition unit 3156 reads data of the surrounding area video corresponding to the attention area video from the storage unit 3505 (S3152).

[0444] In this case, the video composition unit 3156 identifies the surrounding area video by the surrounding area video ID indicated in the header information of the attention area video read in step S150. Alternatively, a surrounding area image including, as header information, the same timing information as the timing information indicated in the header information may be searched for and identified.

[0445] In the above example, it is assumed that the attention area video contains header information. On the other hand, if the attention area video does not contain header information but the surrounding area video does, the surrounding area video may be first read by having the user specify it in step S150, and the attention area video may then be specified and read from the header information in step S152.

[0446] The moving image composition unit 3156 combines frames of the display moving image using frames of the attention area moving image and frames of the surrounding area moving image (S3154). In this case, first, the first frame A1 of the attention area moving image is inserted into the position indicated by the area information 3174 in the first frame B1 of the surrounding area moving image, thereby combining the first frame C1 of the display moving image. As shown in FIG. 71, the moving image composition unit 3156 displays the first frame C1 of the display moving image on the display unit 3506 (S3156).

[0447] The video composition unit 3156 determines whether the next frame of the attention area video is present before the next frame B2 of the surrounding area video (S3158). If the next frame of the attention area video is present (S3158: Yes), the video composition unit 3156 updates the attention area 3172 with the next frames A2 and A3 and maintains the surrounding area 3176 in the previous frame B1 (S3162), thereby combining the next frames C2 and C3 of the displayed video (S162) and displaying them sequentially (S3156).

[0448] On the other hand, if in step S3158 there is no next frame of the attention area video until the next frame B2 in the surrounding area video (S3158), the video synthesis unit 3156 updates the attention area 3172 with the next frame A4 and also updates the surrounding area 3176 with the next frame B2 (S3164), thereby synthesizing (S3162) and displaying (S3156) the next frame C4 of the displayed video.

[0449] As long as there is a next frame of peripheral area 3176 in the peripheral area video (S3160: Yes), steps S154 to S3160 are repeated. If there is no next frame of peripheral area 3176 in the peripheral area video (S3160: No), the video composition unit 3156 searches for whether there is a pair of attention area video and peripheral area video in the unit time next to the unit time of the pair of attention area video and peripheral area video (S3166). For example, the video composition unit 3156 searches for whether there is an attention area video in the same folder in the recording unit 3505, the header information of which includes timing information indicating a timing immediately after the timing indicated by the timing information of the attention area video.

[0450] As long as there is a pair of attention area video and peripheral area video for the next unit time (S3166: Yes), steps S150 to S3166 are repeated. If there is no pair of attention area video and peripheral area video for the next unit time (S3166: No), the operation ends.

[0451] As described above, it is possible to display a smooth moving image of the region of interest 3172 containing the main subject 3171 while reducing the overall amount of data. In step S162, the region of interest 3172 is updated with the next frame as is, and the frames of the displayed image are synthesized. However, this synthesis method is not limited to this. As another example, the boundary line of the main subject 3171 in the region of interest 3172 may be identified by image processing, and the main subject 3171 enclosed by this boundary line may be updated to the next frame, while the previous frame may be maintained for the area outside the boundary line of the main subject 3171 even within the region of interest 3172, and synthesized with the frame of the peripheral region 3176. In other words, the frame rate for the area outside the boundary line of the region of interest 3172 may be reduced to that of the peripheral region 3176. This prevents the boundary line of smoothness in the displayed moving image from appearing unnatural. Furthermore, the frame rate for playback does not need to be the same as the frame rate used during shooting (180 fps for the region of interest and 60 fps for the peripheral region). For example, the region of interest may be 60 fps and the peripheral region may be 20 fps. In this case, the video will be played in slow motion.

[0452] FIG. 74 shows a plan view of the configuration of a pixel region 3700 of the image sensor 3100 and an example of its operation. Note that FIGS. 74 to 77 show projections of each pixel block 3131 in the pixel region 3700 and each memory block 3730 in the memory unit 3114 on the same plane. The pixel blocks 3131 are arranged at regular intervals in the row and column directions throughout the pixel region 3700. Each pixel block 3131 has m×n pixels, where n and m are 2 or greater. The pixel block 3131 may be composed of 32×64 pixels arranged in a matrix. In this example, each memory block 3730 is a memory provided for each pixel block 3131. That is, each pixel block 3131 has a memory block 3730 that corresponds to it one-to-one. Each memory block 3730 is provided in an area of ​​the signal processing chip 3111 that overlaps with the corresponding pixel block 3131.

[0453] The pixel blocks 3131 are grouped into groups of multiple pixel blocks 3131 that are distributed at regular intervals within the pixel region 3700. The memory blocks 3730 corresponding to the pixel blocks 3131 within a group are shared by the pixel blocks 3131 within the group. Sharing means that pixel data of the multiple pixel blocks 3131 can be directly or indirectly read from and written to that memory block 3730. It is preferable to group all pixel blocks 3131 included in the pixel region 3700 so that the distance between the pixel blocks 3131 within one group is maximized. It is also more preferable that the group of pixel blocks 3131 includes multiple pixel blocks 3131 located at the outermost periphery of the pixel region 3700 in the imaging chip 3113. In this case, the control unit 3740 controls the multiple pixel blocks 3131 located at the outermost periphery by fixing the frame rate to a lower frame rate than the high-speed frame rate (in this example, the reference frame rate).

[0454] Here, the position of the pixel block 3131 is represented by coordinates (x, y). In this example, four pixel blocks 3131 located at positions (4,4), (4,1), (1,4), and (1,1) are grouped. The other pixel blocks 3131 are similarly grouped, with each pixel block 3131 spaced at regular intervals.

[0455] Each storage block 3730 corresponding to a pixel block 3131 in a group is shared by all pixel blocks 3131 in the group. This allows pixel data of a high-frame-rate pixel block 3131 to be stored in a storage block 3730 corresponding to a standard-frame-rate pixel block 3131 in the group. In this example, the pixel data of the high-frame-rate pixel block 3131 at the shaded position (4,4) is stored in the standard-frame-rate storage block 3730 in order from among the storage blocks 3730 corresponding to pixel blocks 3131 at the positions (4,4), (4,1), (1,4), and (1,1).

[0456] In other words, if pixel data is already stored in a storage block 3730 corresponding to a pixel block 3131 with a high frame rate, the control unit 3740 stores the pixel data corresponding to the pixel block 3131 in one of the storage blocks 3730 in the same group as the pixel block 3131. Here, as shown in FIG. 68 , the region of interest 3172 is formed by contiguous pixel blocks 3131. Therefore, by grouping multiple pixel blocks 3131 spaced at regular intervals within the pixel region 3700, the probability that a group will contain a mixture of pixel blocks 3131 with a high frame rate and pixel blocks 3131 with a reference frame rate can be increased. This improves memory utilization efficiency without increasing the memory capacity of the storage block 3730. Furthermore, because the groups that share the storage block 3730 are fixed, additional data indicating which pixel block 3131 the pixel data stored in each storage block 3730 corresponds to can be reduced or omitted.

[0457] Fig. 75 is a plan view of an example of another configuration of the image sensor 3100 shown in Fig. 74. The image sensor 3100 of this example differs from the embodiment shown in Fig. 74 in that it includes, instead of the memory unit 3114, a memory unit 3810 that is located outside the pixel region 3700 and that is provided along each of the sides in the row and column directions. Note that the memory unit 3810 may be identical to the memory unit 3114 except for its physical location.

[0458] The memory unit 3810 of this example is composed of multiple memory areas 3812 that are provided in the row and column directions opposite areas that overlap with the pixel blocks 3131 at the outermost periphery of the pixel region 3700. Each memory area 3812 may be composed of 2×2 memory blocks 3730. Each memory block 3730 is a memory area 3812 in a memory provided for each group. The control unit 3740 generates address information based on information about the position, frame rate, and timing of each of the grouped pixel blocks 3131, and writes pixel data to the memory blocks 3730 sequentially.

[0459] In this example, the memory blocks 3730 corresponding to the grouped pixel blocks 3131 form a 2×2 memory area 3812. In other words, because the memory blocks 3730 corresponding to the grouped pixel blocks 3131 are adjacent to each other in one location, there is no need to connect distant memory blocks 3730 via wiring, as is the case when memory blocks 3730 are provided for each area that overlaps with the pixel blocks 3131. This reduces the time required to write / read pixel data due to RC delay. Furthermore, when inputting pixel data to the next-stage arithmetic circuit, only one bus needs to be provided for the memory area 3812. Furthermore, the circuit configuration required to write / read pixel data can be simplified compared to when a memory block is provided for each area that overlaps with the pixel blocks 3131.

[0460] FIG. 76 is a plan view showing another example of the operation of the image sensor 3100 shown in FIG. 74. This example differs from the embodiment shown in FIG. 74 in that it further includes a transmission path 3710 that transmits pixel data between memory blocks 3730 corresponding to adjacent pixel blocks 3131. The transmission path 3710 may be wiring that connects the memory blocks 3730 to each other. The transmission path 3710 connects a control unit 3740 to all of the memory blocks 3730. The control unit 3740 sequentially transfers pixel data corresponding to a pixel block 3131 with a high frame rate to adjacent memory blocks 3730 in synchronization with the high frame rate. Here, "synchronizing with the high frame rate" refers to sequentially storing pixel data in multiple adjacent memory blocks 3730 at the same timing as the pixel block 3131 with a high frame rate captures pixel data.

[0461] Here, we will explain an example where the frame rate of pixel block 3131 at position (4,4) is five times the reference frame rate. If the reference frame rate is 60 fps, the high-speed frame rate is 300 fps. The imaging timing at the high-speed frame rate is as follows: T0 is when time t=0, T1 is when time t=1 / 300 s, T2 is when time t=2 / 300 s, T3 is when time t=3 / 300 s, T4 is when time t=4 / 300 s, and T5 is when time t=5 / 300 s.

[0462] At timing T0, control unit 3740 stores pixel data of the captured subject in storage blocks 3730 corresponding to all pixel blocks 3131. Next, at timing T1, control unit 3740 moves the pixel data stored in storage block 3730 at position (3,4), which has a lower frame rate and is adjacent, to storage block 3730 at position (2,4) on the outer periphery, and moves and stores the pixel data stored in storage block 3730 corresponding to pixel block 3131 at position (4,4) in the now empty storage block 3730 at position (3,4). At the same time, control unit 3740 stores the pixel data of pixel block 3131 at position (4,4) acquired at timing T1 in the corresponding storage block 3730 at position (4,4).

[0463] At timing T2, control unit 3740 moves and stores the pixel data stored in memory block 3730 at position (4,3) to memory block 3730 at position (4,2) on the outer periphery, and moves and stores the pixel data in memory block 3730 corresponding to pixel block 3131 at position (4,4) to the now empty memory block 3730 at position (4,3). At the same time, control unit 3740 stores the pixel data of pixel block 3131 at position (4,4) obtained at timing T2 in the corresponding memory block 3730 at position (4,4).

[0464] At timing T3, control unit 3740 moves and stores the pixel data stored in memory block 3730 at position (5,4) to memory block 3730 at position (6,4) on the outer periphery, and moves and stores the pixel data in memory block 3730 corresponding to pixel block 3131 at position (4,4) to the now empty memory block 3730 at position (5,4). At the same time, control unit 3740 stores the pixel data of pixel block 3131 at position (4,4) obtained at timing T3 in the corresponding memory block 3730 at position (4,4).

[0465] At timing T4, control unit 3740 moves and stores the pixel data stored in storage block 3730 at position (4,5) to storage block 3730 at position (4,6) in the circumferential direction, and moves and stores the pixel data in storage block 3730 corresponding to pixel block 3131 at position (4,4) in now-empty storage block 3730 at position (4,5). At the same time, control unit 3740 stores the pixel data of pixel block 3131 at position (4,4) acquired at timing T4 in storage block 3730 at the corresponding position (4,4). At this time, pixel data from timings T0 to T4 is stored in storage block 3730 at position (4,4) corresponding to pixel block 3131 at position (4,4), and in storage blocks 3730 at positions (3,4), (4,3), (5,4), and (4,5) two-dimensionally surrounding storage block 3730.

[0466] The control unit 3740 may move each piece of pixel data stored in the storage blocks 3730 at positions (3,4), (4,3), (5,4), and (4,5) to the storage block 3730 closest to the edge of the pixel region 3700 among the adjacent storage blocks 3730. In other words, the control unit 3740 may move and store each piece of pixel data stored in the storage blocks 3730 at positions (3,4), (4,3), (5,4), and (4,5) in the storage blocks 3730 at positions (1,4), (4,1), (6,4), and (4,6), which are at the edges of the pixel region 3700.

[0467] At timing T5, the control unit 3740 transfers the pixel data stored in all memory blocks 3730 in the pixel area 3700 to a downstream memory or arithmetic circuit via the bus line. The control unit 3740 updates the frame cycle and repeats the operations from timing T0 to T4 described above.

[0468] The control unit 3740 fixes the frame rates of the pixel blocks 3131, among the multiple pixel blocks 3131, that are located along the outermost periphery of the pixel region 3700, to the reference frame rate. However, if a pixel block 3131 with a high frame rate is located at the edge of the pixel region 3700, the number of adjacent storage blocks 3730 is limited, making it difficult to distribute pixel data two-dimensionally. Therefore, the control unit 3740 prevents the pixel block 3131 with a high frame rate from being located at the outermost periphery of the pixel region 3700. For example, the control unit 3740 fixes the frame rates of the pixel blocks 3131 at the outermost periphery of the pixel region 3700 to the reference frame rate.

[0469] The control unit 3740 simultaneously writes new pixel data to the storage blocks 3730 corresponding to all of the pixel blocks 3131, and transmits the pixel data for each pixel block 3131 collectively to a downstream arithmetic processing circuit. In this way, the control unit 3740 sequentially moves pixel data of pixel blocks 3131 with a high frame rate toward the edge of the pixel region 3700 to storage blocks 3730 corresponding to adjacent pixel blocks 3131, thereby allowing the storage blocks 3730 to be shared by multiple pixel blocks 3131, thereby reducing memory capacity. The pixel data allocated to multiple adjacent storage blocks 3730 may have, as header information, position data of the corresponding pixel block 3131 in the pixel region 3700, and frame data indicating the frame to which the pixel data belongs, as additional data.

[0470] In this example, the control unit 3740 sequentially moves and stores pixel data of pixel blocks 3131 with a high frame rate to storage blocks 3730 corresponding to adjacent pixel blocks 3131, but the pixel data may be moved to every other storage block 3730, or may be moved and stored in storage blocks 3730 in a diagonal direction rather than in a row and column direction. The control unit 3740 may select the storage block 3730 to which pixel data should be moved based on the frame rate information of each pixel block 3131.

[0471] FIG. 77 is a plan view showing another example of the configuration of the image sensor 3100. In this example, pixel data is transmitted between memory blocks 3730 corresponding to adjacent pixel blocks 3131, similar to the image sensor 3100 shown in FIG. 76. However, similar to the image sensor 3100 shown in FIG. 75, the image sensor 3100 of this example includes a memory unit 3810 provided outside the area of ​​the signal processing chip 3111 that overlaps with the pixel area 3700. The memory unit 3810 includes memory areas 3820 divided by the number of pixel blocks 3131 in the row direction (six in this example) and memory areas 3822 divided by the number of pixel blocks 3131 in the column direction (six in this example). The control unit 3740 stores pixel data corresponding to the pixel blocks 3131 of the high frame rate in the predetermined memory areas 3820 and 3822 in synchronization with the high frame rate.

[0472] The control unit 3740 may write pixel data of the pixel block 3131 at position (4,4) with a high frame rate to the storage areas 3820, 3822 associated with the outermost pixel block 3131 with a low frame rate in synchronization with the frame rate. The control unit 3740 may also select storage areas 3820, 3822 associated with pixel blocks 3131 other than the outermost pixel block 3131 based on the frame rate information of each pixel block 3131 and write the pixel data thereto. The storage areas 3820, 3822 are shared by the pixel data of the pixel blocks 3131 with a high frame rate and the pixel data of the pixel blocks 3131 with a low frame rate. In this example, writing and reading only needs to be performed for each storage area 3820, 3822, and there is no need to write and read for each storage block 3730 provided for each pixel block 3131, thereby simplifying the circuit configuration. In this example, the memory areas 3820 and 3822 in the memory unit 3810 have the same memory space size. Furthermore, the locations of the memory areas 3820 and 3822 in the memory space may be fixed within the memory unit 3810, or may change dynamically.

[0473] FIG. 78 shows the configuration and operation of a portion of an image sensor 3100 according to another embodiment. This embodiment differs from the above-described embodiment in that the storage unit 3114 is configured as a multi-layer buffer memory. The storage unit 3114 of this embodiment includes a temporary memory 3850 and a transfer memory 3860. The temporary memory 3850 has storage blocks 3830 corresponding to each pixel block 3131 and is used to control pixel data for the high-data-rate pixel block 3712. The transfer memory 3860 receives pixel data input from the temporary memory 3850 and transfers the pixel data to the next-stage memory or arithmetic circuit. The transfer memory 3860 has a storage area at least equal in size to the total storage area of ​​the multiple storage blocks 3730. Here, the total storage area refers to the size of the memory space provided by the temporary memory 3850. The temporary memory 3850 of this embodiment has the same function and configuration as the storage block 3730 shown in FIG. 76.

[0474] Here, we will explain an example where the frame rate of pixel block 3712 is five times the reference frame rate. If the reference frame rate is 60 fps, the high-speed frame rate is 300 fps. The imaging timing at the high-speed frame rate is as follows: T0 is when time t=0, T1 is when time t=1 / 300 s, T2 is when time t=2 / 300 s, T3 is when time t=3 / 300 s, T4 is when time t=4 / 300 s, and T5 is when time t=5 / 300 s.

[0475] The control unit 3740 stores all pixel data of the object imaged at timing T0 in the memory blocks 3830 corresponding to all pixel blocks 3131. The control unit 3740 transfers the stored pixel data to the transfer memory 3860 at a timing before T1. In other words, the control unit 3740 copies and stores all pixel data of the object imaged at timing T0 in the corresponding memory areas 3870 of the transfer memory 3860 before the next pixel data is input from the pixel block 3712 operating at the high frame rate.

[0476] At timing T1 synchronized with the high-speed frame rate, control unit 3740 stores pixel data from high-speed frame rate pixel block 3712 in corresponding storage block 3853 of temporary memory 3850 via bus 3720. Control unit 3740 moves and stores the pixel data stored in storage block 3853 in adjacent storage block 3854 at timing T2 or a timing before T2. Control unit 3740 stores pixel data from pixel block 3712 in corresponding storage block 3853 of temporary memory 3850 via bus 3720 at timing T2 synchronized with the high-speed frame rate.

[0477] The control unit 3740 moves and stores the pixel data stored in storage block 3853 to adjacent storage block 3855 at timing T3 or before T3. The control unit 3740 stores the pixel data from pixel block 3712 in corresponding storage block 3853 of temporary memory 3850 via bus 3720 at timing T3, in synchronization with the high frame rate. The control unit 3740 moves and stores the pixel data stored in storage block 3853 to adjacent storage block 3856 at timing T4 or before T4. The control unit 3740 stores the pixel data from pixel block 3712 in corresponding storage block 3853 of temporary memory 3850 via bus 3720 at timing T4, in synchronization with the high frame rate.

[0478] The control unit 3740 stores the pixel data stored in storage blocks 3854, 3855, 3856, and 3857 of the temporary memory 3850 in corresponding storage areas 3864, 3865, 3866, and 3867 of the transfer memory 3860 via the bus 3840 at timing T5 or a timing before T5. In other words, after the temporary memory 3850 receives pixel data at the high-speed frame rate immediately before the reference timing, it transfers the pixel data to the transfer memory 3860 until it receives pixel data at the next reference timing.

[0479] The control unit 3740 may further transfer the pixel data stored in memory blocks 3854, 3855, 3856, and 3857 adjacent to memory block 3853 to other adjacent memory blocks in synchronization with the high-speed frame rate. The control unit 3740 transfers all of the pixel data stored in transfer memory 3860 to a subsequent memory or arithmetic circuit.

[0480] According to this embodiment, the memory block 3853 corresponding to the pixel block 3712 with a high frame rate is connected to the memory blocks 3854, 3855, 3856, and 3857 adjacent to the memory block 3853 by the transmission path 3710, eliminating the need to connect all memory blocks via the transmission path 3710. This allows for high-speed pixel data transfer. Furthermore, a cache memory such as an SRAM can be used as the temporary memory 3850, enabling high-speed read / write operations. Furthermore, since the memory block 3830 is not shared in the temporary memory 3850, the circuit configuration required for read / write operations can be simplified. Furthermore, in the transfer memory 3860, the only shared memory area is the memory area adjacent to the memory area 3863 corresponding to the pixel block 3712 with a high frame rate. Therefore, wiring connecting the memory areas 3863 to each other is not required in the transfer memory 3860. Furthermore, although an example has been described in which temporary memory 3850 has the configuration of memory block 3730 shown in FIG. 76, temporary memory 3850 may have any of the configurations of memory block 3730 shown in FIGS. 74 to 77.

[0481] Fig. 79 is a flowchart showing another example of the operation of the imaging device to generate and record a moving image. In Fig. 79, the same operations as those in Fig. 67 are assigned the same reference numerals, and the description thereof will be omitted.

[0482] In the operation of Fig. 79, instead of or in addition to the frame rate of Fig. 67, different thinning rates are used between the region of interest 3172 and the peripheral region 3176. More specifically, in step S3120, the driver 3502 causes pixels thinned out at a low thinning rate in the pixel block 3131 included in the region of interest 3172 to accumulate charge and output image signals, and causes pixels thinned out at a high thinning rate in the pixel block 3131 included in the peripheral region 3176 to accumulate charge and output image signals. For example, a thinning rate of 0 is used in the pixel block 3131 included in the region of interest 3172, meaning that all pixels are read out, and a thinning rate of 0.5 is used in the pixel block 3131 included in the peripheral region 3176, meaning that half the pixels are read out.

[0483] In this case, the driving unit 3502 separately drives the set of reset transistor 3303, transfer transistor 3302 and selection transistor 3305 of the pixel block 3131 included in the peripheral region 3176 and the set of reset transistor 3303, transfer transistor 3302 and selection transistor 3305 of the pixel block 3131 included in the region of interest 3172, thereby obtaining image signals at different thinning rates.

[0484] In step S3110, the video generation unit 3154 generates an attention area video corresponding to the attention area 3172, based on the image signal of the attention area 3172 output at a low thinning rate. Similarly, the video generation unit 3154 generates a surrounding area video corresponding to the surrounding area 3176, based on the image signal of the surrounding area 3176 output at a high thinning rate. Furthermore, in step S3112, the video generation unit 3154 records the attention area video and the surrounding area video in the recording unit 3505, adding information about the respective thinning rates.

[0485] Figure 80 shows an example of pixels 3188 that are read out at a thinning rate of 0.5 for one pixel block. In the example shown in Figure 80, when the pixel block 3132 in the peripheral region 3176 is in a Bayer array, pixels 3188 that are read out and pixels that are not read out are set every other Bayer array unit in the vertical direction, that is, every two rows when viewed pixel by pixel. This allows thinning readout to be performed without disrupting the color balance.

[0486] Fig. 81 is a flowchart showing the operation of the imaging device for playing back and displaying a moving image, corresponding to Fig. 79. In Fig. 81, the same operations as those in Fig. 73 are given the same reference numerals, and the description thereof will be omitted.

[0487] In step S3170 of Figure 81, the moving image synthesis unit 3156 complements the pixels of the frames of the peripheral area moving image to match the resolution of the frames of the attention area moving image, and then synthesizes the frames of the displayed image by fitting the frames of the attention area moving image into the frames of the peripheral area moving image. This makes it possible to obtain a high-resolution image signal from the attention area 3172 containing the main subject 3171, while reducing the amount of data by limiting the peripheral area 3176 to a lower resolution. Therefore, compared to high-speed readout from all pixels, it is possible to reduce the load on driving and image processing, and suppress power consumption and heat generation.

[0488] 61 to 81, the region of interest 3172 is rectangular, but the shape of the region of interest 3172 is not limited to this. The region of interest 3172 may be a convex polygon, a concave polygon, or a doughnut shape with the surrounding region 3176 embedded therein, as long as it follows the boundary line of the pixel block 3131. Furthermore, multiple regions of interest 3172 may be set at intervals. In this case, different frame rates may be set for the regions of interest 3172.

[0489] The frame rates of the region of interest 3172 and the peripheral region 3176 may also be variable. For example, the amount of movement of the main subject 3171 may be detected each time a unit of time elapses, and a higher frame rate may be set for the region of interest 3172 the greater the amount of movement of the main subject 3171. Furthermore, the selection of pixel blocks 3131 to be included in the region of interest 3172 may be updated as needed within the unit of time to follow the main subject 3171.

[0490] Although generation of the moving images in FIGS. 67 and 79 is initiated by the user pressing a record button, and playback of the moving images in FIGS. 73 and 81 is initiated by the user pressing a play button, the start points are not limited to these. As another example, a single button operation by the user may cause the operation of generating a moving image and the operation of playing the moving image to be executed consecutively, and a through image (also referred to as a live view display) may be displayed on the display unit 3506. In this case, a display that allows the user to recognize the attention area 3172 may be superimposed. For example, a frame may be displayed on the display unit 3506 around the boundary of the attention area 3172, or the brightness of the peripheral area 3176 may be reduced or the brightness of the attention area 3172 may be increased.

[0491] In the operation of FIG. 79 , the thinning rate is made different between the region of interest 3172 and the peripheral region 3176. Instead of making the thinning rate different, the number of rows when adding pixel signals of pixels in adjacent rows may be made different. For example, the number of rows in the region of interest 3172 is 1, i.e., pixel signals are output without adding adjacent rows, and the number of rows in the peripheral region 3176 is made larger than that of the region of interest 3172, for example, 2, and pixel signals of pixels in the same column of two adjacent rows are output. This makes it possible to reduce the overall signal amount while maintaining a higher resolution in the region of interest 3172 than in the peripheral region 3176, as in FIG. 79 .

[0492] The moving image synthesizing unit 3156 may be provided in an external display device, for example, a PC, instead of being provided in the image processing unit 3511 of the imaging device 3500. Furthermore, the above embodiment is not limited to being applied to the case of generating moving images, but may also be applied to the case of generating still images.

[0493] Furthermore, in all of the above embodiments, the plurality of pixel blocks 3131 are divided into two regions, the region of interest 3172 and the surrounding region 3176, but this is not limiting and the pixel blocks 3131 may be divided into three or more regions. In this case, the pixel block 3131 corresponding to the boundary between the region of interest 3172 and the surrounding region 3176 may be set as a boundary region, and the boundary region may be controlled using an intermediate value between the control parameter value used for the region of interest 3172 and the control parameter value used for the surrounding region 3176. This makes it possible to prevent the boundary between the region of interest 3172 and the surrounding region 3176 from looking unnatural.

[0494] The charge accumulation time, accumulation count, etc. may be different between the region of interest 3172 and the peripheral region 3176. In this case, the region of interest 3172 and the peripheral region 3176 may be divided based on brightness, and an intermediate region may also be provided.

[0495] 82A and 82B are diagrams illustrating an example of a scene and its region division. FIG. 82A shows a scene captured by the pixel region of the imaging chip 3113. Specifically, the scene simultaneously captures a shadow object 3601 and a middle object 3602 in an indoor environment and a highlight object 3603 in an outdoor environment, observed inside a window frame 3604. When capturing a scene with a large contrast between highlight and shadow, a conventional imaging device would experience crushed shadows in the shadow area if charge accumulation was performed based on the highlight area, while performing charge accumulation based on the shadow area would result in blown-out highlights in the highlight area. In other words, the dynamic range of the photodiode is insufficient for a scene with a large contrast between highlight and shadow areas to output an image signal by uniformly accumulating charge across both the highlight and shadow areas. Therefore, in this embodiment, the scene is divided into partial regions, such as highlight and shadow regions, and the photodiodes corres...

Claims

1. An imaging element including a plurality of stacked semiconductor substrates, a first pixel having a first photoelectric conversion unit that converts light into an electric charge; a second pixel having a second photoelectric conversion unit that converts light into an electric charge; a calculation unit that calculates a first evaluation value for controlling driving of the first pixel and a second evaluation value for controlling driving of the second pixel; Equipped with the first photoelectric conversion unit and the second photoelectric conversion unit are disposed on a first semiconductor substrate among the plurality of semiconductor substrates; the arithmetic unit is disposed on a second semiconductor substrate among the plurality of semiconductor substrates; Image sensor.

2. 2. The imaging device according to claim 1, the calculation unit calculates the first evaluation value using a first signal based on the charges converted by the first photoelectric conversion unit, and calculates the second evaluation value using a second signal based on the charges converted by the second photoelectric conversion unit. Image sensor.

3. 3. The imaging device according to claim 2, a first conversion unit that converts the first signal into a digital signal; a second conversion unit that converts the second signal into a digital signal; Equipped with the calculation unit calculates the first evaluation value using the first signal converted into a digital signal by the first conversion unit, and calculates the second evaluation value using the second signal converted into a digital signal by the second conversion unit. Image sensor.

4. 4. The imaging device according to claim 3, the first conversion unit and the second conversion unit are disposed on the second semiconductor substrate. Image sensor.

5. 5. The imaging device according to claim 3, a first storage unit that stores the first signal converted into a digital signal by the first conversion unit; a second storage unit that stores the second signal converted into a digital signal by the second conversion unit; Equipped with the calculation unit calculates the first evaluation value using the first signal stored in the first storage unit, and calculates the second evaluation value using the second signal stored in the first storage unit. Image sensor.

6. 6. The imaging device according to claim 5, the first storage unit and the second storage unit are disposed on the second semiconductor substrate; Image sensor.

7. 7. The imaging device according to claim 1, the calculation unit includes a first calculation circuit that calculates the first evaluation value and a second calculation circuit that calculates the second evaluation value; Image sensor.

8. The imaging device according to any one of claims 1 to 7, a control unit that controls driving of the first pixel and driving of the second pixel; the control unit controls driving of the first pixel based on the first evaluation value, and controls driving of the second pixel based on the second evaluation value. Image sensor.

9. 9. The imaging device according to claim 8, the control unit controls an accumulation time for accumulating the electric charge converted by the first photoelectric conversion unit as the drive of the first pixel, and controls an accumulation time for accumulating the electric charge converted by the second photoelectric conversion unit as the drive of the second pixel. Image sensor.

10. 10. The imaging device according to claim 8, the control unit controls a frame rate for reading out from the first pixel a signal based on the charge converted by the first photoelectric conversion unit as the drive of the first pixel, and controls a frame rate for reading out from the second pixel a signal based on the charge converted by the second photoelectric conversion unit as the drive of the second pixel. Image sensor.

11. 11. The imaging device according to claim 8, the first pixel is disposed on the first semiconductor substrate and has a first transfer unit that transfers charges converted by the first photoelectric conversion unit; the second pixel is disposed on the first semiconductor substrate and has a second transfer unit that transfers the charges converted by the second photoelectric conversion unit; the control unit controls the first transfer unit and the second transfer unit. Image sensor.

12. 12. The imaging device according to claim 8, the first pixel has a first discharge unit that discharges the charge converted by the first photoelectric conversion unit, and the second pixel has a second discharge unit that discharges the charge converted by the second photoelectric conversion unit, The drive unit controls the first discharge unit and the second discharge unit. Image sensor.

13. The imaging device according to claim 12, the first discharge portion and the second discharge portion are disposed on the first semiconductor substrate; Image sensor.

14. The imaging device according to any one of claims 8 to 13, the control unit is disposed on the second semiconductor substrate; Image sensor.

15. An imaging device comprising the imaging element according to any one of claims 1 to 14.

Images