Imaging element and imaging device

Abstract

This imaging element comprises: a pixel unit in which a plurality of pixels each including a photoelectric conversion part that converts light into electric charge are disposed; an exposure processing unit that calculates, for each pixel, a second accumulation time during which the photoelectric conversion parts of the pixels accumulate electric charge, on the basis of a first output signal output from each of the plurality of pixels by photoelectric conversion in a first accumulation time for each of the pixels in which the photoelectric conversion part of the pixel accumulates electric charge; and a compression unit that compresses a second output signal output from each of the plurality of pixels by photoelectric conversion in the second accumulation time for each pixel calculated by the exposure processing unit.

Description

Image sensor and image pickup device Incorporation by Reference

[0001] This application claims priority from Japanese Patent Application No. 2024-46833, filed on March 22, 2024, the contents of which are incorporated herein by reference.

[0002] The present invention relates to an imaging element and an imaging device.

[0003] An image sensor is known that has a pixel array section in which a plurality of pixels are arranged in a matrix, and a frame memory that records pixel signals from each of the plurality of pixels that make up the pixel array section (see, for example, Patent Document 1). Conventionally, the increasing amount of image data has been a problem.

[0004] JP 2019-186961 A

[0005] The imaging element of the disclosed technique includes a pixel section in which a plurality of pixels are arranged, each including a photoelectric conversion section that converts light into electric charges; an exposure processing section that calculates, for each pixel, a second accumulation time during which the photoelectric conversion section of the pixel accumulates electric charges, based on a first output signal output from each of the plurality of pixels by photoelectric conversion during a first accumulation time for each pixel during which the photoelectric conversion section of the pixel accumulates electric charges; and a compression section that compresses the second output signal output from each of the plurality of pixels by photoelectric conversion during the second accumulation time for each pixel calculated by the exposure processing section.

[0006] FIG. 1 is an exploded perspective view showing an example of an image sensor. FIG. 2 is an explanatory diagram showing an example of a specific configuration of a pixel unit. FIG. 3 is a circuit diagram showing an example of a circuit configuration of a pixel. FIG. 4 is an explanatory diagram showing an example of a specific configuration of a control circuit unit. FIG. 5 is an explanatory diagram showing an example of the internal configuration of a control block. FIG. 6 is an explanatory diagram showing an example of signal transmission between a first semiconductor substrate and a second semiconductor substrate in an image sensor. FIG. 7 is an explanatory diagram showing an example of an X-Z cross section of an image sensor according to this embodiment. FIG. 8 is a timing chart showing Example 1 of an image sensing operation of an image sensor. FIG. 9 is an explanatory diagram showing an example of data compression using exposure time control for each pixel block. FIG. 10 is a block diagram showing an example of a configuration related to data compression of an image sensor. FIG. 11 is a block diagram showing a detailed configuration of a signal processing circuit. FIG. 12 is an explanatory diagram showing an example of generation of an exposure value map. FIG. 13 is a block diagram showing a detailed example of a first compression circuit. FIG. 14 is an explanatory diagram showing an example of color separation by a color separation unit. FIG. 15 is an explanatory diagram showing an example of quantization processing by a quantization unit. FIG. 16 is an explanatory diagram showing an example of encoding processing by an encoding unit. FIG. 17 is a block diagram showing an example of image data conversion processing by the image processing circuit.

[0007] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention as claimed. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0008] In this specification, the X-axis and Y-axis are orthogonal to each other, and the Z-axis is orthogonal to the XY plane. The XYZ-axes form a right-handed system. The direction parallel to the Z-axis may be referred to as the stacking direction of the image sensor 100. In this specification, the terms "up" and "down" are not limited to the up and down directions in the direction of gravity. These terms merely refer to relative directions in the Z-axis direction. Note that in this specification, the arrangement in the X-axis direction will be described as a "row" and the arrangement in the Y-axis direction as a "column," but the matrix direction is not limited to this.

[0009] <Configuration of Image Sensor in Figures 1 to 8> First, the configuration of the image sensor will be described with reference to Figures 1 to 8. The structure of the image sensor may be either a back-illuminated type or a front-illuminated type.

[0010] FIG. 1 is an exploded perspective view showing an example of an image sensor 100. The image sensor 100 captures an image of a subject. The image sensor 100 generates image data of the captured subject. The image sensor 100 includes a first semiconductor substrate 110, a second semiconductor substrate 120, and a third semiconductor substrate 130. As shown in FIG. 1 , the first semiconductor substrate 110 is stacked on the second semiconductor substrate 120, and the second semiconductor substrate 120 is stacked on the third semiconductor substrate 130.

[0011] The first semiconductor substrate 110 has a pixel section 101. The pixel section 101 outputs a pixel signal based on incident light.

[0012] The second semiconductor substrate 120 has a control circuit section 102 and a peripheral circuit section 121 .

[0013] The control circuit unit 102 receives pixel signals output from the first semiconductor substrate 110. The control circuit unit 102 processes the received pixel signals. The control circuit unit 102 is disposed on the second semiconductor substrate 120 at a position facing the pixel unit 101. For example, the control circuit unit 102 is disposed so as to overlap the pixel unit 101 in the direction in which the first semiconductor substrate 110 and the second semiconductor substrate 120 are stacked. The control circuit unit 102 may output a control signal to the pixel unit 101 for controlling the driving of the pixel unit 101.

[0014] The peripheral circuit unit 121 controls the driving of the control circuit unit 102. The peripheral circuit unit 121 is arranged around the control circuit unit 102 on the second semiconductor substrate 120. Specifically, the peripheral circuit unit 121 is arranged in an area of ​​the second semiconductor substrate 120 that is arranged outside the area in which the control circuit unit 102 is arranged. The peripheral circuit unit 121 may also be electrically connected to the first semiconductor substrate 110 and control the driving of the pixel unit 101. The peripheral circuit unit 121 is arranged along two sides of the second semiconductor substrate 120, but the arrangement of the peripheral circuit unit 121 is not limited to this example.

[0015] The third semiconductor substrate 130 has a data processing unit 103. The data processing unit 103 uses the digital data output from the second semiconductor substrate 120 to perform addition processing, thinning processing, and other image processing.

[0016] 2 is an explanatory diagram showing an example of a specific configuration of the pixel unit 101. The pixel unit 101 has a plurality of pixel blocks 200. The plurality of pixel blocks 200 are arranged in the row and column directions in the pixel unit 101. Specifically, the plurality of pixel blocks 200 includes M×N (M and N are natural numbers) pixel blocks 200 arranged in the row and column directions in the pixel unit 101. Although the figure shows a case where M is equal to N, M and N may be different.

[0017] The pixel block 200 has a plurality of pixels 201. The plurality of pixels 201 are arranged in rows and columns in the pixel block 200. The pixel block 200 has m×n (m and n are natural numbers) pixels 201 arranged in the rows and columns. For example, the pixel block 200 has 16×16 pixels 201 arranged in the rows and columns. The number of pixels 201 corresponding to the pixel block 200 is not limited to this. Although the illustration shows a case where m is equal to n, m may be different from n.

[0018] The pixel block 200 has a plurality of pixels 201 connected to a common control line (for example, a transfer control line 311 and a discharge control line 312 described later) in the row direction. For example, each pixel 201 in the pixel block 200 is connected to the common control line so that the pixels 201 are set to the same exposure time. Specifically, for example, every n pixels 201 arranged in the row direction are connected by the common control line.

[0019] On the other hand, between different pixel blocks 200, one pixel block 200 may be set to an exposure time different from that of the other pixel block 200. For example, when one pixel block 200 and the other pixel block 200 are arranged side by side in the row direction, the multiple pixels 201 included in one pixel block 200 and the multiple pixels 201 included in the other pixel block 200 are connected by different control lines. The multiple pixels 201 in the mth row of one pixel block 200 are commonly connected by a control line different from the common control line to which the multiple pixels 201 in the mth row of the other pixel block 200 are connected. Furthermore, when one pixel block 200 and the other pixel block 200 are arranged side by side in the column direction, the multiple pixels 201 included in one pixel block 200 and the multiple pixels 201 included in the other pixel block 200 are connected by different control lines. The pixels 201 in the mth row of one pixel block 200 are commonly connected to a control line that is different from the common control line to which the pixels 201 in the mth row of the other pixel block 200 are connected.

[0020] Furthermore, for example, when one pixel block 200 and the other pixel block 200 are arranged side by side in the row direction, the plurality of pixels 201 included in one pixel block 200 and the plurality of pixels 201 included in the other pixel block 200 are connected by different signal lines 202. The plurality of pixels 201 in the nth column of one pixel block 200 are connected in common by a signal line 202 that is different from the common signal line 202 to which the plurality of pixels 201 in the nth column of the other pixel block 200 are connected. Furthermore, when one pixel block 200 and the other pixel block 200 are arranged side by side in the column direction, the plurality of pixels 201 included in one pixel block 200 and the plurality of pixels 201 included in the other pixel block 200 are connected in common by different signal lines 202. The plurality of pixels 201 in the nth column of one pixel block 200 are connected in common by a signal line 202 that is different from the common signal line 202 to which the plurality of pixels 201 in the nth column of the other pixel block 200 are connected.

[0021] The pixel blocks 200 are arranged corresponding to the control blocks 400 described later. That is, one pixel block 200 is arranged for one control block 400.

[0022] Furthermore, multiple pixel blocks 200 may be arranged for one control block 400. Even when multiple pixel blocks 200 are arranged for one control block 400, different exposure times may be set for each pixel block 200. When two pixel blocks 200 arranged in the column direction are arranged for one control block, the control block 400 controls 2m×n pixels 201. Specifically, for example, the control block 400 controls 32×16 pixels 201. The number of pixels 201 corresponding to the control block 400 is not limited to this.

[0023] 3 is a circuit diagram showing an example of the circuit configuration of a pixel 201. The pixel 201 includes a photoelectric conversion unit 300 and a readout unit 310. The readout unit 310 has a transfer unit 301, a discharge unit 302, an FD (floating diffusion) 303, a reset unit 304, and a pixel output unit 305, and reads out a pixel signal based on the charge converted by the photoelectric conversion unit 300 to a signal line 202. The pixel output unit 305 has an amplifier unit 351 and a selection unit 352. The transfer unit 301, the discharge unit 302, the FD 303, the reset unit 304, the amplifier unit 351, and the selection unit 352 are collectively referred to as the readout unit 310. The readout unit 310 will be described as an N-channel FET, but the type of transistor is not limited to this.

[0024] The photoelectric conversion unit 300 has a photoelectric conversion function of converting light into electric charges, and accumulates the electric charges generated by photoelectric conversion. The photoelectric conversion unit 300 is configured by, for example, a photodiode.

[0025] The transfer unit 301 transfers the charge of the photoelectric conversion unit 300 to the FD 303. The transfer unit 301 controls the electrical connection between the photoelectric conversion unit 300 and the FD 303. The transfer unit 301 is configured, for example, by a transistor. The transfer unit 301 may also be an element that has at least a gate terminal and configures part of a transistor with a part of the photoelectric conversion unit 300 as the source terminal and a part of the FD 303 as the drain terminal. The gate terminal of the transfer unit 301 is connected to a transfer control line 311 for inputting a transfer control signal φTX. The transfer control line 311 will be described later.

[0026] The discharge unit 302 discharges the charge accumulated in the photoelectric conversion unit 300 to a power supply wiring to which a power supply voltage VDD is supplied. The discharge unit 302 controls the connection between the photoelectric conversion unit 300 and the power supply wiring. The discharge unit 302 is configured, for example, by a transistor. The discharge unit 302 may also be an element that has at least a gate terminal and constitutes part of a transistor in which a part of the photoelectric conversion unit 300 serves as a source terminal and a part of a diffusion region connected to the power supply wiring serves as a drain terminal. The gate terminal of the discharge unit 302 is connected to a discharge control line 312 for inputting a discharge control signal φPDRST. Note that although the discharge unit 302 has been described as discharging the charge of the photoelectric conversion unit 300 to a power supply wiring to which a power supply voltage VDD is supplied, the discharge unit 302 may also be discharged to a power supply wiring to which a power supply voltage different from the power supply voltage VDD is supplied.

[0027] The FD 303 accumulates the charges transferred from the photoelectric conversion unit 300 by the transfer unit 301 .

[0028] The reset unit 304 discharges the charge accumulated in the FD 303 to the power supply wiring to which the power supply voltage VDD is supplied. The reset unit 304 resets the potential of the FD 303 to the power supply voltage VDD, which is a reference potential. The reset unit 304 controls the electrical connection between the FD 303 and the power supply wiring. The reset unit 304 is configured, for example, by a transistor. The reset unit 304 may also be an element that has at least a gate terminal and configures part of a transistor with a part of the FD 303 as a source terminal and a part of a diffusion region connected to the power supply wiring as a drain terminal. The gate terminal of the reset unit 304 is connected to a reset control line 313 for inputting a reset control signal φRST. The reset control line 313 will be described later.

[0029] The pixel output unit 305 outputs a pixel signal based on the potential of the FD 303 to the signal line 202. The pixel output unit 305 has an amplifier 351 and a selection unit 352. The amplifier 351 is configured with a transistor. The amplifier 351 has a gate terminal connected to the FD 303, a drain terminal connected to a power supply line to which a power supply voltage VDD is supplied, and a source terminal connected to the drain terminal of the selection unit 352.

[0030] The selection unit 352 controls the electrical connection between the pixel 201 and the signal line 202. When the selection unit 352 electrically connects the pixel 201 and the signal line 202, a pixel signal is output from the pixel 201 to the signal line 202. The selection unit 352 is configured by a transistor. The selection unit 352 may also be an element that has at least a gate terminal and configures part of a transistor with a part of the amplifier unit 351 as a source terminal and a part of a diffusion region connected to the signal line 202 as a drain terminal. The gate terminal of the selection unit 352 is connected to a selection control line 314 that spans multiple pixel blocks 200 and is used to input a selection control signal φSEL. The source terminal of the selection unit 352 is connected to the load current source 306.

[0031] The load current source 306 is connected to the signal line 202 and supplies a current for reading out pixel signals from the pixels 201. This stabilizes the operation of the amplifier unit 351. The load current source 306 is also connected to the signal line 202. The load current source 306 may be provided on the first semiconductor substrate 110 or on the second semiconductor substrate 120.

[0032] The FD 303 and the pixel output unit 305 may be shared with other pixels 201. For example, the FD 303 and the pixel output unit 305 may be shared by a plurality of pixels 201 arranged side by side in the row or column direction. The pixel 201 may also be configured with a plurality of photoelectric conversion units 300 and transfer units 301.

[0033] FIG. 4 is an explanatory diagram showing an example of a specific configuration of the control circuit unit 102. The control circuit unit 102 has multiple control blocks 400. The multiple control blocks 400 are arranged in a row and column direction in the control circuit unit 102. Specifically, the control circuit unit 102 has M×N control blocks 400. When one pixel block 200 is arranged for one control block 400, the control circuit unit 102 has the control block 400 directly below the pixel block 200. One pixel block 200 and one control block 400 have substantially the same shape and size. Furthermore, when multiple pixel blocks 200 arranged in a column direction are arranged for one control block 400, the control circuit unit 102 has one control block 400 directly below the multiple pixel blocks 200 arranged in a column direction.

[0034] The control block 400 is provided corresponding to the pixel block 200. As an example of the correspondence between the control block and the pixel block, for example, the control block 400 is located directly below the pixel block 200 in the direction in which the first semiconductor substrate 110 and the second semiconductor substrate 120 are stacked (stacking direction). The control block 400 is electrically connected to the pixel block 200 via a signal line 202, a transfer control line 311, and a discharge control line 312. Specifically, the control block 400 located directly below the pixel block 200 in the stacking direction is electrically connected to the pixel block 200 directly above it in the stacking direction (hereinafter referred to as the corresponding pixel block 200) via local control lines such as the transfer control line 311 and the discharge control line 312. The control block 400 also receives pixel signals output from the pixels 201 of the corresponding pixel block 200 via the signal line 202.

[0035] The control block 400 controls the driving of the corresponding pixel block 200. For example, the control block 400 controls the exposure time of the pixels 201 included in the corresponding pixel block 200. The control block 400 also has a signal processing unit 402 that processes input signals, and processes pixel signals output from the pixels 201 included in the corresponding pixel block 200. For example, the control block 400 converts analog pixel signals output from the pixels 201 included in the corresponding pixel block 200 into digital signals.

[0036] The control block 400 has a pixel control unit 401 and a signal processing unit 402. The pixel control unit 401 has an autonomous exposure processing unit 411, an exposure control unit 412, and a pixel driving unit 413, and controls the pixels 201 of the pixel unit 101. The signal processing unit 402 has a signal input unit 421, a signal conversion unit 422, and a signal output unit 423, and converts analog pixel signals from the pixel unit 101 into digital signals and transfers them to the pixel control unit 401 and the data processing unit 103.

[0037] The autonomous exposure processing unit 411 is a circuit that calculates the exposure time of the pixels 201 included in the corresponding pixel block 200 based on the pixel signals converted into digital signals by the signal processing unit 402. Details of the autonomous exposure processing unit 411 will be described later.

[0038] The exposure control unit 412 is a circuit that controls the exposure of the pixels 201 included in the corresponding pixel block 200 based on the exposure time calculated by the autonomous exposure processing unit 411. Specifically, the exposure control unit 412 generates a control signal for controlling the exposure time (charge accumulation time of the photoelectric conversion unit 300) of the pixels 201 included in the corresponding pixel block 200. For example, the exposure control unit 412 adjusts the start timing or end timing of exposure of the pixels 201 included in the corresponding pixel block 200 to control the exposure time for each pixel block 200. The exposure control unit 412 is provided in the control block 400, extending in the row direction.

[0039] The pixel driving unit 413 outputs the control signal generated by the exposure control unit 412 to the pixels 201 included in the corresponding pixel block 200. The pixel driving unit 413 is a driving circuit that drives the pixels 201 included in the corresponding pixel block 200. The pixel driving unit 413 drives the pixels 201 in a pixel row selected from the pixels 201 included in the corresponding pixel block 200. The pixel driving unit 413 is provided extending in the column direction. As a result, the pixel driving unit 413 is disposed at a position corresponding to the m pixels 201 arranged in the column direction. The autonomous exposure processing unit 411, exposure control unit 412, and pixel driving unit 413 are arranged in an L-shape in the control block 400, with the pixel driving unit 413 extending in the column direction and the autonomous exposure processing unit 411 and exposure control unit 412 extending in the row direction.

[0040] The signal input unit 421 receives pixel signals output from pixels 201 included in the corresponding pixel block 200. The signal input unit 421 outputs the received pixel signals to the signal conversion unit 422. The signal input unit 421 may be provided for each of n pixels 201 arranged in the row direction in the corresponding pixel block 200. The signal input unit 421 may include a processing circuit that performs signal processing such as noise reduction on the pixel signals output from the first semiconductor substrate 110. The signal input unit 421 may also include a voltage adjustment circuit that adjusts the voltage of the signal line 202 connected to the pixel 201 included in the corresponding pixel block 200 so that it does not fall below a predetermined value. When the load current source 306 is disposed on the second semiconductor substrate, it may be disposed in the signal input unit 421 included in the corresponding control block 400.

[0041] The signal conversion unit 422 converts the pixel signals output from the signal input unit 421 into digital signals. The signal conversion unit 422 sequentially converts into digital signals the pixel signals output from m pixels 201 arranged in the column direction in the corresponding pixel block 200. The signal conversion unit 422 converts into parallel digital signals the pixel signals output from the pixels 201 arranged in n columns in the row direction in the corresponding pixel block 200.

[0042] The signal output unit 423 stores the pixel signals converted into digital signals by the signal conversion unit 422. The signal output unit 423 may have a latch circuit for storing the digital signals. The signal output unit 423 is arranged between the signal conversion unit 422 and the autonomous exposure processing unit 411 in the column direction. The signal output unit 423 outputs the pixel signals converted into digital signals to the outside of the control circuit unit 102. The signal output unit 423 is provided in the control block 400, extending in the row direction. The signal output unit 423 is arranged between the signal conversion unit 422 and the autonomous exposure processing unit 411 in the column direction.

[0043] 5 is an explanatory diagram showing an example of the internal configuration of the control block 400. The signal conversion unit 422 includes n comparators 501 and n storage units 502. The exposure control unit 412 includes a pixel block control unit 503 and a level shift unit 504. A combination of one comparator 501 and a storage unit 502 connected to that comparator 501 forms one ADC (Analog-to-Digital Converter) 500.

[0044] The comparators 501 are provided extending in the column direction in the control block 400. The n comparators 501 are arranged side by side in the row direction. A comparator 501 is arranged for every m pixels 201 arranged in the column direction in the corresponding pixel block 200. The comparators 501 sequentially read out pixel signals from the m pixels 201 arranged in the column direction in the corresponding pixel block 200 and convert them into digital signals.

[0045] The storage unit 502 stores the pixel signals converted into digital signals using the comparator 501. The storage unit 502 is provided on the negative side of the comparator 501 in the Y-axis direction in the signal conversion unit 422. For example, the storage unit 502 has a latch circuit. The storage unit 502 may have a memory configured using an SRAM or the like.

[0046] The pixel block control unit 503 controls the operation of the transfer unit 301 and the discharge unit 302 of the pixel 201 included in the corresponding pixel block 200. Specifically, the pixel block control unit 503 outputs a transfer control signal φTX for controlling the transfer unit 301 of the pixel 201 included in the corresponding pixel block 200, and a discharge control signal φPDRST for controlling the discharge unit 302 of the pixel 201 included in the corresponding pixel block 200. The pixel block control unit 503 is provided extending in the row direction in the control block 400. The pixel block control unit 503 is arranged between the level shift unit 504 and the autonomous exposure processing unit 411 in the column direction.

[0047] The level shift unit 504 adjusts the voltage level of the control signal output from the pixel block control unit 503. Specifically, the level shift unit 504 boosts the voltage level of the transfer control signal φTX output from the pixel block control unit 503. The level shift unit 504 also boosts the voltage level of the discharge control signal φPDRST output from the pixel block control unit 503.

[0048] The transfer unit 301 receives the transfer control signal φTX boosted by the pixel block control unit 503 via a transfer control line 311. The discharge unit 302 receives the discharge control signal φPDRST boosted by the pixel block control unit 503 via a discharge control line 312.

[0049] In this way, the pixel block control unit 503 boosts the transfer control signal φTX and the discharge control signal φPDRST to voltage levels used in the transfer unit 301 and the discharge unit 302 of the readout unit 310 of the pixel 201. The level shift unit 504 is provided in the control block 400, extending in the row direction.

[0050] The level shift unit 504 is provided closer to the outer periphery of the control block 400 than the pixel block control unit 503. The end of the level shift unit 504 on the positive side in the X-axis direction and the end on the negative side in the Y-axis direction are located at the outermost sides of the control block 400. The end of the level shift unit 504 on the negative side in the X-axis direction is in contact with the pixel driving unit 413.

[0051] The level shift unit 504 and pixel drive unit 413 handle the level-shifted signal. On the other hand, the autonomous exposure processing unit 411, pixel block control unit 503, level shift unit 504, and pixel drive unit 413 handle the pixel signal output from the first semiconductor substrate 110.

[0052] Here, each component of the control block 400 is formed in a well region provided in the second semiconductor substrate 120. The well regions are provided separately according to the voltage level of the signals to be handled. The well regions are separated depending on whether the power supply used is a digital power supply or an analog power supply. Furthermore, even if the signal conversion unit 422 uses the same analog power supply, it may be separated from an area that uses another analog power supply from the perspective of noise. Separating the well regions requires well isolation regions spaced apart according to the manufacturing process rules.

[0053] In the control block 400, well regions for forming the level shift unit 504 and the pixel driving unit 413 are separated from other well regions. For example, the level shift unit 504 and the pixel driving unit 413 can be provided in an L-shape, thereby sharing the well regions of the level shift unit 504 and the pixel driving unit 413. Sharing the well region makes it possible to omit a well isolation region, thereby improving layout efficiency.

[0054] The L-shaped pixel control unit 401 forms part of the outer periphery of the control block 400. This allows the well region to be shared with other control blocks 400 adjacent in the row and column directions.

[0055] 6 is an explanatory diagram showing an example of signal transmission between the first semiconductor substrate 110 and the second semiconductor substrate 120 in the image sensor 100. The global driving unit 600 is provided in the peripheral circuit unit 121 arranged on either side of the control circuit unit 102.

[0056] The transfer control line 311a and the discharge control line 312a are each connected to the pixels 201 included in the pixel block 200a. The transfer control line 311a is connected to the gate terminal of the transfer unit 301 included in the pixel 201 included in the pixel block 200a, and the discharge control line 312a is connected to the gate terminal of the discharge unit 302 included in the pixel 201 included in the pixel block 200a. The transfer control line 311a supplies the transfer control signal φTX output from the control block 400a to the transfer unit 301 included in the pixel 201 included in the pixel block 200a. The discharge control line 312a supplies the discharge control signal φPDRST output from the control block 400a to the discharge unit 302 included in the pixel 201 included in the pixel block 200a.

[0057] Similarly, the transfer control line 311b and the discharge control line 312b are each connected to the pixel 201 included in the pixel block 200b. The transfer control line 311b is connected to the gate terminal of the transfer unit 301 included in the pixel 201 included in the pixel block 200b, and the discharge control line 312b is connected to the gate terminal of the discharge unit 302 included in the pixel 201 included in the pixel block 200b. The transfer control line 311b supplies the transfer control signal φTX output from the control block 400b to the transfer unit 301 included in the pixel 201 included in the pixel block 200b. The discharge control line 312b supplies the discharge control signal φPDRST output from the control block 400b to the discharge unit 302 included in the pixel 201 included in the pixel block 200b.

[0058] When there is no need to distinguish between the transfer control lines 311a and 311b, they are referred to as transfer control lines 311. When there is no need to distinguish between the discharge control lines 312a and 312b, they are referred to as discharge control lines 312.

[0059] The transfer control line 311 and the discharge control line 312 are examples of local control lines connected to the first pixel of the pixel block 200. Note that the transfer control line 311 and the discharge control line 312 are commonly connected to n pixels 201 arranged in the row direction in the pixel block 200.

[0060] The global driver 600 outputs a reset control signal φRST, a selection control signal φSEL, and a transfer selection control signal φTXSEL. The global driver 600 is connected to a reset control line 313, a selection control line 314, and a transfer selection control line 603 that output control signals to each pixel block 200.

[0061] The global driving unit 600 supplies a reset control signal φRST and a selection control signal φSEL to the plurality of pixel blocks 200 via a reset control line 313 and a selection control line 314. The global driving unit 600 supplies a transfer selection control signal φTXSEL to the plurality of control blocks 400 via a transfer selection control line 603.

[0062] The transfer selection control signal φTXSEL is supplied from the global drive unit 600 to the control block 400 in order to control the exposure time for each pixel block 200. The control block 400, to which the transfer selection control signal φTXSEL is supplied, outputs the transfer selection control signal φTXSEL to the corresponding pixel block 200. The control block 400 determines whether or not to input the transfer selection control signal φTXSEL to the pixel 201 as the transfer control signal φTX or the discharge control signal φPDRST. As a result, input of the transfer control signal φTX or the discharge control signal φPDRST to the pixel 201 is skipped.

[0063] For example, when the transfer control signal φTX determines the end time of exposure, the control block 400 extends the exposure time by skipping the transfer control signal φTX. Also, when the transfer control signal φTX determines the start time of exposure, the control block 400 can shorten the exposure time by skipping the transfer control signal φTX. In this way, the exposure time of the pixel block 200 can be adjusted by the transfer selection control signal φTXSEL. The same applies when the discharge control signal φPDRST determines the start or end time of exposure.

[0064] The reset control line 313, the selection control line 314, and the transfer selection control line 603 are provided in common to a plurality of pixel blocks 200. The reset control line 313, the selection control line 314, and the transfer selection control line 603 are wired so as to cross the first semiconductor substrate 110 in the row direction. The reset control line 313, the selection control line 314, and the transfer selection control line 603 may also be wired so as to cross the first semiconductor substrate 110 in the column direction.

[0065] For example, the reset control line 313 is connected to the gate terminal of the reset unit 304 of the pixel 201 in the pixel block 200 and supplies the reset control signal φRST. The selection control line 314 is connected to the gate terminal of the selection unit 352 of the pixel 201 in the pixel block 200 and supplies the selection control signal φSEL. The transfer selection control line 603 is connected to each of the multiple control blocks 400 and supplies the transfer selection control signal φTXSEL to the pixel control unit 401.

[0066] Although the global driving unit 600 outputs the transfer selection control signal φTXSEL to the control block 400 from the second semiconductor substrate 120 via the first semiconductor substrate 110, the transfer selection control signal φTXSEL may be output to the control block 400 without passing through the first semiconductor substrate 110. In this case, the transfer selection control line 603 is provided on the second semiconductor substrate 120.

[0067] The bonding portions 610 are provided on the bonding surfaces where the first semiconductor substrate 110 and the second semiconductor substrate 120 are bonded to each other. The bonding portions 610 align the transfer control lines 311, the discharge control lines 312, and the transfer selection control lines 603 between the first semiconductor substrate 110 and the second semiconductor substrate 120. Each of the bonding portions 610 is composed of a pair of conductive bonding pads, and is bonded and electrically connected by applying pressure to the first semiconductor substrate 110 and the second semiconductor substrate 120, for example.

[0068] The image sensor 100 controls the exposure time for each pixel block 200 by changing the timing of at least one of the transfer unit 301 and the discharge unit 302 using local control lines such as the transfer control line 311 and the discharge control line 312. The image sensor 100 can control the exposure time with fewer control lines by combining local control lines such as the transfer control line 311 and the discharge control line 312 with global control lines such as the reset control line 313, the selection control line 314, and the transfer selection control line 603.

[0069] FIG. 7 is an explanatory diagram showing an example of an X-Z cross section of an image sensor 100 according to this embodiment. While FIG. 7 shows a back-illuminated image sensor 100, the image sensor 100 is not limited to a back-illuminated type. The image sensor 100 includes a microlens layer 700, a color filter layer 702, a first semiconductor substrate 110, a second semiconductor substrate 120, and a third semiconductor substrate 130. As shown in the figure, light from a subject is incident in the direction indicated by the outline arrow (the negative Z-axis direction in the figure). The surface of the first semiconductor substrate 110 on which light is incident (the positive Z-axis side in the figure) may be referred to as the front surface, and the surface on the opposite side (the negative Z-axis side in the figure) may be referred to as the back surface.

[0070] The microlens layer 700 has a plurality of microlenses 701. The plurality of microlenses 701 are stacked on the positive side of the Z axis relative to the color filter layer 702. Light is incident on the microlenses 701. The microlenses 701 focus the incident light onto the photoelectric conversion unit 300. A microlens 701 may be provided for each photoelectric conversion unit 300. The optical axis L of the microlens 701 is the stacking direction (direction parallel to the Z axis) of the first semiconductor substrate 110, the second semiconductor substrate 120, and the third semiconductor substrate 130.

[0071] The color filter layer 702 has a plurality of color filters 703 and a passivation film 704. The color filter layer 702 is stacked on the positive side of the Z axis relative to the first semiconductor layer 711. The color filter 703 is an optical filter that transmits light in a specific wavelength region. The color filter 703 is an optical filter having specific spectral characteristics. The multiple color filters 703 have multiple optical filters with different spectral characteristics, and transmit light in different wavelength regions. The multiple color filters 703 are arranged in a specific array (for example, a Bayer array).

[0072] An example of the first semiconductor substrate 110 is a back-illuminated CMOS image sensor. The first semiconductor substrate 110 has a first semiconductor layer 711 and a first wiring layer 712. The first semiconductor layer 711 is provided on the positive side of the Z axis relative to the first wiring layer 712. The first semiconductor layer 711 has a plurality of pixel blocks 200 arranged two-dimensionally in the row and column directions. The first semiconductor layer 711 has a plurality of pixels 201 arranged two-dimensionally in the row and column directions. Each of the plurality of pixels 201 has a plurality of photoelectric conversion units 300 that accumulate charge based on incident light, and a plurality of readout units 310.

[0073] The first wiring layer 712 is provided closer to the second semiconductor substrate 120 (toward the negative side of the Z axis in the drawing) than the first semiconductor layer 711. The first wiring layer 712 has a plurality of wires 713 made of a conductor film (metal film), a plurality of bonding pads 714, and an insulating film (insulating layer).

[0074] The first wiring layer 712 has a plurality of wirings 713 electrically connected to a power supply, a circuit, etc. In the first semiconductor substrate 110, the wirings 713 specifically include, for example, a power supply wiring to which a predetermined power supply voltage is supplied, a signal line 202 that transmits pixel signals from the first semiconductor substrate 110 (pixels) to the second semiconductor substrate 120, a transfer control line 311 that transmits control signals from the second semiconductor substrate 120 to the first semiconductor substrate 110 (pixels), a discharge control line 312, a reset control line 313, a selection control line 314, and a transfer selection control line 603. The first wiring layer 712 may be multi-layered, and may include passive elements and active elements.

[0075] The bonding pads 714 are provided on the surface (the surface on the negative side of the Z axis) of the first wiring layer 712 and are connected to the wiring 713. As will be described later, the bonding pads 714 are also used to assist in connecting layers together. The bonding pads 714 are formed of a conductive material such as copper. The bonding pads 714 may also be formed of gold, silver, or aluminum. An insulating layer (insulating film) is formed between the multiple wirings 713 and between the multiple bonding pads 714.

[0076] The second semiconductor substrate 120 has a second semiconductor layer 721, a second wiring layer 722, and a wiring layer 723. The second wiring layer 722 is provided closer to the first semiconductor substrate 110 than the second semiconductor layer 721 (on the positive side of the Z axis in the drawing). The wiring layer 723 is provided closer to the third semiconductor substrate 130 than the second semiconductor layer 721 (on the negative side of the Z axis in the drawing), and is provided between the second semiconductor layer 721 and the third semiconductor substrate 130. The second semiconductor layer 721 has a control circuit unit 102 and a peripheral circuit unit 121. The control circuit unit 102 has a plurality of control blocks 400 arranged two-dimensionally in the row and column directions.

[0077] Similar to the first semiconductor substrate 110, the second semiconductor substrate 120 has a plurality of wirings 713 provided on the second wiring layer 722, a plurality of bonding pads 714 provided on the second wiring layer 722 and the wiring layer 723, and an insulating film (insulating layer) provided on the second wiring layer 722 and the wiring layer 723.

[0078] The second wiring layer 722 has a plurality of wirings 713 and bonding pads 714 for electrically connecting to a power supply or circuit, transmitting signals from the pixel unit 101 to the control circuit unit 102, and transmitting signals from the control circuit unit 102 to the pixel unit 101. In the second semiconductor substrate 120, the wirings 713 specifically include, for example, a power supply wiring for supplying a predetermined power supply voltage, a signal line 202 for transmitting pixel signals from the first semiconductor substrate 110 (pixels) to the second semiconductor substrate 120, and a transfer control line 311, a discharge control line 312, a reset control line 313, a selection control line 314, and a transfer selection control line 603 for transmitting control signals from the second semiconductor substrate 120 to the first semiconductor substrate 110 (pixels). The second wiring layer 722 may be multi-layered and may include passive and active elements. The wirings 713 and bonding pads 714 may also be provided in the wiring layer 723.

[0079] The second semiconductor substrate 120 further includes through-silicon vias (TSVs) 724 that connect the circuits provided on the front and back surfaces of the second semiconductor substrate 120 to each other. The TSVs 724 are preferably provided in the peripheral region. The TSVs 724 transmit image data and the like generated by the data processing unit 103 to the first semiconductor substrate 110. The TSVs 724 may also be provided in the first semiconductor substrate 110 and the third semiconductor substrate 130.

[0080] The third semiconductor substrate 130 has a third semiconductor layer 731 in which the data processing unit 103 is provided, and a third wiring layer 732. The third wiring layer 732 is provided between the third semiconductor layer 731 and the second semiconductor substrate 120.

[0081] Similar to the first semiconductor substrate 110, the third semiconductor substrate 130 has wiring 713 and a plurality of bonding pads 714 provided on a third wiring layer 732. The third wiring layer 732 has the plurality of wirings 713 and bonding pads 714 for electrical connection to a power supply, a circuit, etc., for transmitting signals from the control circuit unit 102 to the data processing unit 103, and for transmitting signals from the data processing unit 103 to the control circuit unit 102 of the second semiconductor substrate 120.

[0082] The first semiconductor substrate 110, the second semiconductor substrate 120, and the third semiconductor substrate 130 are stacked by electrical connection between the bonding pads 714 provided on each layer and bonding between the wiring layers (insulating layers) of each layer.

[0083] When the first semiconductor substrate 110 and the second semiconductor substrate 120 are stacked, an interface 720 is formed between the negative surface of the first wiring layer 712 along the Z axis and the positive surface of the second wiring layer 722. Similarly, when the second semiconductor substrate 120 and the third semiconductor substrate 130 are stacked, an interface 730 is formed between the negative surface of the wiring layer 723 along the Z axis and the positive surface of the third wiring layer 732. A plurality of bonding pads 714 are arranged on the interface 720 and the interface 730. Specifically, corresponding bonding pads 714 are aligned, and the two layers are stacked, thereby electrically connecting the aligned bonding portions.

[0084] The first semiconductor substrate 110, the second semiconductor substrate 120, and the third semiconductor substrate 130 may be stacked in the form of wafers before being made into chips, and then formed (individuated) by dicing the stacked wafers, or may be formed by dicing each of the first semiconductor substrate 110, the second semiconductor substrate 120, and the third semiconductor substrate 130 wafers and then stacking them.

[0085] Fig. 8 is a timing chart showing an example 1 of imaging operation of the image sensor 100. Fig. 8 shows an example of imaging operation in which the drive of the image sensor 100 is controlled by the transfer control signal φTX, the discharge control signal φPDRST, the reset control signal φRST, and the selection control signal φSEL. In Fig. 8, the discharge control signal φPDRST is locally controlled, and the transfer control signal φTX, the reset control signal φRST, and the selection control signal φSEL are globally controlled. Note that the suffixes <1>, <2>, ..., <m> at the end of each signal on the left side indicate the row number of the pixel 201 within the pixel block.

[0086] The discharge control signal φPDRST controls the timing at which exposure starts. The exposure start timing corresponds to the falling edge of the discharge control signal φPDRST (for example, time T1). That is, before the exposure start time T1, the discharge control signal φPDRST turns on the discharge unit 302 to discharge the charge accumulated in the photoelectric conversion unit 300, and exposure starts at the falling edge of the discharge control signal φPDRST. Because the discharge control signal φPDRST is locally controlled, the exposure time can be adjusted for each pixel block 200.

[0087] The transfer control signal φTX controls the timing to end exposure. At time T3, the transfer control signal φTX turns on the transfer unit 301, thereby transferring the charge accumulated in the photoelectric conversion unit 300 to the FD 303. The timing to end exposure corresponds to the falling edge of the transfer control signal φTX (for example, time T4). Because the transfer control signal φTX is a globally controlled signal, the timing to end exposure is the same for each pixel block 200.

[0088] The reset control signal φRST controls the timing of discharging the charge accumulated in the FD 303. At time T2, the reset control signal φRST turns on the reset unit 304, thereby discharging the charge in the FD 303. By discharging the charge in the FD 303 before the timing of the end of exposure, the influence of the charge remaining in the FD 303 when the charge is transferred from the photoelectric conversion unit 300 can be suppressed.

[0089] The selection control signal φSEL is a signal for selecting an arbitrary pixel 201. The selection control signal φSEL controls the on / off of the selection unit 352. At time T2, the selection control signal φSEL is set to high. At time T3, the pixel 201 for which the selection control signal φSEL is set to high outputs a pixel signal to the signal line 202 in response to the transfer control signal φTX being turned on. On the other hand, the pixel 201 for which the selection control signal φSEL is not set to high does not output a pixel signal.

[0090] The image sensor 100 locally controls the discharge control signal φPDRST to change the exposure start timing for each pixel block 200, thereby controlling the exposure time for each pixel block 200. The image sensor 100 may also locally control the transfer control signal φTX to control the exposure end timing for each pixel block 200. The image sensor 100 may also locally control both the transfer control signal φTX and the discharge control signal φPDRST to control both the exposure start timing and end timing for each pixel block 200.

[0091] <Fig. 9 Data compression using exposure time control for each pixel block 200> Next, data compression using exposure time control for each pixel block 200 in the image sensor 100 will be described. If image data 901 captured by the image sensor 100 is output directly to the outside of the image sensor 100, the amount of output data will increase. For this reason, by implementing a data compression mechanism inside the image sensor 100, the image data generated by the image sensor 100 is compressed within the image sensor 100 and output to the outside of the image sensor 100. This will be described in detail below.

[0092] 9 is an explanatory diagram showing an example of data compression using exposure time control for each pixel block 200. For simplicity, a fixed-point image capture will be used as an example. The image sensor 100 generates image data 901. As shown in the luminance value range 911 of the histogram 910 of the image data 901, light and dark are distributed over a wide range.

[0093] The image sensor 100 generates an exposure value map 902 based on the average luminance value of each pixel block 200 in the image data 901. The exposure value map 902 is a collection of exposure values ​​that indicate the exposure time for each pixel block 200, with larger exposure values ​​indicating longer exposure times. The image sensor 100 brightens the same portion of the exposure value map 902 as the image data 901 becomes darker, thereby lengthening the exposure time for the pixel block 200 in that portion, and darkens the same portion of the exposure value map 902 as the image data 901 becomes brighter, thereby shortening the exposure time for the pixel block 200 in that portion.

[0094] Because this is fixed-point imaging, the image sensor 100 generates image data 903 by controlling exposure for each control block 400 using the exposure value map 902. Specifically, for example, the image sensor 100 brightens the same part of the image data 903 the darker the part of the exposure value map 902 (longer the exposure time), and darkens the same part of the image data 903 the brighter the part of the exposure value map 902 (shorter the exposure time).

[0095] As shown in histogram 930 of image data 903, light and dark are distributed within a certain luminance value range 931 that is narrower than the luminance value range 911 of histogram 910. Therefore, by compressing and encoding such image data 903 using a data compression mechanism 904, it is possible to achieve higher compression efficiency than image data 901 that does not use exposure value map 902.

[0096] In the following description, the image data 901 may be referred to as the previous frame 901 and the image data 903 may be referred to as the current frame 903 .

[0097] 10 is a block diagram showing an example of a configuration related to data compression of the image sensor 100. The image sensor 100 has a pixel unit 101, a data compression mechanism 904 (plurality of first compression circuits 1002), a pixel drive unit 413, a signal processing circuit 1001, an image output unit 1003, and a second compression circuit 1004. The external processing device 1010 has a first decoding circuit 1011, a second decoding circuit 1012, and an image processing circuit 1013.

[0098] The pixel unit 101 outputs image data (previous frame) 901 as an image signal that is the source of generating the exposure value map 902, and the image signal output by each pixel block 200 of the image data (previous frame) 901 is referred to as image signal <901>.

[0099] In addition, the pixel unit 101 outputs the current frame 903 as an image signal generated from the imaging element 100 by exposure control using the exposure value map 902, and the image signal output by each pixel block 200 of the current frame 903 is referred to as image signal <903>.

[0100] In addition, of the exposure value map 902 corresponding to the entire previous frame 901, the map area generated based on the image signal <901> is represented as a map signal <902>.

[0101] In this example, a case will be described in which an image signal <901> is output to a signal processing circuit 1001, and an image signal <903> is output to a first compression circuit 1002 of a data compression mechanism 904.

[0102] First, a case where the previous frame 901 is output to the signal processing circuit 1001 will be described. The pixel unit 101 outputs an image signal <901> to the corresponding signal processing circuit 1001. The signal processing circuit 1001 is provided in the control block 400. That is, there are the same number of signal processing circuits 1001 as there are control blocks 400. The signal processing circuit 1001 includes an autonomous exposure processing unit 411 and an exposure control unit 412. The signal processing circuit 1001 outputs an exposure value map 902 to the data compression mechanism 904 and the second compression circuit 1004. The signal processing circuit 1001 outputs a control signal to the pixel driving unit 413 for controlling the exposure time (charge accumulation time of the photoelectric conversion unit 300) of the pixels 201 included in the corresponding pixel block 200. Details of the signal processing circuit 1001 will be described later with reference to FIG. 11.

[0103] There are the same number of pixel driving units 413 as there are control blocks 400. Based on the map signal <902>, the pixel driving units 413 set pixel driving signals (discharge control signal φPDRST, reset control signal φRST, selection control signal φSEL) to the corresponding pixel blocks 200. The pixel driving units 413 output the pixel driving signals (discharge control signal φPDRST, reset control signal φRST, selection control signal φSEL) to the corresponding pixel blocks 200.

[0104] The second compression circuit 1004 compresses (e.g., run-length encodes) the map signal <902> from the signal processing circuit 1001 and outputs it to the second decoding circuit 1012 of the external processing device 1010. The second compression circuit 1004 acquires and combines the map signals <902> from each signal processing circuit 1001. The second compression circuit 1004 compresses the combined exposure value map 902. The compressed exposure value map 902 is referred to as a compressed exposure value map (902). The second compression circuit 1004 outputs the compressed exposure value map (902) to the second decoding circuit 1012.

[0105] Next, a case where the image signal <903> is output to the first compression circuit 1002 of the data compression mechanism 904 will be described. The pixel unit 101 outputs the image signal <903> to the corresponding first compression circuit 1002.

[0106] A first compression circuit 1002 is provided for each pixel block 200. The first compression circuit 1002 compresses the image signal <903> based on the map signal <902>. The compressed image signal <903> is referred to as a compressed image signal (<903>). The first compression circuit 1002 outputs the compressed image signal (<903>) to the image output unit 1003.

[0107] The image output unit 1003 combines the compressed image signals (< 903 >) to generate compressed image data ( 903 ) and outputs it to the first decoding circuit 1011 .

[0108] Next, the external processing device 1010 will be described. The external processing device 1010 is communicably connected to the image sensor 100. The external processing device 1010 has a first decoding circuit 1011, a second decoding circuit 1012, and an image processing circuit 1013. The first decoding circuit 1011 decodes (decompresses) the compressed image data (903) and outputs the decoded current frame 903 to the image processing circuit 1013. The second decoding circuit 1012 decodes (decompresses) the compressed exposure value map (902) and outputs the decoded exposure value map 902 to the image processing circuit 1013.

[0109] The image processing circuit 1013 is an integrated circuit that performs image processing such as color interpolation, white balance adjustment, edge enhancement, gamma correction, and tone conversion. The image processing circuit 1013 also converts the luminance value of the current frame 903 using the exposure value map 902. In this example, since the image sensor 100 is performing fixed-point imaging, the current frame 903 is converted into image data similar to that of the previous frame 901 through this conversion.

[0110] The external processing device 1010 may be configured without the second compression circuit 1004 and the second decoding circuit 1012. In this case, the signal processing circuit 1001 is connected to the image processing circuit 1013, combines the map signals <902> to generate an exposure value map 902, and outputs the map signal <902> to the image processing circuit 1013.

[0111] 11 is a block diagram showing the detailed configuration of the signal processing circuit 1001. The signal processing circuit 1001 has an autonomous exposure processing unit 411 and an exposure control unit 412. The autonomous exposure processing unit 411 has an average value calculation unit 1101, an exposure value calculation unit 1102, and a map memory 1103. The reference value 1104 is set by, for example, the external processing device 1010.

[0112] The average value calculation unit 1101 calculates the average value of the luminance of the image signal <901> of the previous frame 901. The image signal <901> includes m×n pixel signals output from the pixel block 200. The average value calculation unit 1101 calculates the average value of the luminance values ​​of the m×n pixel signals (hereinafter referred to as the average luminance value).

[0113] The exposure value calculation unit 1102 refers to the reference value 1104 and the map signal <902> before updating, and calculates an exposure value for the pixel block 200 based on the average luminance value in the image signal <901> calculated by the average value calculation unit 1101. The exposure value calculation unit 1102 updates the map signal <902> and stores it in the map memory 1103.

[0114] The exposure control unit 412 converts the exposure value calculated by the exposure value calculation unit 1102 into an exposure time, and outputs it to the pixel driving unit 413 as a control signal.

[0115] The pixel driving unit 413 generates pixel driving signals (φPDRST, φTX) for each pixel 201 in the pixel block 200 based on a control signal from the exposure control unit 412 , and outputs the signals to the pixels 201 in the pixel block 200 .

[0116] 12 is an explanatory diagram showing an example of generating an exposure value map 902. In this example, for simplicity, the number of pixel blocks 200 is assumed to be M=4 and N=4. The data compression mechanism 904 generates the exposure value map 902 when the previous frame 901 is input in the following steps (1) to (7).

[0117] (1) When the image signal <901> of the previous frame 901 is input, the average value calculation unit 1101 in each signal processing circuit 1001 calculates the average luminance value of the image signal <901>. For ease of understanding, an average luminance value map 1201 is used. The average luminance value map 1201 is a map combining the average luminance values ​​of each image signal <901>. For example, the hatched square in the upper left indicates an average luminance value of "300" in the image signal <901> from the upper left pixel block 200 of the pixel unit 101, and the hatched square in the lower right indicates an average luminance value of "1100" in the image signal <901> from the lower right pixel block 200 of the pixel unit 101. Each average value calculation unit 1101 calculates the average luminance value of each square in the average luminance value map 1201.

[0118] (2) When the exposure value calculation unit 1102 in each signal processing circuit 1001 receives the average luminance value from the average value calculation unit 1101, it reads out the map signal <902> from the map memory 1103. For the sake of convenience, Fig. 12 shows an exposure value map 902 combined with the map signal <902> as being stored, but in reality, the map signal <902> is stored in the map memory 1103. Furthermore, exposure value map 902A shows the exposure value map 902 before updating by each exposure value calculation unit 1102, and exposure value map 902B shows the exposure value map 902 after updating by each exposure value calculation unit 1102.

[0119] In the exposure value map 902A, for example, the hatched square in the upper left indicates an exposure value TV=5 corresponding to the image signal <901> from the upper left pixel block 200 of the pixel unit 101, and the hatched square in the lower right indicates an exposure value TV=5 corresponding to the image signal <901> from the lower right pixel block 200 of the pixel unit 101.

[0120] (3) Furthermore, when the exposure value calculation unit 1102 receives the average brightness value from the average value calculation unit 1101, it refers to the upper and lower limits of the average brightness value defined as the reference value 1104. Here, as an example, the upper limit of the average brightness value is set to 1000 and the lower limit of the average brightness value is set to 400.

[0121] (4) Furthermore, the exposure value calculation unit 1102 calculates an exposure value for the pixel block 200 in order to generate an image signal <903> for the next frame (in this example, the current frame 903). Here, an explanation will be given using the previous frame 901, the average brightness value map 1201, and the upper left square of the exposure value map 902A as an example.

[0122] The average brightness value "300" indicated by the top left square of the average brightness value map 1201 is lower than the lower limit of "400" for average brightness values. In this case, the exposure value calculation unit 1102 increases the exposure value to be greater than the exposure value "5" of the top left square of the exposure value map 902A in order to shorten the exposure time of the pixel block 200 corresponding to the top left square. For example, the exposure value calculation unit 1102 updates the exposure value "5" of the top left square to "6." The exposure value calculation unit 1102 stores the exposure value "6", which is the updated map signal <902>, in the map memory 1103 and outputs it to the exposure control unit 412.

[0123] Similarly, the previous frame 901, the average brightness value map 1201, and the bottom right square of the exposure value map 902A will be used as an example. The average brightness value "1100" indicated by the bottom right square of the average brightness value map 1201 exceeds the upper limit of "1000" for average brightness values. In this case, the exposure value calculation unit 1102 increases the exposure value to be greater than the exposure value "5" in the bottom right square of the exposure value map 902A in order to extend the exposure time of the pixel block 200 corresponding to the bottom right square. For example, the exposure value calculation unit 1102 updates the exposure value "5" in the bottom right square to "4." The exposure value calculation unit 1102 stores the exposure value "4," which is the updated map signal <902>, in the map memory 1103 and outputs it to the exposure control unit 412.

[0124] If the average brightness value of the average brightness value map 1201 is within the range of the average brightness value in the reference value 1104, which is equal to or less than the upper limit "1000" and equal to or greater than the lower limit "400", the exposure value of the corresponding square in the exposure value map 902A is not changed.

[0125] (5) The exposure value calculation unit 1102 stores the updated exposure value in the map memory 1103 as a map signal <902>.

[0126] (6) The exposure control unit 412 converts the exposure value from the exposure value calculation unit 1102 into an exposure time by referring to the exposure table 1202. The exposure control unit 412 outputs the converted exposure time to the pixel driving unit 413 as a control signal.

[0127] (7) The exposure control unit 412 generates pixel drive signals (φPDRST, φTX) so as to achieve the exposure time output as a control signal from the exposure control unit 412. For example, for the pixel block 200 that outputs the image signal <901> in the hatched square in the upper left, the average luminance value "300" in the image signal <901> is smaller than the lower limit of the average luminance value "400", so the pixel drive unit 413 drives the pixels 201 in the pixel block 200 so as to make the exposure time longer than the previous time (during long-second exposure).

[0128] For the pixel block 200 that output the image signal <901> in the hatched square at the bottom right, the average brightness value "1100" in the image signal <901> is greater than the upper limit of the average brightness value "1000", so the pixel driving unit 413 drives the pixel 201 in the pixel block 200 so that the exposure time is shorter than the previous time (during short-second exposure).

[0129] 13 and 14 First Compression Circuit 1002> Fig. 13 is a block diagram showing a detailed configuration example of the first compression circuit 1002. The first compression circuit 1002 has a color separation unit 1301, a DCT (Discrete Cosine Transform) unit 1302, a coefficient conversion table 1303, coding tables 1304 (first coding table 1341, second coding table 1342), a quantization unit 1305, an inter-block difference calculation unit 1306, and a coding unit 1307.

[0130] A color separation unit 1301 separates the image signal <903> into R (red), Gr (green), Gb (green), and B (blue). By performing the color separation process, offset component differences between the colors are eliminated.

[0131] FIG. 14 is an explanatory diagram showing an example of color separation by the color separation unit 1301. The image signal <903> is data in which R (red), Gr (green), Gb (green), and B (blue) are arranged in a Bayer pattern. The color separation unit 1301 separates R (red) pixels from the image signal <903> and outputs an image signal <903R>. The color separation unit 1301 separates Gr (green) pixels from the image signal <903> and outputs an image signal <903Gr>. The color separation unit 1301 separates Gb (green) pixels from the image signal <903> and outputs an image signal <903Gb>. The color separation unit 1301 separates B (blue) pixels from the image signal <903> and outputs an image signal <903B>.

[0132] 13 , the DCT unit 1302 performs discrete cosine transform processing on each of the image signals <903R>, <903Gr>, <903Gb>, and <903B>. Specifically, the DCT unit 1302 includes an R processing unit 1321, a Gr processing unit 1322, a Gb processing unit 1323, and a B processing unit 1324.

[0133] The R processing unit 1321 performs a discrete cosine transform on the image signal <903R> and outputs the AC and DC components to the quantization unit 1305. The Gr processing unit 1322 performs a discrete cosine transform on the image signal <903Gr> and outputs the AC and DC components to the quantization unit 1305. The Gb processing unit 1323 performs a discrete cosine transform on the image signal <903Gb> and outputs the AC and DC components to the quantization unit 1305. The B processing unit 1324 performs a discrete cosine transform on the image signal <903B> and outputs the AC and DC components to the quantization unit 1305.

[0134] The coefficient conversion table 1303 is generated by converting the map signal <902> from the map memory 1103 with reference to the exposure table 1202. The coefficient conversion table 1303 is a table that can convert the exposure value TV into a compression coefficient. The coefficient conversion table 1303 will be described later with reference to FIG. 15.

[0135] The coding table 1304 includes a first coding table 1341 and a second coding table 1342. The first coding table 1341 is a table for coding AC components, and the second coding table 1342 is a table for coding DC components.

[0136] The quantization unit 1305 quantizes the AC components and DC components from the DCT unit 1302 for each pixel block 200. A specific example of quantization will be described later with reference to FIG.

[0137] The adjacent block difference calculation unit 1306 calculates, for each pixel block 200, a signal (hereinafter referred to as a DC component inter-block difference signal) indicating the difference between the DC component in the image signal <901> of the previous frame 901 quantized for each pixel block 200 by the quantization unit 1305 and the DC component in the image signal <903> of the current frame 903 quantized for each pixel block 200 by the quantization unit 1305.

[0138] The encoding unit 1307 has a first encoding unit 1371 and a second encoding unit 1372. The first encoding unit 1371 refers to a first encoding table 1341 and encodes the AC components quantized by the quantization unit 1305. The first encoding unit 1371 outputs the encoded AC components to the image output unit 1003.

[0139] The second encoding unit 1372 encodes the DC component inter-block difference signal by referring to the second encoding table 1342. The second encoding unit 1372 outputs the encoded DC component to the image output unit 1003. Details of the encoding unit 1307 will be described later with reference to FIG. 16 .

[0140] The image output unit 1003 outputs the coded frequency signal (<1600>), which is a combination of the coded AC components and DC components from the first compression circuit 1002, as a compressed image signal (<903>) to the first decoding circuit 1011. Note that the image output unit 1003 may output the coded frequency signal (<1600>) to a memory arranged upstream of the first decoding circuit 1011. In this case, the first decoding circuit 1011 decodes the coded frequency signal (<1600>) at any timing (for example, the timing at which image data 903 is displayed on a display or the timing at which an instruction is given by a user operation).

[0141] The external processing device 1010 includes a first decoding circuit 1011, a second decoding circuit 1012, and a setting unit 1330. The setting unit 1330 presets a coefficient conversion table 1303 and an encoding table 1304 in a register (not shown) in the first compression circuit 1002.

[0142] <Fig. 15 Quantization Process of Quantizer 1305> Fig. 15 is an explanatory diagram showing an example of the quantization process of the quantizer 1305. The coefficient conversion table 1303 has a first coefficient conversion table 1501 and a second coefficient conversion table 1502. The first coefficient conversion table 1501 is a table used in lossless processing, and the second coefficient conversion table 1502 is a table used in lossy processing. When there is no need to distinguish between the first coefficient conversion table 1501 and the second coefficient conversion table 1502, they are referred to as the coefficient conversion table 1303.

[0143] The coefficient conversion table 1303 is a table having exposure values ​​TV and compression coefficients. The compression coefficients correspond to exposure times. The first coefficient conversion table 1501 has the same compression coefficient value ("1" in this example) regardless of the exposure value TV. The second coefficient conversion table 1502 has different compression coefficients depending on the exposure value TV. However, the compression coefficients may be the same across multiple exposure values ​​TV. The compression coefficients to be applied in the second coefficient conversion table 1502 are set in advance by the setting unit 1330 for each first compression circuit 1002. This allows the compression rate to be different for each image signal <903>.

[0144] The frequency signal 1503 is a signal composed of a DC component (hatched squares) and an AC component (non-hatched squares) from the DCT unit 1302. The frequency signal 1503 exists for each of R, Gr, Gb, and B.

[0145] When performing lossless compression, the quantization unit 1305 divides the frequency signal 1503 by the compression coefficient “1” in the first coefficient conversion table 1501 to output the AC components of the quantized frequency signal 1504 to the first encoding unit 1371.

[0146] When performing lossy compression, the quantization unit 1305 divides the frequency signal 1503 by the compression coefficient specified in the second coefficient conversion table 1502 (rounding off any fractional part, for example), and outputs the DC component of the quantized frequency signal 1504 to the adjacent block difference calculation unit 1306. Fig. 15 shows the quantization process when the compression coefficient is "2".

[0147] Whether the quantization unit 1305 performs lossy compression or lossless compression is set in advance by the setting unit 1330 (for example, by a user operation).

[0148] <Fig. 16: Encoding Process by Encoding Unit 1307> Fig. 16 is an explanatory diagram showing an example of encoding process by the encoding unit 1307. A quantized frequency signal 1600 is composed of quantized AC components from the quantization unit 1305 and a DC component inter-block difference signal from the adjacent block difference calculation unit 1306. In Fig. 15, the DC component of frequency signal 1503 is "-50", and the DC component of quantized frequency signal 1504 is "-25". Therefore, the DC component inter-block difference signal is "-25". A quantized frequency signal 1600 exists for each of R, Gr, Gb, and B.

[0149] The encoding table 1304 includes input data, Huffman codes, and additional bits. The encoded data is data in which the additional bits are added to the end of the Huffman code.

[0150] The encoding unit 1307 refers to the encoding table 1304, accepts the value of each square of the quantized frequency signal 1600 as input data, and encodes the input data. For example, if the input data is "-1", the corresponding Huffman code is "010" and the corresponding additional bit is "0". Therefore, when the input data "-1" is input to the encoding unit 1307, the encoding unit 1307 outputs encoded data "0100" in which the additional bit "0" is added to the end of the Huffman code "010".

[0151] 17 Image Data Conversion Processing by Image Processing Circuit 1013 Next, a description will be given of image data conversion processing by the image processing circuit 1013 in the external processing device 1010. For example, when image data 901 and 903 are captured using fixed-point shooting, the current frame 903 will be an image similar to the previous frame 901, but the current frame 903 is affected by the exposure value map 902. Therefore, the image processing circuit 1013 converts the luminance value of each pixel in the current frame 903 using the exposure value map 902, thereby restoring the image data to be similar to that of the previous frame 901.

[0152] 17 is a block diagram showing an example of image data conversion processing by the image processing circuit 1013. The external processing device 1010 has a memory 1701, an inverse DCT unit 1702, and a coefficient generation unit 1703.

[0153] The memory 1701 holds the coded frequency signals (<1600>) for each of R, Gr, Gb, and B output from each first compression circuit 1002. The held coded frequency signals (<1600>) for each of R, Gr, Gb, and B are read into the first decoding circuit 1011. The first decoding circuit 1011 decodes the coded frequency signals (<1600>) for each of R, Gr, Gb, and B, and outputs the decoded frequency signals 1600 for each of R, Gr, Gb, and B to the inverse DCT unit 1702.

[0154] The inverse DCT unit 1702 performs an inverse discrete cosine transform on the decoded frequency signals 1600 for each of R, Gr, Gb, and B, demodulating them into quantized DC components output from the quantization unit 1305 to the adjacent block difference calculation unit 1306, synthesizes the converted color-separated image signals <903R>, <903Gr>, <903Gb>, and <903B>, and outputs the synthesis result (corresponding to the current frame 903) to the image processing circuit 1013.

[0155] Specifically, for demodulation of the DC components, the inverse DCT unit 1702 outputs, as a difference signal, the difference value of the DC components for each pixel block 200 sent from the quantization unit 1305. For example, if the DC component of pixel block 200-1 is "10," the DC component of pixel block 200-2 is "6," and the DC component of pixel block 200-3 is "4," the difference signal of pixel block 200-1 is "10," the difference signal between the DC component "10" of pixel block 200-1 and the DC component "6" of pixel block 200-2 is "4" (=10-6), and the difference signal between the DC component "6" of pixel block 200-2 and the DC component "4" of pixel block 200-3 is "2" (=6-4).

[0156] Then, the inverse DCT unit 1702 demodulates the DC component "6" of the pixel block 200-2 by subtracting the difference signal "4" between the DC component "10" of the pixel block 200-1 and the DC component "6" of the pixel block 200-2 from the DC component "10" of the pixel block 200-1.

[0157] Furthermore, the inverse DCT unit 1702 demodulates the DC component "4" of the pixel block 200-3 by subtracting the difference signal "2" between the DC component "6" of the pixel block 200-2 and the DC component "4" of the pixel block 200-3 from the DC component "6" of the pixel block 200-2.

[0158] The coefficient generation unit 1703 generates a coefficient based on the coefficient conversion table 1303 and the exposure value map 902. Specifically, for example, the coefficient generation unit 1703 generates a coefficient that simultaneously divides back both the correction coefficient and the compression coefficient of the exposure value TV. Specifically, for example, if the correction coefficient of the exposure value TV is "2" and the compression coefficient is "4," the correction coefficient multiplied by the compression coefficient multiplied by the luminance value in the synthesis result (corresponding to the current frame 903) is the luminance value of the previous frame 901, and therefore the coefficient "8" is generated.

[0159] The image processing circuit 1013 demodulates the luminance values ​​of the current frame 903 for each pixel block 200 using coefficients that differ for each pixel block 200 generated by the coefficient generation unit 1703. Because the values ​​are different, the image processing circuit 1013 restores the previous frame 901 from the current frame 903 by demodulating each pixel block 200.

[0160] In this example, the image processing circuit 1013 uses coefficients generated by the coefficient generation unit 1703, and therefore performs inverse quantization within the image processing circuit 1013. However, an inverse quantization unit outside the image processing circuit 1013 may perform inverse quantization using the compression coefficients to restore the current frame 903, and the image processing circuit 1013 may then perform a correction to divide the current frame 903 back by the correction coefficient of the exposure value TV to restore the previous frame 901.

[0161] As described above, according to the image sensor 100, by implementing a data compression mechanism 904 inside the image sensor 100, image data generated by the image sensor 100 is compressed within the image sensor 100 and output to the outside of the image sensor 100. This reduces the amount of output data. Specifically, for example, the image sensor 100 uses exposure control for each control block 400 to encode image data, thereby increasing the efficiency of data compression and enabling data transmission at a lower bandwidth. Furthermore, by generating an exposure value map using exposure control for each control block 400, it is possible to change the compression rate for each pixel block 200. This enables optimal compression tailored to the subject.

[0162] The external processing device 1010 described above may be implemented in the imaging device together with the imaging element 100, or may be communicatively connected to the imaging element 100 via a network such as the Internet, a LAN (Local Area Network), or a WAN (Wide Area Network).

[0163] In addition, in the above example, the signal processing circuit 1001 (autonomous exposure processing unit 411, exposure control unit 412) and the pixel driving unit 413 are described as being provided within the control block 400, but they may also be provided in the peripheral circuit unit 121 or the third semiconductor substrate 130.

[0164] The present invention is not limited to the above-mentioned contents, and may be any combination of these. Furthermore, other embodiments that are conceivable within the scope of the technical idea of ​​the present invention are also included in the scope of the present invention.

[0165] 100 Image sensor, 101 Pixel unit, 102 Control circuit unit, 121 Peripheral circuit unit, 200 Pixel block, 201 Pixel, 300 Photoelectric conversion unit, 400 Control block, 401 Pixel control unit, 402 Signal processing unit, 411 Autonomous exposure processing unit, 412 Exposure control unit, 413 Pixel driving unit, 901 Image data (previous frame), 902 Exposure value map, 903 Image data (current frame), 904 Data compression mechanism, 1001 Signal processing circuit, 1002 First compression circuit, 1003 Image output unit, 1004 Second compression circuit, 1010 External processing device, 1011 First decoding circuit, 1012 Second decoding circuit, 1013 Image processing circuit, 1101 Average value calculation unit, 1102 Exposure value calculation unit, 1104 Reference value, 1202 Exposure table, 1301 Color separation unit, 1303 coefficient conversion table, 1304 coding table, 1305 quantization unit, 1307 coding unit

Claims

1. An imaging element having: a pixel section in which a plurality of pixels are arranged, each including a photoelectric conversion section that converts light into an electric charge; an exposure processing section that calculates, for each pixel, a second accumulation time during which the photoelectric conversion section of the pixel accumulates electric charge, based on a first output signal output from each of the plurality of pixels by photoelectric conversion over a first accumulation time for each pixel during which the photoelectric conversion section of the pixel accumulates electric charge; and a compression section that compresses the second output signal output from each of the plurality of pixels by photoelectric conversion over the second accumulation time for each pixel calculated by the exposure processing section.

2. An image sensor according to claim 1, wherein the exposure processing section sets the time length of the second accumulation time to the first time length if the value of the first output signal is within a predetermined range.

3. An image sensor according to claim 2, wherein the exposure processing unit sets the length of the second accumulation time to a second length of time different from the first length of time if the value of the first output signal is outside the predetermined range.

4. An imaging device according to claim 3, wherein the exposure processing unit sets the second time length to be shorter than the first time length if the value of the first output signal is outside the predetermined range and greater than the upper limit of the predetermined range.

5. An imaging device according to claim 3, wherein the exposure processing unit sets the second time length to be longer than the first time length if the value of the first output signal is outside the predetermined range and smaller than the lower limit of the predetermined range.

6. An image sensor according to claim 1, wherein the exposure processing unit outputs a map in which the second accumulation times for each pixel are arranged in a spatial direction.

7. An imaging device according to claim 1, further comprising an imaging control section that controls imaging by the pixel section based on the second accumulation time, and the pixel section outputs the second output signal for each pixel as a result of imaging control by the imaging control section.

8. An imaging device according to claim 1, wherein the range of distribution of the second histogram indicating the number of occurrences of the value of the second output signal for each pixel is narrower than the range of distribution of the first histogram indicating the number of occurrences of the value of the first output signal for each pixel.

9. An imaging element according to claim 1, wherein the number of values ​​of the first output signal whose occurrence count is equal to or greater than a threshold in a second histogram indicating the number of occurrences of values ​​of the second output signal for each pixel is greater than the number of values ​​of the first output signal whose occurrence count is equal to or greater than a threshold in a first histogram indicating the number of occurrences of values ​​of the first output signal for each pixel.

10. An imaging device according to claim 1, wherein the pixel section is configured to have a plurality of pixel blocks arranged therein, each including one or more of the pixels, and the exposure processing section calculates the second accumulation time for each of the pixel blocks based on a statistical value of the second output signal from the one or more pixels included in each of the plurality of pixel blocks.

11. An imaging device according to claim 10, wherein the compression section compresses the second output signal of each pixel in the pixel block for each pixel block based on the second accumulation time for each pixel block.

12. An imaging device according to claim 11, wherein the compression section separates the second output signals of each pixel in the pixel block by color, and compresses the second output signals for each separated color for each pixel block.

13. An imaging device comprising: the imaging element according to claim 1; and a decoding section that decodes the second output signal compressed by the compression section.

Images