Image sensor and imaging device

Abstract

The present invention provides an image sensor capable of detecting the direction and speed of movement of a moving object with low power consumption and high speed. [Solution] An image sensor comprising: a signal acquisition unit that sequentially acquires pixel signals of multiple pixels having different exposure timings from a pixel block containing multiple pixels; and a calculation unit that calculates information regarding the motion of the image indicated by the pixel signals by comparing a first pixel signal acquired from a first pixel block and a second pixel signal acquired from a second pixel block, among the pixel signals acquired from each of the multiple pixel blocks.

Description

Technical Field

[0001] The present invention relates to an imaging device and an imaging apparatus.

Background Art

[0002] An imaging device in which a plurality of pixels are arranged in a two-dimensional array has been described. Conventionally, improvement in image quality has been desired (for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

[0004] One aspect of the present invention is an imaging device including: a signal acquisition unit that sequentially acquires pixel signals of a plurality of pixels having different exposure timings from a pixel block including a plurality of pixels; and a calculation unit that calculates information regarding the movement of an image indicated by the pixel signals by comparing a first pixel signal acquired from a first pixel block and a second pixel signal acquired from a second pixel block among the pixel signals acquired from the plurality of pixel blocks.

[0005] One aspect of the present invention is an imaging apparatus including the above-described imaging device.

Brief Description of the Drawings

[0006] [[ID=4A]] [Figure 1] It is a diagram showing a schematic configuration example of an imaging device according to an embodiment. [Figure 2] It is a diagram showing a configuration example of a pixel unit according to an embodiment. [Figure 3] It is a diagram showing a configuration example of an imaging device according to an embodiment. [Figure 4] It is a diagram showing a read timing when photographing a moving object. [Figure 5]This figure shows example images of different directions of movement for moving objects. [Figure 6] This figure shows an example of an image taken when photographing a moving object. [Figure 7] This figure shows an example of pixel signals between block rows during motion capture according to the embodiment. [Figure 8] This figure shows an example of a method for calculating the direction of a moving object and its image plane velocity according to the embodiment. [Figure 9] This figure illustrates an example of a method for estimating an ROI region according to the embodiment. [Figure 10] This figure shows the ROI region and areas outside the ROI region according to the embodiment, as well as an example of a readout circuit. [Figure 11] This is a flowchart of an example processing procedure including ROI region estimation of an image sensor according to the embodiment. [Figure 12] This figure shows an example of distortion that occurs when photographing a moving object, specifically when the object moves vertically. [Figure 13] This figure shows an example of an image taken when photographing a moving object in a modified case. [Figure 14] This figure shows an example configuration of an imaging device according to the embodiment. [Modes for carrying out the invention]

[0007] Embodiments of the present invention will be described below with reference to the drawings. Note that in the drawings used in the following description, the scale of each component has been appropriately changed to ensure that each component is recognizable.

[0008] <Example of general configuration of the image sensor, example of the configuration of the pixel section> First, we will explain the schematic configuration of the image sensor and the configuration of the pixel section using Figures 1 and 2. Figure 1 is a diagram showing a schematic configuration example of an image sensor according to an embodiment. Image sensor 1 captures an image of a subject. Image sensor 1 generates image data of the captured subject. In Figure 1, the X and Y axes are orthogonal to each other, and the Z axis is orthogonal to the XY plane. The XYZ axes form a right-handed system. The direction parallel to the Z axis is sometimes referred to as the stacking direction of the image sensor 1. In the following explanation, the terms "up" and "down" are not limited to the up and down directions in the direction of gravity. These terms merely refer to relative directions in the Z axis direction. In the following explanation, the arrangement in the X axis direction is described as a "row," and the arrangement in the Y axis direction is described as a "column," but the matrix direction is not limited to these.

[0009] The image sensor 1 comprises a first semiconductor substrate 10 and a second semiconductor substrate 20. The first semiconductor substrate 10 is stacked on the second semiconductor substrate 20. The first semiconductor substrate 10 includes a pixel portion 11. The pixel unit 11 comprises a plurality of pixel blocks 15. The pixel unit 11 outputs a pixel signal based on the incident light. The pixel block 15 comprises multiple pixels 121.

[0010] The second semiconductor substrate 20 includes a control circuit section 21 and a peripheral circuit section 23. The control circuit unit 21 includes a plurality of AD converters (hereinafter also referred to as "ADCs") 25. Pixel signals output from the first semiconductor substrate 10 are input to the control circuit unit 21. The control circuit unit 21 processes the input pixel signals. The control circuit unit 21 is positioned on the second semiconductor substrate 20, for example, opposite the pixel unit 11. The control circuit unit 21 may output control signals to the pixel unit 11 for controlling the driving of the pixel unit 11.

[0011] The peripheral circuit section 23 controls the driving of the control circuit section 21. The peripheral circuit section 23 controls, for example, the signal readout of the pixel 121 included in the pixel section 11. The peripheral circuit section 23 is arranged on the second semiconductor substrate 20, for example, around the control circuit section 21. Alternatively, the peripheral circuit section 23 may be electrically connected to the first semiconductor substrate 10 and control the driving of the pixel section 11. In Figure 1, the peripheral circuit section 23 is arranged along two sides of the second semiconductor substrate 20, but the arrangement of the peripheral circuit section 23 is not limited to this.

[0012] In addition to the first semiconductor substrate 10 and the second semiconductor substrate 20, the imaging device 1 may have a memory chip stacked on the second semiconductor substrate 20. The memory chip performs, for example, image processing according to a signal output from the second semiconductor substrate 20. Further, the structure of the imaging device 1 may be a back-illuminated type or a front-illuminated type.

[0013] FIG. 2 is a diagram showing a configuration example of a pixel portion according to an embodiment. In FIG. 2, an enlarged view of a pixel block 15 included in the pixel portion 11 is shown. The pixel portion 11 includes M×N (M and N are natural numbers) pixel blocks 15 arranged side by side along the row direction and the column direction. In FIG. 2, the case where M is equal to N is illustrated, but M and N may be different.

[0014] The pixel block 15 includes m×n (m and n are natural numbers) pixels 121. For example, the pixel block 15 includes 16×16 pixels 121. The number of pixels 121 included in the pixel block 15 is not limited to this and may be one or more. In FIG. 2, the case where m is equal to n is illustrated, but m may be different from n.

[0015] The pixel block 15 has a plurality of pixels 121 connected to a common control line in the row direction.

[0016] As shown in FIG. 2, the pixel block group 12 includes two pixel blocks 15. In the example of FIG. 2, the pixel block group 12 includes two pixel blocks 15 arranged side by side along the column direction. The number of pixel blocks 15 included in the pixel block group 12 may be one or three or more.

[0017] If the pixel block group 12 comprises multiple pixel blocks 15, each pixel block 15 may be set to a different exposure time. The pixel block group 12 has 2m × n pixels 121. For example, the pixel block group 12 may have, for example, 32 × 16 pixels 121. However, the number of pixels 121 in the pixel block group 12 is not limited to this.

[0018] Each pixel 121 is equipped with a photoelectric conversion unit that converts light into electric charge. The photoelectric conversion unit stores the photoelectrically converted charge. 2m pixels 121 are arranged in n columns in the row direction within the pixel block group 12.

[0019] <Example of image sensor configuration> Next, we will explain an example of an image sensor configuration. Figure 3 shows an example of the configuration of an image sensor according to an embodiment. As shown in Figure 3, the image sensor 1 comprises a pixel unit 11, an image memory 31 (signal acquisition unit), a calculation unit 32, an ROI area control unit 33 (signal acquisition unit), an association unit 34, and a storage unit 35. The calculation unit 32 includes, for example, a filter 321 and an arithmetic unit 322.

[0020] The pixel section 11 comprises multiple pixels 121. The number of pixels in the pixel section 11 is, for example, 4224 pixels in both the vertical and horizontal directions.

[0021] The image memory 31 stores pixel signals, which are obtained by converting the pixel signals captured by the pixel unit 11 and read out from the pixels, from analog signals to digital signals. The image memory 31 is equipped with a readout circuit for reading out the pixel signals.

[0022] The calculation unit 32 performs calculations for motion detection on the accumulated pixel signals to calculate information related to motion. This information related to motion includes the direction of movement and the speed of movement. Filter 321 applies, for example, a differential filter to the image signal. The calculation unit 322 squares the result of the differentiation and determines whether the squared result exceeds a threshold. Determine the relative positional relationship of pixel addresses that exceed a threshold.

[0023] The ROI region control unit 33 extracts the ROI (Region of Interest) region based on the results calculated by the calculation unit 32. The ROI region control unit 33 transmits the extracted region information to the sensor 2 by means of register setting or other means. The ROI region control unit 33 outputs the extracted region information to the association unit 34.

[0024] The association unit 34 associates the captured data with the ROI region and stores it in the storage unit 35.

[0025] The memory unit 35 stores the captured data in association with the ROI region.

[0026] <Method for calculating information related to movement> Next, we will explain how to calculate information about movement. First, we will explain the image when an object in motion is captured by the image sensor 1. Figure 4 is a diagram showing the readout timing when a moving object is captured. Note that Figure 4 shows a part of the pixel section 11. In images g20 and g30, the longitudinal direction is the x-axis direction and the transverse direction is the y-axis direction. Image g20 represents the state in which the image g21 of the object being captured is stationary. Image g30 represents the state in which the image g21 of the object being captured has moved from right to left with respect to the x-axis direction. Note that in images g20 and g30, the pixel block 15 is assumed to have, for example, 16 × 16 pixels. In the following explanation, the pixel block 15 in the row direction will be referred to as block row 17.

[0027] The image sensor 1 reads out the image signals of each pixel block 15 from top to bottom in the y-axis direction. Therefore, at time t1, the pixel signals of pixels g31, g32, and g33 are read out, and at time t2, the pixel signals of pixels g34, g35, and g36 are read out. In this way, when photographing a moving object, multiple pixel signals are read out simultaneously, and the timing of reading out other pixel signals in the same pixel block 15 is different, so the image is distorted diagonally row by row.

[0028] Figure 5 shows examples of images when the direction of movement of the moving object is different. Note that Figure 5 shows a part of the pixel section 11. Image g40 shows the state in which the image g41 of the object being captured is stationary. Image g50 shows the state in which the image g41 of the object being captured has moved from left to right in the x-axis direction. Thus, the tilt direction when capturing a moving object differs depending on the direction of movement of the moving object, as shown in Figures 4 and 5. Furthermore, the angle of the tilt g51 differs depending on the speed of movement of the moving object.

[0029] Figure 6 shows an example image when a moving object is photographed. Note that the example in Figure 6 is an image of an object moving from left to right. When a moving object is photographed, if a portion of the overall image g1 is enlarged, the outline appears jagged with adjacent diagonally distorted areas, as shown in the enlarged image g2. In this embodiment, information about movement is detected by utilizing the row-by-row tilt that occurs when a moving object is photographed in this way.

[0030] Next, we will explain how to calculate information about movement. Figure 7 shows an example of pixel signals between block rows during motion imaging according to this embodiment. Note that Figure 7 shows a part of the pixel section 11. Image g100 is an image of the pixel signals during motion imaging. Region g101 is the region where the moving object was imaged, i.e., the ROI region.

[0031] When a moving object moves left or right, the pixel signals at the boundary (block boundary) between block row g112, which contains the first pixel block g111, and block row g114, which contains the second pixel block g113, have different readout timings, as explained using Figure 4. This results in a tilt in the x-axis direction during motion imaging. In other words, in adjacent row blocks, the position of the pixel signals shifts in the x-axis direction between the bottom row of block row g112, which has the slowest readout, and the top row of block row g114, which has the fastest readout.

[0032] In images g121 to g123, pixel signals are represented as high-level and low-level for simplicity. For example, black squares represent high-level signals, and white squares represent low-level signals. Furthermore, in images g121 to g123, the top row (row N) is the row of the bottommost pixel in block row g112, and the bottom row (row N+1) is the row of the topmost pixel in block row g114.

[0033] Image g121 shows an example of pixel signals at a block boundary when a moving object moves from right to left. As shown in image g121, when a moving object moves from right to left, the pixel signals in the lower row are shifted to the right relative to the pixel signals in the upper row. Image g122 shows an example of pixel signals at a block boundary when a moving object moves from left to right. As shown in image g122, when a moving object moves from left to right, the pixel signals in the lower row are shifted to the left relative to the pixel signals in the upper row. Image g123 is an example of the pixel signal at the block boundary when the moving object moves from left to right faster than in image g122. As shown in image g123, when the moving object moves from left to right faster than in image g122, the amount of displacement is greater than in image g122. In this embodiment, when a moving object moves in the left-right direction, the direction and speed of the object's movement are detected by calculating the amount and direction of the shift in the pixel signals of the rows that straddle the block boundary.

[0034] The first pixel signal is the pixel signal with a relatively late exposure timing (e.g., the bottom row) in the first pixel block (e.g., pixel block g111). The second pixel signal is the pixel signal with a relatively early exposure timing (e.g., the top row) in the second pixel block (e.g., pixel block g113).

[0035] Figure 8 shows an example of a method for calculating the direction of a moving object and the image plane velocity according to this embodiment. In graphs g201 to g206, the horizontal axis represents the horizontal pixel address in the pixel section 11, and the vertical axis represents the brightness value of the pixel. The example in Figure 8 is when the moving object moves from right to left, as shown in image g121 in Figure 7.

[0036] Graph g201 shows the luminance values ​​for the horizontal pixel addresses in the top row (row N) of image g121 in Figure 7. Graph g202 shows the luminance values ​​for the horizontal pixel addresses in the bottom row (row N+1) of image g121 in Figure 7. Filter 321 applies a differential filter horizontally (in the row direction) to the Nth and N+1th rows. Graph g203 shows the result of applying the differential filter horizontally to the Nth row, and graph g204 shows the result of applying the differential filter horizontally to the N+1th row. Note that any well-known differential filter method (e.g., the convolution operation of (0 -1 1)) is acceptable.

[0037] Next, the arithmetic unit 322 squares the result of the differentiation. Graph g205 shows the result of squaring over the Nth row, and graph g206 shows the result of squaring over the N+1th row. Next, the calculation unit 322 determines whether the squared luminance value exceeds the threshold g211. Next, the calculation unit 322 calculates the direction of the moving object and the image plane velocity by determining the relative positional relationship of pixel addresses that exceed the threshold g211 in the Nth and N+1th rows. Note that the calculation method explained using Figure 8 is just one example and is not limited to this.

[0038] For example, Figure 8 illustrates an example where calculations are performed using pixel signals from adjacent rows separated by a block boundary, but the rows used are not limited to these. For example, the combination could be the bottom row (Nth row) of block row g112 (or pixel block g111) in Figure 7 and the second row from the top (N+2th row) of block row g114 (or pixel block g113), or the second row from the bottom (N-1th row) of block row g112 (or pixel block g111) and the top row (N+1th row) of block row g114 (or pixel block g113). In this way, rows within a predetermined distance can be used between two block rows or pixel blocks that straddle a block boundary.

[0039] Alternatively, two or more lines may be used for each pixel block. For example, it may be a combination of the average value of pixel signals of the bottom row (Nth row) and the second row from the bottom (N-1th row) of the block row g112 (or pixel block g111) in FIG. 7, and the average value of pixel signals of the top row (N+1th row) and the second row from the top (N+2th row) of the block row g114 (or pixel block g131).

[0040] <Estimation method of ROI region> Next, an example of the estimation method of the ROI region will be described. FIG. 9 is a diagram for explaining an example of the estimation method of the ROI region according to the present embodiment. Note that the images g310, g320, and g330 show a part of the pixel unit 11.

[0041] First, the ROI region control unit 33 extracts the block boundaries g311 to g314 for which the value obtained by performing the above-described operation on the pixel signals between the blocks exceeds the threshold. The image g310 represents the block boundaries g311 to g314 for which the value obtained by performing the above-described operation on the pixel signals between the blocks exceeds the threshold when the moving object moves from right to left.

[0042] Next, the ROI region control unit 33 extracts the block groups g321 and g322 including the extracted block boundaries. The image g320 represents the block group g321 including the extracted block boundaries g311 and g312, and the block group g322 including the extracted block boundaries g313 and g314.

[0043] Next, the ROI region control unit 33 performs clustering processing on the extracted block groups by a well-known method. For example, when there are a plurality of extracted block groups, the ROI region control unit 33 assumes that those with a distance between the block groups within a predetermined value are one group. The image g330 represents the clustered region, that is, the ROI region g331.

[0044] Next, the ROI area control unit 33 estimates by calculating the ROI area g342 where the moving object will be photographed at the next time based on the ROI area g331 and the moving direction and moving speed calculated by the calculation unit 32. Note that the method example described using FIG. 9 is just an example and is not limited thereto.

[0045] <Reading of ROI Area and Areas Other than ROI Area> Next, reading of the ROI area and areas other than the ROI area will be described. When the ROI area is calculated during moving object photography, the imaging device 1 may read the pixel signals of the ROI area at each time and use the pixel signals already stored without reading the pixel signals of the areas other than the ROI area. That is, the imaging device 1 may read the estimated ROI area during moving object photography and not read the areas other than the ROI area. According to this embodiment, it is possible to reduce the data amount and power consumption during moving object photography and realize high-speed continuous photography. Note that, for example, in the case of continuous photography, the imaging device 1 may output only the data of the ROI area without outputting the data of the areas other than the ROI area. According to this embodiment, it is possible to reduce the data to be output.

[0046] FIG. 10 is a diagram showing an example of the ROI area, areas other than the ROI area, and a reading circuit according to this embodiment. The image g410 represents the estimated ROI area g411. The circuit diagram g420 is an example of a reading circuit. First, the estimated ROI is stored in the storage area g421 arranged in each block in the sensor 2 by register setting or the like. Each pixel block includes, for example, a PD which is a photoelectric conversion unit, a transfer transistor TR for transferring the charge accumulated in the PD, a comparator g422, pixel current sources I1 and I2, a Latch circuit g423 for temporarily storing the AD conversion result, and the storage area g421 and the like.

[0047] The memory area g421 stores the signal values ​​corresponding to the enable state in the read area, i.e., the ROI area. The read circuit of the image memory 31 controls the pixel current sources I1 and I2, comparator g422, and latch circuit g423 to the activated state by controlling SW1 and SW2, etc., according to the stored signals.

[0048] The image sensor 1 controls the pixel current sources I1 and I2, comparator g422, and latch circuit g423 to an inactive state in areas that are not read out, i.e., areas other than the ROI region, by controlling SW1 and SW2, etc. As a result, no steady current flows through this circuit. Also, since the latch circuit g423 is also inactive, according to this embodiment, power is reduced because the digital data transmission line is not charged or discharged (by being fixed low or high) during readout. Furthermore, since the readout area is determined in advance, according to this embodiment, the readout scanning circuit g413, which is located on the outer periphery of the pixel array g412, only needs to scan the rows corresponding to the readout area, thus limiting the rows to be read out. For this reason, according to this embodiment, the scanning time is shortened and the amount of output data can also be reduced.

[0049] <Example of processing procedure> Next, an example of the processing procedure for the image sensor 1 will be explained. Figure 11 is a flowchart of an example of a processing procedure including ROI region estimation of the image sensor according to this embodiment.

[0050] (Step S1) The image sensor 1 captures a reference frame. The reference frame includes, for example, a moving object, the background, etc. The image sensor 1 also initializes the count value N (for example, to 0).

[0051] (Step S2) The calculation unit 32 performs calculations such as filtering and squaring on the pixel signals at the block boundaries to detect information about the moving object (direction of movement, speed of movement, etc.).

[0052] (Step S3) The ROI area control unit 33 determines the ROI area based on the calculated information about the moving object.

[0053] (Step S4) The image sensor 1 reads out the pixel signals of the determined ROI region.

[0054] (Step S5) The ROI area control unit 33 calculates and estimates the area in which the moving object to be photographed at the next time (the ROI area at the next time) based on the pixel signals based on the readout ROI area and the information about the moving object.

[0055] (Step S6) The image sensor 1 adds 1 to the count value N.

[0056] (Step S7) The image sensor 1 determines whether the count value N is greater than or equal to threshold A. If the image sensor 1 determines that N is greater than threshold A (Step S7; YES), it returns to step S3. If the image sensor 1 determines that N is less than or equal to threshold A (Step S7; NO), it returns to step S1.

[0057] Note that the processing procedure shown in Figure 11 is just one example and is not limited to this. The image sensor 1 may, for example, perform several processes simultaneously.

[0058] As described above, in this embodiment, the direction and velocity of the moving object are calculated by comparing the first pixel signal obtained from the first pixel block 15 and the second pixel signal obtained from the second pixel block 15, among the pixel signals obtained from each of the multiple pixel blocks 15, and the ROI area is calculated and the movement of the moving object is estimated to be how it will move in the next frame.

[0059] In contrast, conventional technology, when photographing moving subjects, requires, for example, performing image processing and tracking on multiple images captured using all pixels to estimate the area in which the moving object is captured. This has resulted in challenges such as increased data volume and increased power consumption. Furthermore, increasing resolution and frame rate also presents challenges in terms of increased data volume and power consumption.

[0060] According to this embodiment, by efficiently capturing the difference region with the previous frame, it is possible to output data from the minimum necessary region. As a result, this embodiment makes it possible to reduce the amount of data and power consumption. Furthermore, in this embodiment, power consumption can also be reduced by stopping circuit blocks other than the ROI region, which is a region that does not need to be captured.

[0061] <When the moving object moves up and down> The above example described the case where the moving object moves left and right, but it is not limited to this. Next, we will explain examples of calculation and specification methods when the moving object moves up and down, etc. Figure 12 shows an example of distortion that occurs when a moving object is photographed, specifically when the object moves vertically. As shown in image g510 in Figure 12, when a spherical object obj moves from top to bottom, depending on the exposure timing, two images may be captured, as shown in image g520: image g521 of approximately the lower half of the object and image g522 of approximately the upper half of the object. Note that line g523 is a scan line.

[0062] Furthermore, for example, if a square object moves from top to bottom, it may be captured as if it were a rectangular object. Thus, when photographing a moving object that is moving vertically, the image may be captured in a way that differs from its original shape. For this reason, the image sensor 1 may, for example, detect the ROI region corresponding to the moving object from multiple captured images, read out the pixel signals of the detected ROI region, and estimate and store information about the object's shape. Using this information, the image sensor 1 can calculate the direction and speed of vertical movement.

[0063] Furthermore, the image sensor 1 may calculate both horizontal and vertical information, and use the calculated concessions from both directions to estimate the ROI region in which a moving object will be photographed at the next time, including the vertical and diagonal directions.

[0064] <Variation> Although the above example described an example with multiple blocks, the method of this embodiment can also be applied to, for example, two blocks. Figure 13 shows an example of an image taken when a moving object is photographed in a modified example. Images g601 and g602 are images taken when an object moves from right to left. The dashed rectangle g610 is an example of the readout range when reading out pixel signals from top to bottom using a rolling shutter method. In this way, the image sensor 1 may read out pixel signals in a strip-like manner from top to bottom for two blocks. In this case as well, the image sensor 1 may calculate and estimate the direction of movement, movement speed, and ROI area by performing the above-described calculation processing on the pixel signals of the blocks above and below the block boundary.

[0065] In this modified example as well, when calculating motion information, the image sensor 1 may use the bottom row of the upper block and the top row of the lower block. Alternatively, as in the embodiment described above, the image sensor 1 may use rows from the bottom up within a predetermined range within the upper block and rows from the top up within a predetermined range within the lower block, and multiple rows may be used.

[0066] <Example of imaging device configuration> Here, we will describe an example configuration of an imaging device 700 equipped with the image sensor 1 of each embodiment. Figure 14 shows an example of the configuration of an imaging device according to an embodiment. As shown in Figure 14, the imaging device 700 includes, for example, an image sensor 1, an imaging optical system 701, an image processing unit 703, a work memory 704, an operation unit 705, a display unit 706, a control device 707, and a storage unit 708. Note that the configuration of the imaging device shown in Figure 14 is just one example and is not limited thereto.

[0067] The imaging optical system 701 is composed of multiple lenses and guides the light beam from the field of view to the image sensor 1. The imaging optical system 701 may be integrated with the imaging device 700, or it may be configured to be interchangeable with the imaging device 700. Furthermore, the imaging optical system 701 may have a built-in focus lens or a built-in zoom lens.

[0068] The image processing unit 703 works in cooperation with the work memory 704 to perform image processing on the pixel signals captured by the image sensor 1.

[0069] The work memory 704 temporarily stores, for example, pixel signals before and after image compression, and is also used as a buffer memory for images captured by the image sensor 1.

[0070] The operation unit 705 detects the user's operation result and outputs it to the control device 707. The operation unit 705 is, for example, a touch panel sensor or mechanical switch provided on the display unit 706.

[0071] The display unit 706 is composed of, for example, a liquid crystal display panel, and displays images (still images, videos) and various information captured by the image sensor 1, as well as displaying operation input screens.

[0072] The control device 707 is, for example, a CPU (Central Processing Unit) and controls each part. The control device 107 may include a calculation unit 32 for the image sensor 1, an ROI area control unit 33, and an association unit 34.

[0073] The memory unit 708 stores various data, such as pixel signals acquired in response to imaging instructions, on a storage medium such as a memory card.

[0074] Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention. [Explanation of Symbols]

[0075] 1…Image sensor, 10…First semiconductor substrate, 20…Second semiconductor substrate, 11…Pixel section, 12…Pixel block group, 15,15-1,…,15-m…Pixel block, 21…Control circuit section, 23…Peripheral circuit section, 25…AD converter, 31…Image memory, 32…Calculation section, 33…ROI area control section, 34…Association section, 35…Storage section, 321…Filter, 322…Calculation section, 121…Pixel, 700…Imaging device, 701…Imaging optical system, 703…Image processing section, 704…Work memory, 705…Operation section, 706…Display section, 707…Control device, 708…Storage section

Claims

1. A signal acquisition unit sequentially acquires pixel signals from a pixel block containing multiple pixels, each of which has a different exposure timing. A calculation unit calculates information regarding the motion of an image indicated by a pixel signal by comparing a first pixel signal obtained from a first pixel block with a second pixel signal obtained from a second pixel block, among the pixel signals obtained from each of the multiple pixel blocks. An image sensor equipped with the following features.

2. The calculation unit described above, Based on the time difference between the exposure timing of the first pixel signal and the exposure timing of the second pixel signal, the motion information is calculated. The image sensor according to claim 1.

3. The calculation unit described above, Based on the amount of difference between the image shown by the first pixel signal and the image shown by the second pixel signal, the motion information is calculated. The image sensor according to claim 1.

4. The first pixel signal is a pixel signal with a relatively slow exposure timing in the first pixel block. The second pixel signal is a pixel signal with a relatively early exposure timing in the second pixel block. The image sensor according to claim 1.

5. The first pixel signal is the pixel signal with the latest exposure timing in the first pixel block. The second pixel signal is the pixel signal with the earliest exposure timing in the second pixel block. The image sensor according to claim 4.

6. The first pixel block and the second pixel block each have pixels from which the pixel signal is acquired at the same exposure timing. The image sensor according to claim 1.

7. The aforementioned information regarding the movement is information about the direction of movement of the object. The calculation unit described above, A differential filter is applied to the first pixel signal and the second pixel signal, the result of applying the differential filter is squared, and the relative position between the image corresponding to the first pixel signal and the image corresponding to the second pixel signal where the squared value exceeds a threshold is calculated to calculate the movement direction information of the object. The image sensor according to claim 1.

8. The aforementioned motion information is information about the object's velocity, The calculation unit described above, A differential filter is applied to the first pixel signal and the second pixel signal, the result of applying the differential filter is squared, and the relative position between the image corresponding to the first pixel signal and the image corresponding to the second pixel signal where the squared value exceeds a threshold is calculated to calculate the movement speed information of the object. The image sensor according to claim 1.

9. The aforementioned motion information is region information of the moving object, The calculation unit described above, A differential filter is applied to the first pixel signal and the second pixel signal, the result of applying the differential filter is squared, and the relative position between the pixel corresponding to the first pixel signal and the pixel corresponding to the second pixel signal whose squared value exceeds a threshold is calculated to calculate the region information of the moving object. The image sensor according to claim 1.

10. The calculation unit described above, Based on the region of the moving object, the direction of movement of the object, and the speed of movement of the object, To estimate the position in which the moving object to be imaged in the future will be imaged, The image sensor according to claim 9.

11. The signal acquisition unit does not read out the pixel signals of the pixel blocks that do not correspond to the positions where the estimated future image of the moving object will be captured. The image sensor according to claim 10.

12. An imaging device comprising an image sensor according to any one of claims 1 to 10.

Images