Image pickup device, imaging apparatus, monitoring apparatus, and control method of image pickup device

Abstract

To provide an imaging apparatus that can easily and quickly acquire an image when an event occurs.SOLUTION: An image pick-up device has a plurality of pixels, and electric charges accumulated in the plurality of pixels are periodically reset. In at least part of the plurality of pixels, first read-out processing of reading out the accumulated electric charge with first exposure is performed, and first signal processing based on a first pixel signal read out through the first read-out processing is performed. It is determined, in accordance with a result of the first signal processing, whether to perform second read-out processing of reading out the accumulated electric charges with second exposure and second signal processing based on a second pixel signal read out through the second read-out processing. The first read-out processing, the first signal processing, and the second read-out processing are performed within one cycle of a reset that is periodically performed.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to an imaging device, an imaging apparatus, a monitoring device, and a method for controlling an imaging device.

Background Art

[0002] There has been proposed an imaging apparatus that detects a moving subject from an image captured by an imaging device and changes the operation of the imaging device. Patent Document 1 proposes an imaging apparatus that changes the operation setting of an imaging device based on the presence or absence of a moving subject in an image. The imaging apparatus described in Patent Document 1 has a frame memory that stores the captured pixel signals in time series, compares the images stored in the frame memory, and has a configuration in which the operation state of a pixel is determined by a state determination unit based on the presence or absence of a moving subject in the image. The imaging apparatus determines the operation of an image sensor at the time of acquiring an image after the next frame based on the presence or absence of a moving subject in the image.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The imaging apparatus of Patent Document 1 provides a large frame memory for storing a plurality of images obtained from an imaging device in order to detect a moving subject, and compares the images between frames. Therefore, there are problems of an increase in cost due to the addition of a large frame memory and an increase in power consumption. In addition, since image determination is performed after acquiring one image from the imaging device, there is a problem that it is difficult to improve the responsiveness to an image.

[0005] In view of the above problems, an object of the present invention is to provide an imaging apparatus capable of simply and quickly acquiring an image when an event occurs.

Means for Solving the Problems

[0006] A first aspect of the present invention is An image sensor having multiple pixels, The accumulated charge of the plurality of pixels is reset periodically. In at least some of the pixels of the plurality of pixels, A first readout process is performed to read out the accumulated charge from the first exposure. A first signal processing is performed based on the first pixel signal read out by the first readout process described above. Depending on the result of the first signal processing, it is determined whether or not to perform a second readout process to read out the accumulated charge due to the second exposure, and a second signal processing based on the second pixel signal read out by the second readout process. The first read operation, the first signal processing, and the second read operation are performed within one cycle of a periodically occurring reset. It is an image sensor.

[0007] A second aspect of the present invention is, A control method for an image sensor comprising: a plurality of pixels each including a photoelectric conversion unit, a readout unit for reading the charge accumulated in the photoelectric conversion unit, and a reset unit for resetting the photoelectric conversion unit; and a signal processing unit for performing signal processing on pixel signals output from the plurality of pixels, The steps include periodically resetting the photoelectric conversion unit in the plurality of pixels, The steps include: performing a first readout process to read out the charge accumulated by the first exposure; The steps include: performing a first signal processing based on the first pixel signal read out by the first readout process; Depending on the result of the first signal processing, the steps include determining whether to perform a second readout process to read out the accumulated charge due to a second exposure, and a second signal processing based on the second pixel signal read out by the second readout process, Includes, The first read operation, the first signal processing, and the second read operation are performed within one cycle of a periodically occurring reset. This is a method for controlling the image sensor. [Effects of the Invention]

[0008] According to the present invention, an image when an event occurs can be acquired simply and quickly.

Brief Description of the Drawings

[0009] [Figure 1A] It is a schematic diagram for explaining the configuration of the image pickup device according to the embodiment. [Figure 1B] It is a schematic diagram for explaining the configuration of the image pickup device according to the embodiment. [Figure 1C] It is a schematic diagram for explaining the configuration of the image pickup device according to the embodiment. <^ [Figure 1D] It is a schematic diagram for explaining the configuration of the image pickup device according to the embodiment. [Figure 1E] It is a schematic diagram for explaining the configuration of the image pickup device according to the embodiment. [Figure 2] It is a schematic diagram for explaining the processing flow of the image pickup device according to the embodiment. [Figure 3] It is a schematic diagram for explaining the processing timing of the image pickup device according to the embodiment. [Figure 4A] It is a schematic diagram for explaining the configuration of the image pickup device according to the embodiment in more detail. [[ID=4^0]] [Figure 4B] It is a schematic diagram for explaining the configuration of the image pickup device according to the embodiment in more detail. [Figure 4C] It is a schematic diagram for explaining the configuration of the image pickup device according to the embodiment in more detail. [Figure 4D] [[ID=^0]]It is a schematic diagram for explaining the configuration of the image pickup device according to the embodiment in more detail. [Figure 5] It is a schematic diagram for explaining the processing timing of the image pickup device according to the embodiment. [Figure 6] It is a schematic diagram for explaining the processing flow of the image pickup device according to the second embodiment. [Figure 7] It is a schematic diagram for explaining an image pickup device according to the third embodiment. [Figure 8] It is a schematic diagram for explaining the processing timing of an image pickup device according to the fourth embodiment. [Figure 9] It is a schematic diagram for explaining the processing of an image pickup device according to the fifth embodiment. [Figure 10] It is a schematic diagram for explaining an image pickup device according to the sixth embodiment. [Figure 11] It is a schematic diagram for explaining an image pickup device according to the seventh embodiment. [Figure 12] It is a schematic diagram for explaining an image pickup device according to the eighth embodiment. [Figure 13] It is a schematic diagram for explaining an imaging device according to the ninth and tenth embodiments.

Embodiments for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to these embodiments, and various modifications and changes within the scope of the gist are possible.

[0011] [ (First Embodiment) FIG. 1A is a schematic diagram for explaining the configuration of an image pickup device 100 included in an embodiment of the present invention. The image pickup device 100 includes a photoelectric conversion unit 110, a reset unit 120, a signal readout unit 130, a signal processing unit 140, and a signal output unit 150. The signal readout unit 130 includes an amplifier 131 and a pixel selection unit 132. In FIG. 1A, 200 is incident light, 201 is a charge signal, 202 is a photodetection signal, 203 is a pixel output after signal processing, 204 is an image output signal, and 205 is a photodiode charge reset signal.

[0012] In this embodiment, the periodic charge reset processing of the photoelectric conversion unit 110 is performed. And one cycle During this time, the signal processing unit 140 performs a first readout process to read out the accumulated charge from the first exposure, and a first signal processing based on the first pixel signal read out by the first readout process. The signal processing unit 140 further determines, depending on the result of the first signal processing, whether or not to perform a second readout process to read out the accumulated charge from the second exposure, and a second signal processing based on the second pixel signal read out by the second readout process. The first readout process, the first signal processing, the second readout process, and the second signal processing are performed within one cycle of a periodically performed reset process.

[0013] In Figure 1A, the photoelectric conversion unit 110 is composed of, for example, a photodiode. The photodiode can generate and store electric charge depending on the intensity and duration of the received light. In this specification, the light-receiving unit of the photodiode is called a pixel. One pixel includes the photoelectric conversion unit 110, a signal readout unit 130 that reads the charge stored in the photoelectric conversion unit 110, and a reset unit 120 that resets the photoelectric conversion unit 110. The image sensor 100 has a configuration in which multiple pixels are arranged in one or two dimensions. The charge generated in the photoelectric conversion unit (photodiode) 110 is converted into a voltage signal in the signal readout unit 130, transferred, and output to the signal processing unit 140.

[0014] The signal readout unit 130 is equipped with an amplifier 131 for each pixel that converts electric charge into voltage. The output from each pixel's amplifier 131 is connected to the pixel selection unit 132 and output. The signal readout unit 130 is driven based on a control signal from a control circuit (not shown).

[0015] The signal processing unit 140 performs signal processing on pixel signals output from multiple pixels. The signal processing unit 140 has, firstly, a function to detect the occurrence of an event based on the signal from each pixel (first signal processing function), and secondly, a function to correct the signals received from each pixel and output the signal to the signal output unit 150 (second signal processing function). The detailed operation of the signal processing unit 140 will be explained using Figures 2 and 3.

[0016] The signal output unit 150 outputs all the signals from all pixels received from the signal processing unit 140 to the outside. In this specification, this output signal containing information from all pixels is called an image signal, and one set of image signals is called a frame. The image signal is used outside the image sensor 100 to modify, display, or record an image of one captured frame. At this time, outside the image sensor 100, the signal positions are generally rearranged and the brightness and color are adjusted.

[0017] Furthermore, the signal output unit 150 outputs an image output signal for each frame, usually at a constant frequency fout (=1 / period T). This frequency fout is called the frame rate, and it is an indicator that shows how many sets of frame images are output per second. The unit used for frame rate is fps (frames per second). The specific value of the frame rate varies depending on the number of pixels in the image sensor, but it is generally between a few fps and several hundred fps.

[0018] The accumulated charge of the photoelectric conversion unit 110 is read by the signal readout unit 130. However, if the accumulated charge is not reliably removed after reading, extraneous signals will be mixed into the image signal, degrading the quality (image quality) of the image signal for the next readout. Therefore, the reset unit 120 removes the charge from the photoelectric conversion unit 110 (resets the charge of the photoelectric conversion unit 110). To prevent degradation of image quality, it is desirable that this charge reset process of the photoelectric conversion unit 110 be performed on all pixels at the frame cycle before charge accumulation occurs in the pixels.

[0019] In Figure 1A, for the sake of simplicity, the image sensor 100 is shown as having only one pixel. As shown, the image sensor 100 may include multiple pixels and a signal transfer unit 133 for transferring light detection signals from multiple pixels, as shown in Figure 1B. The signal transfer unit 133 is a means for reading out and transferring light detection signals, and a transfer method used in CMOS sensors is employed. Specifically, the image sensor 100 is used in a configuration where multiple pixels are arranged in two dimensions, a signal transfer unit 133 is arranged for each pixel in the row (horizontal) direction, and the signal transfer unit 133 is shared in the column (vertical) direction by a pixel selection unit 132. The signal transfer unit 133 is connected to a signal processing unit 140, and the output signal from the pixel selected by the pixel selection unit 132 is transmitted to the signal processing unit 140 via the signal transfer unit 133. By sequentially selecting multiple pixels in a time series, signals from multiple pixels can be sequentially extracted using a common signal transfer unit 133.

[0020] Specific configuration examples of the photoelectric conversion unit 110 and the signal readout unit 130 will be described with reference to Figures 4A and 4D. Although only one pixel 101 is shown in Figure 4A for simplicity, in reality, multiple pixels 101 are provided for a single transfer signal line 441, as shown in Figure 4D. Furthermore, Figure 4D shows the configuration for one row of pixels, and the image sensor 100 has multiple rows of the configuration shown in Figure 4D. Note that all transistors described in this specification are MOS transistors.

[0021] In Figure 4A, a photodiode 401 is used as the photoelectric conversion section 110, with its cathode terminal connected to GND. A reset transistor 431 is placed between the positive power supply voltage VDD and the photodiode 401 as the reset section 120. A drive transistor 411 and a load transistor 412 are placed in series between the positive power supply VDD and the negative power supply VSS as the charge-voltage conversion amplifier 131. The gate input GATE of the drive transistor 411 is connected to the photodiode 401. The positive power supply voltage VDD, drive transistor 411, load transistor 412, and negative power supply VSS constitute a source follower amplifier 410. The source follower amplifier 410 has a high input impedance and a low output impedance, and acts as a buffer amplifier that converts the charge accumulated in the input element into a voltage and transmits it to the subsequent circuit.

[0022] A selection transistor 420 is used as the pixel selection unit 132 and is positioned between the drive transistor 411 and the load transistor 412. By turning on the selection transistor 420, the source follower amplifier between the positive power supply VDD and the negative power supply VSS operates, generating VOUT. The signal transfer unit 133 is the transfer signal line 441 and is connected to the VOUT terminal of the source follower amplifier. As shown in Figure 4D, the VOUT terminals of all pixels 101 that share this signal transfer unit 133 are all connected to the same transfer signal line 441. For multiple pixels connected to the same transfer signal line 441, only the selection transistor 420 of one pixel can be selected at a time. By sequentially switching the selected transistor, all outputs can be read out in chronological order from each pixel. Note that the load transistor 412 of the source follower amplifier does not need to be provided for each pixel; only one is needed per transfer signal line 441. Also, the image sensor may have multiple pixel rows and transfer signal lines 441 as shown in Figure 4D.

[0023] The reset unit 120 consists of a reset transistor 431 positioned between the positive power supply VDD and the photodiode 401. By turning on the reset transistor 431 before starting exposure, the charge accumulated in the photodiode 401 can be removed.

[0024] Figure 4B illustrates another configuration example of the photoelectric conversion unit 110 and the signal readout unit 130. Figure 4B shows a configuration in which a diffusion capacitor FD402 and an FD transfer transistor 421 are added to the configuration shown in Figure 4A. The diffusion capacitor FD402 is the gate of the drive transistor 411. It is positioned between the input GATE and the GND terminal. Furthermore, the selection transistor 420 is inserted in the wiring between the cathode terminal of the photodiode 401 and the gate input GATE of the drive transistor 411. In the configuration of Figure 4B, the FD transfer transistor 421 is normally OFF during exposure and is turned ON at the end of exposure to transfer the charge accumulated in the photodiode 401 to the diffusion capacitor FD402, which can then be converted into a voltage signal by the drive transistor 411.

[0025] Furthermore, in Figure 4B, the reset transistor 431 is positioned differently from that in Figure 4A, being located between the positive power supply VDD and the gate input GATE of the drive transistor 411. Therefore, by turning on the reset transistor 431, the accumulated charge in the photodiode 401 and the charge accumulated in the diffusion capacitor FD402 can be reset.

[0026] Figure 4C will be used to explain another configuration. Figure 4C shows an example of a pixel configuration with a global electronic shutter (GS) function that can perform simultaneous transfer of all pixels. This configuration adds one holding capacitance unit 403 and two transistors (a second reset transistor 432 and a GS transfer transistor 422) to the configuration of Figure 4B. For illustrative purposes, the position of the reset transistor 431 has been changed between Figure 4C and Figure 4B.

[0027] When the GS transfer transistor 422 is turned ON, it transfers the charge stored in the photodiode 401 to the holding capacitor 403. When the FD transfer transistor 421 is turned ON, it transfers the charge from the holding capacitor 403 to the FD capacitor 402. In addition, the stored charge in the photodiode 401 can be reset by turning ON the reset transistor 431.

[0028] Next, the processing flow performed by the image sensor 100 in this embodiment will be explained using Figure 2. The following processing is performed based on control signals from the control circuit, but for simplicity of explanation, mention of the control circuit and control signals will be omitted. Also for simplicity of explanation, the processing flow will be explained for one pixel.

[0029] The process shown in Figure 2 begins in step 300 when an imaging instruction is received. In step 310, the reset unit 120 is driven to reset the charge stored in the photoelectric conversion unit 110. At this point, the photodiode 110, whose stored charge has been reset, begins to receive light and starts to store charge again. After a predetermined first time T1 has elapsed since the stored charge was reset, the pixel signal readout process in step 320 is performed.

[0030] In step 320, the signal readout unit 130 reads out a first pixel signal corresponding to the accumulated charge of the pixel (first readout process). In this specification, the period from the start of light reception to the reading out of the pixel signal is referred to as the exposure period. As soon as the reading of the pixel output in step 320 is completed, the pixel starts receiving light again. In other words, in this embodiment, exposure is temporarily stopped for a very short time by the first readout process, but exposure continues during the first signal processing period described later. As a result, the image information to be acquired is not interrupted by the first signal processing, and image information with higher real-time capabilities can be acquired.

[0031] In step 330, the signal processing unit 140 detects whether an event has occurred based on the pixel output signals read out at the same time in step 320 (first signal processing). The signal processing unit 140 determines that an event has occurred when a moving subject in the image meets predetermined conditions. Examples of predetermined conditions include a moving subject suddenly stopping, a stationary subject suddenly starting to move, or a subject moving in a way that is different from an acceptable or expected movement. It is possible.

[0032] In step 340, the signal processing unit 140 branches the processing depending on whether or not an event has been detected.

[0033] If an event is detected (340-YES), the process proceeds to step 350. In step 350, after a predetermined second time T2 has elapsed since step 320, the signal readout unit 130 reads out a second pixel signal corresponding to the accumulated charge of the pixel (second readout). During this second readout period, the signal that was exposed (accumulated) during the time T2 period from the first readout timing to the second readout timing is extracted. In step 360, the signal processing unit 140 corrects the second pixel signal (second signal processing). In step 370, the signal processing unit 140 outputs the corrected frame image.

[0034] On the other hand, if no event is detected (340-NO), the process proceeds to step 371. In step 371, the second readout by the signal readout unit 130 and the second signal processing by the signal processing unit 140 are not performed. Furthermore, the invalid frame image generation unit 151 of the signal output unit 150 generates an invalid frame image and outputs it externally. Here, an "invalid frame image" is simply any information that differs from the second signal obtained assuming the second readout was performed, and in the simplest sense, it is meaningless data. Examples of meaningless data include images in which all pixel values ​​are filled with 0 or other specific values, or images with a specific pattern. In addition, the form in which the invalid frame image is output can be a form in which the entire image with a long update time (frame drop) is output, or a form in which a coarse image obtained in the first readout is output.

[0035] In step 380, it is determined whether to take a picture of the next frame. If a picture is to be taken, the process returns to step 310. If a picture is not to be taken of the next frame, the process ends (390).

[0036] When repeated shooting is performed, the reset process of the photoelectric conversion unit 110's accumulated charge by the reset unit 120 (step 310) is set at a predetermined period for each pixel, and the frequency of this reset is matched to the frame rate. On the other hand, the frequency of the image signal output from the signal output unit 150 (steps 370 and 371) is also set to match the frame rate. Although the reset period is the same for each pixel, the reset timing may be the same for all pixels, or it may differ for each pixel or pixel row.

[0037] In this specification, the exposure and readout operations are collectively referred to as imaging, the first exposure and first readout are collectively referred to as first imaging, and the second exposure and second readout are collectively referred to as second imaging. The first imaging can be referred to as imaging (exposure and readout) for acquiring pixel information for event detection, and the second imaging can be referred to as imaging (exposure and readout) for acquiring pixel information for output image generation. Furthermore, a set of pixel data obtained in the first imaging is referred to as first subframe pixel data, and a set of pixel data obtained in the second imaging is referred to as second subframe pixel data.

[0038] The operating timing of the image sensor 100 according to this embodiment will be explained using Figure 3. For simplicity, the timing chart will be explained for each pixel.

[0039] First, in order to keep the accumulated charge of the photoelectric conversion unit 110 constant, a photodiode charge reset signal 205 is generated as shown in Figure 3(a). This photodiode charge reset signal 205 is a signal that repeats at a constant period, and as mentioned above, it periodically resets all pixels at the same period as the frame period.

[0040] During one reset cycle, first, a first image (first exposure and first readout) is performed at each pixel.

[0041] As shown in Figure 3(b), the photoelectric conversion unit 110 of the reset pixel receives light and begins to accumulate charge (start of exposure period). This exposure period continues until the next first readout signal is generated.

[0042] As shown in Figure 3(c), when the first readout signal is input to the signal readout unit 130, the signal readout unit 130 reads out the accumulated charge from the photoelectric conversion unit 110. The readout signal is called the first pixel signal. The details of this operation differ depending on whether the configuration of the image sensor 100 is as shown in Figures 4A to 4C, but ultimately the charge signal is output as a voltage signal.

[0043] As shown in Figure 3(d), the signal processing unit 140 performs event detection signal processing (first signal processing) on ​​the first pixel signal read out in the first readout process. Simultaneously with the start of this signal processing, as shown in Figure 3(f), the photoelectric conversion unit 110 starts a second exposure period, and the exposure period continues until a second readout signal is generated (Figure 3(g)).

[0044] Figure 3(e) shows the result of the first signal processing, and a signal is output depending on whether an event was detected or not (340-YES or 340-NO). In this embodiment, a high-level signal is output if an event is detected, and a low-level signal is output if no event is detected.

[0045] Depending on the result of the first signal processing, the signal processing unit 140 determines whether or not to perform a second readout process to read out the accumulated charge related to the second exposure, and a second signal processing based on the second pixel signal read out by the second readout process. Specifically, if a predetermined event is detected in the first signal processing (340-YES), the signal processing unit 140 determines to perform the second readout process and the second signal processing. On the other hand, if no predetermined event is detected in the first signal processing (340-NO), the signal processing unit 140 determines not to perform the second readout process and the second signal processing. If the second readout process and the second signal processing are not performed, the invalid frame image generation unit 151 of the signal output unit 150 generates an invalid image signal and outputs an invalid frame image signal (DUM DATA in Figure 3(i)) from the signal output unit 150. Let me explain the concept of "combination".

[0046] As shown in Figure 3(g), when a predetermined event is detected (340-YES), a second readout signal is input to the signal readout unit 130, and the signal readout unit 130 performs a second readout process to read the accumulated charge from the photoelectric conversion unit 110. The readout signal is called the second pixel signal. As shown in Figure 3(h), the signal processing unit 140 performs a correction (adjustment) process (second signal processing) to make the second pixel signal suitable as an image. The second signal processing is the process of generating an image signal for output from the image sensor 100 based on the second pixel signal. In Figure 3(i), the adjusted image signal is output externally as a single image output from the signal output unit 150 (OUT DATA section in Figure 3(h)). ).

[0047] As shown in Figure 3, the period T for outputting the image signal matches the period T for the reset signal. Furthermore, as shown in Figure 3, once the second readout is complete, the pixel information itself has already been acquired, so the reset signal can be generated even during the second signal processing or image output. This reduces wasted time and minimizes the image output period T. can.

[0048] In this embodiment, examples of predetermined events detected by the signal processing unit 140 include the start of movement of a stationary subject and the sudden stopping of a moving subject.

[0049] Image sensors that detect moving objects as events can be used for monitoring suspicious individuals or intruders. Specifically, only when a moving object such as a suspicious person or intruder is detected by the first readout process and first signal processing (event detection process), the second readout process and second signal processing are performed to output an image of the moment the moving object was detected.

[0050] Furthermore, image sensors that detect the sudden stopping of moving subjects as an event can be used for traffic monitoring. That is, for example, they can detect when a car suddenly stops due to a traffic accident and output an image of the moment of detection.

[0051] Event detection can be easily performed, for example, by storing pixel data acquired a short time earlier (a predetermined frame earlier) and detecting the change between that data and newly acquired pixel data. The event detection function of this embodiment is not limited to the above function and method; other functions and methods can be used as long as the detection is based on information obtained through imaging. According to this embodiment, regardless of the specific content of the event detection function, events can be automatically detected by the image sensor 100 alone, and an image of the moment the event occurs can be output in a timely manner. Alternatively, the system may be configured to detect the occurrence of an event and output an image only for that moment. Or, when an event is detected, the system may be configured to perform a second readout process and a second signal processing for a predetermined frame period, and then output an image. By continuing image output for a predetermined frame period, it is possible to continue capturing the situation after the event has occurred.

[0052] In Figure 3, it is explained that the reset performed during one frame period is only one, in order to reset all pixels simultaneously. However, the present invention is not limited to this form, and a sub-reset may be performed one or more times between the completion of the first readout and the start of exposure. In this specification, the drive of the reset unit 120 performed before the start of the first exposure is called a reset (or main reset), and the drive of the reset unit 120 performed before the start of the second exposure is called a sub-reset, and the two are considered to be clearly separated operations or processes.

[0053] Furthermore, while the reset cycle is the same for all pixels, the timing of the reset may be simultaneous for all pixels (global shutter method) or may differ for each pixel row (rolling shutter method).

[0054] As described above, in this embodiment, a first signal is read from the signal readout unit 130 at intervals during the periodic charge reset processing of the photoelectric conversion unit 110, and the first signal processing is performed in the signal processing unit 140. Then, based on the result of the first signal processing, it is determined whether or not to perform a second signal readout. Therefore, according to this embodiment, it is possible to provide an image sensor that can acquire images without delay when an event occurs without performing high-load signal processing or image transfer externally.

[0055] In the explanations for Figures 2 and 3, only the case of a single pixel was explained for the sake of simplicity. If the image sensor has multiple pixels, the operation can be almost the same as described in Figures 2 and 3, provided that there are signal transfer units 133 for each pixel. However, in many cases, the signal transfer unit 133 is shared by multiple pixels, so the readout is performed with a slight timing difference.

[0056] Furthermore, in this embodiment, the first read operation, the first signal processing, the second read operation, and the second signal processing are performed within one cycle of a periodic reset, but this is not limited to this. For example, the first read operation, the first signal processing, and the second read operation may be performed within one cycle of the reset, and the second signal processing may be performed in the next cycle or a subsequent cycle.

[0057] Furthermore, in this embodiment, the signal processing unit 140 was described as not performing a second readout or second signal processing upon receiving the result of event detection (340-YES, 340-NO) of the signal processing (first signal processing). However, the present invention is not limited to this. Adopting a configuration in which a second readout or second signal processing is performed even when 340-NO is determined will generate unnecessary power consumption, but these configurations may be adopted if the increase in power consumption does not pose a practical problem. In this case, since the signal output unit 150 has an invalid frame image generation unit 151, it can perform the same operation as the present invention.

[0058] By adopting the circuit configuration described in Figure 4A or Figure 4B, the timing can be schematically represented as shown in Figure 5. In Figure 5, 501 is the reset signal (main), 502 is the first exposure period, and 503 is the second exposure period. In Figure 5, for the sake of simplicity, only the timing for 8 pixels is schematically shown. Also, due to space limitations in the diagram, only 501, 502, and 503 are shown, but the other signals shown in Figure 3 also operate with slightly shifted timings in a similar manner. Here, Tread1 is the first readout period, and Tcal1 represents the period required for the first signal processing. By adopting this configuration, the degree to which the first readout affects the exposure can be minimized, and the exposure period can be extended. Also, since the second exposure continues during the first signal processing period, there is no need to stop the exposure. Furthermore, by adopting the circuit configuration described in Figure 4C, the reset timing, i.e., the exposure timing, can be made the same for all pixels.

[0059] Furthermore, while Figure 1B shows the signal transfer unit 133 directly connected to the signal processing unit 140, as shown in Figure 1C, a configuration can also be implemented with a digital signal conversion unit 134 between the signal transfer unit 133 and the signal processing unit 140. The digital signal conversion unit 134 can convert the analog output signal from the amplifier 131 of each pixel into a digital signal 207. This allows the digital signal to be directly input to the signal processing unit 140, facilitating signal processing using the digital signal.

[0060] Furthermore, although the signal processing unit 140 and the signal output unit 150 are located inside the image sensor 100 in this embodiment, the signal processing unit 140 and the signal output unit 150 may also be located outside the image sensor 100.

[0061] Furthermore, as shown in Figure 1D, each pixel's signal readout unit 130 is equipped with a digital signal conversion unit 134 that digitizes the output signal of the amplifier 131, and can be configured to transmit the digital signal 207 to the signal transfer unit 133.

[0062] Furthermore, as shown in Figure 1E, the same configuration can be used to improve the signal-to-noise ratio (S / N) by placing an amplification amplifier 135 between the signal transfer unit 133 and the signal processing unit 140, and transmitting the pixel signal as an amplified signal 208.

[0063] (Second embodiment) The second embodiment differs from the first embodiment in that it uses the pixel signals (image information) obtained in the first imaging to output an image from the signal output unit 150, but otherwise it is the same as the first embodiment. The differences from the first embodiment will be mainly described below.

[0064] Figure 6 is a flowchart illustrating the operation of this embodiment. The operation of this embodiment is the same as that of the first embodiment (Figure 2), except that the content of step 360 is changed. In Figure 6, the dotted line 321 shows the flow of pixel signals obtained by the first imaging, and the dotted line 322 shows the flow of pixel signals obtained by the second imaging.

[0065] In this embodiment, in the pixel signal adjustment process (step 365) following the second readout process (step 350), the signal output unit 150 outputs an image using the pixel signal from the first imaging process, which determines the event, in addition to the pixel signal 322 from the second imaging process. Specifically, as shown in Figure 6, in step 365, the signal processing unit 140 generates a frame image based on the pixel signal 321 from the first readout (step 320) and the pixel signal 322 from the second readout (step 350). The frame image is generated, for example, by combining the pixel signal 321 and the pixel signal 322. In step 370, the signal processing unit 140 outputs the generated frame image.

[0066] In this embodiment, the pixel signal obtained by the first readout used for event detection is also used to generate the frame image. Therefore, even when event detection is performed, the total exposure time for shooting is hardly reduced. Accordingly, this embodiment provides the effects of the first embodiment while also enabling the output of frame images with higher image quality.

[0067] (Third embodiment) The third embodiment differs from the first embodiment in that it performs the first readout only on some pixels, and is otherwise the same as the first embodiment. This embodiment may be implemented in combination with the second embodiment.

[0068] In this embodiment, the multiple pixels 600 are divided into a first pixel group and a second pixel group, and the first imaging is performed using only the first pixel group. Hereinafter, pixels belonging to the first pixel group will also be referred to as the first pixels, and pixels belonging to the second pixel group will also be referred to as the second pixels.

[0069] Figure 7A is a schematic diagram illustrating an example of the pixel configuration in this embodiment. As shown in Figure 7A, the image sensor has a first pixel group 601 and a second pixel group 602 arranged in rows (horizontally). That is, the first pixels and the second pixels are arranged in alternating rows, and when viewed in the column direction, the first pixels and the second pixels are arranged alternately.

[0070] The operation of this embodiment is basically the same as that of the first embodiment (Figure 2). However, in the first readout process in step 320, only pixels belonging to the first pixel group 601 are targeted, and pixels belonging to the second pixel group 602 are not read out. The event detection process in step 330 is performed based on the pixel signal obtained from the first pixel group 601. After the event is detected, the processing from step 350 onward is performed on both the first pixel group 601 and the second pixel group 602.

[0071] As described above, according to this embodiment, instead of reading out all pixel signals for event detection, only some pixels are read out, thus shortening the pixel signal transfer time. Therefore, the reduction in exposure time during the second image capture process associated with event detection can be minimized. In addition, since the number of pixels used in the event detection process (first signal processing) is limited, the signal processing load can be minimized, enabling miniaturization and reduced power consumption.

[0072] Furthermore, since the pixels belonging to the first pixel group 601 are scattered throughout the entire screen, event detection can be performed across the entire screen. Here, since sufficient information can often be obtained even with downsampled pixel signals in event detection, this embodiment is widely applicable. This is possible. It is desirable that the number of pixels belonging to the first pixel group 601 is less than the number of pixels belonging to the second pixel group 602. Also, considering the processing load and detection efficiency, it is desirable that the ratio of the number of pixels belonging to the first pixel group 601 to the total number of pixels be around several tens of percent to several percent.

[0073] Furthermore, while it is necessary to add control lines for the first and second imaging of the first pixel, since an entire row is considered the first pixel, the number of control lines relative to the number of first pixels can be minimized.

[0074] As described above, the image sensor according to this embodiment can reduce the signal processing and transfer processing load by performing the first readout process and the first signal processing using only the first pixel group, thereby enabling efficient event detection.

[0075] Note that the pixel arrangement in Figure 7A is merely one example. For example, as shown in Figure 7B, the pixels of the first pixel group 601 can also be arranged in a dot pattern. This increases the number of control lines for the pixels, but it reduces the number of pixel data read out in the first readout process. Therefore, it is possible to configure an image sensor that further reduces power consumption and shortens the time required for event detection. Figure 7B shows a configuration in which the first pixels are arranged individually in a dot pattern, but a configuration in which multiple first pixels are arranged in a tile pattern may also be adopted. Furthermore, the first pixels do not need to be arranged in a grid pattern and may be placed at any position. Figure 7C shows a configuration in which the first pixels are arranged in a tile pattern and at non-periodic intervals. The pixel arrangements that can be adopted in this embodiment are not limited to the examples above and may be arbitrary.

[0076] (Fourth embodiment) The fourth embodiment differs from the third embodiment in that the second readout process for the second pixel group is different, but otherwise it is the same as the third embodiment. The differences from the third embodiment will be mainly described below.

[0077] The image sensor according to this embodiment includes a first pixel group and a second pixel group, similar to the third embodiment. In this embodiment, while the pixels in the first pixel group are performing the first imaging, the pixels in the second pixel group begin exposure for the second imaging.

[0078] The timing of the operation of the second pixel group will be explained using Figure 8. Note that the timing of the operation of pixels belonging to the first pixel group is the same as that explained using Figure 3, so the explanation will be omitted.

[0079] The difference between the operation of the second pixel group (Figure 8) and the operation of the first pixel group (Figure 3) is that, as shown in Figures 8(b) to (d), the second pixel group does not undergo a first exposure period, a first readout, or a first signal processing. In this embodiment, the start timing of the first exposure of the first pixel group and the start timing of the second exposure of the second pixel group are the same, and the exposure period of the first exposure of the first pixel group and the exposure period of the second exposure of the second pixel group overlap in time. However, even if the exposure start timings are not matched, the effects of this embodiment can be obtained by overlapping the exposure periods and making the exposure period of the second exposure of the second pixel group longer in the third embodiment as well.

[0080] In this embodiment, pixels belonging to the second pixel group can use a long exposure period from after the accumulated charge reset signal to the second readout signal, as shown in Figure 8(e). As a result, with respect to the second pixel group, even if an event is detected, it is not necessary to shorten the exposure time during imaging, and the frame image of the pixels in the second pixel group will not be affected by the image quality due to event detection.

[0081] The image sensor according to this embodiment can perform immediate event detection while suppressing the degradation of image quality of acquired images.

[0082] (Fifth embodiment) The fifth embodiment differs from the fourth embodiment in the content of the frame image generation process, but is otherwise the same as the fourth embodiment. The differences from the fourth embodiment will be mainly explained below.

[0083] In this embodiment, the pixel signal obtained from the first pixel during the second imaging period and the pixel signal obtained from the second pixel during the second imaging period are combined (corrected) based on the ratio of the exposure period lengths to generate pixel output data.

[0084] This embodiment will be explained using Figure 9A. In this embodiment, similar to the fourth embodiment, while the pixels in the first pixel group are performing the first imaging, the pixels in the second pixel group begin exposure for the second imaging. Therefore, as shown in Figure 9A, the length of exposure T1 for the pixels in the first pixel group during the second imaging and the length of exposure T2 for the pixels in the second pixel group during the second imaging are different. Specifically, the pixels in the first pixel group undergo additional first imaging and first signal processing compared to the pixels in the first pixel group, resulting in a shorter exposure time during the second imaging. Therefore, if the acquired pixel signals are converted directly into frame images, the pixels in the first image group will appear darker than the pixels in the second pixel group.

[0085] In light of this problem, the signal processing unit 140 in this embodiment performs the following processing as a second signal processing step. First, the signal processing unit 140 adjusts each pixel value of the second pixel signal of the first pixel group by multiplying it by the ratio of the second exposure time of the second pixel group to that of the first pixel group. Then, the signal processing unit 140 generates an output image signal based on the adjusted second pixel signal of the first pixel group and the second pixel signal of the second pixel group.

[0086] Figure 9B is a diagram illustrating the frame image generation process by the signal processing unit 140 in this embodiment. In this embodiment, the signal processing unit 140 has a correction unit 141 that increases the pixel value by applying a predetermined gain to the second pixel signal 221 obtained from the second imaging of the first pixel. The predetermined gain is the ratio (T2 / T1) of the exposure period time length T1 information 211 for the second exposure of the first pixel and the exposure period time length T2 information 212 for the second exposure of the second pixel. By applying the gain T2 / T1 to the pixel signal 221 of the first pixel, the corrected pixel signal 231 becomes the same brightness as the pixel signal 222. The signal processing unit 140 combines the corrected pixel signal 231 by the correction unit 141 with the pixel signal 222 obtained from the second imaging of the second pixel and outputs a frame image 203.

[0087] In this way, the brightness of the first and second pixels can be made nearly uniform, minimizing the change in brightness of the frame image. Although applying gain to the pixel signal of the first pixel increases noise, the number of first pixels can be less than the number of second pixels, so the image quality degradation associated with correcting the pixel signal of the first pixel can be kept to a minimum.

[0088] As described above, the image sensor according to this embodiment can perform immediate event detection while minimizing changes in the brightness of the acquired image quality.

[0089] (Sixth embodiment) The sixth embodiment differs from the fourth embodiment in that it also uses the pixel signal from the first image capture of the first pixel used for event detection to generate the frame image; otherwise, it is the same as the fourth embodiment. The implementation method is the same as the fifth embodiment. This embodiment may also be implemented in combination with the fifth embodiment.

[0090] Figure 10 is a schematic diagram illustrating the signal processing unit 140 of this embodiment. The signal processing unit 140 generates a frame image 203 using the first pixel signal 220 and the second pixel signal 221 of the first pixel group and the second pixel signal 222 of the second pixel group. The first pixel signal 220 obtained from the first imaging of the first pixel is used for event detection and then temporarily stored in the internal memory of the signal processing unit 140. Subsequently, the first pixel signal 220 is added to the second pixel signal 221 obtained from the second imaging of the first pixel by the adder 142. The combined signal is called the pixel signal 232. The signal processing unit 140 adds the pixel signal 232 to the pixel signal 222 obtained from the second imaging of the second pixel to generate a frame image 203.

[0091] According to this embodiment, the exposure time of the pixel signal used to generate the frame image for the first pixel is the sum of the first exposure time T0 (Figure 3(b)) and the second exposure time T1 (Figure 3(e)). The summed exposure time is approximately the same as the exposure period T2 (Figure 8(e)) during the second shooting of the second pixel. Therefore, the difference in brightness between the first and second pixels can be suppressed, and a high-quality frame image can be generated. Furthermore, since the magnification of the pixel signal is not changed as in the fifth embodiment, there is no deterioration in the signal-to-noise ratio (SNR), and thus a higher quality image can be output.

[0092] Therefore, the image sensor according to this embodiment can perform immediate event detection while minimizing the degradation of image quality of acquired images.

[0093] In this embodiment, the description has been based on a configuration in which pixel signals 232 and 222 are simply added together, but the present invention is not limited to this. If there are no problems in practical use, the above configuration may be used, or if higher quality pixels are desired, a configuration combined with the configuration of the fifth embodiment may be adopted. Specifically, the signal processing unit 140 may increase the pixel signal 232, which is the sum of pixel signals 220 and 221, by the ratio of the second exposure time (T2) of the second pixel to the sum of the first exposure time and second exposure time of the first pixel (T0+T1). This makes it possible to correct the difference in exposure time for the first readout time for event detection, and to output a frame image with higher quality.

[0094] (Seventh Embodiment) The seventh embodiment differs from the first embodiment in that the first imaging (exposure and readout) is repeated multiple times within one cycle, but is otherwise the same as the first embodiment. This embodiment may be implemented in combination with any of the second to sixth embodiments.

[0095] Figure 11 is a diagram illustrating the operation of the image sensor in this embodiment. In this embodiment, as shown in Figure 11, the first exposure and first readout processes are repeated multiple times during the first imaging period within one cycle. In Figure 11, three exposures and readouts are performed during the first imaging period, but the number is not particularly limited as long as it is two or more times. By performing multiple exposures and readouts during the first imaging period, event detection processing (first signal processing) can be performed based on multiple first pixel signals, making it possible to more accurately grasp the movement state and direction of the subject. Specifically, the signal processing unit 140 can grasp high-speed movement within a single frame and more accurately detect events.

[0096] The image sensor according to this embodiment can perform event detection more accurately without requiring high-load signal processing or image transfer externally, and can provide an image sensor that can acquire images without delay when an event occurs.

[0097] (Eighth embodiment) The eighth embodiment differs from the first to seventh embodiments in that the image sensor is configured as a stacked sensor; otherwise, it is the same as the first to seventh embodiments. The image sensor according to this embodiment is configured as a stacked type sensor in which a substrate having a photoelectric conversion element (first substrate) and a digital substrate (second substrate) are stacked in the substrate thickness direction.

[0098] Figure 12 is a diagram illustrating the configuration of the image sensor according to this embodiment. The image sensor according to this embodiment has a configuration in which a substrate 651 having a photoelectric conversion unit 110 and a digital substrate 652 are stacked in the substrate thickness direction and connected by a substrate bonding member 653. The substrate 651 having the photoelectric conversion unit is equipped with circuits that are close to the pixels, such as the photoelectric conversion unit 110, a signal readout unit 130, a pixel selection unit 132, and a signal transfer unit 133. On the other hand, the digital substrate 652 is equipped with a digital signal conversion unit 134, a signal processing unit 140, and memory. By connecting these two substrates in the vertical direction, signals from each pixel can be efficiently transferred to the digital unit, making it possible to create a smaller and higher-performance image sensor. Furthermore, the substrate bonding member 653 can be easily realized using general semiconductor manufacturing processes such as solder bumps or direct bonding between substrates.

[0099] In this embodiment, since a stacked type sensor is used, signal processing of pixel signals from a large number of pixels can be performed at high speed, resulting in more accurate and faster event detection.

[0100] (Ninth embodiment) The ninth embodiment is an imaging device using any of the image sensors described in the first to eighth embodiments.

[0101] Figure 13A is a schematic diagram illustrating the imaging device 701 of this embodiment. The imaging device 701 comprises an image sensor 100 according to any of the first to eighth embodiments and a recording device 702. The image sensor 100 outputs an image output 204 to the recording device 702. The image output 204 is a valid frame image if an event has occurred, and an invalid frame image if no event has occurred. The recording device 702 records when a valid frame image is input, and does not record when an invalid frame image is input.

[0102] In this way, by configuring the imaging device 701 using the image sensor 100, it is possible to monitor events in the subject and automatically record the moment an event occurs.

[0103] In addition, the above description describes a configuration in which only the moment an event occurs is recorded. However, the system can also be used in a similar manner to record for a certain period after an event occurs. Furthermore, the imaging device 701 may have a function to transmit the image at the time of event detection to an external source via a network or the like, instead of recording it to the recording device 702, or in addition to doing so.

[0104] (Tenth embodiment) The tenth embodiment is a monitoring device using any of the image sensors described in the first to eighth embodiments.

[0105] Figure 13B is a schematic diagram illustrating the monitoring device 800 of this embodiment. The monitoring device 800 comprises an image sensor 100 according to any of the first to eighth embodiments and an event notification unit 802. The image sensor 100 outputs an image output 204 to the event notification unit 802. The image output 204 is a valid frame image if an event has occurred, and an invalid frame image if no event has occurred. The event notification unit 802 outputs a valid frame When an image is input, the system notifies that an event has occurred; however, if an invalid frame image is input, no notification is given. The specific manner of notification is not particularly limited. For example, the event notification unit 802 may notify an external device that an event has occurred via communication, or it may notify people in the vicinity that an event has occurred by generating sound or light.

[0106] For example, the monitoring device 800 can be installed to photograph locations such as intersections and configured to detect the occurrence of traffic accidents as events. When a traffic accident is detected, the monitoring device 800 can be configured to automatically notify the police and report the situation to a traffic information center. The monitoring device 800 may also directly distribute images to nearby vehicles. This allows drivers to make their own judgments based on the images and take evasive action to avoid danger or congestion.

[0107] Furthermore, the monitoring device 800 can also be used in a hazard prevention system at intersections with poor visibility. The monitoring device 800 is configured to detect the approach of vehicles or pedestrians as events and to warn those nearby of their approach.

[0108] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. [Explanation of Symbols]

[0109] 100: Image sensor 110: Photoelectric conversion unit 120: Reset unit 130: Signal readout unit 140: Signal processing unit

Claims

1. An imaging element having a plurality of pixels, The accumulated charges of the plurality of pixels are periodically reset; In at least some of the plurality of pixels, performing a first readout process for reading out the charges accumulated by the first exposure; performing first signal processing based on first pixel signals read out by the first readout processing; a determination is made according to a result of the first signal processing as to whether or not to perform a second readout process for reading out accumulated charges due to a second exposure and a second signal processing based on a second pixel signal read out by the second readout process; the first readout process, the first signal processing, and the second readout process are performed within one period of a periodically performed reset; Image sensor.

2. the first readout process, the first signal process, the second readout process, and the second signal process are performed within one period of a reset that is periodically performed; 2. The imaging device according to claim 1.

3. the second signal processing is processing for generating an output image signal based on the first pixel signal in addition to the second pixel signal.

3. The imaging device according to claim 1.

4. the plurality of pixels includes a first group of pixels and a second group of pixels; the first readout process is performed on pixels belonging to the first pixel group; the second readout process is performed on pixels belonging to the first pixel group and pixels belonging to the second pixel group; 4. The imaging device according to claim 3.

5. an exposure period of the first exposure for the pixels belonging to the first pixel group and an exposure period of the second exposure for the pixels belonging to the second pixel group overlap in time; 5. The imaging device according to claim 4.

6. a start timing of the first exposure for the pixels belonging to the first pixel group and a start timing of the second exposure for the pixels belonging to the second pixel group are equal; 6. The imaging device according to claim 5.

7. the second signal processing is processing for generating an image signal for output based on the second pixel signal.

7. The imaging device according to claim 4.

8. the second signal processing is a process of adjusting each pixel value of a second pixel signal of a pixel belonging to a first pixel group by a ratio between an exposure time of a second exposure of the pixels belonging to the second pixel group and an exposure time of a second exposure of the pixels belonging to the first pixel group, and generating the image signal for output using the adjusted second pixel signals.

8. The imaging device according to claim 7.

9. the second signal processing is processing for generating the image signal for output using first pixel signals and second pixel signals of pixels belonging to a first pixel group and second pixel signals of pixels belonging to a second pixel group.

9. The imaging device according to claim 8.

10. the second signal processing is a process of adjusting each pixel value, which is a sum of a first pixel signal and a second pixel signal of a pixel belonging to a first pixel group, by a ratio between an exposure time of a second exposure of the pixels belonging to the second pixel group and a total exposure time of the first exposure and the second exposure of the pixels belonging to the first pixel group, and generating the image signal for output using the adjusted image signal; 10. The imaging device according to claim 9.

11. the number of pixels belonging to the first pixel group is less than the number of pixels belonging to the second pixel group; 11. The imaging device according to claim 4.

12. the first exposure and the first readout process are performed multiple times within one reset cycle; 12. The imaging device according to claim 1.

13. the first signal processing is processing for detecting the occurrence of a predetermined event, When the occurrence of the predetermined event is detected, the second readout process and the second signal process are performed.

12. The imaging device according to claim 1.

14. When the occurrence of the predetermined event is detected, the second readout process and the second signal process are performed continuously for a predetermined period of time.

14. The imaging device according to claim 13.

15. the predetermined event includes at least one of a still subject starting to move and a moving subject becoming still; 15. The imaging device according to claim 14.

16. if the occurrence of the predetermined event is not detected in the first signal processing, the second readout processing and the second signal processing are not performed, and an invalid image signal is output.

16. The imaging device according to claim 13.

17. The invalid image signal is a signal representing an image filled with a specific value or an image having a specific pattern.

17. The imaging device according to claim 16.

18. a first substrate having a photoelectric conversion unit and a readout unit that reads out charges stored in the photoelectric conversion unit; a second substrate having a signal processing unit that performs the first signal processing and the second signal processing; Equipped with The first substrate and the second substrate are stacked in the substrate thickness direction.

18. The imaging device according to claim 1.

19. an imaging element according to any one of claims 1 to 18; A recording device; Equipped with the first signal processing is processing for detecting the occurrence of a predetermined event, the recording device records the image signal generated by the second signal processing when the occurrence of the predetermined event is detected. Imaging device.

20. an imaging element according to any one of claims 1 to 18; A notification unit; Equipped with the first signal processing is processing for detecting the occurrence of a predetermined event, the notification unit notifies the occurrence of the predetermined event when the occurrence of the predetermined event is detected. Monitoring equipment.

21. the notification unit notifies the occurrence of the predetermined event, and also notifies the image signal generated by the second signal processing.

21. The monitoring device of claim 20.

22. A method for controlling an image sensor including a plurality of pixels each including a photoelectric conversion unit, a readout unit that reads out charges accumulated in the photoelectric conversion unit, and a reset unit that resets the photoelectric conversion unit, and a signal processing unit that performs signal processing on pixel signals output from the plurality of pixels, comprising: periodically resetting the photoelectric conversion units in the plurality of pixels; performing a first readout process for reading out charges accumulated by the first exposure; performing first signal processing based on first pixel signals read out by the first readout processing; determining whether to perform a second readout process of reading out accumulated charges due to a second exposure and a second signal process based on a second pixel signal readout by the second readout process, depending on a result of the first signal process; Including, the first readout process, the first signal processing, and the second readout process are performed within one period of a periodically performed reset; A method for controlling an image sensor.

Images