Solid-state imaging device with ai function, driving method thereof, and electronic device

Abstract

To provide a solid-state imaging device, a driving method for a solid-state imaging device, and an electronic apparatus that can store output pixel signals from a sensor unit in a frame memory at 1, 2, or 4 times the size of one TV frame, enable readout at 120 Hz or higher, enable selection of a desired band of spatial frequency obtained by the difference in output, and also enable easy determination of the direction of movement of a subject within one TV frame.SOLUTION: A detection unit 430 measures the difference between segments corresponding to the same position for each frame or the continuity of signals between a plurality of adjacent segments within a frame to detect the edges and movement of the subject on the basis of the results by driving and moving a lens or sensor unit by a focusing unit 420, improves the autofocus's ability to track moving objects and extract subjects on the basis of this detection result, and enables tracking.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a solid-state imaging device having an AI function (artificial intelligence) that improves the efficiency of image recognition and artificial intelligence calculations, a driving method for the same, and electronic equipment having the AI ​​function. [Background technology]

[0002] 2. Description of the Related Art CMOS (Complementary Metal Oxide Semiconductor) image sensors are in practical use as solid-state imaging devices (image sensors) that use photoelectric conversion elements that detect light and generate electric charges. CMOS image sensors are widely used as part of various electronic devices such as digital cameras, video cameras, surveillance cameras, medical endoscopes, personal computers (PCs), and portable terminal devices (mobile devices) such as mobile phones.

[0003] CMOS image sensors have a photodiode (photoelectric conversion element) and a floating diffusion (FD) amplifier with a floating diffusion layer for each pixel, and the mainstream readout method is a column-parallel output type that selects a row in the pixel array and reads them out simultaneously in the column output direction.

[0004] Furthermore, a wide variety of pixel signal readout (output) circuits have been proposed for column-parallel output CMOS image sensors. Among them, one of the most advanced circuits is a circuit that includes an analog-to-digital converter (ADC) for each column and extracts pixel signals as digital signals (see, for example, Patent Documents 1 and 2).

[0005] In this CMOS image sensor with column-parallel ADC (column AD type CMOS image sensor), the comparator compares the so-called RAMP wave with the pixel signal, and performs AD conversion by performing digital CDS in the downstream counter.

[0006] However, although this type of CMOS image sensor is capable of high-speed signal transfer, it has the disadvantage of not being able to perform global shutter readout.

[0007] In response to this, a digital pixel sensor has been proposed that places an ADC (and even a memory section) including a comparator in each pixel, making it possible to realize a global shutter that starts and ends exposure at the same timing for all pixels in the pixel array section (see, for example, Patent Documents 3 and 4). [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Japanese Patent Application Laid-Open No. 2005-278135 [Patent Document 2] Japanese Patent Application Laid-Open No. 2005-295346 [Patent Document 3] US 7164114 B2 FIG, 4 [Patent Document 4] US 2010 / 0181464 A1 Summary of the Invention [Problem to be solved by the invention]

[0009] In conventional CMOS image sensors, photoelectrically converted signals are read out in order from the top left of the screen, so pixel signals are read out in chronological order. Therefore, when it is desired to obtain a certain frequency band of an image, such as in an autofocus system, it is necessary to pass the time series signal through a filter and perform an operation.

[0010] Furthermore, when extracting the R / G / B color components, such as for auto white balance, the time-series signal was obtained by passing it through a selector circuit. Therefore, because the images captured by the CMOS image sensor are read out sequentially in time series, all processing that can be easily achieved through spatial processing must first be converted into temporal frequencies, resulting in inefficient processing. Furthermore, since this processing was performed every 60 Hz of a TV frame, it was sometimes impossible to obtain information about a subject that was moving at high speed.

[0011] The present invention provides a solid-state imaging device, a method for driving a solid-state imaging device, and electronic equipment that can store output pixel signals from a sensor unit in a frame memory at a frequency 1, 2, or 4 times that of one TV frame, enable readout at 120 Hz or higher, enable selection of a desired band of spatial frequency obtained by the difference in output, and further enable easy determination of the direction of movement of a subject within one TV frame. [Means for solving the problem]

[0012] A solid-state imaging device according to a first aspect of the present invention comprises a sensor unit in which pixels that perform photoelectric conversion are arranged, and a readout control unit that reads out pixel signals from the pixels of the sensor unit, the readout control unit including a frame memory and capable of storing output pixel signals of the sensor unit in the frame memory at a frequency that is 1, 2, or 4 times that of one TV frame, and capable of reading out at 120 Hz or more; a focusing unit that drives a lens or the sensor unit arranged in a front stage on the light incident side of the sensor unit along an associated optical axis to focus on a subject; and a detection unit that defines a group of multiple (n x n) pixels as a segment, and measures the difference between segments corresponding to the same position for each frame or the continuity of signals with multiple adjacent segments within a frame based on the results of driving and moving the lens or the sensor unit by the focusing unit, thereby detecting the edges and movement of the subject, and based on this detection result, improving the autofocus's ability to track a moving object and extract a subject, and further enabling tracking.

[0013] A second aspect of the present invention is a method for driving a solid-state imaging device that performs readout control to read out pixel signals from pixels of a sensor unit in which pixels that perform photoelectric conversion are arranged, the readout control including: a pixel signal readout step that is capable of storing output pixel signals of the sensor unit in a frame memory at a frequency that is 1, 2, or 4 times that of one TV frame, and that performs readout at 120 Hz or more; a focusing step that drives a lens that is arranged in a front stage on the light incident side of the sensor unit or the sensor unit along an associated optical axis to focus on a subject; and a detection step that defines a plurality of (n x n) pixel sets as segments, and, based on the results of driving and moving the lens or the sensor unit in the focusing step, measures the difference between segments corresponding to the same position for each frame or the continuity of signals with a plurality of adjacent segments within a frame to detect the edges and movement of the subject, and, based on the detection results, improves the moving object tracking ability and subject extraction ability of autofocus and further enables tracking.

[0014] An electronic device according to a third aspect of the present invention comprises a solid-state imaging device and an optical system that forms an image of a subject on the solid-state imaging device, wherein the solid-state imaging device comprises a sensor section in which pixels that perform photoelectric conversion are arranged, and a readout control section that reads out pixel signals from the pixels of the sensor section, wherein the readout control section comprises a pixel signal readout circuit that includes a frame memory and is capable of storing output pixel signals of the sensor section in the frame memory at a frequency that is 1, 2, or 4 times that of one TV frame, and is capable of reading out at 120 Hz or more, a focusing section that drives a lens or the sensor section that is arranged in a front stage on the light incident side of the sensor section along an associated optical axis to focus on the subject, and a detection section that defines a group of multiple (n x n) pixels as a segment, and measures the difference between segments corresponding to the same position for each frame or the continuity of signals with multiple adjacent segments within a frame based on the results of driving and moving the lens or the sensor section by the focusing section, thereby detecting the edges and movement of the subject, and based on this detection result, improving the moving object tracking ability and subject extraction capability of the autofocus and further enabling tracking. [Effects of the Invention]

[0015] According to the present invention, the pixel signal output from the sensor unit can be stored in the frame memory at 1, 2 or 4 times the size of one TV frame, making it possible to read out at 120 Hz or higher. Furthermore, according to the present invention, it is possible to select a desired band of spatial frequencies obtained from the difference in output, and it is also possible to easily determine the direction of movement of a subject within one TV frame. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a block diagram illustrating an example of the configuration of a solid-state imaging device according to an embodiment of the present invention. [Figure 2] 2 is a circuit diagram showing an example of a pixel signal readout system of the solid-state imaging device according to the embodiment of the present invention. FIG. [Figure 3] 1 is a diagram showing the configuration of a pixel PXL, a sensor gate switch SGSW, an AD conversion unit, and a horizontal register HREG of a sensor unit according to an embodiment of the present invention, and a diagram showing an example of the configuration of four frame memories that receive the signals. FIG. [Figure 4] FIG. 10 is a diagram showing the relationship between the on / off state of the sensor gate switch and the output timing in one TV frame. [Figure 5] 1 is a diagram showing an image of a movable image sensor unit as a focusing unit which is a driving device according to an embodiment of the present invention; [Figure 6] FIG. 1 is a first diagram for explaining a process for observing the movement direction and correlation of a subject. [Figure 7] FIG. 2 is a second diagram for explaining the process for observing the movement direction and correlation of the subject. [Figure 8] FIG. 10 is a diagram illustrating correlation between segments. [Figure 9] FIG. 10 is a diagram illustrating an example of the configuration of a spatial frequency band-pass filter (BPF). [Figure 10] FIG. 10 shows an overlay for pixel data correlation at the BPF output and segment end face. [Figure 11] 1 is a diagram showing an example of the configuration of an electronic device having an artificial function to which a solid-state imaging device according to an embodiment of the present invention is applied; DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0018] (Embodiment) FIG. 1 is a block diagram showing an example of the configuration of a solid-state imaging device according to an embodiment of the present invention. FIG. 2 is a circuit diagram showing an example of a pixel readout system of the solid-state imaging device according to the embodiment of the present invention. In this embodiment, the solid-state imaging device 10 is configured by, for example, a CMOS image sensor.

[0019] As shown in FIG. 1, the solid-state imaging device 10 has a sensor section 20 as an imaging section, a vertical scanning circuit (row scanning circuit) 30, a signal processing circuit 40, and a timing control circuit 50 as main components. Of these components, for example, the vertical scanning circuit 30, the signal processing circuit 40, and the timing control circuit 50 constitute a pixel signal readout control unit 60.

[0020] In this embodiment, the solid-state imaging device 10 has a sensor section 20 in which pixels that perform photoelectric conversion are arranged, and a signal processing circuit 40 that, under a read control section 60, reads out pixel signals SPX from the pixels 200 of the sensor section 20, performs predetermined signal processing, and outputs the signal.

[0021] The signal processing circuit 40, under the control of the read control unit 60, is configured to perform, for example, pixel signal read processing, focusing processing, and based on the results of these, measure the difference between segments SGM corresponding to the same position for each frame FRM or the signal continuity with multiple adjacent segments SGM within the frame FRM to detect the edges and movement of the subject, and based on this detection result, improve the autofocus's ability to track moving objects and extract subjects, and even perform detection processing that enables tracking.

[0022] The signal processing circuit 40 of this embodiment has, as one of its main components, a pixel signal readout section 410, a focusing section 420, a detection section 430, and a horizontal register group 440, which are arranged as one of the components of the readout control section 60.

[0023] The pixel signal readout unit 410 is mainly disposed at the connection between the sensor unit 20 and the signal processing circuit 40 . The pixel signal readout unit 410 includes, for example, a photoelectric conversion readout unit 411 as the pixel 200, an AD (analog-digital) conversion unit 412, and a memory unit 413 having a frame memory FRM, and is configured as, for example, a stacked CMOS image sensor. In the solid-state imaging device 10 according to this embodiment, each pixel PXL has an AD (analog-to-digital) conversion function, and the AD conversion unit 412 has a comparator CMP that compares the voltage signal read out by the photoelectric conversion readout unit 411 with a reference voltage, performs analog-to-digital (AD) conversion processing on the readout voltage signal VSL, and outputs a digitized comparison result signal.

[0024] The pixel signal readout unit 410 of this embodiment includes a frame memory FRM, and is capable of storing the output pixel signals of the sensor unit 20 in the frame memory FRM at a frequency that is 1, 2, or 4 times that of one TV frame, and is configured to enable readout at 120 Hz or higher.

[0025] The focusing unit 420 of the signal processing circuit 40 of this embodiment is configured to include a focusing unit 420 as a driving device that drives the lens LNS or sensor unit 20, which is located in the front stage on the light incident side of the sensor unit 20, along the associated optical axis to focus on the subject.

[0026] Furthermore, the detection unit 430 of the signal processing circuit 40 of this embodiment defines a group of multiple (n x n) pixels as a segment, and based on the results of driving and moving the lens LNS or the sensor unit 20 by the focusing unit 420, measures the difference between segments corresponding to the same position for each frame or the continuity of signals with multiple adjacent segments within a frame, thereby detecting the edges and movement of the subject using, for example, an AI function, and is configured to improve the autofocus's ability to track moving objects and extract subjects based on this detection result, and even enable tracking.

[0027] In the detection unit 430 of this embodiment, 3x3, 4x4, 8x8 or 9x9 pixels are defined as one segment, and by measuring the difference between segments corresponding to the same position for each frame FRM or the signal continuity with eight adjacent segments within the frame, the edges and movement of the subject can be detected, for example, using an AI function, thereby improving the autofocus's ability to track moving objects and extract subjects, and even enabling tracking.

[0028] Furthermore, the pixel signal readout unit 410 of this embodiment includes in the sensor unit 20 a pixel signal readout unit 410 that reads out all pixels in the horizontal direction at 60 Hz in the first TV frame, and in the next TV frame mixes the charges of two adjacent pixels or two adjacent pixels of the same color and reads them out at 120 Hz, or mixes the charges of four adjacent pixels or four adjacent pixels of the same color and reads them out at 240 Hz, and it is possible to select a desired band of spatial frequency obtained by the difference between these outputs.

[0029] Below, we will provide a detailed description of the configuration and function of each part of the solid-state imaging device 10 of this embodiment, in particular the configuration and function of the sensor unit 20, pixel signal readout unit 410, focusing unit 420, and detection unit 430, as well as the readout processing associated with them.

[0030] FIG. 2 is a circuit diagram showing an example of the configuration of a pixel signal readout circuit according to an embodiment of the present invention.

[0031] The sensor unit 20 has a plurality of pixels 200 arranged in a matrix of N rows and M columns.

[0032] The pixel signal readout unit 410 according to this embodiment includes a photoelectric conversion readout unit 411, an AD conversion unit (referred to as ADC) 412, and a memory unit 413 having a frame memory. The sensor unit 20 of this embodiment is configured as a stacked CMOS image sensor of a first substrate 110 and a second substrate 120, but in this example, as shown in Figure 2, a photoelectric conversion readout unit 411 is formed on the first substrate 110, and an AD conversion unit 412 and a memory unit 413 are formed on the second substrate 120.

[0033] The photoelectric conversion readout unit 411 of the pixel 200 includes a photodiode (photoelectric conversion element) and one in-pixel amplifier. Specifically, the photoelectric conversion readout unit 411 has, for example, a photodiode PD0 which is a photoelectric conversion element. In the pixel 200 of this embodiment, the photodiode PD0 is connected to a floating diffusion FD serving as an output node ND0.

[0034] The photodiode PD0 accumulates the charge generated by photoelectric conversion during the accumulation period. A transfer transistor TG0-Tr serving as a transfer element is connected between the storage portion PND0 of the photodiode PD0 and the floating diffusion FD, and the storage portion PND0 is connected to a predetermined fixed potential VAAPIX.

[0035] The photoelectric conversion readout unit 411 has a reset transistor RST-Tr as a reset element, a source-follower lower transistor SF-Tr as a source follower element, and a readout node ND1 corresponding to a floating diffusion FD as one output node ND0. The output buffer unit 4111 is configured including the SF-Tr and the readout node ND1.

[0036] In the photoelectric conversion readout section 411 according to this embodiment, a readout node ND1 of an output buffer section 4111 is connected to an input section of an AD conversion section 412. The photoelectric conversion readout unit 411 converts the charge of the floating diffusion FD serving as an output node into a voltage signal according to the amount of charge, and outputs the converted voltage signal VSL to the AD conversion unit 412.

[0037] For example, during the comparison processing period of the AD conversion unit 412, the photoelectric conversion readout unit 411 outputs a voltage signal VSL corresponding to the accumulated charge of the photodiode PD0, which is a photoelectric conversion element, transferred from the photodiode PD0 to the floating diffusion FD, which is an output node, during the accumulation period PI. The photoelectric conversion readout section 411 outputs a readout reset signal (signal voltage) (VRST) and a readout signal (signal voltage) (VSIG) as pixel signals to the AD conversion section 412 during the comparison processing period.

[0038] The photodiode PD0 generates and accumulates signal charges (electrons in this case) in an amount corresponding to the amount of incident light. In the following, the case where the signal charges are electrons and each transistor is an n-type transistor will be described, but the signal charges may be holes and each transistor may be a p-type transistor.

[0039] In each pixel 200, a buried photodiode (PPD) is used as the photodiode (PD). The surface of the substrate on which the photodiode (PD) is formed has interface states due to defects such as dangling bonds, which causes a large amount of charge (dark current) to be generated by thermal energy, making it impossible to read out the correct signal. In a buried photodiode (PPD), the charge storage section of the photodiode (PD) is embedded in the substrate, which makes it possible to reduce the dark current from mixing into the signal.

[0040] The transfer transistor TG0-Tr of the photoelectric conversion readout unit 411 is connected between the storage unit PND0 of the photodiode PD0 and the floating diffusion FD, and is controlled by a control signal TG0 applied to the gate via a control line. The transfer transistor TG0-Tr is selected and turned on during the transfer period PT when the control signal TG0 is at a high (H) level, and transfers the charges (electrons) photoelectrically converted and accumulated in the photodiode PD0 to the floating diffusion FD.

[0041] The source follower transistor SF-Tr as a source follower element has its source connected to the read node ND1, its drain connected to the power supply line Vaapix, and its gate connected to the supply line of the control signal VBNPIX. A signal line LSGN1 between the read node ND1 and the input section of the AD conversion section 412 is driven by a current transistor IC-Tr as a current source element.

[0042] This current transistor IC-Tr, or the current transistor connected to the read node ND1 of the photoelectric conversion readout unit 411 and the signal line LSGN1, functions as a sensor gate switch SGSW.

[0043] The AD conversion unit 412 of the pixel 200 functions to convert the analog voltage signal VSL output by the photoelectric conversion readout unit 411 into a digital signal by comparing it with a reference voltage VREF, which is a ramp waveform that is changed with a predetermined slope or a fixed voltage.

[0044] As shown in FIG. 2, the AD conversion unit 412 includes a comparator (COMP) 4121, a load capacitor CL1 on the output side, and a reset switch SW-RST.

[0045] The comparator 4121 receives the voltage signal VSL output from the output buffer unit 4111 of the photoelectric conversion readout unit 411 to the signal line LSGN1 at its inverting input terminal (-) as its first input terminal, and receives the reference voltage VREF at its non-inverting input terminal (+) as its second input terminal.The comparator 4121 performs AD conversion processing (comparison processing) by comparing the voltage signal VST with the reference voltage VREF and outputting a digitized comparison result signal SCMP.

[0046] The comparator 4121 has a coupling capacitor CC1 connected to its inverting input terminal (-) as its first input terminal, and is configured to achieve low noise and a high SNR at low illuminance by AC coupling the output buffer section 4111 of the photoelectric conversion readout section 411 on the first substrate 110 side and the input section of the comparator 4121 of the AD conversion section 412 on the second substrate 120 side.

[0047] In addition, in the comparator 4121, a reset switch SW-RST is connected between the output terminal and an inverting input terminal (-) serving as a first input terminal, and a load capacitor CL1 is connected between the output terminal and a reference potential VSS.

[0048] Basically, in the AD conversion unit 412, the analog signal (potential VSL) read out from the output buffer unit 4111 of the photoelectric conversion readout unit 411 to the signal line LSGN1 is compared in a comparator 4121 with a reference voltage VREF, for example, a ramp signal RAMP which is a slope waveform that changes linearly with a certain gradient. At this time, a counter (not shown) arranged for each column, similar to the comparator 4121, operates, and the voltage signal VSL is converted into a digital signal by changing the ramp signal RAMP having a ramp waveform and the counter value in one-to-one correspondence. Basically, the AD conversion unit 412 converts a change in the reference voltage VREF (for example, a ramp signal RAMP) into a change in time, and converts this time into a digital value by counting it in a certain cycle (clock). Then, when the analog signal VSL and the ramp signal RAMP (reference voltage VREF) intersect, the output of the comparator 4121 is inverted, stopping the input clock of a counter (not shown), or inputting the clock that had been stopped into a counter (not shown), and the counter value (data) at that time is stored in the memory unit 413 (230), completing the AD conversion. After the AD conversion period described above is completed, the data (signals) stored in the memory unit 413 (230) of each pixel 200 is output from the signal processing circuit 40 to a signal processing circuit (not shown), and a two-dimensional image is generated by predetermined signal processing.

[0049] The memory unit 413 is configured by SRAM or DRAM, and is supplied with a digitally converted signal, which corresponds to a photoconversion code and can be read out by an external IO buffer of the signal processing circuit 40 around the pixel array. In this example, the memory section 413 has four memories 4131 (231), 4132 (232), 4133 (233), and 4134 (234) connected to the output of the comparator 4121.

[0050] The digital data resulting from the comparison processing of the comparator 4121 is stored alternately in four memories 4131 (231), 4132 (232), 4133 (233), and 4134 (234) of the memory section 4130. This makes it possible to achieve a fast read operation.

[0051] The vertical scanning circuit 30 drives the photoelectric conversion readout units 411 of the pixels 200 in the shutter row and readout row through the row scanning control lines in accordance with the control of the timing control circuit 50 . The vertical scanning circuit 30, under the control of the timing control circuit 50, supplies the comparator 221 of each pixel 200 with a reference voltage VREF that is set in accordance with the comparison process. Furthermore, the vertical scanning circuit 30 outputs, in accordance with the address signal, a row selection signal of a read row for reading out a signal and a row address of a shutter row for resetting the charge accumulated in the photodiode PD.

[0052] The signal processing circuit 40 includes, for example, an IO buffer arranged corresponding to the memory output of each pixel 200 of the sensor unit 20, and outputs digital data read from each pixel 200 to the outside.

[0053] The timing control circuit 50 generates timing signals necessary for signal processing in the pixel section 20, the vertical scanning circuit 30, the output circuit 40, and the like.

[0054] In this embodiment, the readout control unit 60 controls the readout of pixel signals from the pixels 200 .

[0055] (Readout control of pixel signals from pixels 200 by the readout control unit 60) Next, the control of reading out pixel signals from the pixels 200 by the readout control unit 60 according to this embodiment will be specifically described.

[0056] First, the basic configuration, various operations, and functions of the sensor unit 20 of the solid-state imaging device (CMOS image sensor) 10 having the above-described configuration and the pixel signal readout circuit 410 will be explained in order with reference to the drawings.

[0057] Figure 3 is a diagram showing the configuration of the pixels (shown as photoelectric conversion elements, photodiodes in Figure 1) PXL, sensor gate switch SGSW, AD conversion unit and horizontal register HREG of the sensor unit 20 according to an embodiment of the present invention, and an example configuration of four frame memories MEM0 to MEM3 (231 to 234) that receive the signals. FIG. 4 is a diagram showing the relationship between the on / off state of the sensor gate switch and the output timing in one TV frame.

[0058] The pixel PXL in Figure 3 is exposed for 1 / 240 seconds and accumulates charge. Then, every 1 / 240 seconds, the sensor gate switch SGSW is turned on, the accumulated charge is A / D converted, and transferred to the horizontal register HREG. The horizontal register HREG is prepared for the number of vertical pixels, and sequentially shifts stored data in the horizontal direction, outputs it, and stores it in the subsequent frame memory MEM. This operation is completed in 1 / 240 seconds after the sensor gate switch SGSW is turned on. During this time, the next 1 / 240 second image is accumulated in pixel PXL in parallel, and once this is complete, the above operation is repeated (Figure 4). In this way, four pieces of image data are output in one TV frame FRM.

[0059] FIG. 5 is a diagram showing an image of a movable image sensor unit as a focusing unit, which is a driving device according to an embodiment of the present invention.

[0060] The image sensor unit 20 is driven by a piezo, voice coil, stepping motor, or the like, to drive the substrate of the image sensor unit 20 horizontally, vertically, or in a rotational direction around the optical axis of the lens LNS. The driving cycle is one TV frame.

[0061] First, the front-stage focus lens is driven for each TV frame to bring the central subject into focus. To improve the efficiency, the front focus lens may be fixed, and instead the image sensor unit 20 may be made movable in the focal direction of the optical axis. Furthermore, if the user wishes to focus on a subject at a specific position on the screen, the contrast of that portion is extracted and the focus is adjusted. Alternatively, if a PDAF or phase difference sensor is available, the focusing operation can be left to that and this mechanism can be omitted.

[0062] 6 and 7 are diagrams for explaining the process for observing the movement direction and correlation of the subject. FIG. 8 is a diagram for explaining the correlation between segments.

[0063] In Fig. 6, in order to observe the movement direction and correlation of the object, for example, a 4x4 pixel segment SGM (Fig. 7) is used to calculate the absolute value of the difference between the first TV frame FRM1 and the second TV frame FRM2. Σ|pij-p(i+3)(j+3)| By adding up 16 pixels and finding a segment with high correlation, i.e., small difference, the direction of movement of the subject can be determined.

[0064] To measure the direction of movement, the image sensor unit 20 is moved to the right between the first and second TV frames for image 1, to the left for image 2, upward for image 3, and downward for image 4 (selecting the point where the correlation of the segments is strongest), and the direction of movement between one TV frame can be easily determined.

[0065] By repeating the above operation over the entire screen, it is possible to determine that the direction in which the object is moving is the direction in which the object is moving, and the arrangement of the point in which the correlation between one corresponding segment is greatest and the point in which the correlation between the adjacent segment is greatest is the direction in which the object is moving, among the directions in which the image sensor unit 20 is moved.

[0066] Furthermore, by tracing this arrangement, the closed surface of the subject is estimated. For example, if the correlation between the 1st TV frame image 1 and the 2nd TV frame image 1 is greatest, it is determined that the subject is moving to the right, and therefore the point where the difference between the values ​​of p14, p24, or p34 between the 1st TV frame p24 and the 2nd TV frame is smallest can be considered to indicate that the brightness of the subject has continued.

[0067] Similarly, p24 of image 1 in the second TV frame is compared with p14, p24, and p34 of image 1 in the third TV frame, and the continuity of the brightness of the subject is observed, allowing the direction of movement of the subject to be ascertained.

[0068] If it is determined that the subject moves to the left during one TV frame (ie, image 3 has the greatest correlation), then p21 of the first TV frame image 3 is compared with p11, p21, and p31 of the second TV frame image 3, for example. If the movement direction between TV frames is upward, p12 of the first TV frame image 4 is compared with p11, p12, and p13 of the second TV frame image 4. Furthermore, if the subject moves downward during one TV frame, p42 of the first TV frame image 2 is compared with p41, p42, and p43 of the second TV frame image 2 to determine the direction in which the brightness of a particular point on the subject continues.

[0069] By operating this, in addition to the direction of movement on the entire screen, detailed information such as the direction and speed of movement on a pixel-by-pixel basis can be obtained.

[0070] If the subject moves faster, the pixel to be compared is shifted by two or several pixels in the direction of the movement.

[0071] The image sensor unit 20 can be driven to rotate clockwise or counterclockwise around the optical axis to observe the direction of movement of the subject.

[0072] FIG. 9 is a diagram showing an example of the configuration of a spatial frequency band pass filter (BPF). FIG. 10 shows an overlay for correlation of pixel data at the BPF output and segment end face.

[0073] For example, if all pixels of the image sensor unit 20 are read out at 60 Hz for the first TV frame, and then for the second TV frame, the first two pixels are read out horizontally at 120 Hz, and the latter four pixels are read out horizontally at 120 Hz, a signal with the bandwidth shown in Figure 9 can be obtained. BPF2 is calculated by subtracting Sig half from Sig full, and BPF1 is calculated by subtracting Sig four from Sig half. These can be used as signals for contrast autofocus, and because BPF2 indicates the edge of the subject, it can also be used as an edge extraction signal. Furthermore, by overlapping it with the closed surface of the subject, highly accurate area extraction becomes possible.

[0074] (Closed curve generation related) For example, if you subtract the image of the first frame read out at 60 Hz in Figure 9 from the image of the second frame read out at 120 Hz, you can obtain contour information like BPF2. By tracing the point in the vicinity of the contour where the correlation of brightness at the end face of the segment in the movement direction between the frames obtained above is highest, it is possible to extract the closed curve formed by the contour of the subject.

[0075] The closed curve obtained in this way is defined as a new object-corresponding segment, and autofocus and tracking according to the object are made possible by calculating the correlation between these object segments for each TV frame.

[0076] Furthermore, if the correlation of the above-mentioned object segments between TV frames decreases by more than a certain amount, it means that another object has invaded a part other than the closed curve part where the correlation is maintained, and a new object segment can be formed for the invading object.

[0077] In this way, even if the subject changes over time or suddenly enters the screen, it is possible to adjust the focus for each subject segment, thereby achieving stable focus.

[0078] <Effects of this embodiment> According to this embodiment, all horizontal pixels of the sensor unit 200 are read within one TV frame, and pixel signals obtained by mixing two pixels are read out at 120 Hz in the first half of one TV frame, and four pixels mixed together are read out at 240 Hz in the second half of one TV frame, and each data is stored in a frame memory, and the 120 Hz data is subtracted from the data of one TV frame, thereby forming a spatial frequency bandpass filter (BPF). This allows for efficient extraction of subject contour information. Similarly, by subtracting 240Hz data from 1TV frame data, or 240Hz data from 120Hz data, it is possible to construct BPFs of various bands, which makes it possible to obtain information useful for autofocusing and image data compression extremely efficiently.

[0079] Furthermore, according to this embodiment, in the sensor unit 200 that is provided with an actuator as a drive device that can drive the substrate of the sensor unit 200 up and down, left and right, or in a rotational direction around the optical axis, and that can output data that combines four pixels horizontally at 240 Hz four times within one TV frame, the image sensor is moved to the left for each frame in the first readout, to the right in the second readout, up in the third readout, and down in the fourth readout, and the pixel data for each segment that corresponds in position each time is compared, thereby making it possible to efficiently estimate the movement direction of the subject within two TV frames.

[0080] Furthermore, according to this embodiment, the detailed movement direction for each subject is calculated by correlating the movement direction determined above with corresponding pixels of the segment.

[0081] Furthermore, according to this embodiment, it is possible to estimate a closed curve formed by the contour of the subject from the correlation of pixel data of the end faces of the corresponding segments in the movement direction.

[0082] In addition, according to this embodiment, a more detailed closed curve can be obtained by repeating the above operations (2) to (4) for a segment corresponding to the contour information obtained by the method for extremely efficiently obtaining information useful for autofocusing and image data compression described above.

[0083] Furthermore, according to this embodiment, object segments can be created corresponding to the closed curve of the object obtained by the above operations, and by calculating the correlation between each object segment for each TV frame, it is possible to achieve focusing and tracking tailored to the desired object.

[0084] The solid-state imaging device 10 described above can be applied as an imaging device to electronic devices such as digital cameras, video cameras, mobile terminals, surveillance cameras, and medical endoscope cameras.

[0085] FIG. 11 is a diagram showing an example of the configuration of an electronic device having an artificial function to which a solid-state imaging device according to an embodiment of the present invention is applied.

[0086] As shown in FIG. 11, the electronic device 300 includes a CMOS image sensor 310 to which the solid-state imaging device 10 according to this embodiment can be applied. Furthermore, the electronic device 300 has an optical system (lens or the like) 320 that guides incident light to the pixel region of the CMOS image sensor 310 (forming an image of a subject). The electronic device 300 includes a signal processing circuit (PRC) 330 that processes the output signal of the CMOS image sensor 310 .

[0087] The signal processing circuit 330 performs predetermined signal processing on the output signal of the CMOS image sensor 310 . The image signal processed by the signal processing circuit 330 can be displayed as a moving image on a monitor such as an LCD display, or output to a printer, or can be recorded directly on a recording medium such as a memory card, and various other forms are possible.

[0088] As described above, by incorporating the above-described solid-state imaging device 10 as the CMOS image sensor 310, it is possible to provide a high-performance, small-sized, and low-cost camera system. This will enable the realization of electronic devices with AI functions, such as surveillance cameras and medical endoscope cameras, that are used in applications where camera installation requirements include restrictions on mounting size, the number of connectable cables, cable length, and installation height. [Explanation of symbols]

[0089] 10···Solid-state imaging device, 20···Sensor unit, PD0···Photodiode, TG0-Tr···Transfer transistor, FD···Floating diffusion, RST-Tr···Reset transistor, RST0-Tr···Reset transistor, SF-Tr···Source follower transistor, SF0-Tr···Source follower transistor, 200···Pixel, 30···Vertical scanning circuit, 40···Signal processing Circuit, 410...pixel signal readout section, 411...photoelectric conversion readout section, 412...AD conversion section, 4121...comparator, 413...memory section, 4131 to 4134...memory, 420...focusing section, 430...detection section, 50...timing control circuit, 60...readout control section, 300...electronic devices, 310...CMOS image sensor, 320...optical system, 330...signal processing circuit (PRC).

Claims

1. a sensor unit in which pixels that perform photoelectric conversion are arranged; a readout control unit that reads out pixel signals from the pixels of the sensor unit, The read control unit a pixel signal readout circuit including a frame memory, capable of storing output pixel signals of the sensor unit in the frame memory at a frequency 1, 2 or 4 times that of one TV frame, and capable of reading at 120 Hz or more; a focusing unit that focuses on a subject by driving a lens or the sensor unit, which is disposed in front of the sensor unit on the light incident side, along a related optical axis; a detection unit that defines a group of multiple (n x n) pixels as a segment, and measures the difference between segments corresponding to the same position for each frame or the continuity of signals with multiple adjacent segments within a frame based on the result of driving and moving the lens or the sensor unit by the focusing unit, thereby detecting the edge or movement of the subject, and based on this detection result, improves the moving object tracking ability and subject extraction ability of the autofocus, and further enables tracking. Solid-state imaging device.

2. The detection unit To observe the movement direction and correlation of the subject, the absolute values ​​of the differences between the first and second TV frames of an n x n pixel segment are summed for all pixels in the segment, and the movement direction of the subject is determined by finding a segment with high correlation and low difference.

2. The solid-state imaging device according to claim 1.

3. The detection unit To measure the direction of movement, the sensor is moved to the right or left between the first and second TV frames for the first image, left or right for the second image, up or down for the third image, and down or up for the fourth image, and the direction of movement between TV frames is determined by selecting the segment with the strongest correlation.

3. The solid-state imaging device according to claim 2.

4. The detection unit The above process is repeated for the entire screen, and the direction in which the alignment of the point where the correlation between one corresponding segment is greatest and the point where the correlation between the adjacent segments is greatest is most similar among the swing directions of the sensor unit, and the direction in which the subject is moving is determined.

4. The solid-state imaging device according to claim 3.

5. The detection unit It is possible to estimate the closed curve formed by the contour of the subject from the correlation of pixel data on the end faces of the corresponding segments in the movement direction.

5. The solid-state imaging device according to claim 4.

6. The pixel signal readout unit In the first TV frame, all horizontal pixels are read at 60Hz, In the next TV frame, the sensor section includes a pixel signal readout circuit that mixes the charges of two adjacent pixels or two adjacent pixels of the same color and reads them out at 120 Hz, or mixes the charges of four adjacent pixels or four adjacent pixels of the same color and reads them out at 240 Hz, and a desired band of spatial frequency obtained by the difference between these outputs can be selected.

3. The solid-state imaging device according to claim 2.

7. The pixel signal readout unit All horizontal pixels of the sensor unit are read within one TV frame, and two pixels are mixed and read out, and the resulting pixel signals are read out at 120 Hz in the first half of one TV frame. In the latter half of one TV frame, the four mixed pixels are read out at 240 Hz, and each data is stored in the frame memory. By subtracting 120 Hz data from one TV frame data, a spatial frequency band pass filter (BPF) is formed.

3. The solid-state imaging device according to claim 2.

8. The pixel signal readout unit All horizontal pixels of the sensor unit are read within one TV frame, and two pixels are mixed and read out, and the resulting pixel signals are read out at 120 Hz in the first half of one TV frame. In the latter half of one TV frame, the four mixed pixels are read out at 240 Hz, and each data is stored in the frame memory. By subtracting 240 Hz data from 1 TV frame data, or subtracting 240 Hz data from 120 Hz data, BPFs of various bands can be formed.

3. The solid-state imaging device according to claim 2.

9. A method for driving a solid-state imaging device that performs readout control to read out pixel signals from pixels of a sensor unit in which pixels that perform photoelectric conversion are arranged, the method comprising: The read control includes: a pixel signal reading step capable of storing an output pixel signal of the sensor unit in a frame memory at a frequency 1, 2, or 4 times that of one TV frame, and reading out the pixel signal at 120 Hz or more; a focusing step of focusing on an object by driving a lens disposed in a front stage on the light incident side of the sensor unit or the sensor unit along a related optical axis; a detection step of defining a plurality of (n×n) pixel sets as a segment, measuring the difference between segments corresponding to the same position for each frame or the continuity of signals with a plurality of adjacent segments within a frame based on the result of driving and moving the lens or the sensor unit in the focusing step, thereby detecting the edge or movement of the subject, and improving the moving object tracking ability and subject extraction ability of the autofocus based on this detection result, and further enabling tracking. A method for driving a solid-state imaging device.

10. a solid-state imaging device; an optical system that forms a subject image on the solid-state imaging device, the solid-state imaging device, a sensor unit in which pixels that perform photoelectric conversion are arranged; a readout control unit that reads out pixel signals from the pixels of the sensor unit, The read control unit a pixel signal readout circuit including a frame memory, capable of storing output pixel signals of the sensor unit in the frame memory at a frequency 1, 2 or 4 times that of one TV frame, and capable of reading at 120 Hz or more; a focusing unit that focuses on a subject by driving a lens or the sensor unit, which is disposed in front of the sensor unit on the light incident side, along a related optical axis; A detection unit defines a group of multiple (n x n) pixels as a segment, and measures the difference between segments corresponding to the same position for each frame or the continuity of signals with multiple adjacent segments within a frame based on the results of driving and moving the lens or the sensor unit using the focusing unit, thereby detecting the edges and movement of the subject, and improving the autofocus's ability to track moving objects and extract subjects based on the detection results, and further enabling tracking. electronic equipment.

Images