Solid-state imaging device having ai function, method for driving same, and electronic apparatus

Abstract

Provided are: a solid-state imaging device capable of storing an output pixel signal of a sensor unit in a frame memory at 1X, 2X, or 4X of a single TV frame, enables reading at 120 Hz or greater, enables selection a desired band of a spatial frequency obtained by the difference in output, and makes it possible to easily obtain the movement direction of a photographic subject between single TV frames; a method for driving the solid-state imaging device; and an electronic apparatus. A detection unit 430 is included that, on the basis of a result of driving and moving a lens or sensor unit by using a focusing unit 420, detects the edges and movement of a photographic subject by measuring the differences between segments corresponding to the same position for each frame or the continuity of signals with a plurality of adjacent segments in a frame, and on the basis of this detection result, enhances the moving-object followability and photographic-subject extraction capability of the autofocus, further enabling tracking.

Description

Solid-state imaging device with AI function, driving method thereof, and electronic device

[0001] The present invention relates to a solid-state imaging device having an AI (artificial intelligence) function that improves the efficiency of image recognition and artificial intelligence calculations, a driving method for the same, and an electronic device having the AI ​​function.

[0002] CMOS (Complementary Metal Oxide Semiconductor) image sensors are currently in practical use as solid-state imaging devices (image sensors) that use photoelectric conversion elements that detect light and generate electric charges. CMOS image sensors are widely used as part of various electronic devices, such as digital cameras, video cameras, surveillance cameras, medical endoscopes, personal computers (PCs), and portable terminal devices (mobile devices) such as mobile phones.

[0003] CMOS image sensors have a photodiode (photoelectric conversion element) and a floating diffusion (FD) amplifier with a floating diffusion layer for each pixel, and the mainstream readout method is a column-parallel output type in which a row in the pixel array is selected and the pixels are simultaneously read out in the column output direction.

[0004] Furthermore, a wide variety of pixel signal readout (output) circuits have been proposed for column-parallel output CMOS image sensors, and one of the most advanced circuits is a circuit that includes an analog-to-digital converter (ADC) for each column and extracts pixel signals as digital signals (see, for example, Patent Documents 1 and 2).

[0005] In this column-parallel ADC-equipped CMOS image sensor (column AD type CMOS image sensor), a comparator compares a so-called RAMP wave with a pixel signal, and a counter at a downstream stage performs digital CDS to perform AD conversion.

[0006] However, although this type of CMOS image sensor is capable of high-speed signal transfer, it has the disadvantage of not being able to perform global shutter readout.

[0007] In response to this, a digital pixel sensor has been proposed that places an ADC (and even a memory unit) including a comparator in each pixel, thereby enabling a global shutter that starts and ends exposure at the same timing for all pixels in the pixel array (see, for example, Patent Documents 3 and 4).

[0008] JP 2005-278135 A JP 2005-295346 A US 7164114 B2 FIG, 4 US 2010 / 0181464 A1

[0009] In conventional CMOS image sensors, photoelectrically converted signals are read out sequentially from the top left of the screen, meaning that pixel signals are read out in time series. Therefore, if you want to obtain a certain frequency band of an image, such as in an autofocus system, you need to pass the time series signals through a filter and perform an operation.

[0010] Furthermore, when extracting R / G / B color components, such as for auto white balance, the time-series signal was obtained by passing it through a selector circuit. Therefore, because the image captured by the CMOS image sensor was read out in time series, even processing that could easily be achieved through spatial processing had to be converted into a time frequency first, resulting in inefficient processing. Furthermore, because this processing was performed every 60 Hz of a TV frame, it was sometimes impossible to obtain information about fast-moving subjects.

[0011] The present invention aims to provide a solid-state imaging device, a method for driving a solid-state imaging device, and electronic equipment that can store output pixel signals from a sensor unit in a frame memory at a frequency that is 1, 2, or 4 times that of one TV frame, enable readout at 120 Hz or higher, enable selection of a desired band of spatial frequency obtained by the difference in output, and further enable easy determination of the direction of movement of a subject within one TV frame.

[0012] a focus adjustment unit that drives a lens or the sensor unit arranged in a front stage on the light incident side of the sensor unit along an associated optical axis to adjust the focus on a subject; and a detection unit that defines a group of multiple (n x n) pixels as a segment, and measures the difference between segments corresponding to the same position for each frame or the continuity of signals between multiple adjacent segments within a frame based on the results of driving and moving the lens or the sensor unit, thereby detecting the edges and movement of the subject, and based on the detection results, improving the autofocus ability to track a moving subject and the ability to extract a subject.

[0013] A second aspect of the present invention is a method for driving a solid-state imaging device that performs readout control to read out pixel signals from pixels of a sensor unit in which pixels that perform photoelectric conversion are arranged, the readout control including: a pixel signal readout step that is capable of storing output pixel signals of the sensor unit in a frame memory at a frequency that is 1, 2, or 4 times that of one TV frame, and that performs readout at 120 Hz or more; a focusing step that drives a lens that is arranged in a front stage on the light incident side of the sensor unit or the sensor unit along an associated optical axis to focus on a subject; and a detection step that defines a plurality of (n x n) pixel sets as segments, and, based on a result of driving and moving the lens or the sensor unit in the focusing step, measures differences between segments corresponding to the same position for each frame or signal continuity with a plurality of adjacent segments within a frame to detect edges and movement of the subject, and, based on this detection result, improves the moving object tracking ability and subject extraction ability of autofocus and enables tracking.

[0014] An electronic device according to a third aspect of the present invention comprises a solid-state imaging device and an optical system that forms an image of a subject on the solid-state imaging device, wherein the solid-state imaging device comprises a sensor section in which pixels that perform photoelectric conversion are arranged, and a readout control section that reads out pixel signals from the pixels of the sensor section, wherein the readout control section comprises a pixel signal readout circuit that includes a frame memory and is capable of storing output pixel signals of the sensor section in the frame memory at a frequency that is 1, 2, or 4 times that of one TV frame, and is capable of readout at 120 Hz or more, a focusing section that drives a lens or the sensor section that is arranged in a front stage on the light incident side of the sensor section along an associated optical axis to focus on the subject, and a detection section that defines a group of multiple (n x n) pixels as a segment, and measures the difference between segments corresponding to the same position for each frame or the continuity of signals with multiple adjacent segments within a frame based on the results of driving and moving the lens or the sensor section by the focusing section, thereby detecting the edges and movement of the subject, and based on this detection result, improving the moving object tracking ability and subject extraction ability of the autofocus and further enabling tracking.

[0015] According to the present invention, the output pixel signals of the sensor unit can be stored in the frame memory at 1, 2, or 4 times the size of one TV frame, enabling readout at 120 Hz or higher. Furthermore, according to the present invention, it is possible to select a desired band of spatial frequency obtained by the difference in output, and further, it is possible to easily determine the direction of movement of a subject within one TV frame.

[0016] FIG. 1 is a block diagram showing an example of the configuration of a solid-state imaging device according to an embodiment of the present invention. FIG. 2 is a circuit diagram showing an example of a pixel signal readout system of a solid-state imaging device according to an embodiment of the present invention. FIG. 3 is a diagram showing the configuration of pixels PXL, sensor gate switches SGSW, an AD conversion unit, and a horizontal register HREG of a sensor unit according to an embodiment of the present invention, and a diagram showing an example of the configuration of four frame memories that receive the signals. FIG. 4 is a diagram showing the relationship between the on / off of a sensor gate switch and output timing in one TV frame. FIG. 5 is a diagram showing an image of a movable image sensor unit as a focusing unit that is a driving device according to an embodiment of the present invention. FIG. 6 is a first diagram for explaining processing for observing the movement direction and correlation of an object. FIG. 7 is a second diagram for explaining processing for observing the movement direction and correlation of an object. FIG. 8 is a diagram for explaining correlation between segments. FIG. 9 is a diagram showing an example of the configuration of a spatial frequency bandpass filter (BPF). FIG. 10 is a diagram showing an overlay for pixel data correlation between BPF output and segment end faces. FIG. 11 is a diagram showing an example of the configuration of an electronic device having an artificial function to which a solid-state imaging device according to an embodiment of the present invention is applied.

[0017] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0018] 1 is a block diagram showing an example of the configuration of a solid-state imaging device according to an embodiment of the present invention. 2 is a circuit diagram showing an example of a pixel readout system of the solid-state imaging device according to an embodiment of the present invention. In this embodiment, the solid-state imaging device 10 is configured, for example, by a CMOS image sensor.

[0019] 1, the solid-state imaging device 10 has, as its main components, a sensor unit 20 as an imaging unit, a vertical scanning circuit (row scanning circuit) 30, a signal processing circuit 40, and a timing control circuit 50. Of these components, for example, the vertical scanning circuit 30, the signal processing circuit 40, and the timing control circuit 50 constitute a pixel signal readout control unit 60.

[0020] In this embodiment, the solid-state imaging device 10 has a sensor section 20 in which pixels that perform photoelectric conversion are arranged, and a signal processing circuit 40 that, under a read control section 60, reads out pixel signals SPX from the pixels 200 of the sensor section 20, performs predetermined signal processing, and outputs the signal.

[0021] The signal processing circuit 40, under the control of the read control unit 60, is configured to perform, for example, pixel signal read processing, focusing processing, and based on the results of these, measure the difference between segments SGM corresponding to the same position for each frame FRM or the signal continuity with multiple adjacent segments SGM within the frame FRM to detect the edges and movement of the subject, and based on this detection result, improve the autofocus's ability to track moving objects and extract subjects, and even perform detection processing that enables tracking.

[0022] The signal processing circuit 40 of this embodiment has as its main components a pixel signal readout section 410, a focusing section 420, a detection section 430, and a horizontal register group 440, which are arranged as components of the readout control section 60.

[0023] The pixel signal readout unit 410 is mainly disposed at the connection between the sensor unit 20 and the signal processing circuit 40. The pixel signal readout unit 410 includes, for example, a photoelectric conversion readout unit 411, an AD (analog-digital) conversion unit 412, and a memory unit 413 having a frame memory FRM as the pixel 200, and is configured as, for example, a stacked CMOS image sensor. In the solid-state imaging device 10 according to this embodiment, each pixel PXL has an AD (analog-digital) conversion function, and the AD conversion unit 412 has a comparator CMP that compares a voltage signal read out by the photoelectric conversion readout unit 411 with a reference voltage, performs analog-to-digital (AD) conversion processing on the readout voltage signal VSL, and outputs a digitized comparison result signal.

[0024] The pixel signal readout unit 410 of this embodiment includes a frame memory FRM, and is capable of storing the output pixel signals of the sensor unit 20 in the frame memory FRM at a frequency that is 1, 2, or 4 times that of one TV frame, and is configured to enable readout at 120 Hz or higher.

[0025] The focusing unit 420 of the signal processing circuit 40 of this embodiment is configured to include a focusing unit 420 as a driving device that drives the lens LNS or the sensor unit 20, which is located in the front stage on the light incident side of the sensor unit 20, along the associated optical axis to focus on the subject.

[0026] Furthermore, the detection unit 430 of the signal processing circuit 40 of this embodiment defines multiple (n x n) pixel groups as segments, and based on the results of driving and moving the lens LNS or the sensor unit 20 by the focusing unit 420, measures the difference between segments corresponding to the same position for each frame or the continuity of signals with multiple adjacent segments within a frame, thereby detecting the edges and movement of the subject using, for example, an AI function, and is configured to improve the autofocus's ability to track moving objects and extract subjects based on this detection result, and even enable tracking.

[0027] In the detection unit 430 of this embodiment, 3x3, 4x4, 8x8, or 9x9 pixels are defined as one segment, and by measuring the difference between segments corresponding to the same position for each frame FRM or the signal continuity with eight adjacent segments within the frame, the edges and movement of the subject are detected, for example, using an AI function, thereby improving the autofocus's ability to track moving objects and extract subjects, and even enabling tracking.

[0028] Furthermore, the pixel signal readout unit 410 of this embodiment includes in the sensor unit 20 a pixel signal readout unit 410 that reads out all pixels in the horizontal direction at 60 Hz in the first TV frame, and in the next TV frame mixes the charges of two adjacent pixels or two adjacent pixels of the same color and reads them out at 120 Hz, or mixes the charges of four adjacent pixels or four adjacent pixels of the same color and reads them out at 240 Hz, and it is possible to select a desired band of spatial frequencies obtained by the difference between these outputs.

[0029] Below, we will provide a detailed overview of the configuration and function of each part of the solid-state imaging device 10 of this embodiment, in particular the configuration and function of the sensor unit 20, pixel signal readout unit 410, focusing unit 420, and detection unit 430, as well as the readout processing associated with them.

[0030] FIG. 2 is a circuit diagram showing an example of the configuration of a pixel signal readout circuit according to an embodiment of the present invention.

[0031] The sensor unit 20 has a plurality of pixels 200 arranged in a matrix of N rows and M columns.

[0032] The pixel signal readout unit 410 according to this embodiment includes a photoelectric conversion readout unit 411, an AD conversion unit (referred to as ADC) 412, and a memory unit 413 having a frame memory. The sensor unit 20 according to this embodiment is configured as a stacked CMOS image sensor of the first substrate 110 and the second substrate 120, but in this example, as shown in FIG. 2 , the photoelectric conversion readout unit 411 is formed on the first substrate 110, and the AD conversion unit 412 and the memory unit 413 are formed on the second substrate 120.

[0033] The photoelectric conversion readout unit 411 of the pixel 200 includes a photodiode (photoelectric conversion element) and one in-pixel amplifier. Specifically, the photoelectric conversion readout unit 411 has, for example, a photodiode PD0 which is a photoelectric conversion element. In the pixel 200 of this embodiment, the photodiode PD0 is connected to a floating diffusion FD which serves as an output node ND0.

[0034] The photodiode PD0 accumulates charges generated by photoelectric conversion during an accumulation period. A transfer transistor TG0-Tr serving as a transfer element is connected between an accumulation portion PND0 of the photodiode PD0 and a floating diffusion FD, and the accumulation portion PND0 is connected to a predetermined fixed potential VAAPIX.

[0035] The photoelectric conversion readout section 411 has a reset transistor RST-Tr as a reset element, a source-follower transistor SF-Tr as a source follower element, and a readout node ND1, each of which corresponds to a floating diffusion FD as one output node ND0. The output buffer section 4111 is configured to include the SF-Tr and the readout node ND1.

[0036] In the photoelectric conversion readout unit 411 according to this embodiment, a readout node ND1 of an output buffer unit 4111 is connected to an input unit of an AD conversion unit 412. The photoelectric conversion readout unit 411 converts the charge of the floating diffusion FD serving as an output node into a voltage signal according to the amount of charge, and outputs the converted voltage signal VSL to the AD conversion unit 412.

[0037] For example, the photoelectric conversion readout unit 411 outputs a voltage signal VSL corresponding to the accumulated charge of the photodiode PD0, which is a photoelectric conversion element, transferred from the photodiode PD0 to the floating diffusion FD as an output node during the accumulation period PI during a comparison processing period of the AD conversion unit 412. During the comparison processing period, the photoelectric conversion readout unit 411 outputs a readout reset signal (signal voltage) (VRST) and a readout signal (signal voltage) (VSIG) as pixel signals to the AD conversion unit 412.

[0038] The photodiode PD0 generates and accumulates signal charge (electrons in this case) in an amount corresponding to the amount of incident light. In the following, the signal charge is electrons and each transistor is an n-type transistor, but the signal charge may be holes or each transistor may be a p-type transistor.

[0039] In each pixel 200, a buried photodiode (PPD) is used as the photodiode (PD). The surface of the substrate on which the photodiode (PD) is formed has interface states due to defects such as dangling bonds, which causes a large amount of charge (dark current) to be generated by thermal energy, making it impossible to read out a correct signal. In a buried photodiode (PPD), the charge storage section of the photodiode (PD) is embedded in the substrate, making it possible to reduce the inclusion of dark current in the signal.

[0040] The transfer transistor TG0-Tr of the photoelectric conversion readout unit 411 is connected between the storage unit PND0 of the photodiode PD0 and the floating diffusion FD, and is controlled by a control signal TG0 applied to its gate through a control line. The transfer transistor TG0-Tr is selected and becomes conductive during a transfer period PT when the control signal TG0 is at a high (H) level, and transfers the charges (electrons) photoelectrically converted and stored in the photodiode PD0 to the floating diffusion FD.

[0041] The source of the source follower transistor SF-Tr as a source follower element is connected to the read node ND1, the drain side is connected to the power supply line Vaapix, and the gate of the front transistor IC-Tr is connected to the supply line of the control signal VBNPIX. The signal line LSGN1 between the read node ND1 and the input part of the AD conversion part 412 is driven by the current transistor IC-Tr as a current source element.

[0042] This current transistor IC-Tr, or the current transistor connected to the read node ND1 of the photoelectric conversion read section 411 and the signal line LSGN1, functions as a sensor gate switch SGSW.

[0043] The AD conversion unit 412 of the pixel 200 functions to convert the analog voltage signal VSL output by the photoelectric conversion readout unit 411 into a digital signal by comparing it with a reference voltage VREF, which is a ramp waveform that is changed with a predetermined slope or a fixed voltage.

[0044] As shown in FIG. 2, the AD conversion unit 412 includes a comparator (COMP) 4121, a load capacitor CL1 on the output side, and a reset switch SW-RST.

[0045] The comparator 4121 receives a voltage signal VSL output from the output buffer unit 4111 of the photoelectric conversion readout unit 411 to the signal line LSGN1 at its inverting input terminal (-) as its first input terminal, and receives a reference voltage VREF at its non-inverting input terminal (+) as its second input terminal.The comparator 4121 compares the voltage signal VST with the reference voltage VREF and performs an AD conversion process (comparison process) to output a digitized comparison result signal SCMP.

[0046] The comparator 4121 has a coupling capacitor CC1 connected to its inverting input terminal (-) as its first input terminal, and is configured to achieve low noise and a high SNR at low illuminance by AC coupling the output buffer section 4111 of the photoelectric conversion readout section 411 on the first substrate 110 side and the input section of the comparator 4121 of the AD conversion section 412 on the second substrate 120 side.

[0047] In addition, the comparator 4121 has a reset switch SW-RST connected between the output terminal and an inverting input terminal (-) serving as a first input terminal, and a load capacitor CL1 connected between the output terminal and a reference potential VSS.

[0048] Basically, in the AD conversion unit 412, the analog signal (potential VSL) read out from the output buffer unit 4111 of the photoelectric conversion readout unit 411 to the signal line LSGN1 is compared by a comparator 4121 with a reference voltage VREF, for example, a ramp signal RAMP having a linearly changing slope waveform with a certain gradient. At this time, a counter (not shown) arranged for each column, similar to the comparator 4121, operates, and the voltage signal VSL is converted into a digital signal by the ramp signal RAMP having a ramp waveform changing in one-to-one correspondence with the counter value. Basically, the AD conversion unit 412 converts the change in the reference voltage VREF (for example, the ramp signal RAMP) into a change in time, and converts that time into a digital value by counting it at a certain cycle (clock). Then, when the analog signal VSL and the ramp signal RAMP (reference voltage VREF) intersect, the output of the comparator 4121 is inverted, the input clock of the counter (not shown) is stopped, or the clock that had been stopped is input to the counter (not shown), and the counter value (data) at that time is stored in the memory unit 413 (230), completing the AD conversion. After the above AD conversion period ends, the data (signal) stored in the memory unit 413 (230) of each pixel 200 is output from the signal processing circuit 40 to a signal processing circuit (not shown), and a two-dimensional image is generated by predetermined signal processing.

[0049] The memory unit 413 is configured with SRAM or DRAM, and is supplied with a digitally converted signal that corresponds to the photoconversion code and can be read out by an external IO buffer of the signal processing circuit 40 around the pixel array. In this example, the memory unit 413 has four memories 4131 (231), 4132 (232), 4133 (233), and 4134 (234) connected to the output of the comparator 4121.

[0050] The digital data resulting from the comparison process by the comparator 4121 is stored alternately in four memories 4131 (231), 4132 (232), 4133 (233), and 4134 (234) of the memory section 4130. This makes it possible to achieve a rapid read operation.

[0051] The vertical scanning circuit 30 drives the photoelectric conversion readout units 411 of the pixels 200 through row scanning control lines in the shutter row and readout row in accordance with the control of the timing control circuit 50. The vertical scanning circuit 30 supplies a reference voltage VREF, which is set in accordance with the comparison process, to the comparator 221 of each pixel 200 in accordance with the control of the timing control circuit 50. Furthermore, the vertical scanning circuit 30 outputs a row selection signal of the row address of the read row from which signals are read out and the shutter row from which the charge accumulated in the photodiode PD is reset in accordance with an address signal.

[0052] The signal processing circuit 40 includes, for example, an IO buffer arranged corresponding to the memory output of each pixel 200 of the sensor unit 20, and outputs digital data read from each pixel 200 to the outside.

[0053] The timing control circuit 50 generates timing signals necessary for signal processing in the pixel section 20, the vertical scanning circuit 30, the output circuit 40, and the like.

[0054] In this embodiment, the readout control unit 60 controls the readout of pixel signals from the pixels 200 .

[0055] (Readout Control of Pixel Signals from Pixels 200 by Readout Control Unit 60) Next, the readout control of pixel signals from the pixels 200 by the readout control unit 60 according to this embodiment will be specifically described.

[0056] First, the basic configuration, various operations, and functions of the sensor unit 20 of the solid-state imaging device (CMOS image sensor) 10 having the above-described configuration and the pixel signal readout circuit 410 will be described in order with reference to the drawings.

[0057] 3 is a diagram showing the configuration of the pixels (shown as photoelectric conversion elements, photodiodes in FIG. 1) PXL, sensor gate switch SGSW, AD conversion unit, and horizontal register HREG of the sensor unit 20 according to an embodiment of the present invention, and a diagram showing an example of the configuration of four frame memories MEM0 to MEM3 (231 to 234) that receive the signals. FIG. 4 is a diagram showing the relationship between the on / off of the sensor gate switch and the output timing in one TV frame.

[0058] The pixel PXL in Figure 3 is exposed for 1 / 240 seconds and accumulates charge. Then, every 1 / 240 seconds, the sensor gate switch SGSW is turned on, the accumulated charge is A / D converted, and transferred to the horizontal register HREG. There are horizontal registers HREG for each vertical pixel, and they sequentially shift the stored data horizontally, output it, and store it in the downstream frame memory MEM. This operation is completed 1 / 240 seconds after the sensor gate switch SGSW is turned on. During this time, the next 1 / 240-second image is accumulated in the pixel PXL in parallel, and once that is complete, the above operation is repeated (Figure 4). In this way, four frames of image data are output in one TV frame FRM.

[0059] FIG. 5 is a diagram showing an image of a movable image sensor unit as a focusing unit, which is a driving device according to an embodiment of the present invention.

[0060] The image sensor unit 20 is driven by a piezo, voice coil, stepping motor, etc., which drives the substrate of the image sensor unit 20 horizontally, vertically, or in a rotational direction around the optical axis of the lens LNS. The driving cycle is every TV frame.

[0061] First, the front-stage focus lens is driven for each TV frame to focus on the central subject. To increase efficiency, the front-stage focus lens may be fixed, and the image sensor unit 20 may be driven instead in the focal direction of the optical axis. If the user wishes to focus on a subject at a specific position on the screen, the contrast of that portion is extracted and focusing is performed. Alternatively, if a PDAF or phase difference sensor is provided, the focusing operation can be left to that device and omitted from this mechanism.

[0062] 6 and 7 are diagrams for explaining the process for observing the movement direction and correlation of the subject, and Fig. 8 is a diagram for explaining the correlation between segments.

[0063] In FIG. 6, in order to observe the movement direction and correlation of the subject, for example, the absolute value of the difference Σ|pij-p(i+3)(j+3)| between the first TV frame FRM1 and the second TV frame FRM2 of a 4×4 pixel segment SGM (FIG. 7) is added up for 16 pixels, and the movement direction of the subject can be determined by finding a segment with a large correlation, that is, a small difference.

[0064] To measure the direction of movement, the image sensor unit 20 is moved rightward between the first and second TV frames for image 1, moved leftward for image 2, moved upward for image 3, and moved downward for image 4 (selecting the point where the correlation between the segments is strongest), thereby easily determining the direction of movement between one TV frame.

[0065] The above operation is repeated over the entire screen, and it is found that the direction in which the object is moving is the most similar to the direction in which the image sensor unit 20 is moved, based on the arrangement of the point where the correlation between one corresponding segment is greatest and the point where the correlation between the adjacent segment is greatest.

[0066] For example, if the first TV frame image 1 and the second TV frame image 1 have the greatest correlation, it is determined that the subject is moving to the right, and the point where the difference between the values ​​of p14, p24, or p34 in the first TV frame p24 and the second TV frame p24 is the smallest can be considered to be the point where the brightness of the subject is continuous.

[0067] Similarly, by comparing p24 of the 2nd TV frame image 1 with p14, p24, and p34 of image 1 of the 3rd TV frame, the continuity of the subject's brightness can be observed and the direction of the subject's movement can be grasped.

[0068] If it is determined that the subject moved to the left during one TV frame (i.e., image 3 has the greatest correlation), then, for example, p21 of first TV frame image 3 is compared with p11, p21, and p31 of second TV frame image 3. If the subject moved upward during one TV frame, then p12 of first TV frame image 4 is compared with p11, p12, and p13 of second TV frame image 4. If the subject moved downward during the first TV frame, then p42 of first TV frame image 2 is compared with p41, p42, and p43 of second TV frame image 2 to determine the direction in which the brightness of a specific point on the subject continues.

[0069] By operating this, in addition to the direction of movement on the entire screen, detailed information such as the direction and speed of movement on a pixel-by-pixel basis can be obtained.

[0070] If the subject moves faster, the pixels to be compared are shifted by two or several pixels in the direction of the movement.

[0071] The image sensor unit 20 can be driven to rotate clockwise or counterclockwise around the optical axis to observe the direction of movement of the subject.

[0072] Fig. 9 is a diagram showing an example of the configuration of a spatial frequency band pass filter (BPF), and Fig. 10 is a diagram showing the BPF output and an overlay for pixel data correlation at the segment end face.

[0073] For example, if all pixels of the image sensor unit 20 are read out at 60 Hz for the first TV frame, and two horizontal pixels are summed for the first readout of the second TV frame and read out at 120 Hz, and four pixels are summed for the latter readout and read out at 120 Hz, a signal with the bandwidth shown in Figure 9 can be obtained. Subtracting Sig half from Sig full yields BPF2, and subtracting Sig four from Sig half yields BPF1. These can be used as contrast autofocus signals, and because BPF2 indicates the edge of the subject, it can also be used as an edge extraction signal. Furthermore, by overlaying it with the closed surface of the subject mentioned above, highly accurate area extraction becomes possible.

[0074] (Related to closed curve generation) For example, subtracting the image of the first frame read out at 60 Hz from the image of the second frame read out at 120 Hz in Figure 9 will yield contour information similar to BPF 2. By tracing the point near the position of that contour where the correlation in brightness at the end face of the segment in the direction of movement between the frames obtained above is highest, it is possible to extract the closed curve formed by the contour of the subject.

[0075] The closed curve obtained in this way is defined as a new segment corresponding to the object, and autofocus and tracking according to the object are made possible by calculating the correlation between these object segments for each TV frame.

[0076] Furthermore, if the correlation of the object segments between TV frames decreases by a certain amount or more, it means that another object has invaded a part other than the closed curve part where the correlation is maintained, and a new object segment can be formed for the invading object.

[0077] In this way, even if the subject changes over time or suddenly enters the screen, it is possible to adjust the focus for each subject segment, thereby achieving stable focus.

[0078] <Effects of the Present Embodiment> According to the present embodiment, all horizontal pixels of the sensor unit 200 are read within one TV frame, and pixel signals obtained by mixing two of these pixels are read at 120 Hz in the first half of the TV frame, and four pixels are mixed and read at 240 Hz in the second half of the TV frame. Each data is stored in a frame memory, and the 120 Hz data is subtracted from the one TV frame data, thereby forming a spatial frequency bandpass filter (BPF). This allows for efficient extraction of subject contour information. Similarly, BPFs of various bands can be formed by subtracting 240 Hz data from one TV frame data or subtracting 240 Hz data from 120 Hz data, thereby enabling extremely efficient acquisition of information useful for autofocusing and image data compression.

[0079] Furthermore, according to this embodiment, in the sensor unit 200 that is provided with an actuator as a drive device that can drive the substrate of the sensor unit 200 up and down, left and right, or in a rotational direction around the optical axis, and that can output data mixed with four pixels in the horizontal direction at 240 Hz four times within one TV frame, the image sensor is moved to the left for each frame in the first readout, to the right in the second readout, up in the third readout, and down in the fourth readout, and the pixel data for each segment that corresponds in position each time is compared, thereby making it possible to efficiently estimate the direction of movement of the subject within two TV frames.

[0080] Furthermore, according to this embodiment, the detailed movement direction for each subject is calculated by correlating the movement direction determined above with corresponding pixels of the segment.

[0081] Furthermore, according to this embodiment, it is possible to estimate a closed curve formed by the contour of the subject from the correlation of pixel data of the end faces of the corresponding segments in the movement direction.

[0082] In addition, according to this embodiment, a more detailed closed curve can be obtained by repeating the above operations (2) to (4) for a segment corresponding to contour information obtained by the method for extremely efficiently obtaining information useful for autofocusing and image data compression described above.

[0083] Furthermore, according to this embodiment, it is possible to create object segments corresponding to the closed curve of the object obtained by the above operation, and by calculating the correlation between each object segment for each TV frame, it is possible to achieve focusing and tracking tailored to the desired object.

[0084] The solid-state imaging device 10 described above can be applied as an imaging device to electronic devices such as digital cameras, video cameras, mobile terminals, surveillance cameras, and medical endoscope cameras.

[0085] FIG. 11 is a diagram showing an example of the configuration of an electronic device having an artificial function to which a solid-state imaging device according to an embodiment of the present invention is applied.

[0086] 11 , the electronic device 300 has a CMOS image sensor 310 to which the solid-state imaging device 10 according to the present embodiment can be applied. The electronic device 300 further has an optical system (lens, etc.) 320 that guides incident light (forming an image of a subject) to the pixel region of the CMOS image sensor 310. The electronic device 300 also has a signal processing circuit (PRC) 330 that processes an output signal from the CMOS image sensor 310.

[0087] The signal processing circuit 330 performs predetermined signal processing on the output signal of the CMOS image sensor 310. The image signal processed by the signal processing circuit 330 can be displayed as a moving image on a monitor such as a liquid crystal display, output to a printer, or directly recorded on a recording medium such as a memory card, and various other modes are possible.

[0088] As described above, it is possible to provide a high-performance, compact, and low-cost camera system by incorporating the above-described solid-state imaging device 10 as the CMOS image sensor 310. This makes it possible to realize electronic devices with AI functions, such as surveillance cameras and medical endoscope cameras, that are used in applications where camera installation requirements are limited by mounting size, the number of connectable cables, cable length, installation height, and the like.

[0089] 10... solid-state imaging device, 20... sensor unit, PD0... photodiode, TG0-Tr... transfer transistor, FD... floating diffusion, RST-Tr... reset transistor, RST0-Tr... reset transistor, SF-Tr... source follower transistor, SF0-Tr... source follower transistor, 200... pixel, 30... vertical scanning circuit, 40... signal processing circuit, 410... pixel signal readout unit, 411... photoelectric conversion readout unit, 412... AD conversion unit, 4121... comparator, 413... memory unit, 4131 to 4134... memories, 420... focusing unit, 430... detection unit, 50... timing control circuit, 60... readout control unit, 300... electronic device, 310... CMOS image sensor, 320... optical system, 330... signal processing circuit (PRC).

Claims

1. A device comprising: a sensor unit in which pixels that perform photoelectric conversion are arranged; and a readout control unit that reads out pixel signals from the pixels of the sensor unit, wherein the readout control unit comprises: a pixel signal readout circuit that includes a frame memory and is capable of storing output pixel signals of the sensor unit in the frame memory at a frequency that is 1, 2 or 4 times that of one TV frame, and is capable of reading out at 120 Hz or more; a focusing unit that drives a lens or the sensor unit, which is arranged in a front stage on the light incident side of the sensor unit, along an associated optical axis to focus on a subject; and a detection unit that defines a group of multiple (n x n) pixels as a segment, and measures the difference between segments corresponding to the same position for each frame or the continuity of signals with multiple adjacent segments within a frame based on the results of driving and moving the lens or the sensor unit using the focusing unit, thereby detecting the edges and movement of the subject, and based on this detection effect, improving the autofocus's ability to track moving objects and extract objects, and further enabling tracking; the detection unit is capable of performing a first detection process In order to observe the direction of movement and correlation of the subject, a second detection process can be performed in which the absolute values ​​of the differences between all pixels of an nxn pixel segment between the first TV frame and the second TV frame are added up to find a segment with high correlation and low difference, thereby determining the direction of movement of the subject; and the detection unit can perform a third detection process in which, to measure the direction of movement, the sensor unit is moved to the right or left between the first TV frame and the second TV frame for the first image and compared, moved left or right for the second image, up or down for the third image, and down or up for the fourth image for comparison, and selected to select the point where the correlation of the segment is strongest, thereby determining the direction of movement between one TV frame.

2. A device comprising: a sensor unit in which pixels that perform photoelectric conversion are arranged; and a readout control unit that reads out pixel signals from the pixels of the sensor unit, wherein the readout control unit comprises: a pixel signal readout circuit that includes a frame memory and is capable of storing output pixel signals of the sensor unit in the frame memory at a frequency that is 1, 2 or 4 times that of one TV frame, and is capable of reading out at 120 Hz or more; a focusing unit that drives a lens or the sensor unit, which is arranged in the front stage on the light incident side of the sensor unit, along the related optical axis to focus on a subject; and a detection unit that defines a group of multiple (n x n) pixels as a segment, and measures the difference between segments corresponding to the same position for each frame or the continuity of signals with multiple adjacent segments within a frame based on the result of driving and moving the lens or the sensor unit using the focusing unit, thereby detecting the edges and movement of the subject, and based on this detection effect, improving the moving object tracking ability and subject extraction capability of the autofocus, and further enabling tracking. The detection unit comprises: In order to observe the direction of movement and correlation of the subject, a second detection process can be performed in which the absolute values ​​of the differences between all pixels in an n x n pixel segment between the first TV frame and the second TV frame are added up to find a segment with high correlation and small difference, thereby determining the direction of movement of the subject; the pixel signal readout circuit includes a pixel signal readout circuit in the sensor unit that reads out all pixels in the horizontal direction at 60 Hz in the first TV frame, and in the next TV frame mixes the charges of two adjacent pixels that are adjacent or of the same color and reads them out at 120 Hz, or mixes the charges of four adjacent pixels or four adjacent pixels of the same color and reads them out at 240 Hz, and is capable of selecting a desired band of spatial frequency obtained by the difference in their outputs.

3. The solid-state imaging device of claim 1, wherein the detection unit repeats at least one of the first, second and third detection processes over the entire screen, and determines that the direction in which the alignment of the point where the correlation between one corresponding segment is greatest and the point where the correlation between an adjacent segment is greatest is most similar among the directions in which the sensor unit is swung is the direction in which the subject is moving.

4. A solid-state imaging device according to claim 2, wherein the detection unit is capable of estimating a closed curve formed by the contour of the subject from the correlation of pixel data of the end faces in the movement direction of the corresponding segments.

5. The solid-state imaging device according to claim 1, wherein the pixel signal readout circuit reads out all horizontal pixels in the first TV frame, reads out data obtained by mixing two horizontal pixels in the first half of the next TV frame and data obtained by mixing four horizontal pixels in the second half of the next TV frame at 120 Hz and 240 Hz, respectively, and forms a spatial frequency bandpass filter by subtracting the 120 Hz and 240 Hz data read out in the next TV frame from the data read out in the first TV frame.

6. A solid-state imaging device as claimed in claim 2, wherein the pixel signal readout circuit reads out all horizontal pixels in the first TV frame, reads out data obtained by mixing two horizontal pixels in the first half of the next TV frame and data obtained by mixing four horizontal pixels in the second half of the next TV frame at 120 Hz and 240 Hz, respectively, and forms a spatial frequency bandpass filter by subtracting the 120 Hz and 240 Hz data read out in the next TV frame from the data read out in the first TV frame.

7. A method for driving a solid-state imaging device that performs read control to read out pixel signals from pixels of a sensor section in which pixels that perform photoelectric conversion are arranged, wherein the read control includes: a pixel signal read step that can store output pixel signals of the sensor section in a frame memory at a frequency 1, 2 or 4 times that of one TV frame, and performs readout at 120 Hz or more; a focusing step that drives a lens or the sensor section arranged in the front stage on the light incident side of the sensor section along the related optical axis to focus on the subject; and a detection step that defines a group of multiple (n x n) pixels as a segment, and based on the results of driving and moving the lens or the sensor section in the focusing step, measures the difference between segments corresponding to the same position for each frame or the continuity of signals with multiple adjacent segments within a frame to detect the edges and movement of the subject, and based on this detection result, improves the moving object tracking ability and subject extraction ability of the autofocus and further enables tracking, wherein the detection section: A method for driving a solid-state imaging device, in which a second detection process can be performed to determine the direction of movement of the subject by adding up the absolute values ​​of the differences between all pixels of an nxn pixel segment between the first TV frame and the second TV frame and finding a segment with high correlation and low difference, in order to observe the direction of movement and correlation of the subject; and the detection unit can perform a third detection process to determine the direction of movement between one TV frame by moving the sensor unit right or left between the first TV frame and the second TV frame for the first image and comparing them, moving it left or right for the second image, up or down for the third image, and down or up for the fourth image and comparing them, in order to measure the direction of movement, selecting the segment with the strongest correlation.

8. A solid-state imaging device comprising: an optical system for forming an image of a subject on the solid-state imaging device; wherein the solid-state imaging device comprises: a sensor section in which pixels that perform photoelectric conversion are arranged; and a readout control section for reading pixel signals from the pixels of the sensor section; wherein the readout control section comprises: a pixel signal readout circuit including a frame memory and capable of storing output pixel signals of the sensor section in the frame memory at a frequency that is 1, 2 or 4 times that of one TV frame, and capable of reading at 120 Hz or more; a focusing section for driving a lens or the sensor section arranged in a front stage on the light incident side of the sensor section along an associated optical axis to focus on the subject; and a detection section for defining a group of multiple (n x n) pixels as a segment, and measuring the difference between segments corresponding to the same position for each frame or the continuity of signals with multiple adjacent segments within a frame based on the results of driving and moving the lens or the sensor section by the focusing section, thereby detecting the edges and movement of the subject, and based on this detection result, improving the moving object tracking ability and subject extraction capability of the autofocus and further enabling tracking. The detection section In order to observe the direction of movement and correlation of the subject, a second detection process can be performed in which the absolute values ​​of the differences between all pixels of an nxn pixel segment between the 1st TV frame and the 2nd TV frame are added up to find a segment with high correlation and low difference, thereby determining the direction of movement of the subject; and the detection unit can perform a third detection process in which, to measure the direction of movement, the sensor unit is moved to the right or left between the 1st TV frame and the 2nd TV frame for the first image and compared, moved left or right for the second image, up or down for the third image, and down or up for the fourth image and compared, selecting the segment with the strongest correlation, to determine the direction of movement between 1 TV frame.

9. A method for driving a solid-state imaging device that performs readout control to read out pixel signals from pixels of a sensor section in which pixels that perform photoelectric conversion are arranged, wherein the readout control includes: a pixel signal readout step that is capable of storing output pixel signals of the sensor section in a frame memory at a frequency 1, 2 or 4 times that of one TV frame, and performs readout at 120 Hz or more; a focusing step that drives a lens or the sensor section arranged in a front stage on the light incident side of the sensor section along an associated optical axis to focus on the subject; and a detection step that defines a group of multiple (n x n) pixels as a segment, and based on the results of driving and moving the lens or the sensor section in the focusing step, measures the difference between segments corresponding to the same position for each frame or the continuity of signals with multiple adjacent segments within a frame to detect the edges and movement of the subject, and based on this detection result, is capable of performing a first detection process that improves the moving object tracking ability and subject extraction ability of the autofocus and further enables tracking, wherein the detection step is In order to observe the movement direction and correlation of the subject, a second detection process can be performed in which the absolute values ​​of the differences between all pixels in an n x n pixel segment between the first TV frame and the second TV frame are added up to find a segment with high correlation and small difference, thereby determining the movement direction of the subject; the pixel signal readout step includes a pixel signal readout circuit in the sensor unit that reads out all pixels in the horizontal direction at 60 Hz in the first TV frame, and in the next TV frame mixes the charges of two adjacent pixels that are adjacent or of the same color and reads them out at 120 Hz, or mixes the charges of four adjacent pixels or four adjacent pixels of the same color and reads them out at 240 Hz, and a desired band of spatial frequency obtained by the difference in their outputs can be selected.

10. A solid-state imaging device comprising: an optical system for forming an image of a subject on the solid-state imaging device; wherein the solid-state imaging device comprises: a sensor section in which pixels that perform photoelectric conversion are arranged; and a readout control section for reading out pixel signals from the pixels of the sensor section; wherein the readout control section comprises: a pixel signal readout circuit including a frame memory and capable of storing output pixel signals of the sensor section in the frame memory at a frequency that is 1, 2 or 4 times that of one TV frame, and capable of reading out at 120 Hz or more; a focusing section for driving a lens or the sensor section arranged in a front stage on the light incident side of the sensor section along an associated optical axis to focus on the subject; and a detection section for defining a group of multiple (n x n) pixels as a segment, and measuring the difference between segments corresponding to the same position for each frame or the continuity of signals with multiple adjacent segments within a frame based on the result of driving and moving the lens or the sensor section by the focusing section, thereby detecting the edges and movement of the subject, and based on this detection effect, improving the moving object tracking ability and subject extraction capability of the autofocus, and further enabling tracking; wherein the detection section is In order to observe the movement direction and correlation of the subject, a second detection process can be performed in which the absolute values ​​of the differences between all pixels in an n x n pixel segment between the first TV frame and the second TV frame are added up to find a segment with high correlation and small difference, thereby determining the movement direction of the subject; the pixel signal readout circuit includes a pixel signal readout circuit in the sensor unit that reads out all pixels in the horizontal direction at 60 Hz in the first TV frame, and in the next TV frame mixes the charges of two adjacent pixels that are adjacent or of the same color and reads them out at 120 Hz, or mixes the charges of four adjacent pixels or four adjacent pixels of the same color and reads them out at 240 Hz, and is capable of selecting a desired band of spatial frequency obtained by the difference in their outputs.

Images