A compact high frame rate image sensor system and imaging method thereof

Abstract

The present invention provides a compact high frame rate image sensor system, comprising stacked CMOS semiconductor circuit chips. The bottom chip includes a photosensitive area array and an image capture structure; the top chip includes image processing and logic circuits to quickly extract the image or main features of the image determined by the boundary detection filter. The image sensor system also includes a compilation tool for providing programming parameters to the programmable filter to optimize the image sensor for a specific motion detection application. The present invention also provides a compact high frame rate image sensor system imaging method.

Description

Technical Field

[0001] The present invention relates to an image sensor, and in particular to a CMOS image sensor using a stacked semiconductor circuit chip and an imaging method thereof. Background Art

[0002] The image capture device includes an image sensor and an imaging lens. The imaging lens focuses light onto the image sensor to form an image, and the image sensor converts the light signal into an electrical signal. The image capture device outputs the electrical signal to other components of the host system. The image capture device and other components of the host system form an image sensor system or an imaging system. The application of image sensors has become very common and can be used in various electronic systems, such as mobile devices, digital cameras, medical devices, or computers.

[0003] A typical image sensor includes multiple photosensitive elements ("pixels") arranged in a two-dimensional array. Such image sensors can be configured to produce color images by forming a color filter array (CFA) on the pixels. The technology used to manufacture image sensors, especially the technology used to manufacture complementary metal oxide semiconductor ("CMOS") image sensors, continues to develop rapidly. For example, the requirements of high resolution and low power consumption have promoted the further miniaturization and integration of such image sensors. However, miniaturization will sacrifice pixel photosensitivity and dynamic range, so new methods are needed to solve this problem.

[0004] As pixel size decreases, the total light absorption depth within the substrate becomes insufficient for some light, especially long wavelength light. This is particularly a problem for image sensors using backside illumination (BSI) technology, when the image light is incident on the back side of the sensor substrate. In BSI technology, the sensor silicon substrate may be only 2 microns thick, which is sufficient to absorb blue light but very insufficient to absorb red light, which may require a thickness of 10 microns to be fully absorbed.

[0005] It is well known to form so-called stacked image sensors. In a typical arrangement of this type, the photodiodes or other photosensitive elements of the pixel array are formed in a first semiconductor die or substrate, and the associated readout circuitry for processing the photosensitive element signals is formed in a second semiconductor die or substrate, wherein the second semiconductor die or substrate directly overlies the first semiconductor die or substrate. The first semiconductor die or substrate and the second semiconductor die or substrate are generally referred to herein as a sensor chip and a circuit chip, respectively. More specifically, after calibration of the relevant inter-wafer electrical connections, the first semiconductor die and the second semiconductor die are formed together with other similar dies on the stacked first semiconductor wafer and the second semiconductor wafer, and are diced or cut into stacked assemblies generally referred to as semiconductor chips. In the image sensor claimed for protection in the present disclosure, the photosensitive element and the readout circuitry use the same substrate. When two chips are stacked, it should be understood that in a typical embodiment, the two wafers are stacked and cut into chips and remain stacked to form an electrical system, for example, a stacked image sensor. However, it is also possible to stack individual chips taken out of the first semiconductor wafer onto other chips still in wafer form, or even stack two chips. In addition, the inter-wafer electrical interconnects coupled to the sensor and circuit wafers can be referred to as inter-chip interconnects, while intra-wafer interconnects and intra-chip interconnects refer to interconnects formed between devices located on the same wafer and chip, respectively. An advantage of this arrangement is that the resulting image sensor system occupies a smaller area than an unstacked arrangement. Another advantage is that each chip can be manufactured using different manufacturing methods and materials to allow independent optimization.

[0006] The two most common methods for reading the image signals generated on the sensor chip are rolling shutter mode and global shutter mode. Rolling shutter mode involves exposing different lines of the sensor array at different times and reading out those lines in a selected order. Global shutter mode involves exposing every pixel at the same time and for the same amount of time, similar to the way a mechanical shutter on a traditional "snapshot" camera operates. Prior art digital imaging systems use either rolling shutter readout mode or global shutter readout mode. However, it would be advantageous to have an imaging system that is capable of having two readout modes, and where the readout mode can be selected by the operator.

[0007] Rolling shutter (RS) mode exposes and reads out adjacent rows of the array at different times, that is, each row will start and end its exposure slightly offset from the adjacent rows. Each row is read out after the exposure is complete, and the charge from each row is transferred to the read node of the pixel. Although each row experiences the same exposure time, the top row will end its exposure at a time that is prior to the end of the exposure of the bottom row of the sensor. This time depends on the number of rows and the time offset between adjacent rows. A potential disadvantage of the rolling shutter readout mode is the spatial distortion caused by the above reasons. The distortion becomes more obvious when large objects move faster than the readout rate. Another disadvantage is that different areas of the exposed image will not be very precise in time, which manifests as image distortion. In order to improve the signal-to-noise ratio of the image signal when it is finally read out, especially to reduce short-lived dark noise, a reference readout method called correlated double sampling (CDS) is performed before each pixel charge is converted to an output signal by an amplifier transistor. The amplifier transistor can usually be a transistor in a source follower (SF) configuration.

[0008] Global Shutter (GS) mode exposes all pixels of the array simultaneously. This helps capture fast moving events to freeze them in time. Before exposure begins, all pixels are reset (RST) to the same apparent dark level by draining all charge. At the start of exposure, each pixel begins collecting charge simultaneously and is allowed to continue collecting for the duration of the exposure. At the end of exposure, each pixel transfers charge to its readout node simultaneously. Global Shutter mode can be configured to operate in a continuous manner, whereby the next exposure can be made while the previous exposure is being read out by the readout storage node of each pixel. In this mode, the sensor has a 100% duty cycle, which optimizes temporal resolution and photon collection efficiency. There are no artifacts in the image from the transient readout cycles that occur in rolling shutter mode. Global shutter is considered necessary when precise temporal correlation is required between different areas of the sensor area. Global shutter is also easily synchronized with light sources or other devices.

[0009] Pixels in global shutter mode contain at least one more transistor or storage component than pixels in rolling shutter mode. These additional components are used to store image charge for readout during a period of time after simultaneous exposure. Again, to improve the signal-to-noise ratio in the image signal, it is necessary to read out not only before each pixel charge is converted to an output signal by an amplifier transistor, but also before the pixel charge is transferred to the additional components of the pixel used to store the image charge.

[0010] In summary, rolling shutter allows for the lowest read noise and is useful for very fast data streaming without synchronization with a light source or peripherals. However, it carries the risk of spatial distortion, especially when imaging relatively large, fast-moving objects. Using a global shutter does not carry the risk of spatial distortion, and it is relatively simple and can achieve faster frame rates when synchronized with fast-switching peripherals. Therefore, it would be very advantageous to be able to conveniently offer both rolling shutter and global shutter modes of operation.

[0011] Real-time image processing is difficult to achieve. This is due to several factors, such as the large data sets that images represent and the complex operations that may need to be performed on the images. At a real-time video rate of 30 frames per second, a single operation performed on each pixel of a color image can equal tens of millions of operations per second. Many image processing applications require multiple operations to be performed on each pixel in the image, resulting in even more operations required per second. Typically, an image signal processor (ISP) is used in the imaging system to achieve this. It provides color interpolation to determine which color each pixel represents and interpolates regions at and near the pixel. It can also control the autofocus, exposure, and white balance of the imaging system. In recent years, lens correction such as halos and chromatic aberrations due to lens imperfections have been added, as well as additions such as HDR reconstruction, noise reduction, filtering, and face or object detection. It can also provide focus component control if required. ISPs usually perform their required functions through an embedded CPU. At the same time, they have little or no reconfigurability and usually must be redesigned and manufactured for each application change. The ISP can be included on the circuit die or as an additional discrete chip.

[0012] An alternative is to use a field programmable gate array (FPGA) as an implementation platform for the desired image processing and imaging control functions, especially real-time video processing. FPGAs consist of a matrix of logic modules connected by a switching network. Both the logic modules and the switching network are reprogrammable, allowing application-specific hardware to be built while maintaining the ability to easily change system functionality. FPGAs therefore offer a compromise between the flexibility of general-purpose processors and the speed of hardware-based application-specific circuits (ASICs). FPGAs also have the ability to perform parallel processing, which improves performance, compared to the serial processing provided by many image signal processor (ISP) circuits. In addition to signal processing functions, FPGAs can also provide control circuit and input / output (I / O) circuit configurability.

[0013] When certain novel circuit elements are used with edge-recognition filters that can be optimized for applications through on-chip programmability, there is an opportunity to improve real-time image processing by increasing the high frame rate capabilities of stacked image sensors. The present invention meets these needs and further provides advantages as described in the following summary of the invention. Summary of the invention

[0014] The following description presents the contributions made by the present invention.

[0015] The image sensor has a pixel unit having a photodiode, a transfer transistor, a source follower amplifier transistor, and a read circuit among its components. The photodiode, the transfer transistor, the source follower amplifier transistor, and the reset transistor are disposed within a first substrate of a first semiconductor chip for accumulating image charge in response to light incident on the photodiode. A read circuit block may also be disposed within the first substrate. The first substrate may also include a memory interface that allows the image sensor to be directly coupled to a memory interface port of a standard memory, such as a DRAM memory or an SRAM memory. A second substrate may be stacked on the image sensor substrate and include a standard or customized memory circuit, such as a DRAM or SRAM memory chip connected to the image sensor on the first substrate. A third substrate may be stacked on the opposite side of the memory circuit substrate, and disposed within the third substrate may be certain other circuits useful in image processing, such as FPGA, I / O, PLL, and ISP circuit blocks.

[0016] The main object of the present invention is to provide an image sensor having a pixel unit which has advantages not suggested by the prior art.

[0017] Another object is to provide an image sensor having pixel units that occupy less area, thereby reducing the pixel array size and manufacturing cost.

[0018] It is another object of the present invention to provide an image sensor having stacked pixels that achieves a readout mode for very high frame rate operation using computer programmable digital registers and single bit edge or boundary extraction techniques applied to image processing filters.

[0019] It is another object of the present invention to provide an image sensor having a frame memory stacked thereon to more quickly capture and retain image frames in the memory, allowing image feature detection filters to be applied to subsequently collected image frames.

[0020] Another object of the present invention is to provide an image sensor system that includes specific triggering areas within the imaging array to quickly identify objects moving into the scene and quickly transmit images defined by edges or boundaries.

[0021] Another object of the present invention is to provide an imaging method for an image sensor system that includes specific trigger areas within an imaging array to quickly identify objects moving into a scene and quickly send images defined by edges or boundaries.

[0022] Other features and advantages of the present invention will become more apparent from the following more detailed description taken in conjunction with the accompanying drawings which illustrate advantages of the present invention by way of example. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a schematic structural diagram of an imaging system in one embodiment of the present invention, the imaging system comprising a pixel array having a stacked image sensor pixel unit included in an integrated circuit system;

[0024] Figure 2 A circuit diagram of a stacked image sensor pixel unit with rolling exposure readout in the prior art;

[0025] Figure 3 is a schematic structural diagram of an image sensor system in one embodiment of the present invention, the image sensor system comprising a pixel array having a stacked image sensor chip, a digital data memory chip and an image signal processor chip;

[0026] Figure 4 A schematic diagram of single-bit image data output by an edge or boundary detection filter in an image sensor system according to an embodiment of the present invention;

[0027] Figure 5 is a schematic structural diagram of an image sensor system in another embodiment of the present invention, the image sensor system comprising a pixel array having a stacked image sensor chip, a digital data memory chip and an image signal processor chip;

[0028] Figure 6 A schematic diagram of an image sensor system in one embodiment of the present invention applied to moving object detection;

[0029] Figure 7 A flowchart showing a high frame rate imaging method of an image sensor system according to a first embodiment of the present invention is shown; and

[0030] Figure 8 A flow chart of a high frame rate imaging method of an image sensor system according to a second embodiment of the present invention is shown. DETAILED DESCRIPTION

[0031] The above figures illustrate the present invention, a compact high frame rate image sensor with a field programmable image feature edge or boundary detection filter block. This article discloses different embodiments of a stacked image sensor system. In the following description, numerous specific details are set forth to provide a more thorough understanding of the present invention. However, those skilled in the relevant art should be aware that the technical content recorded in the present invention can be implemented without one or more specific details, or other methods, components, materials, etc. In other examples, well-known structures, materials or operations are not shown or described in detail to avoid obscuring specific content. A substrate can have a front side and a back side. Any process performed from the front side can be considered as a front side operation, and any process performed from the back side can be considered as a back side operation. Structures or devices such as photodiodes and related transistors can be formed on the front side of the substrate. An alternating layer dielectric stack including a metal wiring layer and a conductive layer is formed on the front surface of the substrate.

[0032] The terms "connection" and "coupling" used in the present invention are defined as follows. "Connection" is used to describe a direct connection between two circuit elements, such as a metal line formed according to ordinary integrated circuit processing technology. In contrast, "coupling" is used to describe a direct or indirect connection between two circuit elements. For example, two coupling elements can be directly connected via a metal line, or indirectly connected via an intermediate circuit element (e.g., a capacitor, a resistor, or the source or drain of a transistor). In the stacked chip arrangement of the present invention, since the electrical interconnection on each chip is formed on the front side of the chip, the front sides of the two chips can be directly connected. However, another common practice is to connect the circuits on the front sides of two stacked substrates via through-substrate vias, with the back side of one substrate located on the front side of another substrate. When a particular circuit element is located on or formed on a substrate, it is generally considered that the circuit is located on the front side of the substrate.

[0033] Figure 1 FIG. 1 is a schematic diagram of an imaging system 100 according to an embodiment of the present invention. The imaging system 100 includes a pixel array 102. The pixel array 102 has a plurality of image sensor pixel units included in an integrated circuit system according to the teachings of the present invention. Figure 1 As shown, in the imaging system 100, the pixel array 102 is coupled to the control circuit 108 and the reading circuit 104, and the reading circuit 104 is coupled to the functional logic unit 106. The control circuit 108 and the reading circuit 104 are also coupled to the state register 110. In one embodiment, the pixel array 102 is a two-dimensional (2D) array of image sensor pixels (e.g., pixels P1, P2, ..., Pn). Figure 1As shown, each pixel can be arranged into rows (such as rows R1 to Ry) and columns (such as columns C1 to Cx) to obtain image data of people, places, objects, etc., which can then be used to present a 2D image of the person, place, object, etc. In one embodiment, after each pixel has acquired its image data or image charge, the image data is read out by the read circuit 104 using the read mode specified by the state register 110, and then transmitted to the functional logic unit 106. In various examples, the read circuit 104 may include an amplifier circuit, an analog-to-digital conversion circuit, etc. The state register 112 may include a digital programming selection system to determine whether the reading mode is through a rolling exposure mode or a global exposure mode. The functional logic unit 106 may only store the image data, or may process the image data according to a post-processing image effect (for example, cropping, rotating, removing red eyes, adjusting brightness, adjusting contrast, or other methods). In one embodiment, the read circuit 104 may read out the image data along the readout column lines (such as Figure 1 The image data may be read out row by row (as shown), or other techniques (not shown) may be used to read out the image data, such as serial readout or fully parallel readout of all pixels at the same time. In one embodiment, the control circuit 108 is coupled to the pixel array 102 to control the operability of the pixel array 102. The operation of the control circuit 108 may be determined by the current setting of the status register 112. For example, the control circuit 108 may generate an exposure signal for controlling image acquisition. In one embodiment, the exposure signal is a global exposure signal, so that all pixels in the pixel array 102 simultaneously acquire image data through a single acquisition window. In another embodiment, the exposure signal is a rolling exposure signal, and each pixel row, column, or group is continuously acquired through a continuous acquisition window.

[0034] Figure 2 1 shows a circuit diagram of an image sensor pixel unit 200 in a rolling exposure readout mode in the prior art. This diagram and exemplary pixel are provided to simplify the description of the pixel operation of the present invention. Figure 2 As shown, each sensor pixel 200 includes a photodiode 210 (eg, a photosensitive element) and a corresponding transfer transistor 215 and a pixel support circuit 211. The photodiode 210 may be a "pinned" photodiode commonly used in existing CMOS image sensors. Figure 2In the example of FIG. 1 , the pixel support circuit 211 includes a reset transistor 220, a source follower (SF) transistor 225, and a row select transistor 230 located on the circuit chip, and the row select transistor 230 is coupled to the transfer transistor 215 and the photodiode 210 located on the sensor chip. An amplifier transistor in a source follower configuration is an amplifier transistor in which a signal is input on the gate and output on the source. In other embodiments not shown, the pixel support circuit 211 includes a row select transistor 230, which is coupled to the reset transistor 220, the source follower (SF) transistor 225, the transfer transistor 215 and the photodiode 210 on the sensor chip of the stacked die system. During operation, the photosensitive element 210 generates photogenerated electrons in response to incident light during exposure. The transfer transistor 215 is coupled to receive a transfer signal TX so that the transfer transistor 215 transfers the charge accumulated in the photodiode 210 to a floating diffusion (FD) node 217. The floating diffusion node 217 is actually the drain of the transfer transistor 215 and the photodiode 210 is the source of the transfer transistor 215. In one embodiment, the transfer transistor 215 is a metal oxide semiconductor field effect transistor (MOSFET). The reset transistor 220 is coupled between the power supply VDD and the floating diffusion node 217 to reset the sensor pixel 200 (e.g., discharge or charge the floating diffusion node 217 and the photodiode 210 to a preset voltage) in response to the reset signal RST. The floating diffusion node 217 is coupled to control the gate of the source follower transistor 225. The source follower transistor 225 is coupled between the power supply VDD and the row select transistor 230 to amplify the signal generated by the charge on the floating diffusion node 217. The row select transistor 230 couples the output of the pixel circuit from the source follower transistor 225 to the readout column or bit line 235 in response to the row select signal RS. The photodiode 210 and the floating diffusion node 217 are reset by the temporarily asserted reset signal RST and the transmission signal TX. The accumulation period or accumulation window (e.g., exposure period) begins when the transmission signal TX is deactivated, so that the incident light is converted into photogenerated electrons in the photodiode 210. As photogenerated electrons accumulate in photodiode 210, its voltage decreases (electrons are negative charge carriers). The voltage or charge on photodiode 210 during exposure represents the intensity of light incident on photodiode 210. After the exposure period is over, reset signal RST is deactivated, turning off reset transistor 220 and isolating floating diffusion node 217 from power supply VDD. Transfer signal TX is asserted, coupling photodiode 210 to floating diffusion node 217. Photogenerated electrons are transferred from photodiode 210 to floating diffusion node 217 through transfer transistor 215, causing the voltage of floating diffusion node 217 to decrease by an amount proportional to the photogenerated electrons accumulated on photodiode 210 during exposure.Since photodiode 210 is still accumulating charge and transferring the charge to floating diffusion node 217 while transfer transistor 215 is active, the accumulation period or exposure window is effectively over when transfer transistor 215 is inactive.

[0035] Image sensors now have many important applications, such as the need to capture images of fast-moving objects without distortion. Example applications include autonomous vehicles, fast robots and unmanned aerial vehicles, and may also include video-based artificial intelligence (AI) applications. However, because only one pixel data can be read out from the image sensor at a time, the sensor readout is slow, especially for larger image arrays. The speed of traditional digital imaging systems is limited by the pixel transfer rate of the pixel data bus, which becomes a bottleneck for data transmission, resulting in high latency and limited frame rates. This is especially a problem when the pixel signals are converted and output as 10-bit and 12-bit high dynamic range signals. A further limitation is that because the image sensor derives pixel data based on a preloaded pixel access mode, it is difficult to change the pixel access mode without stopping the normal full-field pixel array readout, and it is not easy to change the access mode without stopping the readout first. The compact image sensor disclosed in the present invention solves the high latency and has customizable capabilities.

[0036] Figure 3An image sensor system 300 according to a first embodiment of the present invention is shown, which includes three stacked semiconductor integrated circuit chips, including an image sensor array chip 310, a digital data memory chip 320, and a logic or image signal processor chip 330. The image sensor array chip 310 can be a back-illuminated (BSI) image sensor, in which the image being captured is incident on the back of the substrate of the image sensor. The stacking direction is downward in the figure. The digital data memory chip 320 is stacked on the front side of the image sensor array chip 310. The front side or circuit side of the digital data memory chip 320 can face the front side of the image sensor array chip 310, or its back side can face the front side of the image sensor array chip 310. The image signal processor chip 330 is stacked on top of the digital data memory chip 320, with its front side or circuit side facing the digital data memory chip 320. The stacked chip arrangement allows the image sensor to be more compact, and in many cases, the area occupied by the stacked system in the image sensing array area is limited. The close contact between the digital data memory chip 320 and the image sensor array chip 310 allows direct coupling between them to facilitate high-speed pixel reading and alleviate the data transmission bottleneck associated with the use of remote memory. The digital data memory chip 320 can store at least one frame of image data from the image sensor array, and it can at least temporarily store pixel signals of multiple frames. The digital data memory chip 320 can be a DRAM or SRAM, or any other memory capable of storing at least one frame of digitized image signal data. In one embodiment, the digital data memory chip 320 can have a single-ended or dual-ended memory interface. In another embodiment, the single-ended or dual-ended memory interface can be included on the image sensor array chip 310. In yet another embodiment, the data memory chip 320 can include a row driver of the image sensor chip 310 to further reduce the area of ​​the image sensor chip 310. In any case, the stacked chip image sensor system 300 allows a compact single component, in which the image sensor array chip 310 is tightly and directly coupled to the digital data memory chip 320 and the image signal processor chip 330, which facilitates high-speed pixel readout between the image sensor array chip 310 and the image signal processor chip 330.

[0037] The image signal processor chip 330 performs various signal processing, such as filtering, on the pixel signals stored in the memory chip, or can perform signal processing on the pixel signals directly from the image sensor chip 310. In this way, the image sensor can provide high frame rate imaging or low frame rate imaging, as well as the ability to switch quickly. The key feature of the image signal processor chip 330 is the on-chip programmable reconfiguration circuit 340, which allows the filter parameters to be changed, for example, by using an off-chip compiler. By using the on-chip programmable reconfiguration circuit 340, the filter parameters can be appropriately modified, and edge or motion extraction filters such as Isotropic Sobel, Roberts, Prewitt, Laplacian and Canny can be applied to the image data. When motion detection is the main application, edge detection based on single-bit representation of image data can significantly increase the high frame rate capability to reduce image processing time and power consumption. The on-chip programmable reconfiguration circuit 340 can be an edge or boundary detection filter block, and the programmable parameters enable the filter block to extract a single-bit mapping representation of multi-bit data stored in the data memory chip 320.

[0038] Figure 4 Schematic diagram of single-bit image data output by edge or boundary detection filter in image sensor system 300 of the present invention. In one embodiment, the edge or boundary detection filter is a 3*3 array structure. In other embodiments, the edge or boundary detection filter can be a 3*5 array structure or other array forms. The multi-bit image data includes a plurality of multi-bit pixel data. The edge or boundary detection filter calculates the plurality of multi-bit pixel data respectively based on the programming parameters, and compares the result values ​​after the operation with the threshold value respectively, and outputs a plurality of single-bit pixel data according to the comparison result. The plurality of single-bit pixel data are combined to constitute the single-bit image data. In one embodiment, when the result value of the multi-bit pixel data after the edge or boundary detection filter is greater than the threshold value, the single-bit pixel data is "1" to map the first feature of the multi-bit pixel data; and when the result value of the multi-bit pixel data after the edge or boundary detection filter is less than the threshold value, the single-bit pixel data is "0" to map the second feature of the multi-bit pixel data.

[0039] In one embodiment, there may be several edge or boundary detection filters, and after further fusing several single-bit pixel data output by several edge or boundary detection filters, single-bit pixel data is output again for image signal processing. In one embodiment, the fusion is performed by voting. For example, if two of the three edge or boundary detection filters output "1" and one outputs "0" single-bit pixel data, then single-bit pixel data of "1" is output after fusion. If two of the three edge or boundary detection filters output "0" pixel data and one outputs "1", single-bit pixel data of "0" is output after fusion. In another embodiment, the fusion is performed by weighted calculation, that is, each edge or boundary detection filter is assigned a certain weight to perform the final fusion and output the final single-bit pixel data. All single-bit pixel data in the image signal are combined to constitute the single-bit image data.

[0040] Figure 5 An image sensor system 500 according to a second embodiment of the present invention is shown, which includes Figure 3 The three-chip stacked structure shown in , includes an image sensor array chip 510, a digital data memory chip 520, an image signal processor chip 530, and an on-chip programmable reconfiguration circuit 540, but a trigger area 550 is defined within the image sensor array chip 510. The trigger area is a special area for the interest identified by the image sensor system application. For example, if the image sensor system is applied to an autonomous driving vehicle, special attention is paid to pedestrians or bicycles entering the field of view from the right, then the trigger area can be defined on the right side of the image sensor array. When a moving object enters the preset trigger area, an alarm signal can be sent to the image processor, and imaging priority is given to this part, and it can be switched to a high frame rate mode, such as a single-bit filtering mode.

[0041] Figure 6 FIG. 1 is a schematic diagram of the image sensor system of the present invention applied to moving object detection. Figure 6 As shown, in one embodiment, the image signal processor chip 530 quickly determines whether a moving object appears based on the multi-bit image data of the previous and next frames of the image, for example, the moving object 610 appears, and the image signal processor chip 530 further determines whether the moving object 610 enters one or more pre-set triggering areas 650. When the moving object 610 enters the triggering area 650, the image signal processor chip 530 generates an alarm signal and generates a predefined action based on the alarm signal, for example, the alarm signal is sent to the image processor array chip 510 to preferentially obtain the image data of the triggering area 650, that is, the image data of the triggering area 650 is preferentially processed to obtain the priority imaging right.

[0042] Specifically, the multi-bit image data includes a plurality of multi-bit pixel data, and the edge or boundary detection filter calculates the absolute value of the subtraction of the multi-bit pixel data of the previous and next frames in the digital data memory 520 based on the programming parameters, and compares the result values ​​after the calculation with the threshold values ​​respectively, and outputs a plurality of single-bit pixel data according to the comparison results, and the plurality of single-bit pixel data are combined to constitute the single-bit image data. In one embodiment, when the result value of the multi-bit pixel data after the edge or boundary detection filter calculation is greater than the threshold value, the single-bit pixel data is "1", so as to map and indicate that a moving object appears in the image signal; and when the result value of the multi-bit pixel data after the edge or boundary detection filter calculation is less than the threshold value, the single-bit pixel data is "0", so as to map and indicate that no moving object appears in the image signal.

[0043] Figure 7 A flowchart of a high frame rate imaging method 700 of an image sensor according to a first embodiment of the present invention is shown, which includes the following steps.

[0044] Step 702: Acquire an image signal, where the image signal is multi-bit image data.

[0045] Step 704: Store the multi-bit image data.

[0046] In one embodiment, at least one frame of image data may be stored, and pixel signals of multiple frames may be at least temporarily stored so that various signal processing such as filtering may be subsequently performed on the pixel signals. In one embodiment, signal processing may also be performed directly on the pixel signals.

[0047] Step 706: Provide programming parameters to the edge or boundary detection filter.

[0048] In one embodiment, edge or motion extraction filters such as Isotropic Sobel, Roberts, Prewitt, Laplacian, and Canny may be applied to the computation of the image data by using an off-chip compiler to change the filter parameters.

[0049] Step 708: The edge or boundary detection filter converts the multi-bit pixel data into single-bit image data, and the single-bit image data mapping represents the multi-bit image data.

[0050] The multi-bit image data includes a plurality of multi-bit pixel data, and the edge or boundary detection filter calculates the plurality of multi-bit pixel data based on the programming parameters, and compares the result values ​​after the operation with the threshold values, and outputs a plurality of single-bit pixel data according to the comparison results, and the plurality of single-bit pixel data are combined to constitute the single-bit image data. In one embodiment, when the result value of the multi-bit pixel data after the edge or boundary detection filter operation is greater than the threshold value, the single-bit pixel data is "1" to map the first feature of the multi-bit pixel data; and when the result value of the multi-bit pixel data after the edge or boundary detection filter operation is less than the threshold value, the single-bit pixel data is "0" to map the second feature of the multi-bit pixel data.

[0051] In one embodiment, there may be several edge or boundary detection filters, and after further fusing several single-bit image data output by several edge or boundary detection filters, the single-bit image data is output again for image signal processing. In one embodiment, the fusion is performed by voting. For example, if two of the three edge or boundary detection filters output "1" and one outputs "0" single-bit pixel data, then the single-bit pixel data of "1" is output after fusion. If two of the three edge or boundary detection filters output "0" and one outputs "1", then the single-bit pixel data of "0" is output after fusion. In another embodiment, the fusion is performed by weighted calculation, that is, each edge or boundary detection filter is assigned a certain weight to perform the final fusion and output the final single-bit image data. All single-bit pixel data in the image signal are combined to constitute the single-bit image data.

[0052] Step 710: Perform image signal processing based on the single-bit image data.

[0053] In this way, the image sensor can provide high frame rate imaging or low frame rate imaging, as well as the ability to switch rapidly.

[0054] Figure 8 FIG. 8 is a flow chart of a high frame rate imaging method 800 of an image sensor according to a second embodiment of the present invention. Different from the high frame rate imaging method 700 of an image sensor, the following steps are further included.

[0055] Step 802: Define one or more trigger areas.

[0056] The trigger area is a special area for the image sensor system application to identify the interest. For example, if the image sensor system is applied to an autonomous vehicle, a pedestrian or bicycle entering the field of view from the right is of particular interest, then the trigger area can be defined on the right side of the image sensor array chip.

[0057] Step 804: Determine whether there is a moving object based on the multi-bit image data of the previous and next frames.

[0058] In one embodiment, the multi-bit image data of the previous and next frames are subtracted and then input into an edge or boundary detection filter, that is, the edge or boundary detection filter calculates the absolute value of the subtraction of the multi-bit pixel data of the previous and next frames based on the programming parameters, and compares the result value after the calculation with the threshold value respectively, and outputs a number of single-bit pixel data according to the comparison result, and the combination of the single-bit pixel data constitutes the single-bit image data. In one embodiment, when the result value of the multi-bit pixel data after the edge or boundary detection filter is greater than the threshold value, the single-bit pixel data is "1", so as to map and indicate that a moving object appears in the image signal; and when the result value of the multi-bit pixel data after the edge or boundary detection filter is less than the threshold value, the single-bit pixel data is "0", so as to map and indicate that no moving object appears in the image signal.

[0059] Step 806: Determine whether the moving object enters the trigger area.

[0060] In one embodiment, whether a moving object enters the triggering area is determined by performing computational processing on the image data of the triggering area.

[0061] Step 808: When a moving object enters the triggering area, an alarm signal is generated.

[0062] Step 810: Based on the alarm signal, a predefined action is generated.

[0063] In one embodiment, the predefined action includes: obtaining image data of the trigger area for imaging first, and switching to a high frame rate mode, such as a single-bit filtering mode, during imaging. Therefore, when motion detection is the main application, edge detection based on single-bit representation of image data can significantly increase the high frame rate capability to reduce image processing time and power consumption.

[0064] In summary, the present invention provides an image sensor system with stacked pixels and an imaging method thereof, which achieves a read mode of operation with very high frame rate by using computer programmable digital registers applied to image processing filters and single-bit edge or boundary extraction technology; and by providing an image sensor with a frame memory stacked thereon, image frames in the memory can be captured and retained more quickly, allowing image feature detection filters to be applied to subsequently collected image frames; and by including specific trigger areas in the imaging array, objects moving into the scene can be quickly identified and images defined by edges or boundaries can be quickly sent.

[0065] References throughout the specification to "one embodiment," "an embodiment," "an example," or "an example" mean that a particular feature, structure, or characteristic described in conjunction with the embodiment or example is included in at least one embodiment or example of the invention. Therefore, phrases such as "in one embodiment" or "in one example" that appear throughout the specification do not necessarily all refer to the same embodiment or example. In addition, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments or examples. Directional terms such as "top," "downward," "above," "below" are used to refer to the direction of the drawings being described. In addition, unless otherwise specifically stated, the terms "having," "including," "containing," and similar terms are defined to mean "comprising." The particular features, structures, or characteristics may be included in an integrated circuit, an electronic circuit, a combinational logic circuit, or other suitable components that provide the described functionality. In addition, it should be understood that the drawings provided herein are only for the purpose of explaining those of ordinary skill in the art, and that the drawings are not necessarily drawn to scale.

[0066] The above description of the illustrated examples of the present invention (including what is described in the Abstract) is not intended to be exhaustive or limited to the precise form disclosed. Although specific embodiments and examples of the present invention are described herein for illustrative purposes, various equivalent modifications may be made without departing from the broader spirit and scope of the present invention. In fact, it should be understood that specific example structures and materials are provided for the purpose of explanation, and other structures and materials may also be used in other embodiments and examples according to the teachings of the present invention. These modifications may be made to the embodiments of the present invention in accordance with the above detailed description. The terms used in the following claims should not be interpreted as limiting the present invention to the specific embodiments disclosed in the specification and claims. On the contrary, the scope is determined entirely by the following claims, which should be interpreted in accordance with established principles of claim interpretation.

[0067] The present invention also provides an image sensor device including the pixel circuits described in the above embodiments. The image sensor device includes a plurality of pixel circuit arrays provided in the above embodiments arranged in rows and columns. The image sensor device also includes a peripheral circuit, which is mainly used to control and process the output of the pixel circuit.

[0068] The various embodiments provided in the embodiments of the present invention include but are not limited to the explanation and description of the invention contents proposed by the present invention. The above embodiments are only for the purpose of explanation and do not constitute a limitation of the present invention. Reasonable revisions or adjustments made to the various embodiments of the present invention fall within the scope of the content protected by the present invention.

Claims

1. A compact high frame rate image sensor system, characterized in that: include: An image sensor array chip, used to acquire an image signal, wherein the image signal is multi-bit image data; an image signal processor chip stacked and connected to the image sensor array chip, wherein the image signal processor chip includes an edge or boundary detection filter having field programmable image features, and wherein programming parameters enable the edge or boundary detection filter to extract single-bit image data, the single-bit image data mapping represents multi-bit image data to reduce image processing time and power consumption; and on-chip programmable reconfiguration circuitry for providing programming parameters to the edge or boundary detection filter to optimize the image sensor for a particular application; Also included is a digital data memory chip, stacked and connected between the image sensor array chip and the image signal processor chip, for storing multi-bit image data from the image sensor array chip; One or more trigger areas are defined within the image sensor array chip, so that when a moving object enters one of the trigger areas, an alarm signal is forwarded to the image signal processor chip to cause a predefined action within the image signal processor chip, the predefined action comprising extracting the single-bit represented image signal from the trigger area; the image signal processor chip outputs multi-bit image data at a standard video rate or single-bit image data at a high frame rate.

2. The image sensor system according to claim 1, wherein: The multi-bit image data includes a number of multi-bit pixel data. The edge or boundary detection filter calculates the number of multi-bit pixel data based on the programming parameters, and compares the result values ​​after the calculation with a threshold value respectively, and outputs a number of single-bit pixel data according to the comparison result. The number of single-bit pixel data are combined to constitute the single-bit image data.

3. The image sensor system according to claim 2, wherein: When the result value of the multi-bit pixel data after the edge or boundary detection filter operation is greater than the threshold, the single-bit pixel data is "1" to map the first feature representing the multi-bit pixel data; And when the result value of the multi-bit pixel data after the edge or boundary detection filter operation is less than the threshold, the single-bit pixel data is "0" to map the second feature representing the multi-bit pixel data.

4. The image sensor system according to claim 1, wherein: The edge or boundary detection filter is one of a Sobel, Roberts, Prewitt, Laplacian or Canny edge filter.

5. The image sensor system according to claim 1, wherein: The image signal processor chip includes multiple edge or boundary detection filters. The image signal processor chip performs another fusion calculation on the single-bit image data output by the multiple edge or boundary detection filters by means of voting or weighted calculation, and then outputs the final single-bit image data for image signal processing.

6. The image sensor system according to claim 1, wherein: The multi-bit image data includes a plurality of multi-bit pixel data. The edge or boundary detection filter calculates the absolute value of the subtraction of the multi-bit pixel data of the previous and next frames in the digital data memory based on the programming parameters, and compares the calculated result value with a threshold value respectively, and outputs a plurality of single-bit pixel data according to the comparison result. The plurality of single-bit pixel data are combined to constitute the single-bit image data.

7. The image sensor system according to claim 6, characterized in that When the result value of the multi-bit pixel data after the edge or boundary detection filter operation is greater than the threshold, the single-bit pixel data is "1", and the mapping indicates that a moving object appears in the image signal; and when the result value of the multi-bit pixel data after the edge or boundary detection filter operation is less than the threshold, the single-bit pixel data is "0", and the mapping indicates that no moving object appears in the image signal.

8. The image sensor system of claim 1, wherein: The image sensor system is provided in one of an autonomous vehicle, a robot, an unmanned aerial vehicle and a machine vision related system.

9. The image sensor system of claim 1, wherein: The digital data memory chip includes one of a static random access memory or a dynamic random access memory frame buffer circuit.

10. The image sensor system of claim 1, wherein: The image sensor array chip includes one of a single-port or a dual-port memory interface.

11. The image sensor system of claim 1, wherein: The image sensor array chip is a back-illuminated image sensor.

12. The image sensor system of claim 1, wherein: The image sensor array chip is a global shutter readout image sensor or a rolling shutter readout image sensor.

13. The image sensor system of claim 1, wherein: A row driver and a control circuit supporting an image sensor array chip are arranged on the digital data memory chip or the image signal processor chip to reduce the area of ​​the image sensor array chip.

14. A compact high frame rate image sensor system imaging method, characterized in that: include: Acquire an image signal, wherein the image signal is multi-bit image data; providing programming parameters to edge or boundary detection filters; The edge or boundary detection filter converts the multi-bit image data into single-bit image data, the single-bit image data mapping represents the multi-bit image data; and performs image signal processing based on the single-bit image data; storing the multi-bit image data; return The method comprises the following steps: defining one or several trigger areas; Determining whether there is a moving object based on multi-bit image data of previous and next frames in the memory; Determining whether the moving object enters the trigger area; When a moving object enters the trigger area, an alarm signal is generated; Based on the alarm signal, a predefined action is generated, the predefined action comprising extracting the image signal represented by the single bit from the trigger area.

15. The imaging method of the image sensor system according to claim 14, characterized in that: The multi-bit image data includes a number of multi-bit pixel data. The edge or boundary detection filter calculates the multi-bit pixel data based on the programming parameters, compares the calculated result values ​​with the threshold values, and outputs a number of single-bit pixel data according to the comparison results. The single-bit pixel data are combined to constitute the single-bit image data.

16. The imaging method of the image sensor system according to claim 15, characterized in that: When the result value of the multi-bit pixel data after the edge or boundary detection filter operation is greater than the threshold, the single-bit pixel data is "1" to map the first feature representing the multi-bit pixel data; And when the result value of the multi-bit pixel data after the edge or boundary detection filter operation is less than the threshold, the single-bit pixel data is "0" to map the second feature representing the multi-bit pixel data.

17. The imaging method of the image sensor system according to claim 14, characterized in that: The multi-bit image data includes a plurality of multi-bit pixel data. The edge or boundary detection filter calculates the absolute value of the subtraction of the multi-bit pixel data of the previous and next frames in the digital data memory based on the programming parameters, and compares the calculated result values ​​with the threshold values ​​respectively, and outputs a plurality of single-bit pixel data according to the comparison results. The plurality of single-bit pixel data are combined to constitute the single-bit image data.

18. The imaging method of the image sensor system according to claim 17, characterized in that: When the result value of the multi-bit pixel data after the edge or boundary detection filter operation is greater than the threshold value, the single-bit pixel data is "1" to map and indicate that a moving object appears in the image signal; And when the result value of the multi-bit pixel data after the edge or boundary detection filter operation is less than the threshold, the single-bit pixel data is "0" to map and indicate that no moving object appears in the image signal.

19. The imaging method of the image sensor system according to claim 14, wherein: The edge or boundary detection filter is one of a Sobel, Roberts, Prewitt, Laplacian or Canny edge filter.

20. The imaging method of the image sensor system according to claim 14, wherein: Also includes: After the single-bit image data output by the plurality of edge or boundary detection filters are fused and calculated again by voting or weighted calculation, the final single-bit image data is output for image signal processing.

Images