Electronic image-capture and image-processing unit

Abstract

The invention relates to an electronic image-capture and image-processing unit comprising: a sensor region (17); a light-sensitive unit (4); a control region having integrated control circuits (14); an image-processing region having integrated image-processing circuits; an electric memory unit (16); and a switching unit (5), characterised in that the image-processing region (8) comprises a structure of programmable resistor elements and / or a structure of MOSFET transistors, wherein the structure has an interconnection for digital and / or analogue data management.

Description

[0001] Electronic image capture and processing unit

[0002] Description

[0003] The invention relates to an electronic image capture and image processing unit, comprising a sensor area with a light-sensitive unit, a control area with integrated control circuits, an image processing area with integrated image processing circuits, an electrical storage unit and a switching unit.

[0004] The first comprehensive electronic image capture and processing technology is disclosed in US Pat. No. 3,540,011 A (1968). It presents an image sensor consisting of discrete photodiodes with coupled storage capacitors. This design is characterized by the ability of semiconductor components to capture and store optical images.

[0005] This led to a dramatic development of electronic image capture and processing technologies. Fundamental development steps include the invention of the CCD sensor, disclosed in patents US 3,858,232A (1971), US 3,792,322A (1973), and US 4,085,456A (1978), and the CMOS-based Active Pixel Sensor (APS), disclosed in patent US 5,471,515 (1995).

[0006] Image sensors based on CMOS technology have largely replaced CCD sensors. This is due to the inherent design advantages of CMOS sensors, such as lower power consumption, smaller dimensions, fewer external components, lower manufacturing costs, higher frame rates at comparable resolutions, windowing, direct analog-to-digital conversion, and the ability to integrate control and image processing on a single chip.

[0007] The inherent design disadvantages of CMOS sensors, such as the low fill factor, could be compensated by additional developments, such as backside illumination, disclosed in JP 127646 and JP 127647(1975) (originally developed for CCD sensors), and the stacked sensor architecture, disclosed in JP 5773379 (2016).

[0008] At the end of this approximately 50-year development process of electronic image capture and processing technologies, highly sophisticated image sensors are available that have found their way into almost all areas of daily life. The most widespread use of electronic image capture and processing technologies remains photography.

[0009] Currently, particular interest is focused on the development of neuromorphic image sensors used in so-called “event cameras”, disclosed in EP3393122A1 .

[0010] These neuromorphic image sensors or event cameras can be advantageously used, for example, in applications for autonomous driving, general monitoring of manufacturing processes, traffic monitoring, and military applications, e.g., drone control, as well as in many other areas of electronic image capture and processing. The object of the invention is to provide an electronic image capture and processing unit that, compared to existing technologies, enables increased data processing speed, an increase in the amount of data that can be processed, in-memory computing, adaptive learning, and an increase in storage capacity and quality.

[0011] Furthermore, the disclosed invention enables real-time image analysis and thus a reduction in latency due to data reduction during image acquisition. Furthermore, it enables image acquisition with a significantly increased dynamic range and significantly reduced power consumption.

[0012] This object is achieved according to the invention with an electronic image capture and processing unit according to the main claim. Preferred variants can be found in the subclaims, the description, and the drawing.

[0013] According to the invention, the image processing area has a structure of programmable resistance elements and / or a structure of MOSFET transistors, wherein the structure has a circuit for digital (conventional computing) and / or analog (neuromorphic computing) data management.

[0014] The structure of programmable resistance elements can perform all logical functions that are performed by MOSFET transistors in conventional processors.

[0015] The structure of programmable resistance elements can be manufactured using the CMOS process, just like the structure of MOSFET transistors.

[0016] Both structures are compatible in terms of both their functionality and their manufacturing process. However, the structure of programmable resistance elements differs from the structure of MOSFET transistors in that each individual component of the structure of programmable resistance elements, the so-called memristor, can permanently store its specific resistance value based on the previous current flow. This means that the respective resistance value is retained even when the current is switched off.

[0017] This additional property of the memristor offers certain advantages over the conventional MOSFET transistor, the single component of the MOSFET transistor structure. These include, for example, higher storage density, increased energy efficiency, and the possibility of in-memory computing. Furthermore, the ability to store analog resistance values ​​non-volatilely is advantageous for the use of memristors in neuromorphic networks.

[0018] The advantages of the conventional MOSFET transistor lie primarily in its high computing speed and its flexibility, which makes it suitable for a wide range of applications.

[0019] The concrete design of the image processing area results from the respective task areas of image processing and image analysis, such as image compression, image and pattern recognition, image enhancement, and the detection of motion dynamics, to name just a few areas.

[0020] Accordingly, the image processing area or its integrated image processing circuits can consist exclusively of a structure of programmable resistance elements or exclusively of a structure of MOSFET transistors, or, in a preferred embodiment, of a mixed structure variably constructed from the two aforementioned components. The mixed structure allows for optimal utilization of the specific advantages of a structure of programmable resistance elements and the specific advantages of a structure of MOSFET transistors.

[0021] Accordingly, the interconnections in the image processing area or in the integrated image processing circuits can consist exclusively of digital or exclusively of neuromorphic or, in a preferred embodiment, variably of a mixed interconnection constructed from the two aforementioned interconnection options.

[0022] Accordingly, the structure of programmable resistance elements or the structure of MOSFET transistors or the variable mixed structure can be operated digitally (conventional computing) or analogue (neuromorphic computing) or, in a preferred embodiment, in variable proportions both analogue and digital.

[0023] In analog mode, the individual memristors or MOSFET transistors operate below their threshold value. For memristors, the threshold is defined by the minimum voltage that must be applied to significantly change the memristor's resistance. For MOSFET transistors, the threshold is defined by the voltage that must be applied for the MOSFET transistor to become conductive.

[0024] The analog operation of memristors or MOSFET transistors enables power-saving operation because the threshold values ​​of the components are not exceeded. The continuous voltage changes represented by the individual memristors or MOSFET transistors in analog operation are particularly suitable for processing in circuits inspired by the nervous systems of living organisms, i.e., in neuromorphic networks.

[0025] In digital operation, the individual memristors or the individual MOSFET transistors are operated above their threshold value.

[0026] The discrete binary states (0 and 1) generated by the individual memristors or MOSFET transistors in digital operation are particularly suitable for processing in conventional digital circuits.

[0027] Data management encompasses all activities related to data handling, such as the collection, processing, analysis, storage and use of data.

[0028] The specific design of the control area results from the respective tasks such as clock frequency generation, analog / digital conversion, data addressing, output of standard procedures and control of the shared use of the photodiodes by the neuromorphic Dynamic Vision Sensor (DVS) or the digital Active Pixel Sensor (APS) of the light-sensitive unit.

[0029] As already described for the image processing area, the control area can comprise either exclusively a structure of programmable resistance elements or exclusively a structure of MOSFET transistors or, in a preferred embodiment, a variable mixed structure of the aforementioned structures. Likewise, the circuits can consist exclusively of digital or exclusively of neuromorphic circuits or, in a preferred embodiment, of a variable combination of digital and neuromorphic circuits. The respective mode of operation of the structure of programmable resistance elements or the structure of MOSFET transistors or the variable mixed structure is also tailored to the respective task.Accordingly, the respective components implemented can be operated digitally (conventional computing) or analogue (neuromorphic computing) or, in a preferred embodiment, in variable proportions, both analogue and digital.

[0030] Additionally or alternatively, the sensor region, as already described for the image processing region and the control region, can also comprise either exclusively a structure of programmable resistance elements or exclusively a structure of MOSFET transistors, or in a preferred embodiment, a variable mixed structure of the aforementioned structures.

[0031] Likewise, the circuits in the sensor area can consist exclusively of digital or exclusively of neuromorphic circuits or, in a preferred embodiment, in a variable combination of digital and neuromorphic circuits.

[0032] Likewise, the components of the sensor area can be operated digitally (conventional computing) or analogue (neuromorphic computing) or, in a preferred embodiment, in variable proportions, both analogue and digital.

[0033] In a conventional CMOS image sensor, three basic components are found in the sensor area: the photodiode, the capacitor, and the metal-oxide-semiconductor field-effect transistor (MOSFET). These three structures are manufactured using the CMOS process on a silicon substrate. They form the so-called pixel, the smallest unit of a conventional CMOS image sensor.

[0034] In image sensors constructed according to this principle, the pixels are digitally read out after exposure, either in series via a so-called "rolling shutter" or in their entirety simultaneously via a so-called "global shutter." In this way, full images, so-called "frames," of the entire scene are output at a rate specified by an external timer. Since all of the image sensor's pixels are captured for each frame, very large data sets are generated in a short time. The transmission, intermediate storage, processing, and, if necessary, analysis of these data sets quickly overwhelms the corresponding hardware and software.

[0035] This limitation of conventional electronic image capture and processing systems has recently led to the development of so-called neuromorphic image sensors, which are used in event cameras. Event cameras are based on the idea that, for image analysis, it is not necessary to analyze every single pixel of the image sensor. Neuromorphic image sensors only generate a signal when a sufficient change in brightness is registered in the scene. Each pixel operates independently of the other pixels of the image sensor. The data output is therefore asynchronous. The output signal contains the following information: the 1-bit polarity indicating whether the brightness of a pixel has increased ("ON") or decreased ("OFF"), the exact position of the pixel on the sensor (X, Y coordinates), and a timestamp (t).

[0036] Brightness values ​​below and / or above a threshold do not generate an output signal from the sensor. This significantly reduces the data sets to be processed and analyzed. Since the threshold is stored in the respective pixel and its value varies depending on the preceding events, programmable resistor elements are particularly suitable for this task. Alternatively, the circuit can also be implemented using conventional MOSFET transistors.

[0037] The output signal of an event camera is characterized by a variable data rate, a sequence of so-called “events” or “spikes” (analogous to the term “action potential” in neurology), which define the brightness change in the area of ​​a pixel, depending on a predefined threshold, at a specific time and place.

[0038] The main advantages of neuromorphic image sensors or event cameras compared to conventional image sensors are, in addition to the significantly reduced data output, a high temporal resolution and thus a very low inertia (reactions in the microsecond range), a very high dynamic range (up to 140 dB) and very low power consumption.

[0039] In a preferred embodiment, the image signals recorded by the photodiodes of the light-sensitive unit are processed both neuromorphically and digitally.

[0040] The neuromorphic process (neuromorphic computing) is characterized by the fact that the structure of programmable resistance elements, the structure of MOSFET transistors, or the variable mixing structure is operated analogously and represents continuous voltage changes. In a preferred embodiment, the structure of programmable resistance elements, the structure of MOSFET transistors, or the variable mixing structure is neuromorphically interconnected and produces spiked output signals.

[0041] The digital process (conventional computing) is characterized by the fact that the structure of programmable resistance elements, the structure of MOSFET transistors, or the variable mixing structure is operated digitally in a conventional manner and generates clear binary states (0 and 1). In a preferred embodiment, the structure of programmable resistance elements, the structure of MOSFET transistors, or the variable mixing structure is digitally interconnected, and the data output is sequential.

[0042] Due to the joint use of the image information recorded by the photodiodes in a neuromorphic and a digital process, a camera constructed in this way provides conventional full color images of a scene and, in addition, the event data of dynamic processes.

[0043] Particularly in light of the advantages resulting from increasing the image sensor's light sensitivity, but also with regard to increasing data processing speed, it is crucial that the electrical storage unit, the switching unit, and the conductive paths are located on the side of the sensor area facing away from the light. This design, known as a "backside illumination chip," allows for the maximum coverage of the available sensor area with photodiodes and the focus of the incident light onto the photodiodes with the highest possible intensity.

[0044] In almost all electronic image acquisition and processing applications, component miniaturization is desirable. Faster image processing and increased memory capacity are also advantageous in almost all areas. Furthermore, it is desirable to integrate the control elements and as many of the downstream specific image processing processes as possible into the sensor chip. These requirements are addressed and implemented by the concept of "Stacked Sensor Architecture."

[0045] In the so-called "stacked sensor," the light-sensitive pixel layer is manufactured on a separate silicon substrate. This forms the sensor area of ​​the electronic image acquisition and processing unit. The extensive integrated circuit structures for control and image processing, as well as most of the conductive paths, are located on a separate silicon substrate that encompasses the control and image processing areas. The shared silicon substrate of the control and image processing areas is preferably arranged below the sensor plane, i.e., on the side of the sensor area facing away from the light, and is electrically connected to it via special contact points.

[0046] This advantageous stacked chip architecture significantly improves the light output and allows for the integration of significantly more control and image processing elements.

[0047] Since the silicon substrate of the sensor area is separated from the common silicon substrate of the control area and the image processing area, it is necessary to connect the two substrates by contact points.

[0048] In a preferred embodiment, these contact points are implemented using stacked sensor technology. The image information captured by the photodiodes is processed and analyzed in a preferred embodiment using both a digital process (conventional computing) and a neuromorphic process (neuromorphic computing). To enable this, the integrated image processing circuits are configured such that the image information read synchronously from the photodiodes is preferably processed and analyzed using the digital process. The image information read asynchronously from the photodiodes is preferably processed and analyzed in the integrated image processing circuits using the neuromorphic process. Accordingly, the image processing area encompasses both digital and neuromorphic circuits.

[0049] The individual components of the structure of programmable resistance elements, the individual components of the structure of MOSFET transistors, or the individual components of the variable mixed structure, which are assigned to a neuromorphic process (neuromorphic computing), are operated below their threshold value. In this way, the image information contained in the photodiodes is represented as continuous voltage changes. The structure of programmable resistance elements, the structure of MOSFET transistors, or the variable mixed structure is configured in this area for parallel acquisition and processing of spiked signals.

[0050] The individual components of the structure of programmable resistance elements, the individual components of the structure of MOSFET transistors, or the individual components of the variable mixing structure, which are assigned to the digital process (conventional computing), are operated above their threshold voltage. In this way, the image information contained in the photodiodes is represented in the form of discrete states (0 and 1), based on Boolean logic. The structure of programmable resistance elements, the structure of MOSFET transistors, or the variable mixing structure is configured in this area for sequential acquisition and processing of discrete states (0 and 1).

[0051] Further features of the invention will become apparent from the description of an embodiment with reference to a drawing and from the drawing itself.

[0052] The only figure shows a block diagram of an electronic image acquisition and image processing unit.

[0053] List of reference symbols:

[0054] (1) Light

[0055] (2) variable camera optics

[0056] (3) Microlenses

[0057] (4) light-sensitive unit

[0058] (5) Switching unit

[0059] (6) Contact points

[0060] (7) Tax area

[0061] (8) Image processing area

[0062] (9) integrated image processing circuits

[0063] (10) Display

[0064] (11 ) Data storage

[0065] (12) variable actuators

[0066] (13) variable signal outputs

[0067] (14) integrated control circuits

[0068] (15) Capacitor

[0069] (16) electrical storage unit (17) sensor area

[0070] (18) variable filters

[0071] (19) Photodiode

[0072] (20) Amplifier, readout and reset circuits

[0073] Figure 1 shows a schematic representation in block diagram form of an electronic image acquisition and processing unit in a preferred embodiment. For reasons of clarity and clarity of illustration, the conductive paths integrated into the silicon substrate of the chips are not shown.

[0074] In the illustrated embodiment, the incident light 1 in the visible range is focused onto an array of microlenses 3 by an application-specific variable camera lens 2. Variable filters 18 can be arranged below the microlenses 3, also application-specific; preferably, a Bayer filter is located at this position to distinguish colors in the full images generated by the APS sensor.

[0075] Each microlens 3 is assigned a photodiode 19 onto which it focuses the incident light. In the photodiodes 19, the incident light 1 is converted into an electric current. This current is used to charge a capacitor 15 connected downstream of the photodiode 19. According to the invention, amplifier, readout, and reset circuits 20 are implemented in the silicon substrate of the sensor region 17 exclusively as a structure of programmable resistance elements or exclusively as a structure of MOSFET transistors or as a variable mixed structure. In order to achieve the highest possible fill factor in the sensor region 17 and thus ensure optimal use of the incident light 1, in this preferred embodiment, no further control and image processing elements are incorporated into the silicon substrate of the sensor region 17.These extensive integrated circuits are relocated to a silicon substrate of the control area 7 and the image processing area 8, which is arranged below the sensor area.

[0076] The connection required in this stacked chip architecture between the silicon substrate of the sensor area 17 and the silicon substrate of the control area 7 and the image processing area 8 is ensured by contact points 6.

[0077] The integrated control circuits 14 (e.g., clock generator, A / D conversion, readout raster, etc.) and the integrated image processing circuits 9 (e.g., image compression, white balance, image recognition, windowing, image enhancement, image segmentation, image denoising, etc.) are implemented according to the invention either exclusively as a structure of programmable resistance elements or exclusively as a structure of MOSFET transistors, or as a variable combination of the two aforementioned structural elements. They contain the application-specific algorithms, serve as data storage, and, in a preferred embodiment, perform the corresponding computing operations as in-memory computing. Due to their share of neuromorphic circuitry, it is also possible to enable artificial intelligence processes such as machine learning and deep learning.The processed data from the electronic image acquisition and processing unit are routed to appropriately configured variable signal outputs 13. They can be used there in a variety of ways depending on the application; e.g., for direct image viewing via a display 10 or for data storage in an external data storage device 11. Furthermore, the output data can also be used to control application-specific variable actuators 12 in real time.

Claims

Claims 1 . Electronic image capture and image processing unit comprising a sensor area (17) with a light-sensitive unit (4), a control area (7) with integrated control circuits (14), an image processing area (8) with integrated image processing circuits (9), an electrical storage unit (16), a switching unit (5), characterized in that the image processing area (8) comprises a structure of programmable resistance elements and / or a structure of MOSFET transistors, wherein the structure has a circuit for digital and / or analog data management.

2. Electronic image acquisition and image processing unit according to claim 1, characterized in that the control area (7) comprises a structure of programmable resistance elements and / or a structure of MOSFET transistors, wherein the structure has a circuit for digital and / or analog data management.

3. Electronic image capture and image processing unit according to claim 1 or 2, characterized in that the light-sensitive unit (4) comprises a structure of programmable resistance elements and / or a structure of MOSFET transistors, wherein the structure has a circuit for digital and / or analog data management.

4. Electronic image capture and image processing unit according to one of claims 1 to 3, characterized in that a side of the sensor area (17) facing away from the light has at least one electrical storage unit (16), at least one switching unit (5) and conductor tracks.

5. Electronic image capture and image processing unit according to one of claims 1 to 4, characterized in that the control region (7) comprises integrated control circuits (14) and the image processing region (8) comprises integrated image processing circuits (9), which are arranged together on a separate silicon substrate of the control region (7) and the image processing region (8), wherein the silicon substrate of the control region (7) and the image processing region (8) is arranged on a side of the sensor region (17) facing away from the light.

6. Electronic image capture and image processing unit according to one of claims 1 to 5, characterized in that the silicon substrate of the sensor region (17) is connected to the common silicon substrate of the control region (7) and the image processing region (8) by contact points (6).

7. Electronic image acquisition and image processing unit according to one of claims 1 to 6, characterized in that the structure of programmable resistance elements and / or the structure of MOSFET transistors is configured for asynchronous and / or synchronous acquisition and processing of data.

8. Electronic image capture and image processing unit according to one of claims 1 to 7, characterized in that the structure of resistance elements and / or the structure of MOSFET transistors is configured for analog and / or digital operation.

9. Electronic image acquisition and image processing unit according to one of claims 1 to 8, characterized in that the structure of programmable resistance elements and / or the structure of MOSFET transistors is configured for parallel and / or sequential acquisition and processing of data.

Images