Feedback controlled sensor array and sensor unit
By using multiple photoelectric sensors and feedback signal-controlled storage modules in the image sensor, the output storage of the image sensor is optimized, and the image capture difficulty of CMOS image sensors under high dynamic range and low power consumption is solved, and high-quality images are achieved Capture and process.
Patent Information
- Application Number
- CN202280095108.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-04-19
- Filing Date
- 2022-09-02
- Publication Date
- 2024-11-19
AI Technical Summary
CMOS image sensors have scroll shutter effects, image artifacts and dynamic range limitations under specific conditions, making it difficult to capture high dynamic range images under uncontrolled light conditions, especially when moving objects quickly.
Using an imaging sensor unit including multiple photoelectric sensors, the memory module is controlled through local and global feedback signals, the storage time and conditions of sensor output are optimized, independent sensor operation and output storage are realized, and broad-spectrum photoelectric sensors and summing nodes or winners are used. Take-all nodes improve image quality.
It effectively reduces image artifacts and dynamic range limitations, improves image capture capabilities under high dynamic range and low power consumption, and can retain detailed information in scenes where light and dark areas exist simultaneously.
Smart Images

Figure CN118985136A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to a system, apparatus and method for feedback control of a sensor array, in particular an image sensor array, individual sensor cells of an image sensor array, and a method for providing local and global feedback control of image sensor arrays and sensor cells to optimize sensing dynamics. Background of the Invention
[0003] A sensor array is a group of sensors, usually arranged in a regular geometric pattern, used to detect and process a class of signals. Examples of sensor arrays include optical arrays, such as those used in image sensors and cameras; electromagnetic radiation sensor arrays, such as those used in deep space telescopes and ground-based antennas; chemical sensor arrays; pressure sensor arrays; sonar arrays; and seismometer arrays.
[0004] An image sensor or image sensor array or imager is a sensor that detects the spatial distribution of electromagnetic radiation (visible or invisible) impinging on the sensor to form an image that conveys relevant information. Complementary metal oxide semiconductor (CMOS) sensors are examples of active-pixel image sensors used in modern cameras to capture images. These sensors typically include a regular geometric pattern of pixels that are globally controlled.
[0005] However, CMOS image sensors (CIS) have certain disadvantages under certain conditions. CMOS sensors can capture one row of pixels at a time, and therefore the resulting image can experience a rolling shutter effect, in which the image is skewed. Even CMOS sensors using a global shutter may require post-processing to form clear image information of moving objects. However, due to the way the pixel output is captured in the CMOS sensor, this can introduce image artifacts from the pixel output at a previous moment. In addition, in the case of both bright and dark areas of a scene, many details of the image may be lost due to the over / under position of the pixels that include the bright and dark areas.
[0006] Vision (i.e., the technology of processing images) is the sensing technology with the greatest information density, which better describes the world around us. In order to interpret the scene in real-time machine vision applications, intensive digital processing is usually required, which must be performed by a high-performance computing platform, especially when artificial intelligence techniques are used. Machine vision applications usually use cameras with embedded automatic exposure control, pixel correction, analog-to-digital conversion, and standard output video formats. Despite their high performance, one of the main drawbacks of these applications is the fixed dynamic range, which severely limits their use in many outdoor scenes with uncontrolled light conditions and / or high intra-scene dynamics, such as in automotive, surveillance, and industrial process monitoring applications.
[0007] The most common approach to acquiring a scene comprising both bright and dark areas is based on acquiring two or more frames acquired at different exposure times, which are merged into a single high dynamic range (HDR) image using post-processing, software-based techniques. However, this approach can produce artifacts, especially in the case of fast moving objects, due to the misalignment between the images taken at different moments caused by the time lag between the acquisition of the frames. On the other hand, if more sensors operating at different exposure times are used side by side to capture the images to be merged, the problems caused by moving objects can be mitigated, but they cannot be completely solved and further post-processing is required to repair the potential misalignment in the images (the so-called "parallax" problem) at a high cost in terms of space and energy.
[0008] Other methods have been used to obtain single-exposure HDR images with a low probability of producing artifacts:
[0009] o Sensor arrays based on pixels with compressed dynamic responses of current or voltage (e.g., log-log or linear-log), which suffer from information loss in the compressed portion of the dynamic range and / or the presence of significant levels of noise when the signal is weak;
[0010] o Sensor arrays based on pixels with dual photodiodes: one photodiode with high sensitivity and the other with low sensitivity, whose signals are fused after an exposure time. However, this solution requires redundant pixels and careful design to reduce crosstalk between pixels and obtain uniform optical performance; and
[0011] o Sensor arrays based on pixels with lateral spillover which can be very effective in providing single exposure HDR images, but at the expense of increased complexity that can lead to signal-to-noise (S / N) issues.
[0012] In general, the photographic quality and resolution of an image are not that important if the low photographic quality is compensated by the sensor's ability to capture all relevant details of the scene. When the sensor is able to capture all relevant details, the system is able to interpret the scene through early access to visual features that can be effectively extracted from the image, allowing the system to make fast and reliable decisions. For example, an image acquired by a sensor using different sensitivity levels for different areas of a scene may appear distorted, flat, or have wrong colors, but if this is due to a mechanism that enhances the visibility of relevant details included in the scene, the recognition capabilities of a visual system using such an image may be improved.
[0013] In this regard, custom vision sensors designed for machine vision allow for tight and efficient integration of a hybrid analog-digital image processing layer with the photosensitive element, providing a fast and low-power vision processing device. On the other hand, their main disadvantage compared to commercial imagers designed for photography is the larger pixel size, which means that sensors with the same pixel resolution require a larger chip area and higher silicon cost. However, there are large-area applications that rely on vision technology that do not require high resolution but rather high dynamic range and low power consumption, such as human / object monitoring, tracking and detection, surveillance situations, and human-machine interfaces.
[0014] The above considerations also apply to all cases where the physical quantities to be acquired are distributed in space, whereby their spatial distribution is important, and when sensors other than those based on visible light are used to detect their distribution. This applies, for example, to the case of electromagnetic waves that are not in the visible spectrum (i.e., infrared, ultraviolet, terahertz, etc.) or other physical sources (i.e., sound, pressure, etc.).
[0015] In contrast to the above-mentioned CMOS image sensors and artificial sensor arrays, biological vision, exemplified by the human eye, is not based on a globally controlled, uniformly spaced array of identical photosensitive elements, but uses an irregular distribution of different types of photoreceptors (mainly cones, rods, and ganglion cells) to detect incident light in an autonomous manner (i.e., without the need for a global control system). Rods and cones are connected via the optic nerve in a biofeedback system that includes bipolar cells, ganglion cells, and most importantly, the brain. Retinal ganglion cells also show some ability to detect light, as recent results have shown (i.e., their behavior is affected by the light that hits them), receiving input from an average of about a hundred rods and cones. The retinal ganglion collectively transmits imaging and non-imaging visual information in the form of action potentials. Excitation of retinal ganglion cells results in an increase in the firing rate of action potentials, while inhibition results in a decrease in the firing rate of action potentials. This biological system is able to adaptively use simple inputs from rods and cones to discern complex visual information in the visual cortex of the brain. Different cells are excited according to local and global features of the scene presented to the human eye. Furthermore, the human visual system does not introduce artifacts into the image, and as mentioned above, biological vision does not require a regular geometric arrangement of cells and does not rely entirely on global control. Finally, images acquired by the biological visual system appear poor (often blurry, distorted, and even monochrome in some cases) if compared to typical images acquired by a standard CMOS imager. The brain fully restores the image quality and its information content through a feedback-driven process that optimizes the retinal response.
[0016] It has been recognized that sensor arrays inspired by biological vision systems can overcome the problems associated with artificial sensor arrays. Summary of the invention
[0017] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description.
[0018] In a first aspect, the present disclosure provides an imaging sensor unit, the imaging sensor unit comprising: a sensor module, the sensor module comprising a plurality of photoelectric sensors; a memory module, the memory module coupled to the sensor module, the memory module configured to receive sensor output from the sensor module; a local control module, the local control module coupled to the sensor module, the local control module configured to: receive a local feedback signal from the sensor module; receive a global feedback signal; and use the local feedback signal and the global feedback signal to control the memory module to store the sensor output. The sensor module may be referred to as a photoreceptor or a photoreceptor unit.
[0019] A local control module having functions and operations of controlling a plurality of photosensors in this manner allows a sensor unit to operate at least partially independently of other sensor units and allows the sensor output to be stored according to an optimal time or condition set based on a global feedback signal and a local feedback signal. The photosensor may be a photodiode or any light-sensitive element.
[0020] Preferably, the global feedback signal comprises a threshold; and wherein the local control module is configured to: compare the local feedback signal with the threshold of the global feedback signal; and control the memory module to store the sensor output when the local feedback signal satisfies the threshold.
[0021] The global feedback signal may be adjustable and may be provided from an external source such as a global control module or a computer. Various optimization techniques may be used at the computer to determine the optimal threshold.
[0022] Preferably, the plurality of photosensors include: a first photosensor sensitive to a first portion of the visible spectrum, the first photosensor being configured to output a first sensor output component; a second photosensor sensitive to a second portion of the visible spectrum, the second photosensor being configured to output a second photosensor output component; and a third photosensor sensitive to a third portion of the visible spectrum, the third photosensor being configured to output a third photosensor output component; so that the sensor output includes the first photosensor output component, the second photosensor output component and the third photosensor output component.
[0023] Preferably, the imaging sensor unit further comprises: a summing node configured to sum the first photosensor output component, the second photosensor output component and the third photosensor output component to generate a summed value, wherein the local feedback signal comprises the summed value, so that: the local control module is configured to compare the summed value with a threshold of the global feedback signal; wherein when the summed value satisfies the threshold, the local control module is configured to: cause the memory module to store the first photosensor output component, the second photosensor output component and the third photosensor output component.
[0024] As an alternative to the summing node, the imaging sensor unit preferably also includes: a winner-takes-all node, which is configured to: receive the first photosensor output component, the second photosensor output component and the third photosensor output component; determine the winning photosensor output component by identifying the maximum magnitude difference between the first photosensor output component, the second photosensor output component and the third photosensor output component and the initial values of the corresponding first output component, second output component and third output component; wherein the local feedback signal includes the winning photosensor output component, so that: the local control module is configured to compare the winning photosensor output component with the threshold of the global feedback signal, wherein when the winning photosensor output component meets the threshold, the local control module is configured to: cause the memory module to store the first photosensor output component, the second photosensor output component and the third photosensor output component.
[0025] Preferably, the first photosensor senses red light, the second photosensor senses green light, and the third photosensor senses blue light.
[0026] Preferably, the multiple photosensors include a broad-spectrum photosensor, which is configured to detect the brightness of the observed scene and generate a brightness output, wherein the local feedback signal includes the brightness output, so that when the brightness output meets the threshold of the global feedback signal, the local control module is configured to: cause the memory module to store the sensor output.
[0027] The broad spectrum photoelectric sensor can be used in combination with the first photoelectric sensor, the second photoelectric sensor and the third photoelectric sensor, the summing node or the winner-takes-all node. When used with the first photoelectric sensor, the second photoelectric sensor and the third photoelectric sensor, the local feedback signal device using the broad spectrum photoelectric sensor output as the output from the first photoelectric sensor, the second photoelectric sensor and the third photoelectric sensor does not need to be sampled or disturbed before storage.
[0028] Preferably, the plurality of photosensors include a broad-spectrum photosensor configured to detect the brightness of an observed scene and generate a brightness output; wherein the local control module is further configured to receive a second local feedback signal and a second global feedback signal, wherein the second local feedback signal includes the brightness output, and wherein the second global feedback signal includes a second threshold; wherein the local control module is configured to compare the second local feedback signal with the second threshold, so that when the second local feedback signal meets the second threshold, the local control module is configured to: cause the memory module to store the brightness output; wherein the imaging sensor further includes a readout module configured to output one or more of the brightness output, the first photosensor element output, the second photosensor element output, and the third photosensor element output from the imaging sensor based on a comparison result with a third global feedback signal received at the readout module or a function of the third global feedback signal.
[0029] Using a broad spectrum photosensor and either a summing node or a winner-takes-all node allows two contributions to the local feedback signal, each of which can be used to trigger storage of the sensor output. The broad spectrum photosensor can be a high sensitivity photosensor configured to detect the brightness of an observed scene and produce a brightness output even in the presence of low or very low intensity incident light, wherein the broad spectrum photosensor is more sensitive to incident light than the first photosensor, the second photosensor, and the third photosensor.
[0030] Preferably, the broad spectrum photosensor is configured to detect white light.
[0031] Preferably, the broad spectrum photosensor is configured to detect infrared light.
[0032] Preferably, the imaging sensor unit is subject to a maximum exposure time, wherein if the local feedback signal does not satisfy a threshold within the maximum exposure time, the memory module is configured to store the sensor output.
[0033] The imaging sensor unit may form part of an array of multiple imaging sensor units. Each of these units may be globally controlled in a synchronized manner so that each unit is activated to begin sensing at the same time and so that each unit has the same maximum exposure period. However, based on the local control module of each imaging sensor unit, the sensor output stored by each unit may be different and stored at different times. Each of these sensor outputs and the relative time at which they are stored may be read out in a synchronized manner at the end of the maximum exposure period.
[0034] Preferably, the local control module is further configured to: cause the memory module to store timing information indicating a time to store the sensor output.
[0035] Preferably, the memory module is configured to receive a timing signal as part of the global feedback signal, and wherein the timing information is a record of the timing signal when storing the sensor output.
[0036] Storing timing information in this way means that the imaging sensor unit stores the sensor output according to some optimal condition depending on the local feedback signal and the global feedback signal and stores the time of the sensor output, which indicates when the optimal condition was reached. Both parameters are useful in post-processing tasks to obtain absolute sensor values.
[0037] According to a second aspect, there is provided a sensor array comprising a plurality of imaging sensors according to the first aspect, the sensor array further comprising: a global control module configured to globally control the sensor array, wherein the global control module is configured to provide the global feedback signal to each of the plurality of imaging sensors.
[0038] Preferably, the global feedback signal is adjustable to adjust the characteristics of the image captured by the sensor array.The global feedback signal may include a plurality of different signals, such as a timing signal, and a first global feedback signal, a second global feedback signal, and a third global feedback signal.
[0039] Preferably, the sensor array further comprises a readout circuit configured to read out stored sensor outputs from the plurality of imaging sensors.
[0040] Preferably, the readout circuit is further configured to read out timing information for each of the plurality of imaging sensors, wherein the timing information indicates a time at which the stored sensor output of each of the plurality of imaging sensors is stored.
[0041] As described above, the sensor array can be controlled in a globally synchronized manner such that each sensor cell is activated at the same time and the readout process of the sensor cells occurs simultaneously after the global maximum exposure time. However, each sensor cell is also controlled locally, which means that each cell stores sensor output values and timing information that can be acquired before the end of the maximum exposure time according to a certain optimal condition achieved based on the use of the global feedback signal and the local feedback signal for each specific sensor cell.
[0042] Preferably, the sensor array is communicatively coupled to a computer device, wherein the readout circuit is configured to send information relating to at least a sensor output of each of the plurality of imaging sensors to the computer device.
[0043] Preferably, the sensor array further comprises one or more motion sensor units, the one or more motion sensor units comprising: a motion sensor, the motion sensor comprising: a photosensor; a motion sensor memory module, the motion sensor memory module being configured to receive a timing signal; and a motion sensor control module; wherein the motion sensor control module is configured to: receive and compare a sensor output of the photosensor with a global motion threshold; when the sensor output of the photosensor satisfies the global motion threshold, cause the memory module to store timing information from the timing signal, the timing information indicating the time when the global motion threshold is met; wherein the motion sensor memory module comprises a switch, the switch being configured to modify a storage location of the timing information, wherein: during a first active motion sensor phase of the photosensor corresponding to a first frame of an observed scene, the switch is positioned in a first position corresponding to a first storage location; and During a subsequent second active motion sensor phase of the photosensor, corresponding to a second frame of the observed scene, the switch is positioned in a second position corresponding to a second storage position, so that the local control module is configured to: cause the memory module to store first timing information in the first storage position, the first timing information indicating the time when the global motion threshold is met for the first activation motion sensor phase; and cause the memory module to store second timing information in the second storage position, the second timing information indicating the time when the global motion threshold is met for the second activation motion sensor phase; and wherein the local control module is configured to: calculate a difference value corresponding to the difference between the first timing information and the second timing information; compare the difference value with a second global motion threshold; and when the difference value meets the second global motion threshold, output a signal from the motion sensor indicating that motion has been detected between the first frame and the second frame of the observed scene.
[0044] Preferably, the first global motion threshold and the second global motion threshold are adjustable.
[0045] Preferably, the first global motion threshold is a photosensor output threshold and the second global motion threshold is a time threshold.
[0046] Preferably, a signal indicating that motion has been detected between the first frame and the second frame of the observed scene is configured to activate the plurality of imaging sensor units.
[0047] Preferably, the sensor array comprises a plurality of motion sensor units.
[0048] Preferably, a plurality of motion sensor units are arranged at least around the periphery of the sensor array.
[0049] Using a motion detector such as this, which effectively detects brightness changes between frames at a local level, the sensor array becomes "motion activated", whereby activation of one or more sensor cells is dependent upon the excitation of a motion sensor cell. Based on which motion sensor cells are excited, the sensor cells of the sensor array may be excited only in the area of the sensor array where the excited motion sensor is located.
[0050] According to a third aspect, a method for controlling an imaging sensor unit is provided, wherein the imaging sensor unit comprises: a sensor module, the sensor module comprising a plurality of photoelectric sensors; a local control module, the local control module coupled to the sensor module; and a memory module, the memory module coupled to the sensor module; wherein the method comprises: receiving a sensor output from the sensor module at the memory module; receiving a local feedback signal from the sensor module at the local control module; receiving a global feedback signal at the local control module; and using the local feedback signal and the global feedback signal to control the memory module to store the sensor output.
[0051] Preferably, the method comprises operating the imaging sensor unit according to a cycle, the cycle being defined by: an active sensor phase, the active sensor phase being initiated by: activating the sensor module from an initial state so that the sensor module begins to detect and generate the sensor output; and a passive sensor phase being initiated by: resetting the sensor module to an initial state.
[0052] Preferably, the method further comprises entering the active sensor stage; and after storing the sensor output, entering the passive sensor stage.
[0053] Preferably, the duration of the active sensor phase is limited by a maximum exposure time, so that the method further comprises: causing the memory module to store the sensor output during the maximum exposure time; and entering the passive sensor phase, so that the passive sensor phase is entered after the maximum exposure time has elapsed.
[0054] Preferably, the method further comprises: when storing the sensor output, storing timing information indicating the time at which the sensor output is stored.
[0055] Preferably, the method further comprises controlling a plurality of imaging sensor units, the method comprising: synchronously entering an active phase of each imaging sensor unit.
[0056] According to a fourth aspect, there is provided: a motion sensor, the motion sensor comprising: a photoelectric sensor; a motion sensor memory module, the motion sensor memory module being configured to receive a timing signal; and a motion sensor control module; wherein the motion sensor control module is configured to: receive and compare a sensor output of the photoelectric sensor with a global motion threshold; when the sensor output of the photoelectric sensor meets the global motion threshold, cause the memory module to store timing information from the timing signal, the timing information indicating the time when the global motion threshold is met; wherein the motion sensor memory module comprises a switch, the switch being configured to modify a storage position of the timing information, wherein: during a first active motion sensor phase of the photoelectric sensor corresponding to a first frame of an observed scene, the switch is positioned in a first position corresponding to a first storage position; and during a subsequent second active motion sensor phase of the photoelectric sensor During the sensor phase, corresponding to a second frame of the observed scene, the switch is positioned in a second position corresponding to a second storage position, so that the local control module is configured to: cause the memory module to store first timing information in the first storage position, the first timing information indicating the time when the global motion threshold is met for the first activation motion sensor phase; and cause the memory module to store second timing information in the second storage position, the second timing information indicating the time when the global motion threshold is met for the second activation motion sensor phase; and wherein the local control module is configured to: calculate a difference corresponding to the difference between the first timing information and the second timing information; compare the difference with a second global motion threshold; and when the difference meets the second global motion threshold, output a signal from the motion sensor indicating that motion has been detected between the first frame and the second frame of the observed scene.
[0057] According to a fifth aspect, a sensor unit is provided, which includes: a sensor element, which is used to detect a physical property; a local control module, which is coupled to the sensor element, and is configured to locally control the sensor element; a memory module, which is coupled to the sensor element, and is configured to receive a sensor output from the sensor element, wherein the sensor output is time-varying; wherein the local control module is configured to receive and compare a local feedback signal and a global feedback signal, wherein the local feedback signal is based on the sensor output, and wherein the global feedback signal includes a threshold criterion, wherein when the local feedback signal meets the threshold criterion, the local control module is configured to: cause the memory module to store timing information indicating the time when the threshold criterion is met.
[0058] Preferably, the timing information is a measure of a variable electrical signal that varies according to a predetermined relationship to time, the variable electrical signal being input to the memory module as part of the global feedback signal.
[0059] Preferably, the variable electrical signal is initialized at the beginning of the active sensor phase and reset at the end of the active sensor phase, such that a measurement of the variable electrical signal indicates the time at which the threshold criterion is met from the time the active sensor phase is activated.
[0060] Preferably, if the sensor output does not satisfy the threshold criteria during a predetermined maximum exposure period, the local control module is configured, at the end of the maximum exposure period: to cause the memory module to store timing information indicating the time at which the exposure period ended.
[0061] Preferably, the sensor element comprises one or more optoelectronic sensor elements configured to convert / detect electromagnetic radiation.
[0062] Preferably, the one or more photosensor elements are configured to detect visible light.
[0063] Preferably, the one or more photoelectric sensor elements further include: a first photoelectric sensor element sensitive to a first portion of the visible spectrum, the first photoelectric sensor element being configured to output a first photoelectric sensor element output; a second photoelectric sensor element sensitive to a second portion of the visible spectrum, the second photoelectric sensor element being configured to output a second photoelectric sensor element output; and a third photoelectric sensor element sensitive to a third portion of the visible spectrum, the third photoelectric sensor element being configured to output a third photoelectric sensor element output; thereby the sensor output includes the first photoelectric sensor element output, the second photoelectric sensor element output and the third photoelectric sensor element output.
[0064] Preferably, the sensor unit further comprises: a summing node configured to sum the first photosensor element output, the second photosensor element output and the third photosensor element output to generate a summed value, wherein the local feedback signal comprises the summed value, so that: the local control module is configured to compare the summed value of the global feedback signal with the threshold standard; wherein when the summed value satisfies the threshold standard, the local control module is configured to: cause the memory module to store the first photosensor element output, the second photosensor element output and the third photosensor element output.
[0065] Preferably, and alternatively for the summing node, the imaging sensor unit also includes: a winner-takes-all node configured to: receive the outputs of the first photosensor, the second photosensor and the third photosensor elements; determine the winning photosensor element output by identifying which of the first photosensor element output, the second photosensor element output and the third photosensor element output exhibits the maximum magnitude difference from the initial values of the corresponding first photosensor element output, the second photosensor element output and the third photosensor element output; wherein the winning photosensor element output is the local feedback signal, so that: the local control module is configured to compare the winning photosensor element output with a threshold standard of the global feedback signal; wherein once the winning photosensor element output meets the threshold standard, the local control module is configured to: cause the memory module to store the first photosensor element output, the second photosensor element output and the third photosensor element output.
[0066] Preferably, the one or more photosensor elements include a broad spectrum photosensor element configured to detect the brightness of the observed scene and generate a brightness output, wherein the sensor output is the brightness output. The broad spectrum photosensor element can be used with the first photosensor element, the second photosensor element and the third photosensor element, with a summing node, with a winner-takes-all node, or without these features. The broad spectrum photosensor element can also be the only photosensor element in the sensor unit.
[0067] Preferably, the local feedback signal comprises a brightness output of the broad spectrum photosensor element, such that when the brightness output satisfies the threshold criterion, the local control module is configured to cause the memory module to store timing information indicating the time when the threshold criterion is satisfied.
[0068] Preferably, when using the brightness output as indicated in the above paragraph, the one or more photosensor elements also include: a first photosensor element sensitive to a first part of the visible spectrum, the first photosensor element being configured to output a first photosensor element output; a second photosensor element sensitive to a second part of the visible spectrum, the second photosensor element being configured to output a second photosensor element output; and a third photosensor element sensitive to a third part of the visible spectrum, the third photosensor element being configured to output a third photosensor element output; wherein when the brightness output meets the threshold standard, the local control module is configured to: cause the memory module to store the first photosensor element output, the second photosensor element output and the third photosensor element output.
[0069] Preferably, the memory module further comprises a switch configured to modify a storage location of the timing information indicating a time when the threshold criterion is satisfied, wherein: during a first active sensor phase of the broad-spectrum photosensor, corresponding to a first frame of an observed scene, the switch is positioned in a first position corresponding to a first storage location; and during a subsequent second active sensor phase of the broad-spectrum photosensor, corresponding to a second frame of an observed scene, the switch is positioned in a second position corresponding to a second storage location, such that the local control module is configured to: cause the memory module to store first timing information in the first storage location, the first timing information indicating a time when the threshold criterion is satisfied for the first active sensor phase; and cause the memory module to store second timing information in the second storage location, the second timing information indicating a time when the threshold criterion is satisfied for the second active sensor phase; and wherein the local control module is configured to: calculate a difference value corresponding to a difference between the first timing information and the second timing information; compare the difference value to a second global feedback signal, wherein the second global feedback signal comprises a second threshold criterion, and when the difference value satisfies the second threshold criterion, output an indication from the imaging sensor that the difference value has satisfied the second threshold criterion.
[0070] Preferably, when a summing node or a winner-takes-all node is used, the one or more photosensor elements also include a high-sensitivity, broad-spectrum photosensor element configured to detect the brightness of the observed scene and produce a brightness output even in the presence of low or very low intensity incident light; wherein the local control module is also configured to receive and compare a second local feedback signal and a second global feedback signal, wherein the second local feedback signal is the brightness output, and wherein the second global feedback signal includes a second threshold criterion, wherein when the second local feedback signal meets the second threshold criterion, the local control module is configured to: cause the memory module to store second timing information indicating the time when the second threshold criterion is met; and cause the memory module to store the brightness output; wherein the imaging sensor also includes a readout module configured to output the brightness output from the imaging sensor based on a comparison result or a function thereof with a third global feedback signal received at the readout module, one or more of the first photosensor element output, the second photosensor element output and the third photosensor element output, the timing information and the second timing information.
[0071] Preferably, the readout module is configured to compare the second timing information with the third global feedback signal, wherein the third global feedback signal includes a third threshold standard, and wherein the readout module is configured to output the first photosensor element output, the second photosensor element output and the third photosensor element output and the first timing information, or the brightness output and the second timing information from the imaging sensor.
[0072] Preferably, the imaging sensor unit is configured to output the stored sensor output and the stored timing information from the imaging sensor unit.
[0073] Preferably, the first photosensitive element, the second photosensitive element and the third photosensitive element include red light, green light and blue light sensitive photosensitive elements.
[0074] According to a sixth aspect, there is provided a sensor array comprising a plurality of imaging sensors according to the fifth aspect, the sensor array further comprising: a global control module configured to globally control the sensor array, wherein the global control module is configured to provide the global feedback signal to each of the plurality of imaging sensors.
[0075] Preferably, the sensor array further comprises one or more first type imaging sensors according to claim 16, wherein the global control module is configured to receive one or more indications from the first type imaging sensors to determine whether there is motion in the observed scene between the first frame and the second frame.
[0076] Preferably, the sensor array further comprises one or more second type imaging sensors according to claim 9, 10, 15, 17 or 18, wherein the global control module is configured to activate / turn on the second type imaging sensor when motion is determined between the first frame and the second frame.
[0077] According to a seventh aspect, a method for controlling an imaging sensor unit is provided, wherein the sensor unit comprises: a sensor element for detecting a physical property; a local control module coupled to the sensor element; and a memory module, the memory module being coupled to the sensor element, the memory module being configured to receive a sensor output from the sensor element, wherein the sensor output is time-varying; wherein the method comprises: receiving a local feedback signal and a global feedback signal at the local control module, wherein the local feedback signal is based on the sensor output, and wherein the global feedback signal comprises a threshold criterion, comparing the local feedback signal with the threshold criterion; and when the local feedback signal satisfies the threshold criterion, causing the memory module to store timing information indicating the time when the threshold criterion is satisfied.
[0078] Preferably, the method comprises operating the sensor unit according to a cycle, the cycle being defined by: an active sensor phase, which is initiated by: activating the sensor element from an initial state so that the sensor element begins to detect the physical property and generate the sensor output; and a passive sensor phase, which is initiated by: resetting the sensor element to an initial state.
[0079] Preferably, the method comprises: entering an active sensor phase; and after causing the memory module to store timing information indicative of when the threshold criterion is met, entering the passive sensor phase.
[0080] Preferably, the duration of the active sensor stage is limited by a maximum exposure time, so that the method further comprises: at the maximum exposure time, causing the memory module to store timing information indicating the maximum exposure time; and entering the passive sensor stage, so that entering the passive sensor stage is performed after the maximum exposure time has passed.
[0081] Preferably, the method further comprises storing the sensor output when the local feedback signal satisfies a threshold criterion.
[0082] Preferably, the method further comprises controlling a plurality of imaging sensor units, the method comprising: synchronously entering an active phase of each imaging sensor unit.
[0083] Parts of the methods described herein may be performed by software in a machine-readable form on a tangible storage medium, for example, in the form of a computer program including computer program code means, which, when the program is run on a computer, is suitable for performing all the steps of any method described herein, and wherein the computer program may be embodied on a computer-readable medium. Examples of tangible (or non-temporary) storage media include disks, thumb drives, memory cards, etc., and do not include propagation signals. The software may be suitable for execution on a parallel processor or a serial processor, so that the method steps may be executed in any suitable order or simultaneously. The method may be performed by using a device utilizing the concept of "programmable logic", such as a programmable logic array PLA, a complex programmable logic device CPLD, a field programmable gate array FPGA, and a simple programmable logic device SPLD. BRIEF DESCRIPTION OF THE DRAWINGS
[0084] Embodiments of the present invention will be described by way of example with reference to the following drawings, in which:
[0085] Figure 1 is a schematic diagram of a sensor array including a plurality of sensor units according to an embodiment of the present invention;
[0086] Figure 2is a schematic diagram showing a photoreceptor unit according to an embodiment of the present invention;
[0087] Figure 3 is a schematic diagram showing a light intensity invariant function of a photoreceptor unit according to an embodiment of the present invention; and
[0088] Figure 4 It is shown Figure 3 Timing diagram of the working principle of the circuit shown;
[0089] Figure 5 is a schematic diagram showing a color intensity function of a photoreceptor unit according to an embodiment of the present invention;
[0090] Figure 6 It is shown Figure 5 Timing diagram of the working principle of the circuit shown;
[0091] Figure 6 a is a timing diagram showing the operating principle of the circuit according to various embodiments at different times;
[0092] Figure 7 is a schematic diagram illustrating a light intensity invariant function of a photoreceptor unit according to an embodiment of the present invention;
[0093] Figure 8 It is shown Figure 7 Timing diagram of the working principle of the circuit shown;
[0094] Fig. 9 is a schematic diagram of another exemplary circuit according to an embodiment of the present invention;
[0095] Fig.10 is another schematic diagram of another exemplary circuit according to an embodiment of the present invention;
[0096] Fig.11 is a schematic diagram of another exemplary circuit according to an embodiment of the present invention;
[0097] Fig.12 is a schematic diagram of a sensor array according to an embodiment of the present invention; and
[0098] Fig.13 According to an embodiment of the present invention Fig.12 Another schematic diagram of a sensor array.
[0099] The same reference numerals are used throughout the drawings to denote similar features. DETAILED DESCRIPTION
[0100] In the following description, the terms "sensor", "pixel", "photodiode", "photosensitive element" and "photodiode group" are used interchangeably in the application of detecting light. It should be understood that reference to one of these terms can replace any other of these terms.
[0101] Figure 1 A schematic diagram of a plurality of sensor units 100a to 100n defining a sensor array 200 is shown. Each sensor unit 100 of the sensor array 200 includes a local control module 102, a plurality of sensor elements 104, and a memory module 106. The local control module 102 has three inputs: a global feedback signal 108, a local feedback signal 110, and a global clock signal 112. The local control module 102 is configured to control the function of the sensor element 104. The sensor element 104 is configured to detect or otherwise sense a particular physical signal and provide an output to the memory module 106. The memory module 106 is configured to provide a sensor unit output 114, which can then be stored in a computer or memory of the sensor array 200.
[0102] The sensor array 200 is built on a substrate, which is typically composed of some bulk material suitable for supporting all the electronic components of the sensor (including also all necessary signal and power lines). The specific characteristics of the substrate depend on the manufacturing technology.
[0103] In operation, the global control module 202 of the sensor array 200 is configured to provide global control of the sensor units 100a to 100n. This global level of control is supplemented by local level control at each of the sensor units 100a to 100n provided by the local control modules 102. The global control module 202 is configured to provide a global feedback signal 108 to each sensor unit 100, and the local control modules 102 are configured to locally control each respective sensor unit 100 using the local feedback 110.
[0104] The global control module 202 is configured to control / regulate the operation of each sensor unit 100a to 100n in cycles. Each cycle is started by a first pulse signal generated by the global control module 202. The first pulse signal is carried on the same line as the global clock signal 112 and is sent to each local control module 102 of each sensor unit 100a to 100n to start the sensor capture phase. When the local control module 102 receives the first pulse, the local control module 102 sends a first control signal to the sensor element 104. The first control signal causes the sensor element 104 to leave the reset / standby / low power state and enter the detection / activation state to start the sensor capture phase.
[0105] When the end of the sensor capture phase is reached, the cycle is complete. This is notified to the sensor units 100a to 100n by the global control module 202, which writes a second pulse on the same line carrying the global clock signal 112 to each local control module 102. Data is then collected from the sensor elements 104 via the sensor unit outputs 114 during the sensor readout phase. The second pulse triggers the local control modules 102 to send a reset / stop signal to the sensor elements 104, forcing them into a low power / no operation state until a new cycle is started.
[0106] In some embodiments, once it is determined that the optimal sensor state is reached by satisfying certain conditions / criteria in each individual sensor unit 100, the memory module 106 of the sensor unit 100 is configured to send an end signal using the same line used to send the local feedback signal 110 back to the control module 102 of the sensor unit 100. The end signal triggers the local control module 102, which sends a reset / stop signal to the sensor elements 104, forcing them to enter a low power / no operation state until a new cycle is started. The satisfaction of the conditions / criteria is explained in more detail below.
[0107] Each sensor element in the plurality of sensor elements 104 may be a sensor element of the same type. For example, each of the plurality of sensor elements 104 may be an optical sensor element, whereby each optical sensor element is sensitive to visible light of different wavelengths, is designed to be used in the presence of strong or weak light signals, which may provide fast or slow light-to-voltage conversion schemes, is optimized to reduce electromagnetic or thermal noise, etc. In another example, one or more of the plurality of sensor elements 104 may be an optical sensor element sensitive to wavelengths in the visible wavelength range, and / or one or more of the plurality of sensor elements 104 may be a sensor element sensitive to wavelengths in different wavelength ranges in the electromagnetic spectrum, such as the infrared range of the electromagnetic spectrum. It should be understood that sensor elements sensitive to the gamma, x-ray, ultraviolet, microwave, and radio wave ranges of the electromagnetic spectrum may also be employed when such sensors are technically compatible and may be implemented on the same substrate.
[0108] Alternatively, plurality of sensor elements 104 may each include different groups of sensor elements that are each configured to detect a different type of signal or a different type of information to extract from an incoming signal.
[0109] Combinations of optical / electromagnetic radiation sensors, thermal sensors, pressure sensors, chemical sensors, etc. may be used together in multiple sensor elements 104 when the following may be implemented on the same substrate.
[0110] like Figure 1The sensor units 100a to 100n shown may be identical sensor units 100, including the same plurality of sensor elements 104. Alternatively, one or more of the sensor units 100a to 100n may include different sensor units having different combinations of sensor elements 104 as described above. This allows different sensor units 100a to 100n in the sensor array 200 to perform different functions and tasks. Each of the sensor units 100a to 100n is at least partially controlled locally by the local control module 102 according to the specific functions of each of the sensor units 100a to 100n.
[0111] Since the sensor units 100a to 100n are at least partially controlled locally by the control module 102 of each sensor unit 100, the sensor units 100a to 100n are configured to operate in an asynchronous manner, depending on the functionality of each sensor unit 100a to 100n.
[0112] The functionality of the sensor units 100a to 100n will now be described. As described above, the operation of the sensor unit 100 is based on the repetition of a sequence of actions arranged in cycles. Each cycle begins with a first pulse signal carried on the same line as the global clock signal 112 to initiate the sensor capture phase. When the local control module 102 receives the first pulse, the local control module 102 sends a first control signal to the sensor element 104. The first control signal causes the sensor element 104 to leave the reset / standby / low power state and enter the detection / activation state.
[0113] In the detection state, sensing begins, and the sensor element 104 is configured to provide a sensing output to the memory module 106. The sensor element 104 is configured to detect a physical signal, for example, by converting the physical signal into an electrical signal. How each sensor element of the sensor element 104 reacts to the physical signal will vary according to their specific characteristics. The electrical signals generated by each sensor element 104 together form a sensing output, which is continuously monitored by the memory module 106. For example, the electrical signal can be a voltage or a current.
[0114] The memory module 106 is configured to provide a local feedback signal 110 to the local control module 102, whereby the local feedback signal 110 is based on the sensed output from the sensor element 104. The local feedback signal 110 may be equivalent to the sensed output, which means that the memory module 106 simply passes the sensed output from the sensor element 104 back to the local control module 102 as the local feedback signal 110. The memory module 106 sends the local feedback signal 110 to the control module 102 using the electrical signal from the sensor element 104. The local feedback signal 110 therefore depends on the type, strength or characteristics of the physical signal that is converted into the electrical signal.
[0115] The global feedback signal 108 is generated by the global control module 202 and is also provided to the local control module 102. The control module 102 combines the local feedback signal 110 generated by the memory module 106 with the global feedback signal 108 generated by the global control module 202 to generate a second control signal. The second control signal is used to adjust the operating parameters and / or conditions (e.g., threshold voltages, control voltages, configuration signals, etc.) of the sensor element 104 and indicates to the memory module 106 when an optimal sensor state is reached or when the end of the sensor capture phase is reached so that data from the sensor element 104 can be captured and used to generate the sensor unit output 114. As described above, the global control module 202 indicates the end of the sensor capture phase by a second pulse on the line carrying the global clock signal 112. The end of the sensor capture phase can correspond to a maximum sensor exposure time, which is a programmable time interval.
[0116] How and when to generate the second control signal depends on the implementation of the local control module 102. Specifically, the local control module 102 uses the global feedback signal 108 and the local feedback signal 110 in calculations, comparisons, or other calculations to determine how or when to control the sensing element 104 and / or the memory module 106 with the second control signal. When the global feedback signal 108 and the local feedback signal 110 meet a certain condition or criterion based on the comparison or calculation, the local control module 102 generates the second control signal. Satisfaction of the condition or criterion indicates that the optimal sensor state has been achieved.
[0117] Once the condition or criterion is met, the second control signal is configured to cause the sensor element 104 and / or the memory module 106 to perform an action. For example, the second control signal causes the memory module 106 to generate a sensor unit output 114 for storing the sensor data generated by the sensor element 104. The sensor unit output 114 is then directed to, for example, a global memory or an external computer.
[0118] It should be understood that although the global clock signal 112 is described above as being transmitted to the local control module 102 via a line, the global clock signal 112 may be implemented via more than one line. For example, the global clock signal 112 may include a first line (i.e., a conductor) that carries a first pulse and a second pulse for switching the sensor unit 100 between a low power / non-operating state and an activated / detecting state, and one or more additional lines that define the parameters or type of activation to be initiated or performed by the sensor unit 100. References to the global clock signal 112 include both possibilities.
[0119] While the memory module 106 stores or captures the sensor data to produce the sensor unit output 114, the memory module 106 also stores or captures the global clock signal 112 or a subset thereof. In particular, the global clock signal 112 includes a timing signal V(t) that depends on time or a function thereof. Capturing or recording the value of the timing signal V(t) while capturing data from the sensor element 104 provides an indication of the time at which the data was captured. This time data as well as the sensor data is used to generate the sensor unit output 114. Thus, an external computer or global memory to which the sensor unit output 114 is directed will have access to the data captured from the sensor element 104 as well as the time of capture. Effectively, each sensor unit 100 uses the timing signal of the global clock signal 112 to indicate the local time at which the criteria / conditions for data capture are met at the local control module 102 of the sensor unit 100, and therefore indicates when to capture and store data from the sensor element 104. This mechanism enables optimization of sensing in terms of signal-to-noise ratio or any other quality metric used in the application based on local characteristics of the incident light rather than based on global parameters / characteristics (e.g., integration / exposure time) as in most CMOS sensors.
[0120] When the end of the sensor capture phase is reached, the cycle is complete. This is communicated to the sensor unit 100 by the global control module 202, which writes a second pulse on the line carrying the global clock signal 112 to the local control module 102, as described above.
[0121] In one example, the memory module 106 includes a sample and hold circuit that reads the electrical signal generated by each of the sensor elements 104 when the sensor elements 104 reach an optimal sensor state or when the end of the sensor capture phase is reached, stores the electrical signal, and makes it available via the sensor unit output 114.
[0122] Figure 2 Shown according to Figure 1 A specific implementation of the sensor unit 100. Figure 2, the sensor unit 100 is a photoreceptor unit 100. The photoreceptor unit 100 is a visual or imaging sensor unit. The photoreceptor unit 100 includes a local control module 102, a plurality of sensor elements, wherein the sensor elements are photosensitive elements 104a, and a memory module 106. The local control module 102 has three inputs: a global feedback 108, a local feedback 110, and a global clock 112. The local control module 102 is configured to control the function of the photosensitive element 104a. The photosensitive element 104a is configured to detect light or a component thereof and provide a sensing output to the memory module 106. The memory module 106 is configured to provide a photoreceptor unit output 114 for storage in a global memory or use in a computer, for example. Thus, Figure 2 The components of the photoreceptor unit 100 are Figure 1 , and the sensor element 104 is a photosensitive element 104a. However, it should be understood that different types of sensors may be used in place of the photosensitive element 104a. Therefore, the following description relates to all types of sensors and the photosensitive element 104a (sensor) specified below.
[0123] We will now describe Figure 2 104a and the memory module 106. The local control module 102 uses the global feedback signal 108, the global clock 112 and the local feedback 110 to control the operation of the photosensitive elements 104a and the memory module 106. The sensed output from the photosensitive elements 104a is a collection of voltages (PixOut) corresponding to a single voltage from each photosensitive element 104a. These voltages are sent to the memory module 106. The memory module 106 (which may be a sample and hold circuit) is configured to store the voltages when the conditions / criteria are met, as generally described above. The memory module 106 makes the stored voltages available as the photoreceptor unit output 114. This output can be read and sent to an output interface of a device such as a computer or external memory for final use.
[0124] The operation of the photoreceptor unit 100 is based on the above reference Figure 1The same repetition of the cycle discussed. Each cycle begins with a first pulse to start the image acquisition phase. When the local control module 102 receives the pulse, the local control module 102 sends a first control signal to the photosensitive element 104a. This causes the photosensitive element 104a to leave the reset / standby / low power state and enter the detection state. In the detection state, the photosensitive element 104a is configured to detect light by converting incident light into a time-varying voltage. For a photosensor such as the photosensitive element 104a, the varying voltage is typically represented by some discharge curve caused by the incident photons. The voltage generated by the photosensitive element 104a forms a sensing output and is continuously monitored by the memory module 106. The memory module 106 uses the voltage from the photosensitive element 104a to send a local feedback signal 110 to the control module 102. In some implementations, the local feedback signal 110 is proportional to or equal to the voltage generated by the photosensitive element 104a. The local feedback signal 110 and the global feedback signal 108 generated by the global control module 202 are provided to the control module 102. Then, depending on the specific implementation of the photoreceptor unit 100, the local feedback signal 110 and the global feedback signal 112 are used in comparisons or calculations to determine when a particular condition / criteria is met, indicating that an optimal sensor state is achieved. Once the condition / criteria is met, the local control module 102 generates a second control signal that is configured to control the photosensitive element 104a, and / or controls the memory module 106 to store data from the photosensitive element 104a by outputting a sensor unit output 114. The memory module 106 may be a sample and hold circuit, in which case the satisfaction of the condition / criteria causes the local control module 102 to control the sample and hold circuit to store a voltage corresponding to the sensed output of the photosensitive element 104a for use in generating the sensor unit output 114.
[0125] The sensor unit output 114 is then directed to, for example, a global memory or computer. While the memory module 106 stores the voltage values, the memory module 106 also stores or captures the global clock signal 112 or a subset thereof. In particular, the global clock signal 112 includes a timing signal V(t) that depends on or is a function of time. Capturing or recording the value of the timing signal V(t) while capturing data from the photosensitive element 104a provides an indication of the time at which the data was captured. This time data, as well as the voltage data, is used to generate the sensor unit output 114. Therefore, the computer or global memory to which the sensor unit output 114 is directed can access the voltage data (sensor data) as well as an indication of the time at which it was sampled.
[0126] The photoreceptor unit 100 may be formed as follows Figure 1One of several photoreceptor units 100 of a portion of the array 200 shown. Each photoreceptor unit output 114 from each of the plurality of these photoreceptor units 100 together forms an acquired image / frame. The acquired image / frame is transmitted out of the array 200 via output port lines 1 and 2. The global control module 202 of the array 200 is configured to control each photoreceptor unit 100 in a cyclic sequence. These cycles include an image acquisition phase as described above. At the end of the image acquisition phase for the entire array 200, an image readout operation is performed. The duration of the image acquisition phase is defined by a programmable time interval that fixes the maximum exposure time of each photoreceptor unit 100 and each photosensitive element 104a. The time interval is stored in a register of the global control module 202. The time required for the readout phase depends on the size of the array, the speed of the data transmission line, and other operating parameters. At the beginning of each cycle, the image acquisition phase is activated by the global control module 202 by generating a first pulse on the line of the global clock signal 112, as described above. When the first pulse is detected, each photoreceptor cell 100 starts converting the incident light in an independent and autonomous manner, so that the global control module 202 and the other photoreceptor cells 100a to 100n do not have to be directly involved in acquiring data from a specific photoreceptor cell 100. In order to inform all photoreceptor cells 100a to 100n that the image acquisition phase has ended (and the maximum exposure time has expired), the global control module 202 generates another second pulse for the global clock signal 112 on the line. At the beginning of the image acquisition phase, the global control module 202 also generates a global feedback signal 108 to be used by each photoreceptor cell 100. The global feedback signal 108 may be different for different types of photoreceptor cells 100. In particular, there may be more than one type of photoreceptor cell 100 in the same array 200. Upon completion of the image acquisition phase, the global control module 202 begins the readout phase, in which the voltage values stored by each memory module 106 of each photoreceptor unit 100 are used to generate a corresponding sensor unit output 114, which is sent to the output port (via Line 1 and Line 2). As described above, the voltage values stored by each memory module 106 reflect the data captured at the optimal state of each corresponding photoreceptor unit 100. By adding appropriate readout logic and circuits, the image acquisition phase and the readout phase can optionally be arranged according to a pipeline structure and executed in parallel to speed up the number of images / frames generated per unit time.
[0127] Furthermore, if the global control module 202 is equipped with appropriate processing circuitry, by reading / accessing the output values generated by each photoreceptor unit 100, it can regulate the behavior of the photoreceptor units 100a to 100n by generating a global feedback signal 108 at the beginning of each cycle. The global feedback signal 108 can be a static signal that does not change during the image acquisition phase, or it can be a variable signal that can change during the image acquisition phase. An example of a static global feedback signal is represented by Figure 3 and Figure 5 The global threshold voltages used by the photoreceptor cells in the image sensor array 200 are represented. These static signals are determined starting from the average illumination of the scene to be captured in order to minimize noise and improve resolution. The global control module 202 may also work in conjunction with an external computer, microcontroller, programmable logic, microprocessor, etc. to generate an optimal global feedback signal. In this case, the external computer unit monitors / reads / accesses the image / frame stream generated by the device including the photosensitive array 200, processes them, and generates an optimal global feedback signal, which is then stored in an internal memory unit of the global control module 202 and sent to the photoreceptors 100 according to a programmable time scale during the image acquisition phase.
[0128] The timing signal V(t) included as a component of the global clock signal 112 can also be considered a global feedback signal. The timing signal V(t) is discussed in more detail herein. The timing signal V(t) is a signal that can be used to measure the time of an event occurring in a circuit (as described above). The timing signal V(t) can also be considered a global feedback signal because it is generated and tuned by the global control module 202 to set an appropriate time resolution for the measured data of the photoreceptor unit 100 that is more appropriate to the characteristics of the observed scene. This will be explained in more detail later with reference to specific examples. However, in general, the timing signal V(t) included in the global clock signal 112 is an electrical signal that exhibits a known relationship to time during the image acquisition phase (the integration time period between the beginning and end of each acquisition cycle) so that by sampling the timing signal V(t), a simple inversion operation can be performed to calculate the time t at which the memory module 106 of a particular photoreceptor unit 100 recorded data. In the following example, the timing signal V(t) is implemented with a rising linear ramp that starts from a low value at the beginning of the image acquisition phase and reaches its maximum value at the end of the image acquisition phase. This shape of the timing signal V(t) is only for exemplary purposes, and in general any signal shape (linear or non-linear, increasing or decreasing, etc.) that allows the calculation of a time value (when sampling the timing signal V(t)) can be used. For this reason, the timing signal V(t) is the signal of the source of the measured event time. At the same time, the shape of the function of the timing signal V(t) can be modified to tune the resolution of the time measurement. In the case where the shape of the timing signal V(t) is chosen so that at a specific time t, a large change Δt in t corresponds to a small change ΔV in the timing signal V(t), the time resolution at this specific time is low because a small increase in the measured electrical signal timing signal V(t) is equal to a large elapsed time Δt. This means that when the timing signal V(t) is inverted to calculate t, any small noise on the timing signal V(t) is amplified. Vice versa, when the shape of the timing signal V(t) is chosen so that a small change Δt in time corresponds to a large change ΔV in the timing signal V(t), the time resolution is high because the large change ΔV is equal to the small period Δt, and in this case, when the timing signal V(t) is inverted to calculate t, the noise on the timing signal V(t) is attenuated. In an exemplary implementation, the timing signal V(t) included in the global clock signal 112 is described by a linear shape / function function, which means that the time resolution is constant over the integration period, while in other implementations, the function describing the timing signal V(t) is non-linear, which means that the time resolution can vary over the integration period and be more or less accurate at some times than at other times.
[0129] Other global feedback signals may be generated to improve the overall performance of a device including photosensitive array 200 based on global operating conditions (e.g., temperature and power consumption of the sensor). Figure 3 The comparator in Figure 5 The voltage sources of the components of the winner-take-all (WTA) circuit can be changed based on this feedback signal to improve speed, accuracy or power consumption. These circuits and figures are explained in more detail below. Figure 1 The local feedback signals generated by the memory module 106 in are combined with the global feedback signals to adjust the performance of each photoreceptor unit 100 .
[0130] The photosensitive array 200 may improve the signal-to-noise ratio and provide energy saving benefits, as will be discussed below.
[0131] Now refer to Figures 3 to 10 Specific examples of the photosensitive array 200 and the photoreceptor unit 100 are described.
[0132] Figure 3 is a schematic diagram illustrating a first exemplary implementation of a photoreceptor unit 100-1. In this implementation, the local control module 102 includes a comparison circuit module 102-1, the photosensitive element 104 is a group of photodiodes 104-1, including a red light sensitive photodiode 104-1a, a green light sensitive diode 104-1b, and a blue light sensitive diode 104-1c. The memory module 106 is a sample and hold circuit 106-1, the global feedback signal 108 is a global threshold 108-1, the local feedback signal 110 is the sum 110-1 of the sensor outputs from the photodiodes 104-1, and the global clock signal is a voltage signal V(t) 112-1 as a function of time. The sensor unit output 114 is not shown, but is also described below. The comparison circuit module 102-1 is a component of the local control module 102. It should be understood that Figure 3 Other circuits not shown in the figure may also be included in the local control module, for example, using the global clock signal 112 to control the local operation of the photoreceptor 100-1. The local control module 102 can be configured to locally generate a reset signal from the global clock signal for initiating a reset of the photodiode 104-1. The local control module 102 can also generate a "store" signal for sending an instruction to capture and store data at the sample and hold circuit 106-1 when the integration phase has expired (as indicated by the global clock signal 112).
[0133] Figure 3Also shown is a circuit 150-1 that can be used during the readout phase. Specifically, when a particular photoreceptor cell 100 in position (I, j) of the sensor array 200 must be read, the select SEL signal corresponding to that photoreceptor cell 100 is asserted, causing the corresponding line buffer (at position (i, j)) connected to the supply voltage VDDR to be read. Figure 3 The Rth, Gth, Bth and Vth signals of the photosensitive cells (shown as transistors in FIG. 2 ) are copied to corresponding bit lines blr, big, bib and blv. These bit lines are grouped with the bit lines of other photosensitive cells of the same array 200 and are usually arranged in a column.
[0134] In accordance with Figure 3 In operation of the first exemplary implementation of the photoreceptor unit 100 of FIG. 1 , during an image acquisition phase, the voltages of the red photodiode 104-1a, the green photodiode 104-1b, and the blue photodiode 104-1c are sampled at the sample and hold circuit 106-1 and summed together at a summing node to form a summation 110-1 that at least partially represents a local feedback signal. The sum 110-1 of the three voltages from the photodiodes 104-1 is fed to the comparison circuit module 102-1. The global threshold 108-1 is also fed to the comparison circuit module 102-1. The comparison circuit module 102-1 compares the sum 110-1 to the global threshold 108-1 using a comparator. Once the sum 110-1 reaches or crosses the global threshold 108-1, the comparison circuit module 102-1 proceeds along the Figure 3 The line 116-1 shown transmits the second control signal mentioned above. The second control signal controls the sample and hold circuit 106-1 to store the current value of the voltage of the red photodiode 104-1a, the green photodiode 104-1b and the blue photodiode 104-1c, and the voltage value of the timing signal V(t) 112-1. This is done, for example, by turning on Figure 3 The switches in the sample and hold circuit 106-1 are shown to store the voltage value of each corresponding signal. Therefore, when the global threshold 108-1 is crossed, the action of the comparison circuit module 102-1 is responsible for initiating the capture of sensor data. In one example, Figure 3 The local feedback signal in the case of the summed value 110-1 is the discharge curve of the voltage that decreases during the integration phase. Therefore, when this discharge curve is equal to or below the global threshold 108-1, the comparator changes its output.
[0135] With respect to the general description above, the condition / criterion is that the sum 110-1 exceeds the global threshold 108-1. As described above, the sample and hold circuit 106-1 is also controlled by a global signal generated by the global control module 202, or by a local signal generated upon receipt of a second pulse indicating the end of the image acquisition phase (i.e., the integration phase) via a line carrying the global clock signal 112 from the global control module 202. This local signal is generated to override the output of the comparator in the comparison circuit module 116-1 to store the voltages of the three photodiodes 104-1 and the timing signal V(t) at the end of the image acquisition phase, even if the sum 110-1 did not cross the global threshold 108-1 during the image acquisition phase.
[0136] Thus, when the sum 110 - 1 crosses the global threshold 108 - 1 , or when the end of the image acquisition phase is reached, the sample and hold circuit 106 - 1 is instructed to store and capture the voltage value of each photodiode 104 - 1 .
[0137] When the sample and hold circuit 106-1 captures the voltage data and timing signal V(t) 112-1 of each photodiode 104-1, it can also be configured to send an end signal using the same line used to send the local feedback signal 110-1 back to the comparison circuit module 102-1 and related circuits of the sensor unit 100. The end signal triggers the local control module 102 including the comparison circuit module 102-1 and related circuits to send a reset / stop signal to the photodiodes 104-1, forcing them to enter a low power / no operation state until a new cycle begins in response to the first pulse sent by the global control module 202.
[0138] We will now explain in more detail Figure 3 First, during the low power / off state, or in other words, during the reset phase (RES=H), the photodiode 104-1 is pre-charged to a voltage V r Once the first control signal is provided to the photodiode 104-1 after resetting, in order to start the image acquisition phase, the photodiode 104-1 starts to integrate the incident light, so that the voltage across each photodiode 104-1 decreases almost linearly with the intensity of the light impinging thereon, as described in the following expression:
[0139] V(k) = (l K / C K )t; where i = R, G, B; (1)
[0140] Among them C k is the reverse-biased photodiode capacitance, I kis the photogenerated current, and t represents the photodiode exposure / integration time.
[0141] After the reset phase, and therefore at the start of the image acquisition phase, a timing signal V(t) forming part of the global clock signal 112-1 is activated, keeping track of the time elapsed since the start of the image acquisition phase. The voltages across the three photodiodes 104-1 are monitored by a comparison circuit module 102-1 which calculates the sum 110-1 (S) of the three voltages received from each of the red photodiode 104-1a, green photodiode 104-1b and blue photodiode 104-1c as shown in (2) and compares it to a global threshold 108-1 (TH):
[0142] s = (V r -V R ) + (V r – v G ) + (V r -v B ). (2)
[0143] Where V R 、V G , and V B are the sampled voltages of the red 104-1a, green 104-1b, and blue 104-1c photodiodes, respectively. During the exposure / integration time, the value of S increases from zero because the voltage at each photodiode 104-1 decreases over time as it discharges. At time T0, when the sum 110-1 (S) reaches the global threshold 108-1 (TH), the sample and hold circuit 106-1 controlled by the comparison circuit module 102-1 stores the voltage signals of the red photodiode 104-1a, the green photodiode 104-1b, and the blue photodiode 104-1c together with the voltage signal V(t) 112-1. At the end of the operation, the photoreceptor unit 100 transmits three signals Rth, Gth, and Bth associated with the red 104-1a, green 104-1b, and blue 104-1c photodiodes, respectively. These values are normalized relative to the global threshold 108-1 TH:
[0144] R(T0) + G(T0) + B(T0) = Rth + Gth + Bth = TH. (3)
[0145] The fourth voltage signal Vth is related to the time (T0) at which the sample and hold circuit 106-1 stores the signal. In this exemplary implementation, the global threshold 108-1 (TH) acts as a global signal to set the local exposure time of the photodiode and therefore the local exposure time of the pixels of the array 200, while the local parameter is represented by the sum 110-1 (S). The second global signal is the timing signal V(t), the signal value of which is stored in the sample and hold circuit 106-1 as T0 and represents the voltage ramp / curve of the pixel timestamp.
[0146] Once the image acquisition phase is complete, the values stored at the sample and hold circuit 106-1 are output via the sensor cell output 114 and read out in the readout phase. The readout phase obtains all of the sensor cell outputs 114 from each photoreceptor cell 100a to 100n so that, for the entire scene, the RGB values from each pixel in the array 200 are available for post-processing or other end use. In addition to the RGB values, a value of V(T0) is also obtained, representing the time T0 at which the global threshold 108-1 (TH) was passed for each corresponding photoreceptor cell 100. Knowing V(T0), each set of RGB values corresponding to the photoreceptor cell 100 can be used to calculate the absolute RGB value of each photoreceptor cell 100. The absolute RGB value can be calculated using the following formula:
[0147] ΔV x * T i / T0 Where ΔV x = (V r -V x )(4)
[0148] Where x = R, G, or B, such that Vx corresponds to an RGB voltage value obtained from the sensor unit output 114, and where Ti is the global integration / exposure time. T0 can be calculated by inverting the timing signal V(T0) as described above. This allows the array 200 and any device that includes it to discern useful color information in both bright and dark areas of a scene. For example, in a dark area of a scene captured by the first photoreceptor unit 100, the value of the timing signal V(T0) can be relatively large, meaning that it takes a longer amount of time T0 for the sum 110-1(S) to reach the global threshold 108-1(TH). In a bright area of the same scene captured by the second photoreceptor 100, the value of the timing signal V(T0) can be relatively small, meaning that it takes a shorter amount of time T0 for the sum 110-1(S) to reach the global threshold 108-1(TH). By normalizing these values using the different values of T0 returned by each photoreceptor in the sensor array, an image can be obtained with RGB values that truly represents the full dynamic range of light conditions present in the scene even when very bright and very dark areas exist simultaneously.
[0149] Figure 4 The above reference shows Figure 3 1. A timing diagram of the process described herein is shown. In particular, after the reset phase, each of the red photodiode 104-1a, the green photodiode 104-1b, and the blue photodiode 104-1c is set to an initial voltage V(r). When the image acquisition phase begins, the timing signal V(t) carried by / on the line of the global clock signal 112-1 is initiated, and the voltage at each photodiode 104-1 begins to discharge in a substantially linear manner, as shown in Equation 1 above. Figure 4 At the same time, the sum 110-1(S) calculated according to the above equation 2 increases according to the sum of the voltage reduction amplitudes in the RGB voltage values. At time T0, the sum 110-1S reaches the global threshold 108-1TH. At this time (T0), the comparison circuit module 102-1 instructs the sample and hold circuit 106-1 to capture and store the R, G, and B voltage values from the red 104-1a, green 104-1b, and blue 104-1c photodiodes, respectively. These values are given by Figure 4 At the same time, the sampling and holding circuit 106-1 also stores the value of the timing signal V(t) at t=T0, as shown in FIG. Figure 4 Each of the values Rth, Gth, Bth and Vth from a portion of the sensor unit output 114 is then read out in a readout phase following the image acquisition phase.
[0150] Figure 5 1 is a schematic diagram showing a second exemplary implementation of a photoreceptor unit 100-2. In this implementation, the local control module 102 includes a comparison circuit module 102-2, the photosensitive element 104 is a group of photodiodes 104-2, including a red light photosensitive photodiode 104-2a, a green light photosensitive diode 104-2b, and a blue light photosensitive diode 104-2c. The memory module 106 is a sample and hold circuit 106-2, the global feedback signal 108 is a global threshold 108-2, the local feedback signal 110 is a winner-take-all signal 110-2 of the sensor output from the photodiode 104-1, and the timing signal V(t) 112-2 is a signal carried by a global clock signal line and is a function of time. The sensor unit output 114 is not shown, but is also described below.
[0151] Figure 5 Also shown is a circuit 150-2 that can be used during the readout phase. Specifically, when a particular photoreceptor cell 100 in position (I, j) of the sensor array 200 must be read, the select SEL signal corresponding to that photoreceptor cell 100 is asserted, causing the corresponding line buffer (at position (i, j)) connected to the supply voltage VDDR to be read. Figure 5 The Rth, Gth, Bth and Vth signals of the photosensitive cells (shown as transistors in FIG. 2 ) are copied to corresponding bit lines blr, big, bib and blv. These bit lines are grouped with the bit lines of other photosensitive cells of the same array 200 and are usually arranged in a column.
[0152] In accordance with Figure 5 In operation of the second exemplary implementation of the photoreceptor unit 100 of FIG. 1 , during the image acquisition phase, the voltages of the red photodiode 104-2a, the green photodiode 104-2b, and the blue photodiode 104-2c are sampled at the sample and hold circuit 106-2, and a winner-take-all operation is performed at the winner-take-all node (WTA), wherein the signal from the photodiode 104-2 having the largest voltage change from its initial reset voltage V(r) is passed to the comparison circuit module 102-2. The "winning" voltage signal forms the local feedback signal 110-2. Therefore, the local feedback signal 110-2 is effectively equivalent to the voltage signal of one of the red photodiode 104-2a, the green photodiode 104-2b, and the blue photodiode 104-2c that exhibits the largest change since reset. This local feedback signal 110-2 is fed to the comparison circuit module 102-1. The global threshold 108-2 is also fed to the comparison circuit module 102-2. The comparison circuit module 102-2 uses a comparator to compare the winner-take-all signal 110-2 with the global threshold 108-2. As described above, the WTA node is a circuit that compares the voltage signals of the three photodiodes 104-2 and sends or copies the photodiode voltage signal with the largest change (i.e., the voltage of the photodiode that discharges faster) to the output of the WTA node, and this forms the local feedback signal 110-2. The combination of the WTA node and the comparator 102-2 is equivalent to a circuit that compares all input signals with the global threshold 108-2 at the same time.
[0153] Once the output of the winner-take-all 110-2 reaches or crosses the global threshold 108-2, the comparison circuit module 102-2 follows Figure 5 The line 116-2 shown transmits the second control signal mentioned above. The second control signal controls the sample and hold circuit 106-2 to store the current value of the voltage of the red photodiode 104-2a, the green photodiode 104-2b and the blue photodiode 104-2c, and the voltage value of the voltage signal V(t) 112-2. This is done, for example, by turning on Figure 5The switches in the sample and hold circuit 106-2 are shown to be done to store the voltage value of each corresponding signal. Therefore, when the global threshold 108-2 is crossed, the action of the comparison circuit module 102-2 is responsible for initiating the capture of sensor data. With respect to the general description above, the condition / criterion is that the winner takes all local feedback signal 110-2 exceeds the global threshold 108-2. In addition to this, Figure 5 The features of the second exemplary implementation shown are similar to Figure 3 The features of the first exemplary implementation shown are similar or identical. This is also true with respect to the fact that the sample and hold circuit 106-2 also forms part of the local control module 102. The local control module 102 is controlled by a global signal generated by the global control module 202 of the array 200, or by a local signal generated when a second pulse generated by the global control module 202 is received locally via the global clock line to override the output of the comparator of the comparison circuit module 102-2 and store the voltages of the three photodiodes and the timing signal V(t) at the end of the image acquisition phase if the "win" signal 110-1 does not cross the threshold 108-2 during the image acquisition phase.
[0154] When the sample and hold circuit 106-2 captures the voltage data and voltage signal V(t) 112-2 of each photodiode 104-2, it can also be configured to send an end signal back to the local control module 102 including the comparison circuit module 102-2 and the associated circuits of the sensor unit 100 using the same line used to send the local feedback signal 110-2. The end signal triggers the local control module 102, the comparison circuit module 102-2 and the associated circuits to send a reset / stop signal to the photodiodes 104-2, forcing them to enter a low power / no operation state until a new cycle begins.
[0155] We will now explain in more detail Figure 5 First, during the low power / off state, or in other words, during the reset phase (RES=H), the three photodiodes 104-2 are pre-charged to a voltage V r After the reset is released from the first control signal, the image acquisition phase begins. This causes the three photodiodes 104-2 to begin integrating light, as described above with reference to Figure 4 and Figure 5As explained. The voltage across each photodiode 104-2 begins to drop almost linearly with the intensity of light impinging upon it. After reset, the voltage signal V(t) 112-2 is activated, keeping track of the time elapsed from the end of the reset phase and therefore from the beginning of the image acquisition phase. The comparison circuit module 102-2 monitors the three photodiode voltages during the exposure time. Once one of the three signals (R, G, B voltages) reaches the global threshold 108-2 (Vth), the voltage signals R, G, B of the red photodiode 104-2a, the green photodiode 104-2b, and the blue photodiode 104-2c are stored by the sample and hold circuit 106-2 together with the voltage signal 112-2V(t). Figure 5 The photoreceptor cell 100 shown implements a winner-take-all (WTA) operation among the three photodiodes 104-2. This means that the brightest photodiode is the first photodiode to reach the global threshold 108-2 (Vth). The other two non-winner voltage signals avoid saturation because their voltage levels are always greater than TH. The three signals (Rth, Gth, Bth) from the photodiodes 104-2 are stored and used to form the sensor cell output 114. These signals are not normalized. In particular, it is assumed that R (red) is the brightest signal and Rth is equal to the global threshold 108-2 (TH). The pixel timestamp is represented by Vth. As shown in Figure 3 As in the first exemplary implementation, using the knowledge of V(T0), i.e., the timing signal V(t) when the WTA output signal (local feedback signal 110-2) reaches the global threshold 108-2, each set of RGB values corresponding to the photoreceptor unit 100 can be used to calculate the absolute RGB value of each photoreceptor unit 100. In fact, the absolute RGB value can be calculated using Equation 4 as described above.
[0156] Figure 6 The above reference shows Figure 5 1. In particular, after the reset phase, each of the red photodiode 104-2a, the green photodiode 104-2b, and the blue photodiode 104-2c is set to an initial voltage V1. When the image acquisition phase begins, the global clock signal represented by the timing signal V(t) is started, and the voltage at each photodiode 104-2 begins to discharge in a substantially linear manner according to equation 1 above, as Figure 6 As shown in the R, G and B lines in FIG. At time T0, the winning signal reaches the global threshold 108-2 TH. Figure 6In the example of , the winning signal is the red signal R. At this time (T0), the comparison circuit module 102-2 instructs the sample and hold circuit 106-2 to capture and store the R, G, and B voltage values from the red photodiode 104-2a, the green photodiode 104-2b, and the blue photodiode 104-2c, respectively. These values are represented by Figure 6 At the same time, the sampling and holding circuit 106-1 also stores the value of V(t) at t=T0, as shown in Figure 6 Each of the values Rth, Gth, Bth and Vth then comes from a portion of the sensor cell output 114 that is read out in a readout phase.
[0157] Compared with standard techniques, Figure 5 and Figure 6 The second exemplary implementation of the photoreceptor cell 200 shown is advantageous because it prevents the photodiodes 104-2 from experiencing saturation through a local mechanism regardless of their illumination level. Figure 6 This feature is described in a. Figure 6a shows an example of two photoreceptors 100-2 of the same type with the same chrominance components but under different intensity conditions. In particular, the first of the two photoreceptors is brighter than the second photoreceptor. The first photoreceptor is associated with values R1th, G1th, and B1th, while the second photoreceptor is associated with values R2th, G2th, and B2th. In the first photoreceptor and the second photoreceptor, R is the main chrominance component, and such R value of each of the first photoreceptor and the second photoreceptor first reaches the threshold value TH by the winner-takes-all. This triggers the sample and hold circuit to store the signals: R1th, G1th, B1th, and the timing signal V(T0) for the first photoreceptor and R2th, G2th, B2th, and the timing signal V(T1) for the second photoreceptor. In this case, the second receiver has a time taken for the RGB value to reach the threshold value TH that is greater than the time taken by the first photoreceptor, so that T1>T0. In other words, the first photoreceptor is subjected to brighter light than the second photoreceptor. The ratio T1 / T0 is proportional to the ratio of the light intensities impinging on the two photoreceptors. Furthermore, it can be seen that the relationship between the two triplets R1th, G1th, B1th and R2th, G2th, B2th is the same. This means that the chromaticity information of the first photoreceptor and the second photoreceptor is preserved regardless of the different light intensities encoded by V(T0) and V(T1) of the first photoreceptor and the second photoreceptor. Therefore, even if the first photoreceptor is subjected to much brighter light than the second photoreceptor, RGB information can be easily obtained for both photoreceptors, making it easy to compare the true color information for both photoreceptors. Therefore, only four signals R, G, B, V(T) are needed to retrieve the absolute information of each photoreceptor over a wide dynamic range.
[0158] Figure 71 is a schematic diagram showing a third exemplary implementation of a photoreceptor unit 100-3. In this implementation, the local control module 102 includes a comparison circuit module 102-3, and the photosensitive element 104 is a group of photodiodes 104-3, including: a red light sensitive photodiode 104-3a, a green light sensitive diode 104-3b, a blue light sensitive diode 104-3c, and another photodiode 104-3d sensitive to white light. The memory module 106 is a sample and hold circuit 106-3, the global feedback signal 108 is a global threshold 108-3, and the global clock signal is a voltage signal V(t) 112-3 as a function of time. The sensor unit output 114 is not shown, but is also described below. The third exemplary implementation is different from the first and second exemplary implementations described above in that the local feedback signal 110 is not sampled or taken directly from any of the red light sensitive photodiode 104-3a, the green light sensitive diode 104-3b, or the blue light sensitive diode 104-3c. Instead, another photodiode 104-3d is used. The other photodiode 104-3d is sensitive to white light, and its voltage is sampled to form the local feedback signal 110-3.
[0159] Figure 7 Also shown is a circuit 150-3 that can be used during the readout phase. Specifically, when a particular photoreceptor cell 100 in position (I, j) of the sensor array 200 must be read, the select SEL signal corresponding to that photoreceptor cell 100 is asserted, causing the corresponding line buffer (at position (i, j)) connected to the supply voltage VDDR to be read. Figure 7 The Rth, Gth, Bth and Vth signals of the photosensitive cells (shown as transistors in FIG. 2 ) are copied to corresponding bit lines blr, blg, blb and blv. These bit lines are grouped with the bit lines of other photosensitive cells of the same array 200 and are usually arranged in a column.
[0160] The operation of the third exemplary implementation of the photoreceptor unit 100-3 is the same as the operation of the first and second exemplary implementations described above, except that there is a white light photodiode 104-3d. During the image acquisition phase, the voltages of the red photodiode 104-3a, the green photodiode 104-3b, and the blue photodiode 104-3c are not directly sampled, but only the white photodiode 104-3d is sampled to provide a local feedback signal. The voltage 110-3 of the white light photodiode 104-3d is fed to the comparison circuit module 102-3. The global threshold 108-3 is also fed to the comparison circuit module 102-3. The comparison circuit module 102-3 uses a comparator to compare the white light photodiode 104-3d signal 110-3 with the global threshold 108-3. Once the white light photodiode 104-3d signal 110-3 reaches or crosses the global threshold 108-3, the comparison circuit module 102-3 follows the Figure 3 The line 116-3 shown transmits the above-mentioned second control signal. The second control signal controls the sample and hold circuit 106-3 to store the current values of the voltages of the red photodiode 104-3a, the green photodiode 104-3b and the blue photodiode 104-3c, and the voltage value of the voltage signal V(t) 112-3. Alternatively, the voltage value of the white light photodiode 104-3d is also stored. This is done, for example, by turning on Figure 7 The switches in the sample and hold circuit 106-3 shown are completed to store the voltage value of each corresponding signal. Therefore, when the global threshold 108-3 is crossed, the action of the comparison circuit module 102-3 is the action responsible for initiating the capture of sensor data. With respect to the general description above, the condition / criteria is that the white light photodiode 104-3d signal 110-3 passes the global threshold 108-3. Similar to the first exemplary implementation and the second exemplary implementation described above, the sample and hold circuit 106-3 is also controlled by the global control signal generated by the global control module 202, or by the local signal generated at the end of the image acquisition phase (end of the integration phase) in response to the second pulse generated by the global control module 202. As described above, the second pulse is received locally through the global clock line to override the output of the comparator 116-3 and store the voltages and timing signals V(t) of the three photodiodes at the end of the acquisition phase, even if the white signal 110-3 did not cross the threshold 108-2 during the image acquisition phase.
[0161] When the sample and hold circuit 106-3 captures the voltage data and voltage signal V(t) 112-3 of each photodiode 104-3, it can also be configured to send an end signal using the same line used to send the local feedback signal 110-3 back to the comparison circuit module 102-3 and related circuits of the sensor unit 100. The end signal triggers the comparison circuit module 102-3 and related circuits to send a reset / stop signal to the photodiodes 104-3, forcing them to enter a low power / no operation state until a new cycle begins.
[0162] During the reset phase (RES=H) of the photoreceptor 100 - 3 , the four photodiodes 104 - 3 including the white light photodiode 104 - 3 d are precharged to a voltage V r . After reset has been released (RES=L), the four photodiodes 104-3 begin to integrate light, and the voltage across each photodiode begins to drop almost linearly with the intensity of light impinging upon it, as described above. The spectral response of the white light photosensitive diode 104-3d can be approximated as the integral of the spectral responses of the red light sensitive photodiode 104-3a, the green light sensitive diode 104-3b, and the blue light sensitive diode 104-3c. Therefore, it can be assumed that the voltage of the white photosensitive diode 104-3d reaches the global threshold 108-3TH when the sum of the signals of the red photodiode 104-3a, the green photodiode 104-3b, and the blue photodiode 104-3c will also reach the global threshold 108-3TH. Therefore, the third exemplary implementation is similar to the above reference Figure 3 The first exemplary implementation is described, except that this third implementation attempts to replicate the operation of the first implementation by using white light photodiodes as approximate "physical adders" of the red, green, and blue components of the incident light without directly sampling the red photodiode 104-3a, the green photodiode 104-3b, and the blue photodiode 104-3c. Finally, as in the first and second exemplary implementations, using knowledge of V(T0), i.e., V(t) when the white signal 110-3 reaches the threshold 108-3, the absolute RGB value of each photoreceptor cell 100 can be calculated according to Equation 4 using each set of RGB values corresponding to the photoreceptor cell 100-3.
[0163] Figure 8 The third exemplary implementation method and Figure 71. A timing diagram of the process described herein. In particular, after the reset phase, each of the red photodiode 104-3a, the green photodiode 104-3b, the blue photodiode 104-3c, and the white photodiode 104-3d is set to an initial voltage V(r). When the image acquisition phase begins, the global clock signal represented by the voltage signal V(t) is started, and the voltage at each photodiode 104-3 begins to discharge in a substantially linear manner according to equation 1 above, as Figure 8 As shown in the R, G, B and W lines in FIG. At time T0, the white 104-3d photodiode signal 110-3 reaches the global threshold 108-3TH. At this moment (T0), the comparison circuit module 102-3 instructs the sample and hold circuit 106-3 to capture and store the R, G and B voltage values from the red 104-3a, green 104-3b and blue 104-3c photodiodes, respectively. These values are represented by Figure 8 At the same time, the sampling and holding circuit 106-1 also stores the value of V(t) at t=T0, as shown in Figure 8 Each of the values Rth, Gth, Bth and Vth then comes from a portion of the sensor cell output 114 that is read out in a readout phase.
[0164] Using white light photodiodes 104-3d instead of red 104-3a, green 104-3b, and blue 104-3c photodiodes can be advantageous because the red 104-3a, green 104-3b, and blue 104-3c photodiodes are not subject to any manipulation in the local feedback loop before being stored by the sample and hold circuit 106-3 to form part of the sensor unit output signal 114. In fact, the red 104-3a, green 104-3b, and blue 104-3c photodiodes are not subject to any manipulation in the local feedback loop before being stored by the sample and hold circuit 106-3 to form part of the sensor unit output signal 114. Figure 5 WTA module in or Figure 3 This can result in better accuracy of the stored values of the signals produced by red photodiode 104-3a, green photodiode 104-3b, and blue photodiode 104-3c, and therefore a more accurate representation of the RGB values of the scene.
[0165] Fig. 9is a schematic diagram showing a third exemplary implementation of the photoreceptor unit 100-4. This fourth implementation is actually a combination of features from the first exemplary implementation, the second exemplary implementation, and the third exemplary implementation described above. In the fourth exemplary implementation, the local control module 102 includes a first comparison circuit module 102-4a and a second comparison circuit module 102-4b. Although drawn separately, it should be noted that the first comparison circuit module 102-4a and the second comparison circuit module 102-4b can be implemented in the same circuit block portion of the local control module 102. The photosensitive element 104 is a group of photodiodes 104-4, including: a red light photosensitive photodiode 104-4a, a green light photosensitive diode 104-4b, a blue light photosensitive diode 104-4c, and a high-sensitivity white light photosensitive diode 104-4d. The white light photosensitive diode 104-4d is a broad spectrum photosensor. The memory module 106 includes a sample and hold circuit 106-4.
[0166] The first comparison circuit module 102-4a of the local control module 102 operates in the same manner as described with reference to the first exemplary implementation and the second exemplary implementation described above. In particular, the first local feedback signal 110-4a from the sample and hold circuit 106-4 includes the sum of the RGB voltage values from the photodiode 104-4 (as in the first exemplary implementation), or a winner-takes-all WTA signal including the RGB values (as in the second exemplary implementation). The first local feedback signal 110-4a is compared with the first global threshold 108-4a at the comparator in the first comparison circuit module 102-4a. When the first global threshold 108-4a is met, the first comparison circuit module 102-4a of the local control module 102 is configured to instruct the sample and hold circuit in 106-4 to store the first value of the R, G, B signals and the global timing signal 112-4a. Fig. 9 In FIG. 1 , these values are represented by the stored RGB values Rth, Gth and Bth and the stored first timing signal value Vth1.
[0167] The second comparison circuit module 102-4b operates in the same manner as described with reference to the third exemplary embodiment. In particular, the second local feedback signal 110-4b is the value of the white light photodiode 104-4d. This value is compared to the second global threshold 108-4b at a comparator included in the second comparison circuit module 102-4b portion of the local control module 102. The white light photodiode can be a high sensitivity "white" photodiode, such as a pinned photodiode or a single photon detector or other CMOS compatible high sensitivity photodetector. When the second global threshold 108-4b is met, the second comparison circuit module 102-4b of the local control module 102 is configured to instruct the sample and hold circuit 106-4 to store the W value and the second value of the timing signal 112-4a. Fig. 9 In , these values are represented by the stored W value Wth and the stored second timing signal value Vth2.
[0168] It should therefore be understood that more than one global threshold may be used to compare the data for each photoreceptor unit 100 - 4 .
[0169] The operation of the fourth exemplary implementation is the same as the first, second and third exemplary implementations described above. At the end of the image acquisition phase (integration phase), the sample and hold circuit 106-4 stores or stores the values of the red, green and blue photodiode voltages Rth, Bth and Gth together with the first value Vth1 of the timing signal V(t) 112-4a, which indicates the time when the first local feedback signal 110-4a corresponding to the combination of the R, G, B signals (subject to summation or WTA) crosses the first global threshold 108-4a. Alternatively, if the first global threshold 108-4a is not crossed during the allocated maximum exposure time period, the first value Vth1 of the timing signal 112-4a will indicate the time when the image acquisition phase ends (end of the integration phase). The sample and hold circuit 106-4 will also store or have stored the value Wth of the voltage of the white light photosensitive photodiode 104-4d together with the second value Vth2 of the timing signal V(t) 112-4a when the second local feedback signal 110-4b corresponding to the voltage of the white light photosensitive diode crosses the second global threshold 108-4b, or at the end of the integration phase (if the second global threshold 108-4b is not crossed before the end of image acquisition).
[0170] The sample and hold circuit will eventually store six values: the R, G, B signals Rth, Gth, Bth and W signal Wth of the four photodiodes 104-4 and 104-4d, the first value Vth1 of the timing signal 112-4a and the second value Vth2 of the timing signal 112-4a. These values can be transmitted out of the photoreceptor unit 100-4 as the photoreceptor unit output during the readout phase and can be used, for example, in an external computer or memory.
[0171] Typically, the readout circuit is configured to transmit the photoreceptor output to an external unit / device / application and reads all photoreceptor cells 100-4 of the sensor array 200 sequentially or with some degree of parallelism and sends the data to the external application using appropriate I / O channels connected to the photoreceptor cells 100-4, in particular to the sample and hold circuit 106-4.
[0172] A first example of a read block 300 is Fig. 9 . The readout block 300 includes a logic circuit having six inputs corresponding to the six values Rth, Bth, Gth, Wth, Vth1 and Vth2 obtained during the image acquisition phase. The readout block 300 can be implemented by one or more circuits designed and implemented to filter the six values Rth, Bth, Gth, Wth, Vth1 and Vth2 generated by each photoreceptor unit 100-4 to reduce the bandwidth required to transmit such data to an external computer, memory or application. The logic implemented in the readout block 300 can decide to send only the R, G, B photodiode values Rth, Gth, Bth with the corresponding first value Vth1 of the timing signal 112-4a V(t), or only the W photodiode value Wth with the corresponding second value Vth2 of the timing signal 112-4a V(t) based on the results of the comparison operation.
[0173] When all six values stored by the sample and hold circuit 106-4 are available, the readout block 300 is activated during the readout phase after the end of the image acquisition / integration phase. During this readout phase, the following process occurs in the readout block 300. First, if the second value Vth2 of the timing signal 112-4a (which defines the time when the second local feedback signal 110-4b corresponding to the voltage W of the white light photosensitive diode 104-4d reaches the second global threshold 108-4b, or defines the time when the white light photosensitive diode 104-4d reaches the end of the integration phase without crossing the second global threshold 108-4b) is lower than the third global threshold 108-4c, the plurality of switches 302 in the readout block 300 are configured to connect the corresponding output lines OutR, OutG, OutB and OutV to the input lines 304 corresponding to the R, G, B photodiode voltage values Rth, Bth and Gth stored by the sample and hold circuit 106-4 and the value Vth1 of the first timing signal 112-4a. Otherwise, if the second value Vth2 of the timing signal 112-4a is greater than the third global threshold 108-4c, the switches are configured to connect the output lines OutR, OutG, OutB to the input line 304 corresponding to the value Wth of the white light photodiode 104-4d stored by the sample and hold circuit 106-4, and the output line OutV to the second value Vth2 of the timing signal 112-4a. The input line 304 corresponding to the value Wth is connected at three points in the readout block 300, so that each of the plurality of switches can be switched between the value Wth and one of the values of Rth, Gth, and Bth, respectively.
[0174] Since the white light photodiode 104-4d is more sensitive than the red, green and blue light photodiodes 104-4a, 104-4b, 104-4c, the second local feedback signal 110-4b is more likely to reach and satisfy the second global threshold 108-4b before the first local feedback signal 110-4a corresponding to WTA or the sum of the RGB photodiodes 104-4 may reach the first global threshold 108-4a. Therefore, the second value Vth2 of the timing signal 112-4a is selected for comparison with the third global threshold 108-4c using the comparator, rather than selecting the first value Vth1 of the timing signal 112-4a.
[0175] When the intensity of the light incident on the photoreceptor cell 100-4 is very low, for example when a dark area of the scene is being viewed, the discharge speed of the red, green and blue photodiodes 104-4a, 104-4b, 104-4c is low and there is even a high probability that the first global threshold 108-4a will not be reached within the image acquisition phase and therefore within the allocated maximum exposure time. However, the discharge speed of the white photodiode 104-4d is always faster because this photodiode is more sensitive and therefore the probability that the second global threshold 108-4b will be reached within the image acquisition phase is higher.
[0176] The third global threshold 108-4c used in the readout block 300 is useful for determining the best set of output signals OutR, OutB and OutG in terms of quality or signal-to-noise ratio. If the second value Vth2 of the timing signal 112-4ab stored by the sample and hold circuit 106-3 passes the third global threshold 108-4c, this means that the intensity of the incident light is very low, and the readout block 300 will emit OutR, OutB, OutG using the Wth value collected by the more sensitive white photodiode. Therefore, under low light conditions, the OutR, OutB and OutG values are all equal to the value Wth, which means that color information is lost and the final image will be a grayscale image. In contrast, when the intensity of the incident light is not low, the readout block 300 will emit OutR, OutB and OutG collected by the color set of R, G, B photodiodes 104-4a, 104-4b and 104-4c. In both cases, the OutV value produced by the readout block 300 includes the timing signal V(t) 112-4a sampled when the corresponding signal group has been sampled. In other words, in the presence of bright incident light, OutR, OutB, and OutG will include a color representation of the incident light corresponding to the stored values Rth, Gth, and Bth, and OutV will provide a first value Vth1 of the timing signal 112-4a, which is sampled when the output of the summing or WTA circuit 110-4a reaches the first global threshold 108-4a. In the presence of dark incident light, OutR, OutB, and OutG will provide a grayscale representation of the incident light corresponding to the stored value Wth, and OutV will provide a second value Vth2 of the timing signal 112-4a sampled when the output of the white photodiode reaches the second global threshold 108-4b.
[0177] In another variation of the fifth exemplary implementation, Fig.10As shown in , the alternative readout block 400 is controlled by a more complex function of a first stored value Vth1 for the timing signal 112-4a, a second stored value Vth2 for the first timing value 112-4a, and a third global threshold 108-4c for more general management of the output signal. Using a function of these values, the signal-to-noise ratio can be optimized by using a sensor that is more suitable for very dark and very bright light conditions. Therefore, it should be understood that although the comparison between the second value Vth2 of the timing signal 112-4a and the third global threshold 108-4c is performed according to the fifth exemplary implementation, this is not required, and different functions can be used to determine which of the six stored values Rth, Gth, Bth, Wth, VTh1 and Vth2 are filtered and / or sent to the readout circuit of the sensor array 200. In some examples, the function can be optimized for a specific application domain and can be generated by a neural network trained according to the specific application domain. The function can also use spatial information, such as the location of the photoreceptor unit in the array, to change the behavior of the readout block. For example, in industrial applications or in video surveillance applications where the sensor array is mounted in a fixed position, different standards can be applied by the function to switch between less sensitive color vision and more sensitive grayscale vision in a manner that depends on the specific characteristics of the scene. The function can also be implemented by an external complex device such as a computer. The computer can replace the readout block, and the stored signal input to the readout block can be read out from each photoreceptor by the computer. In this way, the computer can access the Rth, Bth, Gth, Wth, Vth1 and Vth2 values.
[0178] It should be understood that photodiodes capable of acquiring light with high sensitivity, such as photodiodes capable of detecting single photons, are devices that occupy more silicon area than standard photodiodes. For this reason, the fifth implementation proposed herein minimizes the complexity and area requirements of a sensor that provides a good signal-to-noise ratio not only under normal light conditions but also in very dark scenes.
[0179] The behavior of this fifth implementation is very similar to the behavior that exists in the human retina, where the red, blue and green cones are activated when the intensity level of light is good, but are disabled when the intensity of the incident light is low, and when the cones are disabled, the rods, which are very sensitive photoreceptors, are only activated in the presence of dark scenes. The fifth implementation described above effectively simulates this behavior using the readout logic circuits in the readout blocks 300 and 400.
[0180] Finally, for the general architecture 100, the shape / function of the global feedback signals 108-4a, 108-4b, and 108-4c and the timing signal 112-4a can be tuned by the global control block 202 based on the characteristics of the observed scene. As previously described, the first timing signal 108-4a and the second timing signal 108-4b are on the lines carrying the global clock signal 112 along with other signals used to synchronize and arrange operations during the image acquisition cycle ( Fig. 9 and Fig.10 (not shown) together.
[0181] The sample and hold circuit 106-4 may be implemented as one or two circuits and represents the memory module 106 or a portion thereof. The comparator and WTA / summing circuit forming part of the first comparison circuit module 102-4a and the second comparison circuit module 102-4b are part of the local control module 102. A portion of the local control module 102 is also circuitry for operation and information received via the global clock line, while the local feedback line 102 includes the signal generated by the photosensitive element 104a.
[0182] It should be understood that the above-described readout blocks 300 and 400 may be used with any or all of the exemplary implementations described above.
[0183] Fig.11 is a schematic diagram showing an example of how additional sensors / circuits may be used to supplement any of the examples provided above. Specifically, circuit 500 may detect temporal changes in incident light, which may be used to activate Figure 1 The blocks of the architecture make Figure 1 The architecture is only used when the light intensity changes. This technique can be used to effectively add motion detection features to the device and / or improve Figure 1 energy efficiency of the overall architecture depicted in the figure. Instead of continuously processing Figure 1 The blocks of the structure provide a lot of signals and it is even more convenient to process them only when some light changes are detected in the scene. Fig.11 The circuit described in can be used to construct Fig.11 Blocks or units beyond the blocks and photoreceptor units described in the general architecture depicted in, or these blocks or units may be included in the global control module 202 to provide other functionality that can be used to change the global behavior of the sensor device. Fig.11 A circuit 500 is shown that includes a first comparison circuit module 102 - 5 a , an additional sensor 502 , a first global feedback signal 108 - 5 a , a first local feedback signal 110 - 5 a , and a bidirectional switch included in a sample and hold circuit 106 - 5 .
[0184] The circuit 500 further includes a second comparison circuit module 102-5b, a second global feedback value 108-5b, a second local feedback signal 110-5b, and an output circuit 504. The operation of these components is described below.
[0185] The general cycles and phases described above also apply here. The additional sensor 502 may be, for example, an infrared (IR) sensor or a visible white light photodiode. Figure 7 The white light photodiode 104-3d and Fig. 9 and Fig.10The additional sensor 502 is used in the same manner as described above. After the reset phase (RES), the additional sensor 502 starts integrating light or any other physical signal it is designed to detect, and its output voltage 110-5a starts to decrease linearly over time. This signal forms the first local feedback signal 110-5a and is fed to the first comparison circuit module 102-5a. The first global threshold 108-5a is also fed to the first comparison circuit module 102-5a. When the first local feedback signal 110-5a from the additional sensor 502 reaches the first global threshold 108-5a TH, the first comparison circuit module 102-5a changes its output voltage from logic LOW to HIGH using the comparator, thereby instructing the sample and hold module 106-5 to store the timing signal V(t) 112-5 in one of the two memory modules S&H1, S&H2 at time T0, thereby updating the output values V1th and V2th according to the state of the bidirectional switch controlled by the global signal SD, which generates the signals SW1 and SW2. During the next image acquisition cycle, for the next frame recorded on the array 200, the process is repeated, and the value of the timing signal V(t) 112-5 corresponding to time T1 is stored on the other memory module 106-5 S&H1 or S&H2 according to the value of the global control signal SD. Thus, for example, if in a first cycle the value of the timing signal 112-5 is stored in the S&H1 memory module, in the next cycle it will be stored in the S&H2 memory module, then in the next cycle again in the row memory module S&H1, and so on. The global signal SD is used to select the outputs SW1 and SW2 of the bidirectional switches included in the memory module 106-5 at each image acquisition cycle. In this way, at each image acquisition cycle, the values V1th and V2th stored at the output of the sample and hold module 106-5 will instead represent the current value of the timing signal 112-5 and the value at the previous image acquisition cycle when the global threshold TH108-5a has been crossed by the output 110-5a of the additional sensor 502. The absolute value of the difference between the two analog voltages V1th and V2th is then used as a second local feedback signal 110-5b and is fed to the second comparison circuit module 102-5b. The second comparison circuit module 102-5b compares the second local feedback signal 110-5b with the second global threshold 108-5b to generate an output signal DIFF on the output line 504. When the second global threshold 108-5b is not crossed, the DIFF signal on the output line 504 will be low, and when the second global threshold 108-5b is crossed, the DIFF signal on the output line 504 will be high.
[0186] In this way, two switches Sx and Sy can be controlled, which can activate two current sources IY and IX to inject some fixed amount of current into their respective output lines, as described below. Fig.11 In the example, the two current sources IX and IY of a common pixel with coordinates (i, j) in the sensor array 200 are short-circuited to the row bit line (IXi) and the column bit line (IYj), respectively, only when the comparator of the second comparison circuit module 102-5b has detected a significant change in light intensity (DIFF=HIGH). In this case, the two switches Sx and Sy connect the two current sources IX and IY to the corresponding bit lines IXi and IYj, otherwise, if DIFF=LOW, the two current sources are disconnected from the bit lines and do not contribute to the total current flowing in the bit lines involved. Therefore, the circuit 500 is equivalent to a photosensitive element, which generates a low voltage on the output line 504 when the intensity difference of the incident signal (e.g., light) between two consecutive acquisition cycles does not reach a given threshold (in this case, lower than the value TH1 of the second global threshold 108-5b), otherwise it generates a high voltage on the output line 504. In the case of imaging applications, rapid changes in light intensity detected by the additional sensor 502 are usually caused by moving objects passing through the scene. In this way, circuit 500 may be viewed as a “motion / event detector” pixel or “motion / event pixel,” and such pixels may be added / incorporated in sensor array 200 or in global control module 202 to build a sensor that is capable of detecting moving objects in a scene or that can react to the presence of moving objects.
[0187] For example, in Fig.11 In the case where the sensor array 200 includes only motion / event pixels 500 instead of sensor cells 100 or the sensor array 200 includes an array of motion / event pixels 500 and an array of sensor cells 100 sensors, the current flowing through the row and column bit lines connected to the motion / event pixels 500 is proportional to the amount of motion generated by moving objects in the scene during the image acquisition phase. More generally, significant changes in intensity between frames recorded by the circuit 500 can be used to infer motion detected by the circuit 500. Multiple such circuits 500 can be positioned throughout the array 200 to track motion of the entire array 200. In addition, the image acquisition phase of the photoreceptor cells 100 can be based on the detection of such motion in the field of view of the array 200. In this way, the circuit 500 can be combined with the above Figures 1 to 10 Any of the examples provided in can be used in combination to provide a motion activated sensor array.
[0188] Circuit 500 is therefore advantageous because it can be used to detect motion and, for example, can be used to activate array 200 when some motion is detected, or to store the voltage produced by the photodiodes in memory module 106 when some motion is detected, or to stop / reset photodiode operation when there is no motion in the scene. All of these exemplary mechanisms can result in significant energy savings during operation and / or can be beneficial in applications where motion detection is important, such as in automated video surveillance.
[0189] although Figure 1 A uniform sensor array 200 of sensor cells 100a to 100n is shown, but it will be appreciated that this is not required and that the sensor array 200 may have an irregular layout depending on its intended use / function. For example, the sensor array may include an inner array of small form factor sensor cells designed to acquire a central region of a scene with high resolution. Such a central region may be surrounded by a frame including larger sensor cells designed to detect high speed moving objects on the periphery of the scene with high sensitivity. Thus, this exemplary mechanism will simulate the so-called peripheral vision of the human eye focused on motion detection. Fig.12 An example of such a sensor architecture 1100 based on two different types of photoreceptors (a first photoreceptor type E and a second photoreceptor type S) is shown. Fig.12 In the embodiment, the two photoreceptor types E and S have the same size and the frame is formed by only one peripheral layer of photoreceptors of the first type E. However, in general, the first type of photoreceptors E and the second type of photoreceptors S have different geometries and / or sizes, and the frame formed by the first type of photoreceptors E includes any number of horizontal / vertical lines and layers of the first type of photoreceptors E. The second type of photoreceptors S forms the inner part of the array and may include, for example, the photoreceptors described above and in Figures 3 to 10 The second type of photoreceptors E placed on a frame around the central portion of the array 1100 may include, for example, Fig.11 1100, and may be configured, for example, to detect motion events generated by moving objects entering or leaving a scene observed by array 1100. Fig.11 As shown in another exemplary implementation of FIG. 1 , the motion of an object can be detected by measuring the temporal variation of the intensity of light irradiated on the photoreceptor E.
[0190] For the sake of clarity, Fig.13 Representation and Fig.12 The same sensor structure 1100 is shown Fig.111 and 2. The current sources IX and IY of each circuit 500 described in FIG. 1 are summed in the X and Y directions of the array to allow detection of light intensity variations. In this case, each circuit 500 is represented by a cell E, which is shown only by two switches controlled by a signal DIFF and two current sources IX and IY.
[0191] The output signals from the array of second photoreceptors S are transmitted via line 1102. The motion information is detected and transmitted along four output lines (ET, EL, ER, EB). Fig.10 Each of the four sides of the array 1100 is shown as output. Under normal operation, the interior of the array including the second type photoreceptors S is usually OFF, while the first type photoreceptors E at the periphery of the array 1100 are always ON to continuously detect motion along the periphery of the array 1100. Once the first type photoreceptors E detect a significant change in light intensity, the interior array of second type photoreceptors S is activated to acquire an image with higher resolution and higher dynamic range.
[0192] Although operation of the second type of photoreceptors S requires considerable power and data bandwidth, the motion detection task performed with the first type of photoreceptors is simpler, energy efficient and less expensive in terms of data bandwidth requirements. The above-described separation of operations, whereby the motion detection task turns on the more densely populated second type of photoreceptors S, minimizes power consumption and data bandwidth to the external computer, emulating the operation of the human retina.
[0193] By the flow Fig.12 and Fig.13 The motion information represented by the current of the EL, ET, EB, and ER lines in the global control module 202 can be used as a feedback signal by the global control module 202 for the logic implemented in the global control module 202 to activate a specific action. One example is the above-mentioned power saving mechanism when such information is used to turn on or off the internal array of the second type photoreceptors S. In another example, these 4 signals can be aggregated into a single "motion detection" signal, which can be forwarded to an external application. In this case, for example, when the "motion detected" signal is high (i.e., a moving object has been detected), it can be used to wake up an external controller, which in turn can process and store the image generated by the sensor, and when the "motion detected" signal is low (i.e., when there is no moving object in the scene), the external controller can enter a standby / sleep / power saving mode, which will be optimized at the system level for saving energy usage and space for the relevant video / image. In another example, the current values flowing through the EL, ET, EB, and ER lines can be sampled and sent directly to the external application using dedicated output lines.
[0194] Although the above description focuses on imaging sensors and the use of photodiodes as sensor elements 104, it should be understood that any sensor may be used in the same global / local feedback configuration. For example, a thermal sensor, a pressure sensor, or any other type of sensor that can be used in an array may be implemented in place of a photodiode.
[0195] The sensor array 200 described above can be implemented in many different devices and apparatuses. For example, if the sensor array 200 includes an optical or infrared sensor 104, it can be included in a camera or surveillance device. It can also be incorporated into a telescope, satellite, etc. for observing a scene. In the case where the sensor is a sensor designed to detect different wavelengths in the electromagnetic spectrum, the sensor array 200 can be used in satellites or other spacecraft and observatories. For example, in the case where the sensor 104 includes a pressure sensor or a touch sensor, the sensor array 200 can be used in a smart floor panel, etc.
[0196] An important field of application of the sensor array 200 is represented by artificial or machine vision, where the images produced by the sensor array 200 are post-processed by some computing device to extract information that is considered relevant to the specific application. Some examples have been given in the above paragraphs (e.g. in security or video surveillance applications, where the presence / absence of motion in the scene is one type of important information). More generally, an important feature achieved by the present invention is the ability of the sensors and / or receivers forming the sensor array 200 to adapt themselves (i.e. locally) to the intensity of the incident signal so as to maximize the signal-to-noise ratio even when the incident signal is very strong or very weak, and even when such light conditions are present simultaneously in the acquired image. Thus, the sensor array can have different responses at different locations within it, at different receivers, based on the differences in the incident signal. In addition, a flexible global feedback mechanism implemented by the global control module 202, which can be implemented in software in whole or in part by an external computer, can also optimize the response of the sensor when used in a specific application. For example, in an automotive application, a global control module can analyze the image generated by the sensor array and adjust the timing signal V(t) or the global threshold used in the different modules as described above to increase the acquisition accuracy of the interesting area of the image (e.g., the area where there is a road sign and the application needs to read it), while other areas (e.g., areas including the sky or the road surface) can be acquired with lower accuracy. This behavior is similar to what happens in human vision, where the brain decides which part of the observed scene must be acquired with high accuracy and sends signals to the eyes to "configure" the eyes to do so.
[0197] To tune the sensors or photosensors of the sensor array described above, one or more neural networks may be used to generate or modify the timing signal V(t) or any global threshold. One or more neural networks may be trained for a specific application. In the above example, the application may involve driving a vehicle.
[0198] The control of the sensor array 200 may be performed by a computer or computing device. The computer or computing device may communicate with or form part of the target control module 202. As described above, the global control module 202 may adjust various parameters of the embodiment, including global thresholds and global clock signals and other related signals. These signals may be adjusted dynamically and in real time in response to changes in the characteristics of the scene or other sensory environment being observed. The characteristics of the changes may be observed by observing the differences between frames or discrete exposures of the sensor 104. The signals may be adjusted by a computer by applying a neural network or any suitable machine learning algorithm.
[0199] In the above description, computing and / or electronic devices may be used to implement the performance of the method and various functions of the system. Such a device may include one or more processors, which may be a microprocessor, a controller, or any other suitable type of processor for processing computer executable instructions to control the operation of the device so as to collect and record routing information. In some examples, such as where a system-on-chip architecture is used, the processor may include one or more fixed function blocks (also referred to as accelerators) that implement a portion of the method in hardware (rather than software or firmware). Platform software including an operating system or any other suitable platform software may be provided at a computing-based device to enable application software to be executed on the device.
[0200] The various functions described herein can be implemented in hardware, software, or any combination thereof. If implemented in software such as a computer program, these functions can be stored on a computer-readable medium or transmitted on a computer-readable medium as one or more instructions or codes on a computer-readable medium. Computer-readable media may include, for example, computer-readable storage media. Computer-readable storage media may include volatile or non-volatile, removable or non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media may be any available storage medium accessible by a computer. As an example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, flash memory or other storage devices, CD-ROM or other optical disk storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store the required program code in the form of an instruction or data structure and can be accessed by a computer. The disks and discs used herein include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray (RTM) discs (BDs). In addition, propagation signals are not included in the scope of computer-readable storage media. Computer-readable media also include communication media, which includes any medium that facilitates the transfer of computer programs from one place to another. For example, a connection can be a communication medium. For example, if the software is sent from a website, server or other remote source using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio and microwaves, the software is included in the definition of communication media. Combinations of the above should also be included within the scope of computer-readable media.
[0201] Alternatively or additionally, the functions described herein may be at least partially performed by one or more hardware logic components. For example, but not limited to, the hardware logic components that may be used may include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), etc.
[0202] Although shown as a single system, it should be understood that the computing device may be a distributed system. Thus, for example, several devices may communicate via a network connection and may jointly perform tasks described as being performed by the computing device.
[0203] Those skilled in the art will recognize that storage devices for storing program instructions can be distributed over a network. For example, a remote computer can store examples of processes described as software. A local or terminal computer can access the remote computer and download some or all of the software to run the program. Alternatively, a local computer can download software fragments as needed, or execute some software instructions at a local terminal and some software instructions at a remote computer (or computer network). Those skilled in the art will also recognize that all or part of the software instructions can be executed by a dedicated circuit, such as a DSP, a programmable logic array, etc., by utilizing conventional techniques known to those skilled in the art.
[0204] In addition, the actions described herein may include computer executable instructions that can be implemented by one or more processors and / or stored on one or more computer readable media. Computer executable instructions may include routines, subroutines, programs, execution threads, etc. In addition, the results of the actions of these methods may be stored in computer readable media, displayed on a display device, etc.
[0205] As used herein, the term "computer" or "external memory" is intended to encompass computer-readable data storage configured with computer-executable instructions that, when executed by a processor, cause certain functions to be performed. Computer-executable instructions may include routines, functions, etc. It should also be understood that a computer may be located on a single device or distributed across several devices.
[0206] Furthermore, as used herein, the terms "exemplary" or "example" are intended to mean "serving as an illustration or example of something."
[0207] Furthermore, to the extent the term "including" is used in any of the detailed description, the term is intended to be inclusive in a manner similar to the term "comprising."
[0208] Although these methods are shown and described as a series of actions performed in a particular order, it should be understood and recognized that these methods are not limited to the order of the order. For example, some actions can occur in an order different from the order described herein. In addition, an action can occur simultaneously with another action. In addition, in some cases, not all actions are required to realize the methods described herein.
[0209] The order of operations of the methods described herein is exemplary, but the operations may be performed in any suitable order, or simultaneously where appropriate. In addition, operations may be added or replaced in any method, or individual operations may be deleted from any method, without departing from the scope of the subject matter described herein. Aspects of any one of the examples described above may be combined with aspects of any one of the other examples described to form additional examples without losing the effects sought.
[0210] It should be understood that the above description of the embodiments is given as an example only, and various modifications may be made by those skilled in the art. The foregoing includes examples of one or more embodiments. Of course, it is not possible to describe every conceivable modification and change of the above-mentioned apparatus or method for the purpose of describing the aforementioned aspects, but those skilled in the art may recognize that many further modifications and permutations of various aspects are possible.
Claims
1. An imaging sensor unit, comprising: A sensor module, wherein the sensor module includes a plurality of photoelectric sensors; a memory module coupled to the sensor module, the memory module configured to receive a sensor output from the sensor module; a local control module, the local control module being coupled to the sensor module, the local control module being configured to: receiving a local feedback signal from the sensor module; receiving a global feedback signal; and The memory module is controlled using the local feedback signal and the global feedback signal to store the sensor output.
2. The imaging sensor unit of claim 1 , wherein the global feedback signal comprises a threshold; and wherein the local control module is configured to: comparing the local feedback signal with the threshold of the global feedback signal; and When the local feedback signal meets the threshold, the memory module is controlled to store the sensor output.
3. The imaging sensor unit according to claim 2, wherein the plurality of photosensors comprises: a first photosensor sensitive to a first portion of the visible spectrum, the first photosensor configured to output a first sensor output component; a second photosensor sensitive to a second portion of the visible spectrum, the second photosensor configured to output a second photosensor output component; as well as a third photosensor sensitive to a third portion of the visible spectrum, the third photosensor configured to output a third photosensor output component; The sensor output includes the first photosensor output component, the second photosensor output component and the third photosensor output component.
4. The imaging sensor unit according to claim 3, wherein the imaging sensor unit further comprises: a summing node configured to sum the first photosensor output component, the second photosensor output component, and the third photosensor output component to generate a summed value, wherein the local feedback signal includes the summed value, such that: The local control module is configured to compare the summed value with the threshold of the global feedback signal; When the sum value satisfies the threshold, the local control module is configured as follows: The memory module is caused to store the first photosensor output component, the second photosensor output component, and the third photosensor output component.
5. The imaging sensor unit according to claim 3, wherein the imaging sensor unit further comprises: A winner-takes-all node, wherein the winner-takes-all node is configured as: receiving the first photosensor output component, the second photosensor output component and the third photosensor output component; A winning photosensor output component is determined by identifying which of the first photosensor output component, the second photosensor output component, and the third photosensor output component exhibits a maximum magnitude difference from initial values of the respective first, second, and third output components; wherein the local feedback signal includes the winning photosensor output component, such that: The local control module is configured to compare the winning photosensor output component to the threshold value of the global feedback signal; Wherein once the winning photosensor output component satisfies the threshold, the local control module is configured to: The memory module is caused to store the first photosensor output component, the second photosensor output component, and the third photosensor output component.
6. The imaging sensor unit according to any one of claims 3-5, wherein the first photosensor is sensitive to red light, the second photosensor is sensitive to green light, and the third photosensor is sensitive to blue light.
7. The imaging sensor unit according to any one of claims 3 to 6, wherein the plurality of photosensors comprises a broad-spectrum photosensor configured to detect brightness of an observed scene and generate a brightness output, wherein the local feedback signal comprises the brightness output, such that when the brightness output satisfies the threshold of the global feedback signal, the local control module is configured to: The memory module is caused to store the sensor output.
8. The imaging sensor unit according to any one of claims 3 to 6, wherein the plurality of photosensors include a broad-spectrum photosensor for detecting light under low-light conditions, the broad-spectrum photosensor having a higher sensitivity than the first photosensor, the second photosensor, and the third photosensor, wherein the broad-spectrum photosensor is configured to detect brightness of an observed scene and generate a brightness output; wherein the local control module is further configured to receive a second local feedback signal and a second global feedback signal, wherein the second local feedback signal includes the brightness output, and wherein the second global feedback signal includes a second threshold; The local control module is configured to compare the second local feedback signal with the second threshold, so that when the second local feedback signal meets the second threshold, the local control module is configured to: causing the memory module to store the brightness output; The imaging sensor also includes a readout module, which is configured to output the brightness output from the imaging sensor, one or more of the first photosensor element output, the second photosensor element output and the third photosensor element output based on a comparison result with a third global feedback signal received at the readout module or a function of the third global feedback signal.
9. An imaging sensor unit according to claim 7 or 8, wherein the broad spectrum photosensor is configured to detect white light.
10. An imaging sensor unit according to claim 7 or 8, wherein the broad spectrum photosensor is configured to detect infrared light.
11. The imaging sensor unit according to any one of claims 2 to 10, wherein the imaging sensor is subject to a maximum exposure time, wherein if the local feedback signal does not meet the threshold within the maximum exposure time, the memory module is configured to store the sensor output.
12. The imaging sensor unit according to any one of the preceding claims, wherein the local control module is further configured to: The memory module is caused to store timing information indicating a time to store the sensor output.
13. The imaging sensor unit of claim 12, wherein the memory module is configured to receive a timing signal, and wherein the timing information is a record of the timing signal when storing the sensor output.
14. A sensor array, comprising the imaging sensor unit according to any one of claims 1 to 13, the sensor array further comprising: A global control module is configured to globally control the sensor array, wherein the global control module is configured to provide the global feedback signal to each of the plurality of imaging sensor units.
15. The sensor array of claim 14, wherein the global feedback signal is adjustable to adjust the characteristics of an image captured by the sensor array.
16. The sensor array of claim 14 or 15, further comprising a readout circuit configured to read out the stored sensor outputs from the plurality of imaging sensor cells.
17. The sensor array of claim 16, wherein the readout circuit is further configured to read out timing information for each of the plurality of imaging sensor units, wherein the timing information indicates a time at which the stored sensor output is stored for each of the plurality of imaging sensor units.
18. A sensor array according to claim 16 or 17, wherein the sensor array is communicatively coupled to a computer device, wherein the readout circuit is configured to send information related to at least the sensor output of each of the plurality of imaging sensor units to the computer device.
19. The sensor array according to any one of claims 15 to 18, further comprising one or more motion sensor units, the one or more motion sensor units comprising: Photoelectric sensors; a motion sensor memory module configured to receive a timing signal; as well as Motion sensor control module; The motion sensor control module is configured as follows: receiving and comparing a sensor output of the photosensor to a global motion threshold; as well as When the sensor output of the photosensor satisfies the global motion threshold; causing the memory module to store timing information from the timing signal, the timing information indicating a time when the global motion threshold is satisfied; wherein the motion sensor memory module comprises a switch configured to modify a storage location of the timing information, wherein: positioning the switch in a first position corresponding to a first stored position during a first active motion sensor phase of the photosensor corresponding to a first frame of the observed scene; and positioning the switch in a second position corresponding to a second stored position during a subsequent second activation motion sensor phase of the photosensor corresponding to a second frame of the observed scene; The local control module is configured as follows: causing the memory module to store first timing information in the first storage location, the first timing information indicating when the global motion threshold is met for the first activate motion sensor phase; and causing the memory module to store second timing information in the second storage location, the second timing information indicating when the global motion threshold is met for the second activate motion sensor phase; and The local control module is configured as follows: calculating a difference value corresponding to a difference between the first timing information and the second timing information; comparing the difference to a second global motion threshold; and When the difference satisfies the second global motion threshold, a signal is output from the motion sensor indicating that motion has been detected between the first frame and the second frame of the observed scene.
20. The sensor array of claim 19, wherein the first global motion threshold and the second global motion threshold are adjustable.
21. A sensor array according to claim 19 or 20, wherein the first global motion threshold is a photosensor output threshold and the second global motion threshold is a time threshold.
22. A sensor array according to any one of claims 19 to 21, wherein the signal indicating that motion has been detected between the first and second frames of the observed scene is configured to activate the plurality of imaging sensor units.
23. A sensor array according to any one of claims 19 to 22, comprising a plurality of motion sensor units.
24. The sensor array of claim 23, wherein the plurality of motion sensor units are arranged at least around a periphery of the sensor array.
25. A method of controlling an imaging sensor unit, wherein the imaging sensor unit comprises: A sensor module, wherein the sensor module includes a plurality of photoelectric sensors; a local control module, the local control module coupled to the sensor module; as well as a memory module coupled to the sensor module; The method comprises: receiving, at the memory module, a sensor output from the sensor module; receiving, at the local control module, a local feedback signal from the sensor module; receiving a global feedback signal at the local control module; and The memory module is controlled using the local feedback signal and the global feedback signal to store the sensor output.
26. The method according to claim 25, further comprising: The imaging sensor unit is operated according to a cycle, the cycle being defined by: Active sensor phase, which is initiated by: activating the sensor module from an initial state so that the sensor module begins to detect and generate the sensor output; and Passive sensor stage, which is initiated by: The sensor module is reset to the initial state.
27. The method according to claim 26, comprising: Entering the active sensor stage; as well as After storing the sensor output, the passive sensor phase is entered.
28. The method of claim 27, wherein the duration of the active sensor phase is limited by a maximum exposure time, such that the method further comprises: causing the memory module to store the sensor output during the maximum exposure time; as well as The passive sensor phase is entered such that the passive sensor phase is entered after the maximum exposure time has elapsed.
29. The method according to any one of claims 25 to 28, further comprising: When the sensor output is stored, timing information indicating the time at which the sensor output is stored is stored.
30. The method according to any one of claims 25 to 29, further comprising controlling a plurality of imaging sensor units, the method comprising: The active phase of each of the imaging sensor units is entered synchronously.
31. A motion sensor, comprising: Photoelectric sensors; a motion sensor memory module configured to receive a timing signal; as well as Motion sensor control module; The motion sensor control module is configured as follows: receiving and comparing a sensor output of the photosensor to a global motion threshold; as well as When the sensor output of the photosensor satisfies the global motion threshold; causing the memory module to store timing information from the timing signal, the timing information indicating the time when the global motion threshold is satisfied; wherein the motion sensor memory module comprises a switch configured to modify a storage location of the timing information, wherein: positioning the switch in a first position corresponding to a first stored position during a first active motion sensor phase of the photosensor corresponding to a first frame of the observed scene; and positioning the switch in a second position corresponding to a second stored position during a subsequent second activation motion sensor phase of the photosensor corresponding to a second frame of the observed scene; The local control module is configured as follows: causing the memory module to store first timing information in the first storage location, the first timing information indicating when the global motion threshold is met for the first activate motion sensor phase; and causing the memory module to store second timing information in the second storage location, the second timing information indicating when the global motion threshold is met for the second activate motion sensor phase; and The local control module is configured as follows: calculating a difference value corresponding to a difference between the first timing information and the second timing information; comparing the difference to a second global motion threshold; and When the difference satisfies the second global motion threshold, a signal is output from the motion sensor indicating that motion has been detected between the first frame and the second frame of the observed scene.
