Configurable sensor unit and sensor device

Abstract

Configurable sensor units and sensor devices are provided. A configurable sensor unit comprises a sensor device, in a circuit, for generating an electrical signal in response to a stimulus detected by the device, and a programmable non-volatile memory element capable of operating as a load resistor for the sensor device in the circuit, so that the resistance of the load resistor depends on the programmed state of the memory element. The sensor unit has an output for providing an output signal in response to the aforementioned electrical signal and programmed state. A sensor device may comprise a plurality of such configurable sensor units. Such a sensor device may be configured to perform in-sensor computational operations for a neural network architecture. A further sensor device comprises a configurable sensor unit and a controller for programming the memory element of the sensor unit into a programmed state in accordance with the operating requirements of the sensor unit.

Description

[Background technology]

[0001] The present invention generally relates to configurable sensor units and sensor devices incorporating such sensor units.

[0002] Sensors are employed in numerous applications in science, technology, and the Internet of Things (IoT). Sensors (such as image, audio, motion, temperature, chemical, and tactile sensors) can be deployed in multiple environments and are used to detect a wide range of stimuli (light, sound, motion, etc.) for many purposes. The output of such sensors can be processed in various ways and is often processed in neural network (NN) architectures designed to perform cognitive tasks. As an exemplary example, static or video images may be analyzed by an NN system designed to detect objects, people, human emotions, etc. Processing sensor outputs can involve various types of computations. In an NN system, for example, the sensor output is converted into a digital signal, which is then propagated across weighted connections in the network. Signal propagation typically involves computations such as multiply-accumulate (MAC) and matrix-vector multiplication (MVM) operations, where the signal is multiplied by the network weights according to a particular network architecture. Such calculations involve multiple data transfers between memory and processing units and require significant processing resources.

[0003] An in-memory compute (IMC) architecture has been proposed in which specific computational tasks, such as MAC and MVM operations, can be performed in situ within a computational memory unit employing an array of memory cells. This mitigates processing bottlenecks caused by data movement between memory and processing units and improves computational efficiency. In a similar vein, two-dimensional material-based image sensors have been proposed for in-sensor MVM operations (see "Ultrafast machine vision with 2D material neural network image sensors," Mennel et al., Nature 579, 62-66 (2020)) and for convolution operations for edge detection (see "Programmable black phosphorous image sensor for broadband optoelectronic edge computing," Seokhyeong et al., Nature Communications 13, 1485 (2022)). In these image sensors, NN weights are stored by adjusting the photosensitivity of the 2D material photosensor device using field effects, thereby tuning channel doping with charges stored within the device's multi-gate electrode or gate dielectric layer. These systems require advanced device architectures and lack the scalability necessary for image sensor applications.

[0004] Improvements to the sensors, as well as sensor devices with computing capabilities, are highly desirable. [Overview of the Initiative]

[0005] Configurable sensor units and sensor devices are provided. A configurable sensor unit comprises a sensor device, in a circuit, for generating an electrical signal in response to a stimulus detected by the device, and a programmable non-volatile memory element capable of operating as a load resistor for the sensor device in the circuit, so that the resistance of the load resistor depends on the programmed state of the memory element. The sensor unit has an output for providing an output signal in response to the aforementioned electrical signal and programmed state. A sensor device may comprise a plurality of such configurable sensor units. Such a sensor device may be configured to perform in-sensor computational operations for a neural network architecture. A further sensor device comprises a configurable sensor unit and a controller for programming the memory element of the sensor unit into a programmed state in accordance with the operating requirements of the sensor unit.

[0006] Another aspect of the present invention provides a sensor device comprising a plurality of configurable sensor units as described above. The plurality of sensor units may be arranged in a crossbar array having row and column lines for addressing each row and column of the sensor unit and acquiring the output signal from each unit. [Brief explanation of the drawing]

[0007] [Figure 1] This is a schematic representation of one embodiment of a configurable sensor unit.

[0008] [Figure 2] This is a schematic representation of another embodiment of the sensor unit.

[0009] [Figure 3] This is a diagram showing an example of a programmable non-volatile memory element in a sensor unit.

[0010] [Figure 4]It is a diagram of an embodiment of a sensor unit having a switching circuit configuration for programming operations.

[0011] [Figure 5] It is a diagram of another embodiment having a switching circuit configuration for selective connection of different memory elements in a sensor circuit.

[0012] [Figure 6] It is a schematic representation of a conventional image sensor array.

[0013] [Figure 7] It is a schematic representation of a computing sensor device embodying the present invention.

[0014] [Figure 8] It is a diagram of the stage of the convolution operation in CNN.

[0015] [Figure 9] It is a diagram of switching between memory elements in the sensor unit of the device of FIG. 7 for implementing a convolution operation.

[0016] [Figure 10] It is a diagram of the MAC operation for the convolution operation in an embodiment of the sensor device.

[0017] [Figure 11] It is a schematic representation of a further embodiment of a configurable sensor unit.

[0018] [Figure 12] It is a schematic diagram of the configuration of the device layer in an integrated sensor device embodying the present invention.

[0019] [Figure 13] It is a diagram of the configuration of additional circuitry for a sensor unit embodying the present invention. [Figure 14] It is a diagram of the configuration of additional circuitry for a sensor unit embodying the present invention. [Figure 15] This is a diagram showing the configuration of an additional circuit for a sensor unit embodying the present invention.

[0020] [Figure 16] This is a schematic diagram illustrating the signal readout operation for convolution in an embodiment of a sensor device. [Figure 17] This is a schematic diagram illustrating the signal readout operation for convolution in an embodiment of a sensor device.

[0021] [Figure 18] This is a diagram illustrating the computation of an MLP network.

[0022] [Figure 19] This is a diagram of the array structure in an embodiment of a computational sensor device that implements an MLP layer.

[0023] [Figure 20] This diagram shows the structure and operation of an adaptive sensor device embodying the present invention. [Modes for carrying out the invention]

[0024] A first aspect of the present invention provides a configurable sensor unit. The sensor unit comprises a sensor device, which in a circuit generates an electrical signal in response to a stimulus detected by the device, and a programmable non-volatile memory element capable of operating as a load resistor for the sensor device in the circuit, such that the resistance of the load resistor depends on the programmed state of the memory element. The sensor unit has an output for providing an output signal in response to the aforementioned electrical signal and programmed state.

[0025] A sensor unit embodying the present invention, using a programmable non-volatile memory element as a load resistor for a sensor device, can be configured in a cleverly simple manner and can be programmed once or dynamically for various purposes. Using a memory element as a load resistor allows calculations to be performed directly on the signal generated by the sensor device, thereby enabling both detection (signal generation) and calculation (by the programmable load resistor) to be performed locally and without data conversion. The response speed of the sensor unit can be configured as desired and then stored in the non-volatile state of the load resistor for subsequent sensor operation. Using this simple and efficient device structure, the sensor unit can be easily adapted to various operating conditions and / or application requirements. This structure also provides a foundation for efficient and highly scalable computing sensor devices capable of performing in-sensor calculations such as MAC and MVM operations in NN architectures.

[0026] The sensor unit may include a switching circuit configuration for selectively connecting memory elements to the sensor circuit configuration and to a controller for programming the memory elements to the required programmed state. Using the same basic unit structure, various sensor units may be programmed once for various operating conditions and / or application requirements, or the unit may be programmed dynamically to adapt, for example, to fluctuating operating conditions or to in-sensor computation applications.

[0027] In particularly advantageous embodiments, the sensor unit includes a plurality of programmable non-volatile memory elements capable of acting as load resistors for the sensor device within the circuit. The memory elements may be selectively connectable within the circuit to a unit that includes a switching circuit configuration for connecting selected memory elements within the circuit in response to a control signal. These embodiments provide switching between various, individually programmable load resistors to adapt the sensor unit to various operating conditions / application requirements and / or to implement more complex in-sensor computations for, for example, convolutional NN (CNN) architectures.

[0028] Another aspect of the present invention provides a sensor device comprising a plurality of configurable sensor units as described above. The plurality of sensor units may be arranged in a crossbar array having row and column lines for addressing each row and column of the sensor unit and acquiring the output signal from each unit. This provides a highly efficient system architecture for various types of sensor devices, such as image sensors or tactile sensors, where the sensor units are spatially distributed. The sensor device may include a controller for controlling the addressing of the sensor units by the row and column lines and for programming the memory elements of the sensor units.

[0029] Using a sensor unit having multiple memory elements capable of operating as load resistors provides a computational sensor device for accelerating the processing of sensor signals in a CNN. A controller in such a device can program the memory elements of each sensor unit to pre-programmed states corresponding to the respective kernel weights of the CNN layers, and can control the addressing of the sensor units so that the output signals of the sensor units yield the results of the convolution operations in the CNN layers. In an advantageous embodiment, the output signals of each sensor unit are provided on the column lines of the sensor unit's crossbar array. The controller can control the addressing of the sensor units and the selective connection of each of the memory elements within the circuit of the unit by the control signals for each sensor unit so that the signals on the column lines of the array yield the results of the multiplication-accumulation stage of the convolution operations in the CNN layers. This allows in-sensor computations, by switching between load resistors within the sensor unit, to switch in to the kernel weights required for convolution, and MAC operations can be performed in parallel by addressing the units. Multiple MAC operations may be performed in parallel in some architectures. The sensor unit may have a capacitor for each memory element to store charge in accordance with the electrical signal from the sensor device and the programmed state of the memory element, and the capacitor can be selectively connected to the output for providing the output signal of the sensor unit. This allows for temporary storage of the output signal and greater flexibility for processing the signal in the order necessary for convolution.

[0030] Figure 1 shows the basic circuit components of a configurable sensor unit (CSU) embodying the present invention. CSU1 is a sensor device 2, which has a sensor device 2 in the sensor circuit for generating an electrical signal in response to a stimulus detected by the device. In this example, sensor device 2 comprises a photosensor, here a photodiode, which generates an electrical signal in response to incident light. CSU is a load resistor R to sensor device 2. L It also includes a programmable non-volatile memory element 3 that can operate within the circuit. Load resistor R L The resistance depends on the programmed state of memory element 3. L The value of can be set via a programming operation, as schematically shown in the figure. In this embodiment, the memory element 3 is connected in series with the sensor 2 via an amplifier 4 for amplifying the sensor output. The CSU has an output 5 for providing an output signal in accordance with the electrical signal from the sensor 2 and the programmed state of the memory element 3. In this example, the signal at output 5 is due to the (amplified) photocurrent generated by the photodiode 2 and the load resistor R L The voltage across the terminals is determined by the voltage drop. (The elements shown in Figure 1 are indicated by the same reference numerals.) Figure 2 shows a similar embodiment of the CSU. In this CSU6, the amplifier 4 is positioned at the output and connected to the load resistor R L The voltage generated across both ends is amplified.

[0031] The programmable memory element 3 can be implemented using various known memory technologies. Figure 3 shows a particularly useful example, where the memory element is implemented by a mushroom-type PCM cell. The PCM cell comprises a amount of phase-change material, such as GST (germanium antimonitorl), between the top and bottom electrodes of the cell. The cell can be programmed into different analog resistance states by applying voltage pulses ("write" pulses) through the electrodes, which vary the ratio of amorphous and crystalline phases with different resistance characteristics within the phase-change material. The resistance state of the cell can be read out during programming by applying a "read" pulse of a sufficiently low voltage that does not affect the cell state and measuring the resulting current flowing through the cell. By applying such a programming signal, the PCM memory element 3 can achieve a desired resistance value R L The memory element 3 may be programmed using other non-volatile memory devices, such as NAND flash cells, FeFET cells, and resistive RAM (RRAM, or ReRAM) cells including conductive bridge RRAM cells, oxide or metal oxide RRAM cells, and carbon RRAM cells, as well as magneto-resistive random access memory (MRAM) cells or ferroelectric random access memory (FeRAM) cells. One or more of these cells may be placed in various circuit configurations to provide the programmable resistive memory element 3.

[0032] During programming, the memory element 3 can be disconnected from the sensor device 2. This is illustrated in Figure 4 with respect to the basic circuit configuration of Figure 1. Here, the CSU includes a switching circuit configuration 8 for selectively connecting the memory element 3 to the sensor circuit and to the controller 9 for programming the memory element to the required state.

[0033] By providing the programmable memory element 3 as a load resistor for the sensor device, the response speed of the CSU can be adjusted in a particularly simple manner. The memory element 3 can be programmed to the required state once or dynamically, allowing individual CSUs to be configured / reconfigured as desired for various purposes. Exemplary applications are described in more detail below. The response speed of the CSU can be configured as needed and then stored in the non-volatile state of the memory element 3 for subsequent sensor operation.

[0034] A CSU embodying the present invention may include a plurality of programmable non-volatile memory elements, each capable of operating as a load resistor for a sensor device within a sensor circuit. Figure 5 shows one such implementation, again relating to the basic circuit configuration of Figure 1. This CSU 10 has a plurality of memory elements 3 that can be selectively connected within the sensor circuit as load resistors for a sensor device 2. A switching circuit configuration 11 responds to a switching control signal from a controller (not shown in this figure) that connects the selected memory elements 3 within the sensor circuit. The switching circuit configuration 11 may selectively connect each memory element to a controller for programming the individual elements to a desired resistance value. Thus, various load resistors can be switched into the sensor circuit to adapt the CSU operation to various requirements.

[0035] Although a simple CSU circuit has been described above, a CSU may include other circuit elements, such as additional resistors, capacitors, and / or bias voltages, and may have a variety of other circuit devices. Further examples of CSU circuits are described below. In general, a CSU may use any type of sensor device, such as sound, motion, temperature, pressure, chemical (e.g., gas), and tactile sensors, as well as other photosensor devices, such as phototransistors.

[0036] A CSU embodying the present invention may be a component of a sensor device comprising an assembly of such units. Figure 6 is a simplified schematic diagram of a conventional image sensor. This sensor comprises an array of pixel units arranged in a crossbar configuration, each having row lines and column lines (alternatively "word lines" and "bit lines") for addressing each row and column of the pixel units. Each pixel unit accumulates charge in response to incident light. When a given pixel is addressed (by applying row and column selection signals to the corresponding row and column lines), the accumulated charge is converted into a voltage by the pixel unit's amplifier, generating a current on the column line. This current is amplified at the sensor output and converted into a digital signal indicating pixel brightness by an ADC (analog-to-digital converter). A CSU embodying the present invention may be similarly configured in a crossbar arrangement, and an example of such a device is illustrated in Figure 7.

[0037] The schematic diagram in Figure 7 shows an image sensor 15 comprising multiple CSUs 16 configured in a crossbar topology similar to that in Figure 6 (here, a simplified 4x4 array). As shown in Figure 6, the crossbar array has row and column lines for addressing each row and column of the CSUs 16 and acquiring output signals from each unit. In this example, the sensor 15 has an output amplifier and an ADC for outputting a digital signal for further processing. In an alternative embodiment, the column lines may be connected to a subsequent processing unit in a different way, as illustrated by the example below.

[0038] The structure of each CSU16 is schematically illustrated in the enlarged view of Figure 7. This CSU16 is based on the overall structure of Figure 5, and includes a photosensor circuit 17 (having a photosensor and amplifier as shown in Figure 5), a switching circuit configuration 18, and a load resistor R for the photosensor. LThe photosensor circuit includes a plurality of programmable memory elements (PCM elements, as shown in Figure 3) that can be selectively connected. In this embodiment, the output signal of the CSU is provided to the column line of the array when the CSU is addressed. The sensor device provides row and column selection signals for addressing the CSUs, and control signals ("R") for controlling the selection of a specific load resistor by the switching circuit configuration 18 within each CSU. L Includes an array controller 19 which generates along with the selection. The controller 19 also generates write and read pulses to program the individual PCM elements of each CSU to the required resistance values.

[0039] The sensor device 15 can be used, for example, as a computational sensor to implement in-sensor computational functions for processing images acquired within a NN architecture for inference in cognitive tasks. Using a PCM element as a load resistor for the photosensor allows computation to be performed directly on the signal generated by the photosensor, thereby allowing both detection (signal generation) and computation (by the programmable load resistor) to be performed locally and without data conversion. In particular, the change in the response speed of the CSU due to the programmable load resistor is analogous to synaptic weighting in a neural network. Using multiple memory elements within each CSU enables efficient implementation of in-sensor computational functions for CNNs.

[0040] FIG. 8 is a schematic diagram illustrating a convolution operation on an image in the input layer of a CNN. The synaptic layer of the CNN comprises d weight kernels, each of which comprises k×k weights as shown in the figure. An input image (of size n pixels×n pixels) is processed to generate a set of image vectors (“image slices”) for further processing using the convolution kernels. These image slices are generated by sliding an image window of size k pixels×k pixels over the positions of the input image. Assuming a stride of one pixel (window shift), in that case the image window can take (n - k + 1) positions in both the horizontal and vertical directions in the input image. This results in (n - k + 1) 2 image slices each comprising a vector of pixels of dimension k 2 ×1. For each image slice, the MVM operation is performed using a matrix of kernel weights formed from d vectors each comprising k 2 weights of the respective kernel as shown in the figure. A dot product is calculated between each image slice and each weight vector within the kernel matrix to compute the corresponding point within the output image volume. This results in an output image of (n - k + 1) 2 points and depth d as shown in the figure. The points of this output image are mapped to neurons within the next network layer for further processing in the CNN.

[0041] FIG. 9 shows how the convolution operation can be performed within a computing sensor device having the overall architecture of FIG. 7. For this convolution, the kernel dimension k = 2, the stride s = 1, and there are four kernels each having four weights as illustrated by the kernel weight vectors in the figure. Each pixel can take one of four positions within an image slice (apart from pixels at the edges of the input image). These positions, denoted A, B, C, and D, are illustrated by the shaded pixels at the top of the figure.

[0042] To implement this convolution, each CSU 16 contains 16 PCM cells. The controller 19 programs these cells to a state corresponding to their respective kernel weights. Thus, each PCM cell stores one of the 16 kernel weights, as shown in the matrix at the bottom left of the figure. For each pixel of the input image, the photosensor signal from the corresponding CSU must be multiplied by each of the 16 kernel weights. This can be achieved by switching each of the PCM cells to the sensor circuit by the switching circuit configuration 18 of the CSU. For example, to perform multiplication on the sensor output at pixel position A in the image slice, the CSU uses four kernel weights a, as shown by the dashed rectangle in the figure. 11 a 12 a 21 and a 22 Switch between cells that store the weight (b). Similarly, the weight (b 11 ~b 22 ), (c 11 ~c 22 ), and (d 11 ~d 22 Switching between cells that store ) performs multiplication on pixel positions B, C, and D, respectively.

[0043] By controlling the addressing of the CSUs and switching between kernel weights in individual CSUs, the output signals of the CSUs can be combined to produce the result of the convolution operation. For a given image slice in the example shown, the MAC operation for dot product using each kernel is performed using the kernel weights in the four CSUs at pixel positions A, B, C, and D in that slice (a 11 ~a 22 ), (b 11 ~b 22 ), (c 11 ~c 22 ), and (d 11 ~d 22This can be performed by controlling parallel switching between different sets of ). The signals on the column line are then accumulated to obtain the result of the multiplication-accumulation stage of the convolution operation. Thus, MAC / dot product operations can be performed in parallel by addressing multiple CSUs switched to the required weights. Multiple MAC operations may be performed in parallel in some implementations. This is illustrated with respect to a simple example below.

[0044] The schematic diagram in Figure 10 shows an array 20 of 4×4 CSUs 16. To perform a convolution with a 2×2 kernel and stride s=2, as indicated by the dashed window with respect to the shaded CSUs, the shaded CSUs are addressed by row lines 1 and 2 and column lines 1-4. Memory cells to store the weights of a given kernel are selected from within the four CSUs of each window. Thus, from the two CSUs connected to column line 1, o 11 and o 21 The output signal indicated by is accumulated on that column line. Similarly, the output signals from the two CSUs in the next column o 12 and o 22 This is accumulated on column line 2. Column lines 1 and 2 are connected in this embodiment, so that the column signals are accumulated and the MAC result (o 11 +o 21 +o 12 +o 22 This results in the dot product operation between the corresponding image slice and kernel weight vector. The corresponding result for the second image window is provided in parallel on the combined output of column lines 3 and 4. The resulting signal is then output for further processing in the MVM operation. In this implementation, the analog signal can be supplied directly to the word line of the in-memory compute unit 21 for further processing, without conversion. In effect, the compute sensor array thus constitutes an additional core (or "tile") for the IMC unit, accelerating computation in the input layer of the CNN.

[0045] While specific convolution examples have been described above, it will be understood that convolution calculations can be performed in numerous other ways through appropriate addressing of CSUs, switching schemes for time-division switching between kernel weights, and accumulation / further processing of output signals on a set of array column lines. Since each CSU can be modestly sized (e.g., 10 × 10 μm), multiple kernels can be encoded in multiple load resistors without excessive penalty in terms of areal density. It should also be noted that using a stride equal to the kernel vector dimension, i.e., (s=k), limits the number of PCM cells required per CSU to k.

[0046] In some embodiments, the CSU may store output signals to facilitate convolution calculations. In particular, the CSU may include a capacitor for accumulating charge in accordance with the photosensor signal and the weights stored within the element, with respect to each memory element. Figure 11 shows an exemplary embodiment. As schematically illustrated here, each memory element of the CSU 25 has an associated capacitor 26. When the memory elements are connected within the photosensor circuit 27, the associated capacitors are connected to the memory elements by a switching circuit configuration 28. Thus, the capacitors accumulate charge in accordance with the weighted photosensor signals. Each capacitor can be selectively connected to the output of the CSU in response to a further control signal "C select" supplied to the switching circuit configuration 28 by the array controller 29. Thus, the stored signals can be output to the associated column lines in a desired order, providing greater flexibility in convolution calculations.

[0047] It will be understood that the crossbar array of a CSU can be configured in various ways. In a preferred embodiment, the memory elements of various CSUs can be configured within one or more memory arrays (tiles) that can be integrated using standard fabrication methods and topologies used for sensor arrays such as image sensors. For example, modern back-illuminated photosensor arrays are fabricated using hybrid bonding techniques. The schematic diagram in Figure 12 illustrates how memory arrays can be readily embedded within the integrated layer structure of such sensors, providing a densely integrated computing sensor that is compatible with hybrid bonding techniques.

[0048] Various other CSU circuits may be conceived, and specific examples are shown in Figures 13–15 (here having a single PCM element for simplicity). In Figure 13, a photodiode (here operating in bias mode with bias voltage V) acts as a high-impedance current source, and the voltage dropped across the PCM cell is amplified by a gain coefficient determined by fixed load resistors R1 and R2. The output voltage of the amplifier generates a current through fixed load resistor R3, which contributes to MAC calculations on the column lines of the sensor array. Figure 14 shows another configuration in which the PCM cell implements weighted calculations by variable gain at the output node of the amplifier. Figure 15 shows an alternative configuration in which the photodiode generates a voltage scaled by the value of feedback resistor R1. The output voltage of the amplifier generates a current through the PCM element, which can be accumulated on the bit lines of the PCM array for MAC calculations.

[0049] The addressing of the CSU can be controlled in various ways so that the output signal of the CSU yields the result of the convolution operation. Figure 16 illustrates a readout scheme for convolution using a computing sensor device that performs MAC operations on the column lines of a photosensor array. For a 2x2 pixel image window and a stride of s=2, convolution on the entire row of the image sensor can be performed in parallel by a diagonal readout scheme as schematically shown in the figure. At any given moment, convolution can be performed using a single kernel, here K1. Convolution using multiple kernels requires switching between PCM elements within the CSU. Further kernels K2, K3… can be stored in a diagonal pattern below the one for K1 in the shown PCM array, yielding MAC operations on consecutive kernels on the same column line. Figure 17 shows an alternative readout scheme in which MAC operations are performed on the bit lines of a PCM crossbar array using the CSU circuit of Figure 15. Here, convolution can be performed in parallel for all image windows in a row using different kernels encoded in different columns of the PCM tile. This allows convolutions using multiple kernels to be executed in parallel, at the cost of providing separate PCM tiles for each image window.

[0050] The CSU embodying this invention can be applied to other computational topologies such as multilayer perceptrons (MLPs). Figure 18 illustrates signal propagation at the input of an exemplary MLP. Network input, here n of an n×n image. 2Each pixel represents the output signal of each (conceptual) neuron in the first neuron layer L1. Each of these signals is propagated via weighted connections to neurons in the second layer L2 of the network, in this case all m neurons. This requires multiplying each pixel value by each of the m weights. The weighted signals received by each L2 neuron are then accumulated to produce the input signal to that neuron. This involves m MAC operations for MVM computation between an n×1 vector of image pixels and an n×m matrix of network weights.

[0051] The above calculation can be implemented within a computational image sensor, as schematically illustrated in Figure 19. This sensor comprises a crossbar array of CSUs, each having a programmable memory element for storing each weight of an n × m weight matrix. The CSUs of the main array are grouped into rows and columns of subarrays, each containing m CSUs, as shown in the figure. An array controller (not shown) programs the memory elements of the CSUs in each subarray to store their respective weights, so that the output signals of the CSUs in each subarray can be operated to correspond to weighted signals transmitted by each neuron in layer L1 to m neurons in layer L2. The MAC operation stage for MVM calculation can be performed by appropriate addressing of the CSUs. In particular, the output signals of each CSU can be provided to the column lines of the crossbar array, and the addressing of the CSUs is controlled so that the signals on the column lines yield the result of the multiplication and accumulation stage, which can then be further processed in a subsequent processing unit to obtain the final result of the MVM calculation.

[0052] The CSU embodying this invention will be understood to provide an efficient and easily configurable computing sensor device having in-sensor computing capabilities to accelerate computations in the input layer of an NN architecture (where computations are most intensive). The described operation provides reduced latency and improved energy efficiency due to reduced data movement and transformation, and can be integrated with an IMC or other processing unit for efficient and ultrafast end-to-end performance. This offers significant advantages in many applications, such as autonomous driving and edge computing applications in general.

[0053] Further embodiments of the present invention provide a sensor device comprising a CSU generally as described above, and a controller for programming the memory elements (or each of them) of the CSU into programmed states (or states) according to the operating requirements of the CSU. This allows sensor units having the same basic structure within various sensor devices to be programmed to suit various application / operating environments for various operating requirements. The CSU of such a device may have one or more individually programmable and individually selectable memory elements for additional configuration options. Programming may be a one-time operation to suit a particular sensor deployment, or the CSU may be adaptively programmed based on the variable operating requirements of the sensor device. Such an adaptive sensor device may include an array of individually programmable CSUs. In some embodiments, additional adaptability may be provided by adjusting the gain to the sense amplifiers of the individual CSUs and / or the output amplifiers of the array.

[0054] An exemplary example of an adaptive sensor device is described with reference to Figure 20. The plot on the left side of this figure represents the resistor R LFigure 1 shows variations in the photosensitivity of CSUs with different values ​​of . Higher resistance results in a more sensitive CSU, while lower resistance provides an increased dynamic range. This can be utilized to provide a configurable range of selectable dynamic range and sensitivity for photodetection for each pixel in an image sensor array. For example, units in different regions of the array can be programmed to various resistance values, as exemplified by the array on the right side of the figure. Here again, the programming can be adaptive to variable operating requirements, such as fluctuating ambient conditions. For example, in bright environmental conditions, large dynamics can be achieved (at the expense of sensitivity) through adaptive programming, while under dark conditions, high sensitivity can be achieved (at the expense of dynamic range) for the detection of weak signals. Notably (and surprisingly), this is a phenomenon that mimics the human eye. Using the contracted structure of the pupil, the eye achieves a similar signal modulation function. Such adaptive sensor devices provide optimal detection in dynamically changing environments. Using PCM elements in such a device allows the cells to be reprogrammed and maintain their characteristics for 400,000 cycles.

[0055] It will be understood that many changes and modifications may be made to the specific embodiments described. For example, while the CSU is described with particular reference to a photosensor, the CSU may be based on any other type of sensor device. Various other forms of sensor arrays comprising multiple CSUs may be contemplated. In general, features described with reference to one embodiment may be applied to other embodiments as appropriate.

[0056] The descriptions of various embodiments of the present invention are presented for illustrative purposes only and are not intended to be exhaustive or limitful to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used herein have been selected to best describe the principles, practical applications, or technical improvements to the technologies available on the market, or to enable other those skilled in the art to understand the embodiments disclosed herein.

Claims

1. A sensor device, which generates an electrical signal in a circuit in response to a stimulus detected by the device; A programmable non-volatile memory element capable of operating as a load resistor for the sensor device within the circuit, wherein the resistance of the load resistor depends on the programmed state of the memory element; and Output for providing an output signal according to the aforementioned electrical signal and the programmed state. A configurable sensor unit comprising the above.

2. The configurable sensor unit according to claim 1, wherein the sensor unit comprises a plurality of programmable non-volatile memory elements capable of operating as the load resistor for the sensor device within the circuit.

3. The aforementioned sensor unit is: A plurality of programmable non-volatile memory elements that can be selectively connected within the circuit as load resistors for the sensor device; and Switching circuit configuration for connecting a memory element selected in response to a control signal into the circuit. A configurable sensor unit according to claim 1, comprising:

4. The configurable sensor unit according to claim 1, further comprising a switching circuit configuration for selectively connecting the memory element to the circuit and to a controller for programming the memory element to the programmed state.

5. A sensor device comprising multiple configurable sensor units, wherein the configurable sensor units are: A sensor device, which generates an electrical signal in a circuit in response to a stimulus detected by the device; A programmable non-volatile memory element capable of operating as a load resistor for the sensor device within the circuit, wherein the resistance of the load resistor depends on the programmed state of the memory element; and Output for providing an output signal according to the aforementioned electrical signal and the programmed state. A sensor device having the following features.

6. The sensor device according to claim 5, wherein each sensor unit has a plurality of programmable non-volatile memory elements capable of operating as the load resistor for the sensor device of the unit within the circuit.

7. Each sensor unit is: A plurality of programmable non-volatile memory elements that can be selectively connected within the circuit as load resistors for the sensor device of the unit; and Switching circuit configuration for connecting a memory element selected in response to a control signal into the circuit. The sensor device according to claim 5, having the following features.

8. The sensor device according to claim 5, wherein each sensor unit has a switching circuit configuration for selectively connecting the memory element to the circuit and to a controller for programming the memory element to the programmed state.

9. The sensor device according to claim 5, wherein the plurality of sensor units are arranged in a crossbar array having row and column lines for addressing each row and column of the sensor unit and acquiring the output signal from each unit.

10. The sensor apparatus according to claim 9, wherein the sensor device of each sensor unit includes a photosensor.

11. The sensor device according to claim 9, further comprising a controller for controlling the addressing of the sensor unit by the row and column lines and for programming the memory elements of the sensor unit.

12. The sensor device according to claim 11, wherein each sensor unit has a switching circuit configuration for selectively connecting the memory element to the circuit and to the controller for programming the memory element to the programmed state.

13. The sensor device according to claim 11, wherein each sensor unit has a plurality of programmable non-volatile memory elements capable of operating as load resistors for the sensor device of the unit within the circuit.

14. The plurality of memory elements of each sensor unit can be selectively connected within the circuit as load resistors for the sensor device of that unit; Each sensor unit has a switching circuit configuration for connecting a selected memory element into the circuit in response to a control signal; The controller is operable to generate the control signals for each sensor unit. The sensor device according to claim 13.

15. The aforementioned controller is: The memory elements of each sensor unit are programmed to a pre-programmed state corresponding to the respective kernel weights of the convolutional neural network layer; The addressing of the sensor unit is controlled so that the output signal of the sensor unit yields the result of a convolution operation in the neural network layer. The sensor device according to claim 13, which is operable in such a way.

16. The output signals from each sensor unit are provided to the column lines of the crossbar array, and the controller: The memory elements of each sensor unit are programmed to a pre-programmed state corresponding to the respective kernel weights of the convolutional neural network layer; The addressing of the sensor units and the selective connection of each of the plurality of memory elements within the circuit of the unit are controlled by the control signals for each sensor unit such that the signals on the column lines of the array result in the multiplication and accumulation stage of the convolution operation in the neural network layer. The sensor device according to claim 14, which is operable in such a way.

17. The sensor device according to claim 16, wherein each sensor unit has a capacitor for storing charge in accordance with the electrical signal and the programmed state of the memory element, and the capacitor is selectively connectable to the output for providing the output signal of the sensor unit.

18. The sensor units within the crossbar array are grouped into rows and columns of subarrays, each containing a plurality of m sensor units; The controller programs the memory elements of the sensor units in each subarray into programmed states corresponding to the respective weights of the neural network, and as a result, the output signals of the sensor units in each subarray are operated to correspond to weighted signals transmitted by each neuron in the first layer of the network to neurons in the second layer of the network. The sensor device according to claim 11.

19. The sensor device according to claim 18, wherein the output signals of each sensor unit are provided to the column lines of the crossbar array, and the controller is operable to control the addressing of the sensor units so that the signals on the column lines of the array yield the result of the multiplication and accumulation stage of a matrix-vector multiplication operation in the neural network.

20. A configurable sensor unit, where the configurable sensor unit is: A sensor device, which generates an electrical signal in a circuit in response to a stimulus detected by the device; A programmable non-volatile memory element capable of operating as a load resistor for the sensor device within the circuit, wherein the resistance of the load resistor depends on the programmed state of the memory element; and Output for providing an output signal according to the aforementioned electrical signal and the programmed state. Having; and A controller for programming the memory elements of the sensor unit into a programmed state according to the operating requirements of the sensor unit. A sensor device equipped with the following features.

21. The sensor device according to claim 20, wherein the controller is operable to adaptively program the memory element to various programmed states in accordance with variable operating requirements relating to the sensor unit.

22. An array of configurable sensor units, where the configurable sensor units are: A sensor device, which generates an electrical signal in a circuit in response to a stimulus detected by the device; A programmable non-volatile memory element capable of operating as a load resistor for the sensor device within the circuit, wherein the resistance of the load resistor depends on the programmed state of the memory element; and Output for providing an output signal according to the aforementioned electrical signal and the programmed state. Having; Output for providing output signals according to the electrical signals and the programmed states; and A controller for programming the memory elements of the sensor unit to their respective programmed states according to the operating requirements of the sensor device. A sensor device equipped with the following features.

23. The sensor device according to claim 22, wherein the controller is operable to program memory elements of various groups of sensor units into various programmed states according to various operating requirements relating to those sensor units.

24. The sensor device according to claim 22, wherein the controller is capable of adaptively programming the memory elements of the sensor unit to various programmed states according to variable operating requirements relating to the sensor device.

25. The sensor apparatus according to claim 22, wherein the sensor device of each sensor unit includes a photosensor.

Images