Configurable sensor unit and sensor device

Abstract

A configurable sensor unit and a sensor device are provided. A configurable sensor unit comprising: a sensor device for generating an electrical signal in an electrical circuit as a function of a stimulus sensed by the device; and a programmable non-volatile storage element operable in the circuit as a load resistor for the sensor device, whereby a resistance of the load resistor depends on a programming state of the storage element. The sensor unit has an output for providing an output signal dependent on the aforementioned electrical signal and the programming state. A sensor device may include a plurality of these configurable sensor units. Such a sensor device may be configured to perform in-sensor computing operations of a neural network architecture. Additional sensor devices include a configurable sensor unit and a controller for programming storage elements of the sensor unit to a programmed state in accordance with operating requirements of the sensor unit.

Description

Technical Field

[0001] The present invention generally relates to a configurable sensor unit and a sensor device incorporating such a sensor unit. Background Art

[0002] Sensors are used in many applications in science, technology, and the Internet of Things (IoT). Sensors (e.g., image sensors, audio sensors, motion sensors, temperature sensors, chemical sensors, and tactile sensors) can be deployed in a variety of environments and used to detect a variety of stimuli (light, sound, motion, etc.) for a variety of purposes. The output of such sensors can be processed in various ways and is typically processed in a neural network (NN) architecture designed to perform cognitive tasks. As an illustrative example, a still image or video image can be analyzed by a NN system designed to detect objects, people, human emotions, etc. The processing of sensor output can involve various types of computations. For example, in an NN system, the sensor output is converted into a digital signal, which is then propagated through weighted connections in the network. Signal propagation typically involves computations such as multiply-accumulate (MAC) and matrix-vector multiplication (MVM) operations, in which the signal is multiplied by the network weights according to the specific network architecture. Such computations involve multiple data transfers between memory and processing units and require a large amount of processing resources.

[0003] In-memory computing (IMC) architectures have been proposed, in which specific computing tasks such as MAC and MVM operations can be performed in situ in computing memory cells using memory cell arrays. This alleviates the processing bottleneck caused by data movement between memory and processing units, thereby improving computing efficiency. Similarly, image sensors based on two-dimensional materials have been proposed for in-sensor MVM operations (see Mennel et al., “Ultrafast machine vision with 2D material neural network imagesensors”, Nature 579, 62-66 (2020)) and convolution operations for edge detection (see Seokhyeong et al., “Programmable black phosphorous image sensor for broadband optoelectronic edge computing”, Nature Communications 13, 1485 (2022)). In these image sensors, the photoresponsivity of the two-dimensional material photosensor device is adjusted by using field effects to modulate the charge stored in the multi-gate electrode or gate dielectric layer of the device to modulate the channel doping, thereby storing NN weights. These systems require complex device architectures and lack the scalability required for image sensor applications.

[0004] Improvements in sensors and sensor devices with computing capabilities would be highly desirable. Summary of the Invention

[0005] A configurable sensor unit and a sensor device are provided. A configurable sensor unit comprises: a sensor device for generating an electrical signal in a circuit according to a stimulus sensed by the device; and a programmable non-volatile memory element capable of operating in the circuit as a load resistor for the sensor device, wherein the resistance of the load resistor depends on the programming state of the memory element. The sensor unit has an output terminal for providing an output signal that depends on the aforementioned electrical signal and the programming state. The sensor device may include a plurality of these configurable sensor units. Such a sensor device may be configured to perform in-sensor computational operations of a neural network architecture. Another sensor device includes a configurable sensor unit and a controller for programming the memory element of the sensor unit to a programming state that depends on the operational requirements of the sensor unit.

[0006] Another aspect of the present invention provides a sensor device comprising a plurality of configurable sensor cells as described above. The plurality of sensor cells may be arranged in a crossbar array having row and column lines for addressing respective rows and columns of sensor cells to obtain an output signal from each cell. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is a schematic representation of one embodiment of a configurable sensor unit;

[0008] Figure 2 is a schematic representation of another embodiment of a sensor unit;

[0009] Figure 3 An example of a programmable non-volatile storage element of a sensor unit is shown;

[0010] Figure 4 An embodiment of a sensor cell having a switching circuit for programming operation is shown;

[0011] Figure 5 Another embodiment of a switching circuit having selective connection of different storage elements in a sensor circuit is shown;

[0012] Figure 6 is a schematic representation of a conventional image sensor array;

[0013] Figure 7 is a schematic representation of a computational sensor device embodying the present invention;

[0014] Figure 8 Shows the steps of convolution operation in CNN;

[0015] Figure 9 Shown in Figure 7 Switching between storage elements in a sensor unit of the device to implement a convolution operation;

[0016] Figure 10 shows a MAC operation for a convolution operation in an embodiment of a sensor device;

[0017] Figure 11 is a schematic representation of another embodiment of a configurable sensor unit;

[0018] Figure 12 is a schematic diagram of the arrangement of device layers in an integrated sensor device embodying the present invention;

[0019] Figures 13 to 15 shows additional circuit configurations of sensor units embodying the present invention;

[0020] Figure 16 and Figure 17 is a schematic diagram of a convolution signal readout operation in a corresponding embodiment of a sensor device;

[0021] Figure 18 The operation of the MLP network is shown;

[0022] Figure 19 shows an array structure in an embodiment of a computational sensor device implementing an MLP layer; and

[0023] Figure 20 The structure and operation of an adaptive sensor device embodying the present invention are shown. DETAILED DESCRIPTION

[0024] A first aspect of the present invention provides a configurable sensor cell. The sensor cell includes a sensor device configured to generate an electrical signal in a circuit based on a stimulus sensed by the device; and a programmable non-volatile memory element capable of operating in the circuit as a load resistor for the sensor device, wherein the resistance of the load resistor depends on a programmed state of the memory element. The sensor cell has an output terminal for providing an output signal dependent on the electrical signal and the programmed state.

[0025] By using a programmable non-volatile memory element as a load resistor for the sensor device, a sensor unit embodying the present invention can be configured in an extremely simple manner and can be programmed once or dynamically for a variety of purposes. The use of a memory element as a load resistor allows the signal generated by the sensor device to be calculated directly, so that sensing (signal generation) and calculation (via the programmable load resistor) are performed locally without the need for data conversion. The responsiveness of the sensor unit can be configured as required and then saved in the non-volatile state of the load resistor for subsequent sensor operation. Using this simple and effective device structure, the sensor unit can be easily adapted to different operating conditions and / or application requirements. This structure also provides the basis for an efficient, highly scalable computational sensor device that has the ability to perform in-sensor calculations, such as MAC and MVM operations in neural network architectures.

[0026] The sensor cell can include switching circuitry for selectively connecting a memory element in the sensor circuit and for connecting to a controller to program the memory element to a desired programming state. Using the same basic cell structure, different sensor cells can be programmed once for different operating conditions and / or application requirements, or cells can be programmed dynamically, for example, to adapt to changing operating conditions or for in-sensor computing applications.

[0027] In a particularly advantageous embodiment, the sensor unit includes a plurality of programmable non-volatile memory elements capable of operating in a circuit as load resistors for the sensor device. The memory elements can be selectively connected in the circuit, and the unit includes a switching circuit for connecting selected memory elements in the circuit in response to a control signal. These embodiments allow switching between different, individually programmable load resistors to adapt the sensor unit to different operating conditions / application requirements and / or implement more complex in-sensor computations, such as for use in convolutional neural network (CNN) architectures.

[0028] Another aspect of the present invention provides a sensor device comprising a plurality of configurable sensor cells as described above. The plurality of sensor cells can be arranged in a crossbar array having row and column lines for addressing individual rows and columns of sensor cells to obtain an output signal from each cell. This provides an efficient system architecture for various types of sensor devices (e.g., image sensors or tactile sensors) in which the sensor cells are spatially distributed. The sensor device can include a controller for controlling the addressing of the sensor cells via the row and column lines and for programming the memory elements of the sensor cells.

[0029] A computational sensor device is provided for accelerating the processing of sensor signals in a CNN using a sensor cell having multiple storage elements operable as load resistors. A controller of such a device is operable to program the storage element of each sensor cell to a programming state corresponding to the corresponding kernel weight of a CNN layer, and to control the addressing of the sensor cells so that the output signal of the sensor cell provides the result of a convolution operation in the CNN layer. In a preferred embodiment herein, the output signal of each sensor cell is provided to a column line of a crossbar array of sensor cells. The controller is operable to control the addressing of the sensor cell and the selective connection of each storage element in the circuitry of the cell via a control signal for each sensor cell, so that the signal on the column line of the array provides the result of the multiply-accumulate step of the convolution operation in the CNN layer. This allows in-sensor computation to be performed by switching between load resistors in the sensor cell to switch the kernel weights required for the convolution, and MAC operations can be performed by addressing the cells in parallel. In some architectures, multiple MAC operations can also be performed in parallel. For each storage element, the sensor cell may also include a capacitor for storing a charge dependent on the electrical signal from the sensor device and the programming state of the storage element, with the capacitor being selectively connectable to an output terminal to provide the output signal of the sensor cell. This allows temporary storage of the output signal and provides greater flexibility for processing the signal in the order required for convolution.

[0030] Figure 1 The basic circuit components of a configurable sensor unit (CSU) embodying the present invention are shown. The CSU 1 has a sensor device 2 for generating an electrical signal in a sensor circuit in response to a stimulus sensed by the device. In this example, the sensor device 2 comprises a photosensor, here a photodiode, which generates an electrical signal in response to incident light. The CSU also includes a programmable non-volatile memory element 3, which is capable of operating in the circuit as a load resistor R for the sensor device 2. L . Load resistor R L The resistance of R depends on the programming state of the storage element 3. L The value of can be set by a programming operation as schematically indicated in the figure. In this embodiment, a storage element 3 is connected in series with the sensor 2 via an amplifier 4 for amplifying the sensor output. The CSU has an output 5 for providing an output signal that depends on the electrical signal from the sensor 2 and the programming state of the storage element 3. In this example, the signal at the output 5 is supplied by a load resistor R L The voltage drop across the terminals is determined by the (amplified) photocurrent generated by the photodiode 2 . Figure 2 (in Figure 1 ) shows a similar embodiment of a CSU. In this CSU 6, an amplifier 4 is located at the output to amplify the current across the load resistor R L The voltage generated across the terminals.

[0031] The programmable memory element 3 can be implemented using various known memory technologies. Figure 3 A particularly convenient example is shown in FIG. Here, the memory element is implemented as a mushroom-shaped PCM cell. The PCM cell comprises a volume of phase change material, such as germanium antimony telluride (GTS), between the top and bottom electrodes of the cell. The cell can be programmed to different analog resistance states by applying voltage pulses (“write” pulses) via the electrodes to change the proportions of amorphous and crystalline phases with different resistance characteristics within the phase change material. During programming, the resistance state of the cell can be read by applying “read” pulses of sufficiently low voltage so that the cell state is not affected, and measuring the resulting current through the cell. By applying such a programming signal, the PCM memory element 3 can be programmed to the desired resistance value R L The storage element 3 may also be implemented using other non-volatile memory devices, such as NAND flash memory cells, FeFET cells, and resistive RAM (RRAM or ReRAM) cells, including conductive bridge RRAM cells, oxide or metal oxide RRAM cells, carbon RRAM cells, and magnetoresistive random access memory (MRAM) cells or ferroelectric random access memory (FeRAM) cells. One or more of these cells may be arranged in various circuit configurations to provide a programmable resistive storage element 3.

[0032] During programming, the storage element 3 can be decoupled from the sensor device 2. Figure 1 The basic circuit configuration, in Figure 4 Here, the CSU comprises a switching circuit 8 for selectively connecting the storage element 3 in the sensor circuit and to a controller 9 for programming the storage element to a desired state.

[0033] Providing a programmable memory element 3 as a load resistor for the sensor device allows for a particularly simple adjustment of the CSU's responsivity. The memory element 3 can be programmed to a desired state either once or dynamically, allowing individual CSUs to be configured / reconfigured as needed for different purposes. An exemplary application is described in more detail below. The CSU's responsivity can be configured as desired and stored in the non-volatile state of the memory element 3 for subsequent sensor operation.

[0034] A CSU embodying the present invention may include a plurality of programmable non-volatile memory elements, each programmable non-volatile memory element being capable of operating as a load resistor for a sensor device in a sensor circuit. Figure 5 One such embodiment is shown, also for Figure 1 The CSU 10 has a plurality of storage elements 3 that can be selectively connected as load resistors for the sensor devices 2 in the sensor circuit. A switching circuit 11 responds to a switch control signal from a controller (not shown in this figure) to connect selected storage elements 3 in the sensor circuit. The switching circuit 11 also selectively connects each storage element to the controller for programming the individual elements to a desired resistance value. Thus, different load resistors can be switched into the sensor circuit to adapt the CSU operation to different requirements.

[0035] While a simple CSU circuit is described above, the CSU may include other circuit elements, such as additional resistors, capacitors, and / or bias voltages, and may have various other circuit arrangements. Other examples of CSU circuits are described below. Furthermore, the CSU may generally utilize any type of sensor device 2, such as sound sensors, motion sensors, temperature sensors, pressure sensors, chemical (e.g., gas) sensors, and tactile sensors, as well as other optoelectronic sensor devices such as phototransistors.

[0036] A CSU embodying the present invention may be part of a sensor device comprising an assembly of a plurality of such units. Figure 6is a simplified schematic diagram of a conventional image sensor. The sensor comprises an array of pixel cells arranged in a crossbar configuration having row and column lines (alternatively referred to as "word lines" and "bit lines") for addressing corresponding rows and columns of pixel cells. Each pixel cell stores an electrical charge that is dependent on the incident light. When a particular pixel is addressed (by applying row select and column select signals to the corresponding row and column lines), the stored charge is converted by the pixel cell's amplifier into a voltage, thereby generating a current on the column line. This current is amplified at the sensor output and converted by an analog-to-digital converter (ADC) into a digital signal indicating the pixel intensity. A CSU embodying the present invention may similarly be configured in a crossbar arrangement, an example of such a device being Figure 7 shown.

[0037] Figure 7 The image sensor 15 is schematically shown and includes a plurality of Figure 6 The CSU 16 is configured in a crossbar topology (here a simplified 4x4 array). Figure 6 As in the example, the crossbar array has row and column lines for addressing the corresponding rows and columns of CSUs 16 to obtain the output signal from each unit. In this example, the sensor 15 has an output amplifier and ADC for outputting a digital signal for further processing. In alternative embodiments, the column lines can be connected to subsequent processing units in other ways, as shown in the following examples.

[0038] Each CSU 16 is structured in Figure 7 The CSU 16 is based on Figure 5 The general structure of the invention includes a photoelectric sensor circuit 17 (having Figure 5 The photoelectric sensor and amplifier in the switching circuit 18, and a plurality of programmable storage elements (here as Figure 3 These programmable memory elements can be selectively connected as a load resistor R of the photosensor in the photosensor circuit. L In this embodiment, when a CSU is addressed, the output signal of the CSU is provided to the column lines of the array. The sensor device includes an array controller 19 that generates row select signals and column select signals for addressing the CSUs, as well as control signals ("R ") for controlling the selection of a particular load resistor by the switch circuit 18 in each CSU. L The controller 19 also generates write and read pulses for programming the individual PCM elements of each CSU to a desired resistance value.

[0039] The sensor device 15 can be used as a computational sensor for implementing in-sensor computational functions to process acquired images in a NN architecture, such as for inference in cognitive tasks. The use of PCM elements as load resistors for the photosensors allows computations to be performed directly on the signals generated by the photosensors, whereby both sensing (signal generation) and computation (via programmable load resistors) can be performed locally without the need for data conversion. In particular, the change in the responsiveness of the CSU due to the programmable load resistors is analogous to the synaptic weighting operations in a neural network. The use of multiple storage elements in each CSU enables efficient implementation of in-sensor computational functions for CNNs.

[0040] Figure 8 Is a schematic diagram showing the convolution operation on an image at the input layer of a CNN. The synaptic layer of a CNN includes d weight kernels, each of which includes k×k weights, as shown in the figure. The input image (n pixels by n pixels in size) is processed to generate a set of image vectors ("image slices") for further processing using the convolution kernel. These image slices are generated by sliding an image window of size k pixels by k pixels across various positions of the input image. Assuming the step size (window movement amount) is one pixel, the image window can occupy (n-k+1) positions both horizontally and vertically in the input image. This will produce (n-k+1) 2 image slices, each containing a dimension k corresponding to the corresponding window position 2 For each image slice, the MVM operation is performed using a kernel weight matrix formed by d vectors, each matrix including the k values ​​of the corresponding kernel. 2 weights as shown in the figure. The dot product between each image slice and each weight vector in the kernel matrix is ​​calculated to calculate the corresponding point in the output image volume. This will produce a volume containing (n-k+1) 2 The output image of points and depth d is shown in the figure. The points of the output image are mapped to the neurons in the next network layer for further processing in CNN.

[0041] Figure 9 Shows how to have Figure 7 A convolution operation is performed in a computational sensor device with the general structure of FIG. For this convolution, the kernel dimension is k = 2, the stride is s = 1, and there are four kernels, each with four weights, as shown by the kernel weight vectors in the figure. Each pixel can be in one of four positions in the image slice (except for pixels at the edges of the input image). These positions, represented by A, B, C, and D, are illustrated for the shaded pixels at the top of the figure.

[0042] To implement this convolution, each CSU 16 includes sixteen PCM cells. The controller 19 programs these cells to states corresponding to the respective kernel weights. Thus, each PCM cell stores a corresponding one of the sixteen kernel weights, as shown in the matrix at the lower left of the figure. For each pixel of the input image, the photosensor signal from the corresponding CSU must be multiplied by each of the sixteen kernel weights. This can be achieved by switching each PCM cell into the sensor circuit via the switching circuit 18 of the CSU. For example, to perform a multiplication on the sensor output at pixel position A in the image slice, the CSU stores four kernel weights a 11 、a 12 、a 21 and a 22 Similarly, when storing weights (b 11 to b 22 )、(c 11 to c 22 ) and (d 11 to d 22 ) performs multiplication operations on pixel positions B, C, and D, respectively.

[0043] By controlling the addressing of the CSUs and switching between the kernel weights in the various CSUs, the output signals of the CSUs can be combined to provide the result of the convolution operation. For a given image slice in the example shown, by controlling different sets of kernel weights (a 11 to a 22 )、(b 11 to b 22 )、(c 11 to c 22 ) and (d 11 to d 22 ) can perform MAC operations for dot product calculations with each core. The signals on the column lines are then accumulated to provide the results of the multiply-accumulate step of the convolution operation. In this way, MAC / dot product operations can be performed by addressing multiple CSUs switched to the required weights in parallel. In some implementations, multiple MAC operations can also be performed in parallel. This is illustrated below with a simple example.

[0044] Figure 10 An array 20 of 4 by 4 CSUs 16 is shown schematically. To perform a convolution operation using a 2×2 kernel with a stride s=2, the hatched CSUs are addressed via row lines 1 and 2 and column lines 1 to 4, as shown by the dashed window of hatched CSUs. The memory cells storing a given kernel weight are selected in each of the four CSUs in the window. Thus, the output signals o from the two CSUs connected to column line 1 are11 and o 21 are accumulated on the column line. Similarly, the output signals o from the two CSUs in the next column 12 and o 22 is accumulated on column line 2. In this embodiment, column lines 1 and 2 are connected, whereby the column signals are accumulated to provide the MAC result ( 11 +o 21 +o 12 +o 22 ). This provides the result of the dot product operation between the corresponding image slice and the kernel weight vector. The corresponding result of the second image window is provided in parallel on the combined output of column lines 3 and 4. The resulting signal is then output for further processing in the MVM operation. In this implementation, the analog signal can be provided directly to the word line of the in-memory computation unit 21 without conversion for further processing. Thus, in effect, the computation sensor array constitutes an additional core (or "tile") of the IMC unit, thereby accelerating the computation at the input layer of the CNN.

[0045] While a specific convolution example has been described above, it will be appreciated that the convolution computation can be performed in many other ways by appropriate addressing of the CSUs, employing a switching scheme for time-multiplexed switching between kernel weights, and accumulating / further processing the output signals on the column line groups of the array. Since each CSU can be of modest size (e.g., 10 by 10 μm), multiple kernels can be encoded in multiple load resistors without significantly impacting areal density. It should also be noted that using a step size equal to the kernel vector dimension (i.e., s=k) limits the number of PCM cells required per CSU to k.

[0046] In some embodiments, the CSU may store the output signal to facilitate convolution calculations. Specifically, for each storage element, the CSU may include a capacitor for storing a charge that depends on the photosensor signal and the weight stored in the element. Figure 11 An exemplary embodiment is shown. As schematically illustrated, each storage element of a CSU 25 has an associated capacitor 26. When a storage element is connected to a photosensor circuit 27, the associated capacitor is connected to the storage element via a switch circuit 28. The capacitor thus stores a charge that depends on the weighted photosensor signal. In response to another control signal, "Cselect," provided to the switch circuit 28 by an array controller 29, each capacitor is selectively connected to an output of the CSU. Thus, the stored signals can be output to the associated column lines in a desired order, providing greater flexibility for convolution calculations.

[0047] It should be understood that the crossbar array of the CSU can be configured in a variety of ways. In a preferred embodiment, the memory elements of the various CSUs can be configured in one or more memory arrays (tiles) that can be integrated with standard manufacturing methods and topologies used for sensor arrays (such as image sensors, etc.). For example, modern back-illuminated photosensor arrays are manufactured using hybrid bonding technology. Figure 12 It is schematically shown how a memory array can be easily incorporated into the integrated layer structure of such a sensor, providing a densely integrated computational sensor that is compatible with hybrid bonding techniques.

[0048] Various other CSU circuits are also contemplated. Figures 13 to 15 A specific example is shown in FIG (a single PCM element is used here for simplicity). Figure 13 In Figure 1, the photodiode (here biased by voltage V in bias mode) acts as a high-impedance current source, and the voltage drop across the PCM cell is amplified by a gain factor determined by fixed load resistors R1 and R2. The amplifier output voltage generates a current through fixed load resistor R3, which participates in the MAC operation on the column lines of the sensor array. Figure 14 Another arrangement is shown in which the PCM cells implement a weighted operation via a variable gain at the output node of the amplifier. Figure 15 An alternative arrangement is shown in which the photodiode generates a voltage scaled by the value of the feedback resistor R1. The amplifier output voltage produces a current through the PCM element, which can be accumulated on the bit lines of the PCM array for MAC operations.

[0049] The addressing of the CSU may be controlled in various ways such that the output signal of the CSU provides the result of the convolution operation. Figure 16 A readout scheme for convolution operations using a computational sensor device that performs a MAC operation on the column lines of a photosensor array is shown. For an image window of 2×2 pixels and a step size of s=2, convolution operations on entire rows of the image sensor can be performed in parallel using a diagonal readout scheme, as schematically shown in the figure. At any moment, a convolution operation can be performed with a single kernel (here K1). Performing convolution operations using multiple kernels requires switching between PCM cells in the CSU. Additional kernels K2, K3, ... can be stored in a diagonal pattern, below the diagonal pattern of K1 in the PCM array shown, thereby providing MAC operations of consecutive kernels on the same column line. Figure 17 An alternative readout scheme is shown, in which the Figure 15The CSU circuitry performs MAC operations on the bit lines of the PCM crossbar array. Here, convolution operations can be performed in parallel on all image windows in a row using different kernels encoded in different columns of the PCM tile. This allows convolution operations using multiple kernels to be performed in parallel at the expense of providing a separate PCM tile for each image window.

[0050] A CSU embodying the present invention may be applied to other computational topologies, such as a multilayer perceptron (MLP). Figure 18 The signal propagation at the input of an exemplary MLP is shown. The network input (here n by n image) 2 pixels) represents the output signal of the corresponding (conceptual) neuron in the first neuron layer L1. Each of these signals is propagated through weighted connections to the neurons in the second layer L2 of the network, here all m neurons. This requires multiplying each pixel value by each of the m weights. The weighted signals received by each L2 neuron are then summed to provide the input signal to that neuron. This involves n for image pixels × m MAC operations for MVM computation between the n×m matrix of network weights and the vector 1.

[0051] The above calculation can be done in Figure 19 The schematic diagram shows a computational image sensor. The sensor includes a crossbar array of CSUs, each CSU having a programmable storage element for storing the corresponding weights of an n×m weight matrix. As shown in the figure, the CSUs of the main array are grouped into rows and columns of subarrays, each subarray including m CSUs. The array controller (not shown) is operable to program the storage elements of the CSUs in each subarray to store the corresponding weights, so that the output signal of the CSU in each subarray corresponds to the weighted signal transmitted by the corresponding neuron in layer L1 to the m neurons in layer L2. The steps of the MAC operation for the MVM calculation can be performed by appropriate addressing of the CSUs. Specifically, the output signal of each CSU can be provided to the column lines of the crossbar array, and the addressing of the CSUs is controlled so that the signal on the column line provides the result of a multiplication-accumulation step, which can be further processed in a subsequent processing unit to obtain the final result of the MVM operation.

[0052] As can be seen, the CSU embodying the present invention provides an efficient, easily configurable computational sensor device with in-sensor computation capabilities for accelerating computations at the input layer of the NN architecture (where the computation is most intensive). The described operation provides reduced latency and enhanced energy efficiency due to reduced data movement and conversion, and can be integrated with the IMC or other processing units to achieve efficient, ultra-fast end-to-end performance. This provides significant advantages in many applications, such as general autonomous driving and edge computing applications.

[0053] Other embodiments of the present invention provide a sensor device comprising a CSU as described above and a controller for programming the (or each) memory element of the CSU to a programming state that is dependent on the operational requirements of the CSU. This allows sensor units having the same basic structure in different sensor devices to be programmed for different operational requirements, such as to suit different applications / operating environments. The CSU of such a device may have more than one individually programmable and individually selectable memory element to provide additional configuration options. Programming may be a one-time operation suitable for deploying a particular sensor, or the CSU may be adaptively programmed according to the variable operational requirements of the sensor device. Such an adaptive sensor device may comprise an array of individually programmable CSUs. In some embodiments, additional adaptability may be provided by adjusting the gain of the sense amplifiers of the individual CSUs and / or the output amplifiers of the array.

[0054] The following will refer to Figure 20 An illustrative example of an adaptive sensor device is described. The left graph in the figure shows Figure 1 The photosensitivity of the CSU varies with the resistor R L Higher resistance provides a more sensitive CSU, while lower resistance provides an increased dynamic range. This can be used to provide configurable selection of dynamic range and sensitivity to light detection for each pixel in the image sensor array. For example, as illustrated by the array on the right side of the figure, cells in different areas of the array can be programmed to different resistance values. Similarly, programming can be adapted to variable operating requirements, such as changing environmental conditions. For example, in bright environmental conditions, large dynamics can be achieved through adaptive programming (at the expense of sensitivity), and in dark conditions, high sensitivity can be achieved for weak signal detection (at the expense of dynamic range). It is worth noting (and significant) that this is a phenomenon that mimics the human eye. Using the contraction structure of the pupil, the human eye implements a similar signal modulation function. This adaptive sensor device can provide optimal detection in dynamically changing environments. The use of PCM elements in such a device allows the cells to be reprogrammed more than 400,000 times and maintain their characteristics.

[0055] However, it should be understood that various variations and modifications may be made to the specific embodiments described. For example, although the CSU is described with particular reference to a photosensor, the CSU may be based on any other type of sensor device. Various other sensor arrays comprising multiple CSUs are also contemplated. In general, features described with reference to one embodiment may be applied to other embodiments as appropriate.

[0056] The description of various embodiments of the present invention is for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, practical applications, or improvements over existing technologies in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A configurable sensor unit comprising: a sensor device for generating an electrical signal in a circuit based on a stimulus sensed by the device; a programmable non-volatile storage element capable of operating in the circuit as a load resistor for the sensor device, whereby the resistance of the load resistor is dependent on a programmed state of the storage element; as well as An output terminal is configured to provide an output signal that is dependent on the electrical signal and the programming state.

2. The configurable sensor cell of claim 1, comprising a plurality of programmable non-volatile storage elements operable in the circuit as load resistors for the sensor device.

3. The configurable sensor unit of claim 1 , comprising: a plurality of programmable nonvolatile storage elements selectively connectable in the circuit to act as load resistors for the sensor device; as well as A switch circuit is provided for connecting a selected storage element in the circuit in response to a control signal.

4. The configurable sensor unit of claim 1, comprising a switching circuit for selectively connecting the storage element in the circuit and to a controller for programming the storage element to the programmed state.

5. A sensor device comprising a plurality of configurable sensor units, wherein the configurable sensor units comprise: a sensor device for generating an electrical signal in a circuit based on a stimulus sensed by the device; a programmable non-volatile storage element capable of operating in the circuit as a load resistor for the sensor device, whereby the resistance of the load resistor is dependent on a programmed state of the storage element; as well as The output terminal is configured to provide an output signal that is dependent on the electrical signal and the programming state.

6. The sensor arrangement of claim 5, wherein each sensor cell comprises a plurality of programmable non-volatile storage elements capable of operating in the circuit as load resistors for the cell's sensor device.

7. The sensor device of claim 5 , wherein each sensor unit comprises: a plurality of programmable nonvolatile storage elements selectively connectable in the circuit to act as load resistors for the sensor device of the cell; as well as A switch circuit is provided for connecting a selected storage element in the circuit in response to a control signal.

8. The sensor device of claim 5, wherein each sensor unit includes a switching circuit for selectively connecting a storage element in the circuit and connected to a controller to program the storage element to the programming state.

9. The sensor device of claim 5, wherein the plurality of sensor cells are arranged in a crossbar array having row and column lines for addressing corresponding rows and columns of sensor cells to obtain the output signal from each cell.

10. The sensor apparatus of claim 9, wherein the sensor device of each sensor unit comprises a photosensor.

11. The sensor device of claim 9, comprising a controller for controlling addressing of sensor cells via the row and column lines and for programming storage elements of the sensor cells.

12. The sensor device of claim 11, each sensor unit comprising a switching circuit for selectively connecting a storage element in the circuit and connected to the controller to program the storage element to the programming state.

13. The sensor arrangement of claim 11, wherein each sensor cell comprises a plurality of programmable non-volatile storage elements capable of operating in the circuit as load resistors for the cell's sensor device.

14. The sensor device of claim 13, wherein: The plurality of storage elements of each sensor cell are selectively connected in the circuit to act as load resistors for the sensor device of the cell; Each sensor unit includes a switching circuit for connecting a selected storage element in the circuit in response to a control signal; as well as A controller is operable to generate a control signal for each sensor unit.

15. The sensor device of claim 13, wherein the controller is operable to: programming the storage element of each sensor cell to a programmed state corresponding to a corresponding kernel weight of a convolutional neural network layer; and Addressing of the sensor units is controlled such that an output signal of the sensor units provides a result of a convolution operation in the neural network layer.

16. The sensor device of claim 14, wherein the output signal of each sensor unit is provided to a column line of a crossbar array, and wherein the controller is operable to: programming the storage element of each sensor cell to a programmed state corresponding to a corresponding kernel weight of a convolutional neural network layer; and Control signals for each sensor cell control addressing of the sensor cell and selective connection of each of a plurality of storage elements in the circuitry of the cell such that the signal on the column lines of the array provides a result of a multiply-accumulate step of a convolution operation in the neural network layer.

17. A sensor device as claimed in claim 16, wherein for each storage element, each sensor unit includes a capacitor for storing a charge dependent on the electrical signal and the programming state of the storage element, and the capacitor can be selectively connected to the output terminal to provide the output signal of the sensor unit.

18. The sensor device of claim 11, wherein: The sensor units in the crossbar array are grouped into rows and columns of sub-arrays, each sub-array including a plurality of m sensor units; as well as The controller is operable to program the storage elements of the sensor cells in each subarray to programmed states corresponding to respective weights of the neural network such that output signals of the sensor cells in each subarray correspond to weighted signals transmitted by respective neurons in a first layer of the network to neurons in a second layer of the network.

19. A sensor device as claimed in claim 18, wherein the output signal of each sensor unit is provided to a column line of the crossbar array, and wherein the controller is operable to control the addressing of the sensor units so that the signals on the column lines of the array provide the results of the multiplication-accumulation steps of the matrix-vector multiplication operations in the neural network.

20. A sensor device comprising: A configurable sensor unit, wherein the configurable sensor unit comprises: a sensor device for generating an electrical signal in a circuit based on a stimulus sensed by the device; a programmable non-volatile storage element operable in the circuit as a load resistor for the sensor device, whereby the resistance of the load resistor is dependent on a programmed state of the storage element; and an output terminal for providing an output signal dependent on the electrical signal and the programming state; and A controller is configured to program the storage element of the sensor unit to a programming state according to an operation requirement of the sensor unit.

21. The sensor device of claim 20, wherein the controller is operable to adaptively program the storage element to different programming states according to variable operational requirements of the sensor unit.

22. A sensor device comprising: An array of configurable sensor units, wherein the configurable sensor units comprise: a sensor device for generating an electrical signal in a circuit based on a stimulus sensed by the device; a programmable non-volatile storage element operable in the circuit as a load resistor for the sensor device, whereby the resistance of the load resistor is dependent on a programmed state of the storage element; and an output terminal for providing an output signal dependent on the electrical signal and the programming state; an output terminal for providing an output signal dependent on the electrical signal and the programming state; and A controller is configured to program the storage element of the sensor unit to a corresponding programming state according to an operation requirement of the sensor device.

23. The sensor device of claim 22, wherein the controller is operable to program the storage elements of different groups of sensor cells to different programming states according to different operational requirements of the different groups of sensor cells.

24. The sensor device of claim 22, wherein the controller is operable to adaptively program the storage elements of the sensor cells to different programming states according to variable operational requirements of the sensor device.

25. The sensor apparatus of claim 22, wherein the sensor device of each sensor unit comprises a photosensor.

Images