Integrated sensing and machine learning processing device

Abstract

The present disclosure relates to a semiconductor device with integrated sensing and processing capabilities. The semiconductor device includes a sensing module configured to generate a plurality of analog sensing signals, one or more crossbar arrays configured to process the analog sensing signals to generate analog pre-processed sensing data, an analog-to-digital converter (ADC) configured to convert the analog pre-processed sensing data into digital pre-processed sensing data, and a machine learning processing unit configured to process the digital pre-processed sensing data using one or more machine learning models. The machine learning processing unit, the crossbar array, and the ADC are integrated on a processor wafer of the semiconductor device. The sensing module is integrated on a sensor wafer stacked on the processor wafer. [Representative image] Figure 4B

Description

[Technical Field]

[0001] TECHNICAL FIELD Embodiments of the present disclosure relate generally to computing devices, and more particularly to integrated sensing and machine learning processing devices. [Background technology]

[0002] Machine learning (ML) is widely used in facial recognition, speech recognition, natural language processing, image processing, and more. ML typically involves analyzing large amounts of sensing data based on complex machine learning models. Traditional edge devices (local devices in close proximity to sensors that collect sensing data) lack the computational power to perform such analyses. As a result, the sensing data generated by the sensors may need to be digitized and transmitted to a remote computing device (e.g., a data center) with ML processing capabilities. This involves digitizing large amounts of data and may require advanced communication capabilities, as well as significant energy and time consumption, to transfer the digitized sensing data. Transferring raw data from sensors to remote devices may raise privacy concerns, and encrypting the raw sensing data for secure data transfer may further increase the computational costs required for ML. Furthermore, some applications (e.g., medical applications that utilize ML) may require real-time data processing. Therefore, it may be desirable to run machine learning models locally on edge devices. However, traditional edge devices lack the integrated sensing and processing capabilities to locally extract information and features from analog sensing data provided by local sensors and perform ML processing. Summary of the Invention

[0003] The following is a simplified summary of the disclosure to provide a basic understanding of some aspects of the disclosure. This summary is not an extensive overview of the disclosure. It is not intended to identify key features or critical elements of the disclosure or to define the scope of the claims or particular embodiments of the disclosure. Its purpose is to present some concepts of the disclosure in a simplified form as a prelude to the more detailed description that is presented later.

[0004] According to one or more aspects of the present disclosure, there is provided a semiconductor device capable of functioning as an integrated sensing and machine learning processing device. The semiconductor device may include: a sensing module configured to generate a plurality of analog sensing signals; one or more crossbar arrays configured to process the analog sensing signals to generate analog pre-processed sensing data; an analog-to-digital converter (ADC) configured to convert the analog pre-processed sensing data into digital pre-processed sensing data; and a machine learning processing unit configured to process the digital pre-processed sensing data using one or more machine learning models. The machine learning processing unit is fabricated on a processor wafer of the semiconductor device.

[0005] In some embodiments, the sensing module is fabricated on a sensor wafer, and the sensor wafer is connected to the processor wafer via a first interconnect layer.

[0006] In some embodiments, one or more crossbar arrays are fabricated on a processor wafer.

[0007] In some embodiments, the ADC is fabricated on a processor wafer.

[0008] In some embodiments, the sensing module includes an image sensor array and the plurality of analog sensing signals includes a plurality of analog image signals.

[0009] In some embodiments, the analog pre-processed data corresponds to a plurality of features extracted from the analog sensing signal, and the machine learning processing unit performs machine learning using the extracted features.

[0010] In some embodiments, the semiconductor device further includes a packaging substrate, and the processor wafer is connected to the packaging substrate via the second interconnect layer.

[0011] In some embodiments, the machine learning processing unit is powered using an analog sensing signal.

[0012] In some embodiments, the semiconductor device further includes a transceiver configured to transmit predicted outputs generated by the machine learning processing unit based on the one or more machine learning models.

[0013] In some embodiments, the analog pre-processed sensing data represents a convolution of the analog sensing signal with a kernel.

[0014] In some embodiments, the conductance values ​​of multiple crosspoint devices in one or more crossbar arrays are programmed to values ​​representing the kernel.

[0015] In some embodiments, the sensing module includes a two-dimensional sensor array, and a plurality of crosspoint devices of the one or more crossbar arrays are configured to receive as inputs analog sensing signals generated by the two-dimensional sensor array.

[0016] In some embodiments, the one or more crossbar arrays include multiple crossbar arrays arranged in multiple different planes.

[0017] According to one or more aspects of the present disclosure, there is provided a semiconductor device capable of functioning as an integrated sensing and machine learning processing device. The semiconductor device includes a sensing module configured to generate a plurality of analog sensing signals and a machine learning processor configured to generate predicted outputs by processing the analog sensing signals using one or more machine learning models. The machine learning processor includes a plurality of crossbar arrays configured to generate a plurality of analog outputs representative of the predicted outputs, and an analog-to-digital conversion unit configured to convert the plurality of analog outputs representative of the predicted outputs into digital signals representative of the predicted outputs.

[0018] In some embodiments, the semiconductor device further includes a transceiver configured to transmit a signal representing the predicted output generated by the machine learning processing unit to the computing device.

[0019] In some embodiments, the transceiver may receive instructions from the computing device to perform an operation based on the predicted output.

[0020] In some embodiments, the sensing module is fabricated on a sensor wafer, the machine learning processor is fabricated on a processor wafer, and the sensor wafer is connected to the processor wafer via a first interconnect layer.

[0021] In some embodiments, the semiconductor device further includes a packaging substrate, and the processor wafer is connected to the packaging substrate via the second interconnect layer.

[0022] In some embodiments, the sensing module includes an image sensor array and the plurality of analog sensing signals includes a plurality of analog image signals.

[0023] In some embodiments, the sensing module includes a two-dimensional sensor array, and in some embodiments, a plurality of crosspoint devices of the plurality of crossbar arrays are configured to receive as inputs the analog sensing signals generated by the two-dimensional sensor array.

[0024] In some embodiments, the multiple crossbar arrays are arranged in different planes. [Brief explanation of the drawings]

[0025] Various embodiments of the present disclosure will be more fully understood from the following detailed description and the accompanying drawings, which, however, are not intended to limit the disclosure to particular embodiments but are for purposes of explanation and understanding.

[0026] [Figure 1A] FIG. 1A is a schematic diagram illustrating an example of a processing device with integrated sensing and processing capabilities, according to some embodiments of the present disclosure. [Figure 1B] FIG. 1B is a schematic diagram illustrating an example of a processing device with integrated sensing and processing capabilities, according to some embodiments of the present disclosure. [Figure 2] FIG. 2 is a diagram illustrating an example of a crossbar array according to some embodiments of the present disclosure. [Figure 3] FIG. 3 is a schematic diagram illustrating an example of a three-dimensional crossbar array, according to some embodiments of the present disclosure. [Figure 4A] FIG. 4A is a schematic diagram illustrating an example of a semiconductor device that can function as a machine learning processor, according to some embodiments of the present disclosure. [Figure 4B] FIG. 4B is a schematic diagram illustrating an example of a semiconductor device that can function as a machine learning processor, according to some embodiments of the present disclosure. [Figure 5A] FIG. 5A illustrates a cross-sectional view of an example image sensor wafer, according to some embodiments of the present disclosure. [Figure 5B]FIG. 5B illustrates a cross-sectional view of an example image sensor wafer, according to some embodiments of the present disclosure. [Figure 6A] FIG. 6A is a schematic diagram illustrating example functional components of a processor wafer, according to some embodiments of the present disclosure. [Figure 6B] FIG. 6B is a schematic diagram illustrating example functional components of a processor wafer, according to some embodiments of the present disclosure. [Figure 7A] FIG. 7A is a schematic diagram illustrating a cross-sectional view of an example semiconductor device, according to some embodiments of the present disclosure. [Figure 7B] FIG. 7B is a schematic diagram illustrating a cross-sectional view of an example semiconductor device, according to some embodiments of the present disclosure. [Figure 7C] FIG. 7C is a schematic diagram illustrating a cross-sectional view of an example semiconductor device, according to some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0027] Aspects of the present disclosure provide a processing device with integrated sensing and machine learning capabilities and a method for manufacturing the same. The processing device according to the present disclosure can include a sensing module and a machine learning (ML) processor integrated on the same semiconductor device using three-dimensional (3D) chiplet integration or monolithic 3D integration. The sensing module can include a sensor array that generates analog sensing data (e.g., analog image signals generated by an image sensor). The ML processor can process the analog sensing data using one or more machine learning models.

[0028] In one implementation, the ML processor may include a preprocessing unit capable of preprocessing analog sensing data for ML processing by feature extraction, dimensionality reduction, image processing, etc. The preprocessing unit may include one or more crossbar arrays capable of preprocessing the analog sensing data in the analog domain. Each crossbar array may be a circuit structure having interconnected electrically conductive lines sandwiching a resistive switching material at their crosspoints. The resistive switching material may include, for example, memristors (also known as resistive random access memory (RRAM) or ReRAM)). The analog sensing data may be provided as an input signal to the crossbar array. The crossbar array may generate an analog output signal representing the preprocessed sensing data. The analog output signal is then converted to a digital signal representing the preprocessed sensing data for subsequent machine learning processing by the ML processor. By preprocessing the analog sensing data in the analog domain and digitizing the preprocessed sensing data rather than the raw sensing data, the ML processor described herein enables significant data reduction because only a small amount of information (e.g., the preprocessed sensing data) is digitized and transmitted from the edge sensing module to the next layer of the network.

[0029] In another implementation, the ML processor can run a machine learning model on the analog sensing data and generate analog signals representing the predicted output of the ML processing (e.g., classification results, labels assigned to the analog sensing data, outputs of layers of a neural network, decisions made based on the ML model, etc.). For example, the ML processor can utilize a crossbar array to implement a multi-layer neural network. The analog output signal generated by the crossbar array represents the predicted output and can be converted to a digital output and transmitted to another computing device.

[0030] In some embodiments, the processing device can be implemented using 3D chiplet integration. For example, a chiplet implementing the processing device can include a processor wafer and a sensor array stacked on the processor wafer. The sensor wafer can include a sensing module. The processor wafer can include an ML processor. The sensor wafer can be connected to the processor wafer via an interconnect layer utilizing through-silicon-via (TSV) stacking, hybrid metal junctions, in-pixel hybrid junctions, etc. Sensing signals generated by the sensor array can be provided to the ML processor via the interconnect layer. In this way, all CMOS (complementary metal-oxide semiconductor) components and integrated circuits required for machine learning can be integrated into a single wafer piece to process analog sensing signals generated by sensing modules embedded in the sensor wafer.

[0031] The chiplets described herein can provide 3D heterogeneous integration of sensing and processing capabilities, enabling desirable hardware processing capabilities such as near-field sensing processing, in-memory computing, analog computing, and parallel computing. Chiplets can be used to implement 3D neural network hardware, resulting in increased device density, less complex connectivity, and reduced communication losses. In some embodiments, the processing devices described herein can also provide a two-dimensional interface (a cross-section of a 3D neural network) for communicating with a 2D sensor array (e.g., an image sensor array), allowing sensing data generated by the 2D sensor array to be directly input into a 3D neural network for processing without the need for signal storage or reconstructing it into one-dimensional data (e.g., vectors that serve as input to a traditional 2D neural network).

[0032] Furthermore, most sensing signals can be viewed as some form of energy (e.g., temperature, mechanical force, photons, vibration, chemicals, electromagnetic waves, etc.) and can be easily converted into an electrical signal (e.g., 0.5 V) using an emerging device. These electrical signals can serve not only as analog data to be processed, but also as a potential source of power to self-power the sensing module and other components of the processing device. In some embodiments, the processing device can thus wake up only in the presence of a sensing signal, enabling event-driven applications to be implemented, further reducing the amount of data collected for ML processing and energy consumption.

[0033] 1A and 1B are schematic diagrams illustrating example processing devices 100a and 100b with integrated sensing and processing capabilities, according to some embodiments of the present disclosure.

[0034] 1A, processing device 100a may include a sensing module 110, a machine learning (ML) processor 120, and a communication module 130. ML processor 120 may further include a pre-processing unit 121, an analog-to-digital conversion (ADC) unit 123, and a machine learning (ML) processing unit 125. Sensing module 110, ML processor 120, and communication module 130 may be integrated into a chiplet structure, such as the chiplet structure described in connection with FIGS. 4A-4B below.

[0035] The sensing module 110 can include one or more sensor arrays. Each sensor array can include one or more sensors capable of detecting and / or measuring a physical property and generating an electrical signal representative of the physical property. Examples of sensors include image sensors, sound sensors, chemical sensors, pressure sensors, heat sensors, temperature sensors, vibration sensors, microbial fuel cells, electromagnetic sensors, etc. In some embodiments, the multiple sensor arrays of the sensing module 110 can include various types of sensors. In some embodiments, the sensing module 110 can include one or more image sensors, as described in connection with Figures 5A-5B below. In some embodiments, the sensing module 110 can generate analog sensing data in the form of an analog sensing signal (e.g., a voltage signal, a current signal, etc.), such as an analog image signal generated by an image sensor array (e.g., a CMOS image sensor) capable of detecting light and generating an analog image signal representative of the detected light.

[0036] In some embodiments, the sensors of sensing module 110 can obtain enough energy to operate ML processor 120 and / or processing device 100a without requiring an external power source. For example, the electrical signals generated by sensing module 110 can be used to power ML processor 120 and / or processing device 100a.

[0037] The ML processor 120 can process the analog sensing data generated by the sensing module 110 using one or more machine learning models. For example, the preprocessing unit 121 can process the analog sensing data and generate analog preprocessed sensing data. The preprocessing unit 121 can perform any suitable operation on the analog sensing signal to prepare the analog sensing data for subsequent processing by the ML processing unit 125. For example, the preprocessing unit 121 can perform feature extraction on the analog sensing data, and the extracted features of the analog sensing data can be used in subsequent ML processing. As another example, the preprocessing unit 121 can perform dimensionality reduction on the analog sensing data to reduce the amount of data processed in subsequent ML processes. As a further example, the preprocessing unit 121 can perform one or more convolution operations (e.g., a two-dimensional convolution operation, a depthwise convolution operation, etc.) on the analog sensing data. As yet another example, the preprocessing unit 121 can normalize the analog sensing data, rescale and / or resize the analog sensing data, denoise the analog sensing data, etc. In some embodiments, where the analog sensing data includes analog image signals, the pre-processing unit 121 may process the analog sensing data using suitable image processing techniques.

[0038] The pre-processing unit 121 may include one or more crossbar arrays capable of processing analog sensing signals in the analog domain. Each crossbar array may include a plurality of interconnected electrically conductive wires (e.g., row wires, column wires, etc.) and crosspoint devices fabricated at the intersections of the electrically conductive wires. The crosspoint devices may include, for example, memristors, phase-change memory devices, floating gates, spintronic devices, and / or other suitable devices with programmable resistance. In some embodiments, the crossbar array may include one or more crossbar arrays described in connection with Figures 2-3 below.

[0039] As an example, the crossbar array can receive an input voltage signal V and generate an output current signal I. The relationship between the input voltage signal and the output current signal can be expressed as I=VG, where G represents the conductance value of the crosspoint device. In this manner, the input signal is weighted by its conductance at each crosspoint device according to Ohm's law. The weighted currents are output via each bit line and can accumulate according to Kirchhoff's current law. The conductance values ​​of the crosspoint devices can be programmed to values ​​and / or weights representing one or more matrices used to perform the preprocessing of the analog sensing data described above (e.g., feature extraction, dimensionality reduction, convolution, image processing, etc.). The crossbar array can receive as input the analog sensing signals generated by the sensors of the sensing module 110 and generate an analog output signal (e.g., a current signal) representing the preprocessed analog sensing data.

[0040] In some embodiments, the sensing module 110 may include sensors arranged as a two-dimensional (2D) sensor array. Because each sensor may generate an analog sensing signal, the output of the 2D sensor array may be considered a 2D output including the analog sensing signals generated by the sensors (e.g., m×n analog sensing signals generated by m×n sensors). The pre-processing unit 121 may include a three-dimensional (3D) crossbar array including multiple 2D crossbar arrays arranged in a 3D manner. For example, the 2D crossbar arrays may be arranged in different planes (e.g., parallel planes perpendicular to a substrate on which the 3D crossbar array is fabricated). The cross-section of the 3D crossbar array is 2D, allowing the 2D output (e.g., m×n analog sensing signals) generated by the sensing module 110 to be received and / or processed without converting it into one-dimensional data (e.g., vectors representing the sensing signals generated by the sensors). The 3D crossbar array circuit may be and / or include the 3D crossbar array circuit described in connection with FIG. 3 below. In some embodiments, the 3D crossbar array circuit can be fabricated using techniques described in connection with U.S. patent application Ser. No. 16 / 521,975, entitled "Crossbar Array Circuit with 3D Vertical RRAM," the entirety of which is incorporated herein by reference.

[0041] The analog-to-digital converter (ADC) 123 may include any suitable circuitry for converting analog pre-processed sensing data into digital pre-processed sensing data. In some embodiments, the ADC unit 123 may include one or more ADCs 250, which are described in connection with FIG. 2 below.

[0042] The ML processing unit 125 may include circuitry for processing the digital preprocessed sensing data using one or more machine learning models. In some embodiments, the ML processing unit 125 may include a digital signal processor. The ML processing unit 125 may generate predicted outputs by running machine learning models trained using the digital preprocessed sensing data. The predicted outputs may represent, for example, classification results (e.g., class labels assigned to the sensing data), decisions made based on the machine learning models, etc. A machine learning model may refer to a model artifact generated by a processing device using training data that includes known training inputs and corresponding known outputs (correct answers for each training input). The processing device may find patterns in the training data that map known inputs to known outputs (predicted outputs) and provide a machine learning model that captures these patterns.

[0043] The machine learning model may include a machine learning model configured with a single level of linear or nonlinear operations (e.g., a support vector machine), a neural network configured with multiple levels of nonlinear operations, etc. The neural network may include an input layer, one or more hidden layers, and an output layer. The neural network may be trained by adjusting the weights of the neural network according to, for example, a backpropagation learning algorithm. In some embodiments, the crossbar array of the preprocessing unit 121 may implement one or more layers of the neural network. For example, the analog preprocessed sensing data generated by the preprocessing unit 121 may represent the output of the input layer or hidden layer of the neural network.

[0044] Communications module 130 may include any suitable hardware and / or software for facilitating communication between processing device 100a and one or more other computing devices. For example, communications module 130 may include one or more transceivers capable of transmitting and / or receiving RF (radio frequency) signals. Communications module 130 may include components for implementing one or more other wireless transmission protocols (e.g., Wi-Fi, BLUETOOTH, ZIGBEE, cellular, etc.). In some embodiments, communications module 130 may include one or more antennas, which may be integrated into processor wafer 420 or package substrate 430 of FIGS. 4A-4B . Communications module 130 may forward the output of ML processor 120 to another computing device (e.g., a cloud computing device) for further processing. In some embodiments, communications module 130 may further receive instructions from a computing device to perform an operation based on the predicted output (e.g., turn on a display based on the facial recognition result, transmit data to one or more other processing devices, present media content, etc.).

[0045] 1B, processing device 100b may include a sensing module 110, a machine learning (ML) processor 140, and a communication module 130. Sensing module 110 and communication module 130 may be the same as their respective counterparts described in connection with FIG. 1A above.

[0046] The ML processor 140 can process the analog sensing data generated by the sensing module 110 using one or more machine learning models. The ML processor 140 can include a machine learning (ML) processing unit 141 and an ADC 143. The ML processing unit 141 can process the analog sensing data generated by the sensing module 110 using one or more machine learning models to generate an analog predicted output. The analog predicted output can include one or more analog signals.

[0047] In some embodiments, the ML processing unit 141 may include one or more crossbar arrays, each including the crossbar array described in connection with FIG. 2 below. In some embodiments, the ML processing unit 141 may include a 3D crossbar array described in connection with FIG. 3 below. In some embodiments, the crossbar array may implement a neural network that runs a machine learning algorithm. The output signal of the crossbar array may represent the output of the neural network. The neural network may include multiple convolution layers, each performing a specific convolution operation (e.g., 2D convolution, depthwise convolution, etc.). Each layer of the neural network may be implemented using one or more crossbar arrays. For example, one or more first crossbar arrays may implement a first layer (e.g., input layer) of the neural network. The first crossbar array may receive analog sensing data generated by the sensing module 110 as input and perform one or more convolution operations on the analog sensing signals. Performing a convolution operation on the analog sensing data may include convolving different portions of the sensing data with one or more kernels. For example, 2D convolution can be performed by applying a single convolution kernel to analog sensing data. More specifically, the convolution kernel can be used to scan each portion of the sensing input data that is the same size as the convolution kernel to generate a convolution result. As another example, performing depthwise convolution on sensing data includes convolving each channel of the sensing data with a respective kernel and stacking the convolved outputs. For example, the conductance values ​​of multiple crosspoint devices in a first crossbar array can be programmed to values ​​that represent the 2D convolution kernel. An analog sensing signal can be provided as an input signal to the first crossbar array. The first crossbar array can output a current signal that represents the convolution of the analog sensing signal and the 2D convolution kernel.In some embodiments, the crossbar array can store multiple 2D convolution kernels by mapping each of the 2D convolution kernels to multiple crosspoint devices of the first crossbar array. The first crossbar array can output multiple output signals (e.g., current signals) representing the convolution results. The outputs of the first crossbar array can be provided to one or more second crossbar arrays implementing a second layer of a neural network for processing. The outputs of the second crossbar array (e.g., analog current signals) can represent the outputs of the second layer of the neural network. The outputs of the second crossbar array can be provided to one or more third crossbar arrays implementing a subsequent layer (e.g., a second hidden layer) of the neural network for processing. The one or more third crossbar arrays can implement an output layer of the neural network. The outputs of the third crossbar array (e.g., analog current signals) can represent the outputs of the neural network. In some embodiments, the neural network may be implemented using a crossbar array as disclosed in U.S. patent application Ser. No. 16 / 125,454, entitled "Implementation of a Multi-Layer Neural Network Using a Crossbar Array," which is incorporated herein by reference in its entirety.

[0048] The ADC 143 may include any suitable circuitry for converting the analog output of the ML processing unit 141 to a digital output. The digital output may represent a predicted output. In some embodiments, the ADC 143 may include the ADC 250 of FIG. 2.

[0049] The communications module 130 can forward the output of the ML processor 140 to another computing device (e.g., a cloud computing device) for further processing. In some embodiments, the communications module 130 can further receive instructions from the computing device to perform an operation based on the predicted output (e.g., turn on a display based on the facial recognition result, send data to another processing device, present media content, etc.).

[0050] In some embodiments, the processing devices 100a, 100b are self-powered and can operate without an external power source. For example, the sensing module 110 can provide power to components of the processing devices 100a, 100b. The ML processors 120, 140 and their components can be powered and operate as described herein using the analog output generated by the sensing module 110.

[0051] FIG. 2 illustrates an example crossbar array 200 according to some embodiments of the present disclosure. As shown, the crossbar array 200 may include a plurality of interconnected electrically conductive wires, such as one or more row wires 211a, 211b, ..., 211i, ..., 211n and column wires 213a, 213b, ..., 213j, ..., 213m, for an n-row by m-column crossbar array. The crossbar array 200 may further include crosspoint devices 220a, 220b, ..., 220z, etc. Each crosspoint device may connect a row wire to a column wire. For example, crosspoint device 220ij may connect a row wire 211i to a column wire 213j. The number of column wires 213a-m and the number of row wires 211a-n may or may not be the same. The crossbar array 200 may further include wordline (WL) logic 205 connected to the crosspoint devices via the row wires 211a-n. The WL logic 205 may include any suitable components (e.g., one or more digital-to-analog converters (DACs), amplifiers, etc.) for applying input signals to selected cross point devices via row wires 211a-n. Each input signal may be a voltage signal, a current signal, etc. The input signals may correspond to the analog sensing signals generated by the sensing module 110 of FIGS. 1A-1B.

[0052] The row wires 211 may include a first row wire 211 a, a second row wire 211 b, ..., 211 i, ..., and an nth row wire 211 n. Each of the row wires 211 a, ..., 211 n may be and / or include any suitable electrically conductive material. In some embodiments, each row wire 211 a-n may be a metal wire.

[0053] The column wires 213 may include a first column wire 213 a, a second column wire 213 b, ..., and an mth column wire 213 m. Each of the column wires 213 a-m may be and / or include any suitable electrically conductive material. In some embodiments, each column wire 213 a-m may be a metal wire.

[0054] Each cross point device 220a-z may be and / or include any suitable device with tunable resistance, such as a memristor, a phase change memory (PCM) device, a floating gate, a spintronic device, a ferroelectric device, an RRAM device, etc.

[0055] Each of the row wires 211 a-n can be connected to one or more row switches 231 (e.g., row switches 231 a, 231 b, ..., 231 n). Each row switch 231 a-n can include any suitable circuit structure capable of controlling the current through the row wires 211 a-n. For example, the row switches 231 can be and / or include CMOS switch circuits.

[0056] Each of the column wires 213a-213m can be connected to one or more column switches 233 (e.g., switches 233a-233m). Each column switch 233a-233m can include any suitable circuit structure capable of controlling the current through the column wires 213a-213m. For example, the column switches 233a-233m can be and / or include CMOS switch circuits. In some embodiments, one or more of the switches 231a-231n and 233a-233m can further provide fault protection, electrostatic discharge (ESD) protection, noise reduction, and / or any other suitable functionality for one or more portions of the crossbar array 200.

[0057] The output sensor 240 may include any suitable components (e.g., one or more TIAs (transimpedance amplifiers) 240a-n) for converting the currents flowing through the column wires 213a-n into output signals. Each TIA 240a-n may convert the currents flowing through a respective column wire into a respective voltage signal. Each ADC 250a-250m may convert the voltage signal generated by the corresponding TIA into a digital output. In some embodiments, the output sensor 240 may further include one or more multiplexers (not shown).

[0058] Programming circuit 260 can program cross point devices 220a-z selected by switches 231 and / or 233 to appropriate conductance values. For example, programming a cross point device can include applying an appropriate voltage or current signal to the cross point device. The resistance of each cross point device can be electrically switched between a high resistance state and a low resistance state. Setting a cross point device can include switching the resistance of the cross point from a high resistance state to a low resistance state. Resetting a cross point device can include switching the resistance of the cross point from a low resistance state to a high resistance state.

[0059] The crossbar array 200 can perform parallel weighted voltage multiplication and current summation. For example, an input voltage signal can be applied to one or more rows (e.g., one or more selected rows) of the crossbar array 200. The input signal can flow through the crosspoint devices in the row of the crossbar array 200. The conductance of the crosspoint devices can be adjusted to a specific value (also called a "weight"). According to Ohm's law, the input voltage multiplies the crosspoint conductance, generating a current from the crosspoint device. According to Kirchhoff's law, the sum of the currents flowing through the devices on each column generates a current as an output signal, which can be read from the column (e.g., the output of an ADC). According to Ohm's law and Kirchhoff's current law, the input-output relationship of the crossbar array can be expressed as I = VG, where I represents the output signal matrix as a current, V represents the input signal matrix as a voltage, and G represents the conductance matrix of the crosspoint device. In this way, the input signal is weighted by its conductance at each crosspoint device according to Ohm's law. The weighted currents are output through each column wire and can be accumulated according to Kirchhoff's current law, enabling in-memory computing (IMC) through parallel multiplications and additions performed in the crossbar array.

[0060] The crossbar array 200 can be configured to perform vector-matrix multiplication (VMM). A VMM operation can be expressed as Y=XA, where Y, X, and A each represent a respective matrix. More specifically, for example, an input vector X can be mapped to an input voltage V of the crossbar array 200. The matrix A can be mapped to a conductance value G. The output current I can be read and mapped back to an output result Y. In some embodiments, the crossbar array 200 can be configured to implement a portion of a neural network by performing a VMM.

[0061] In some embodiments, the crossbar array 200 can perform convolution operations. For example, performing a 2D convolution on input data may include applying a single convolution kernel to the input signal. Performing a depthwise convolution on input data may include convolving each channel of the input data with a respective kernel corresponding to that channel and stacking the convolved outputs. A convolution kernel may have a specific size defined by multiple dimensions (e.g., width, height, channels, etc.). A convolution kernel may be applied to portions of the input data of the same size to generate an output. The output may be mapped to an element of the convolution result at a position corresponding to the position of the portion of the input data.

[0062] The programming circuitry 260 can program the crossbar array 200 to store convolution kernels for performing 2D convolution operations. For example, the convolution kernel can be converted into a vector and mapped to multiple crosspoint devices of the crossbar array connected to specific bit lines. In particular, the conductance values ​​of the crosspoint devices can be programmed to values ​​representing the convolution kernel. In response to an input signal, the crossbar array 200 can output, via specific bit lines, current signals representing the convolution of the input signal and the 2D convolution kernel. In some embodiments, the crossbar array 200 can store multiple 2D convolution kernels by mapping each of the 2D convolution kernels to a crosspoint device connected to a respective bit line. The crossbar array 200 can output multiple output signals (e.g., current signals) representing the convolution results via column wires 213.

[0063] FIG. 3 is a schematic diagram illustrating an example of a 3D crossbar array circuit 300 according to some embodiments of the present disclosure.

[0064] As shown, the 3D crossbar array circuit 300 can include a first crossbar array 310, a second crossbar array 320, and a third crossbar array 330 arranged in different planes. In some embodiments, the first crossbar array 310, the second crossbar array 320, and the third crossbar array 330 can be arranged in a first plane, a second plane, and a third plane, respectively. The first plane, the second plane, and the third plane can be parallel to one another. In some embodiments, the first plane, the second plane, and the third plane can be perpendicular or parallel to the substrate on which the first crossbar array 310, the second crossbar array 320, and the third crossbar array 330 are formed. Each of the first crossbar array 310, the second crossbar array 320, and the third crossbar array 330 can include one or more 2D crossbar arrays as described in connection with FIG. 2. Although three crossbar arrays are shown in FIG. 3, the 3D crossbar array circuit 300 may include any suitable number of 2D crossbar arrays integrated into the 3D crossbar circuit.

[0065] The first crossbar array 310 may include crosspoint devices 315 connecting a first plurality of wordlines (e.g., WL1_1, WL2_1, WL3_1) and a first plurality of bitlines (e.g., BL1_1, BL2_1, BL3_1). The second crossbar array 320 may include crosspoint devices 325 connecting a second plurality of wordlines (e.g., WL1_2, WL2_2, WL3_2) and a second plurality of bitlines (e.g., BL1_2, BL2_2, BL3_2). The third crossbar array 330 may include crosspoint devices 335 connecting a third plurality of wordlines (e.g., WL1_3, WL2_3, WL3_3) and a third plurality of bitlines (e.g., BL1_3, BL2_3, BL3_3).

[0066] The 3D crossbar array circuit 300 may further include transistors 340. Each transistor 340 may be connected to a respective gate line (GL1, GL2, GL3, etc.) via its gate region. For example, gate line GL1 may be connected to the gate region of a first transistor in the first crossbar array 310, the gate region of a second transistor in the second crossbar array 320, and the gate region of a third transistor in the third crossbar array 330. The source region of each transistor 340 may be connected to a word line. It should be noted that FIG. 3 schematically illustrates the components of the 3D crossbar array circuit 300 and their connections. The schematic diagram shown in FIG. 3 does not represent the physical layout of the components of the 3D crossbar array circuit 300. The components of the 3D crossbar array circuit 300 may be physically arranged in any suitable manner to implement the 3D crossbar arrays described herein. For example, in the physical layout of the 3D crossbar array circuit 300, the transistors 340 may be located on the same layer of the substrate and connected to corresponding gate lines, bit lines, and word lines through vertical vias.

[0067] To select the crosspoint device located at the intersection of WL3_3 and BL3_3, a voltage V G By applying a voltage V to GL3 and grounding the other GLs, the transistor channel on GL3 opens. D The voltage V can be applied to the drain region of the transistor connected to WL3_3, while the drain regions of other transistors located on the same horizontal layer are grounded, so that current flows only through WL3_3. ground can be applied to BL3_3 while maintaining Vs on other BLs that cross WL3_3. Vs can be equal to Vd-Vds, where Vs represents the voltage on the source region of the transistor and Vds represents the voltage drop between the drain and source regions of the transistor. Therefore, only one device on a WL can have both Vs and V groundOther devices on the same WL (WL3_3) will not be programmed due to the lack of voltage difference on those devices.

[0068] Because the crossbar array in the 3D crossbar array circuit 300 is arranged in a 3D manner, a cross section of the 3D crossbar array circuit 300 can be considered as a 2D crossbar array. The 3D crossbar array circuit 300 can receive and process 2D inputs (e.g., m×n analog input signals) without storing them or converting them into one-dimensional data (e.g., vectors representing the 2D inputs). For example, the crosspoint devices located at WL1_1, WL1_2, WL1_3, etc., selected as described above, can receive and process analog sensing signals generated by the 2D sensor array.

[0069] 4A and 4B are schematic diagrams illustrating example semiconductor devices 400a and 400b that can function as machine learning processors in accordance with some embodiments of the present disclosure. The semiconductor devices 400a-b may also be referred to as chiplets.

[0070] As shown, semiconductor device 400a may include a sensor wafer 410, a processor wafer 420, and a packaging substrate 430. Each of sensor wafer 410 and processor wafer 420 may be implemented as multiple wafers in some embodiments. For example, processor wafer 420 may include multiple wafers stacked in a 3D manner as described herein.

[0071] The sensor wafer 410 can include a sensing module (e.g., the sensing module 110 described in connection with FIGS. 1A-1B) that includes one or more sensor arrays. In some embodiments, the sensor wafer 410 can include one or more image sensor wafers, as described in connection with FIGS. 5A-5B below.

[0072] Processor wafer 420 can include one or more wafers incorporating CMOS elements for implementing ADCs, crossbar arrays, driver ICs (integrated circuits), transceivers, and / or any other suitable components for implementing machine learning processing. ML processor 120 of FIG. 1A and / or ML processor 140 of FIG. 1B can be fabricated on processor wafer 420. Processor wafer 420 can be and / or include processor wafer 600a-b, described in connection with FIGS. 6A-6B below.

[0073] The processor wafer 420 and the sensor wafer 410 may be connected through a first interconnect layer 440. The interconnect layer 440 may include one or more metal interconnects (e.g., metal vias, metal pads, etc.). In some embodiments, a portion of the interconnect layer 440 may be considered part of the processor wafer 420. The processor wafer 420 and the sensor wafer 410 may be connected to each other using through-silicon-via (TSV) packing technology, hybrid bond-metal (HBM) packaging technology, in-pixel hybrid bonding (IPHB) technology, and / or any other suitable chip packaging technology for packaging and / or stacking multiple wafers.

[0074] Package substrate 430 may include an antenna, connectors, a power supply, etc. In some embodiments, package substrate 430 does not include CMOS components. Processor wafer 420 may be connected to package substrate 430 through a second interconnect layer 450. For example, interconnect layer 450 may include ball grid array (BGA) bumps. In some embodiments, package substrate 430 may be connected to a PCB (printed circuit board) substrate (not shown).

[0075] Referring to FIG. 4B, in some embodiments, multiple processor wafers and sensor wafers can be stacked on a package substrate 430. As shown, a first sensor wafer 410a can be stacked on a first processor wafer 420a and connected to the first processor wafer 420a via interconnect layer 440-1. A second sensor wafer 410b can be stacked on a second processor wafer 420b and connected to the second processor wafer 420b via interconnect layer 440-2. Similarly, a third sensor wafer 410c can be stacked on a third processor wafer 420c and connected to the third processor wafer 420c via interconnect layer 440-3. Processor wafers 420a, 420b, and 420c can be connected to the package substrate 430 via interconnect layers 450a, 450b, and 450c, respectively. In some embodiments, sensor wafers 410a-c include various types of sensors and can sense different signals. In such an embodiment, processor wafers 420a-c can process different sensing signals generated by different sensor wafers 410a-c. Each of sensor wafers 410a-c, processor wafers 420a-c, interconnect layers 440-1, 440-2, ..., 440-3, and interconnect layers 450a-c can be and / or include their respective counterparts described in connection with FIG. 4A (i.e., sensor wafer 410, processor wafer 420, interconnect layer 440, and interconnect layer 450). While a specific number of wafers are shown in FIGS. 4A and 4B, this is for illustrative purposes only. Any suitable number of sensor wafers and processor wafers can be stacked on package substrate 430 as described herein.

[0076] 5A and 5B illustrate cross-sectional views of example image sensor wafers 500a and 500b, each of which may also be referred to as a CMOS image sensor (CIS) wafer, according to some embodiments of the present disclosure.

[0077] 5A, the image sensor wafer 500a may include an image sensor including a microlens 511, a color filter 513, and a photodiode 515a. The photodiode 515a may be fabricated on a substrate 505 (e.g., a silicon substrate). A metal trace 520a may be located between the color filter 513 and the photodiode 515a.

[0078] Incident light is collected through a microlens 511 and can be separated into multiple color components by a color filter 513. For example, a red filter 513a, a green filter 513b, and a blue filter 513c can separate the red, green, and blue components of the incident light, respectively. The photodiode 515a accumulates photon charges when exposed to light and can convert the charges into an electrical signal (voltage signal).

[0079] Referring to FIG. 5B, image sensor wafer 500b includes a backside illumination structure in which photodiode 515b is disposed behind color filter 513 and metal wiring 520b is disposed behind photodiode 515b. Image sensor wafer 500b can be fabricated by fabricating photodiode 515b and metal wiring 520b on a front-side silicon substrate 505, then flipping substrate 505, thinning the back-side substrate, and fabricating color filter 513 and microlens 511 on the backside. As shown, metal wiring 520b is disposed behind photodiode 515b, while metal wiring 520a is disposed in front of photodiode 515a. Light can enter interface 531a in image sensor wafer 500a and interface 531b in image sensor wafer 500b, respectively. In this manner, light can reach photodiode 515b without passing through metal wiring 520b, allowing photodiode 515b to capture more optical signals than photodiode 515a.

[0080] 6A and 6B are schematic diagrams illustrating example functional components of a processor wafer, according to some embodiments of the present disclosure. As shown in FIG. 6A, processor wafer 600a may include ML processor 120 as described in connection with FIG. 1A. As shown in FIG. 6B, processor wafer 600b may include ML processor 140 as described in connection with FIG. 1B. Processor wafers 600a and 600b may further include CMOS components, ICs, etc. (not shown) for implementing other functions of the processing devices described herein. For example, processor wafers 600a and 600b may include CMOS components for implementing one or more functions of communications module 130 of FIGS. 1A-1B.

[0081] 7A, 7B, and 7C are schematic diagrams illustrating examples of semiconductor devices 700a, 700b, and 700c integrating a sensor wafer and a processor wafer according to some embodiments of the present disclosure.

[0082] Referring to FIG. 7A, the sensor wafer 410 and the processor wafer 420 can be connected via a through-silicon via (TSV) stack 440a. The TSV stack 440a can connect a portion of the metal interconnects 411 of the sensor wafer 410 to a portion of the metal interconnects 421 of the processor wafer 420. Referring to FIG. 7B, the sensor wafer 410 and the processor wafer 420 can be connected via a hybrid bond metal (HBM) connection 440b. Referring to FIG. 7C, the sensor wafer 410 and the processor wafer 420 can be connected via an in-pixel hybrid junction (IPHB) connection 440c. The sensor wafer 410 and the processor wafer 420 can be connected at the pixel level to further facilitate high-speed data processing.

[0083] For ease of explanation, the methods of the present disclosure are depicted and described as a series of steps. However, steps in accordance with the present disclosure can be performed in various orders and / or simultaneously with other steps not shown and described herein. Moreover, not all disclosed steps may be required to implement a method in accordance with the disclosed subject matter. In addition, those skilled in the art will understand and appreciate that the method can alternatively be represented as a series of interrelated states via a state diagram or events.

[0084] As used herein, the terms "approximately," "about," and "substantially" can mean within normal tolerances in the art, e.g., within two standard deviations of the mean, in some embodiments, within ±20% of the target dimension, in some embodiments, within ±10% of the target dimension, in some embodiments, within ±5% of the target dimension, in some embodiments, within ±2% of the target dimension, in some embodiments, within ±1% of the target dimension, and in some embodiments, within ±0.1% of the target dimension. The terms "approximately" and "about" can include the target dimension. Unless otherwise stated or apparent from the context, all numerical values ​​described herein are modified by the term "about."

[0085] As used herein, ranges include all values ​​within that range. For example, the range of 1 to 10 can include any number of the digits 1, 2, 3, 4, 5, 6, 7, 8, 9, and 10, combinations of digits, subranges, and fractions thereof.

[0086] In the foregoing description, many details are set forth. However, it will be apparent that the disclosure may be practiced without these specific details. In some instances, well-known structures and devices are shown in block diagram form, rather than in detail, in order to avoid obscuring the disclosure.

[0087] As used herein, terms such as "first," "second," "third," "fourth," etc. are meant as labels to distinguish between different elements and do not necessarily have an ordinal meaning according to their numerical designation.

[0088] The word "example" or "exemplary" is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as "example" or "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects or designs. Rather, use of the word "example" or "exemplary" is intended to present concepts in a concrete manner. The term "or" as used in this application is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from the context, "X includes A or B" is intended to mean any of the natural inclusive permutations. That is, if X includes A; if X includes B; or if X includes both A and B, then X includes A or B, as in any of the foregoing examples. Additionally, the articles "a" and "an," as used in this application and the appended claims, should be construed generally to mean "one or more" unless otherwise specified or clear from the context that the singular form is intended. Throughout this specification, a reference to an "implementation" or "one implementation" means that a particular feature, structure, or characteristic described in connection with that implementation is included in at least one implementation. Thus, the appearances of the phrase "implementation" or "one implementation" in various places throughout this specification are not necessarily all referring to the same implementation.

[0089] As used herein, when an element or layer is referred to as being "on" another element or layer, the element or layer can be located directly on the other element or layer, or there can be intervening elements or layers. In contrast, when an element or layer is referred to as being "directly on" another element or layer, there are no intervening elements or layers.

[0090] Although many variations and modifications of the disclosure will no doubt become apparent to those skilled in the art after reading the foregoing description, it should be understood that while particular embodiments have been shown and described by way of illustration, they are not intended to be limiting. Accordingly, references to details of various embodiments are not intended to limit the scope of the claims, which will recite only those features that are deemed to be disclosed.

[0091] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application claims the benefit of U.S. Patent Application No. 17 / 932,432, filed September 15, 2022, entitled "INTEGRATED SENSING AND MACHINE LEARNING PROCESSING DEVICES," which is incorporated herein by reference in its entirety.

Claims

1. A semiconductor device comprising: a sensing module configured to generate a plurality of analog sensing signals; one or more crossbar arrays configured to process the analog sensing signals to generate analog pre-processed sensing data; an analog-to-digital converter (ADC) configured to convert the analog pre-processed sensing data into digital pre-processed sensing data; a machine learning processing unit configured to process the digital pre-processed sensing data using one or more machine learning models; Equipped with A semiconductor device, wherein the machine learning processing unit is fabricated on a processor wafer of the semiconductor device.

2. the sensing module is fabricated on a sensor wafer; 10. The semiconductor device of claim 1, wherein the sensor wafer is connected to the processor wafer via a first interconnect layer.

3. The semiconductor device of claim 2 , wherein the one or more crossbar arrays are fabricated on the processor wafer.

4. The semiconductor device of claim 3 , wherein the ADC is fabricated on the processor wafer.

5. the sensing module includes an image sensor array; The semiconductor device of claim 2 , wherein the plurality of analog sensing signals comprises a plurality of analog image signals.

6. the analog pre-processed sensing data corresponds to a plurality of features extracted from the analog sensing signal; The semiconductor device of claim 2 , wherein the machine learning processing unit performs machine learning using the extracted features.

7. Further comprising a package substrate; 3. The semiconductor device of claim 2, wherein the processor wafer is connected to the package substrate through a second interconnect layer.

8. The semiconductor device of claim 2 , wherein the machine learning processing unit is powered using the analog sensing signal.

9. It also has a transceiver, The transceiver includes: sending predicted outputs generated by the machine learning processing unit based on the one or more machine learning models to a computing device; The semiconductor device of claim 1 , configured to receive instructions from the computing device to perform an operation based on the predicted output.

10. The semiconductor device of claim 1 , wherein the analog pre-processed sensing data represents a convolution of the analog sensing signal with a kernel.

11. 11. The semiconductor device of claim 10, wherein conductance values ​​of a plurality of cross point devices of the one or more crossbar arrays are programmed to values ​​representative of the kernel.

12. the sensing module includes a two-dimensional sensor array; The semiconductor device of claim 1 , wherein a plurality of crosspoint devices of the one or more crossbar arrays are configured to receive as inputs the analog sensing signals generated by the two-dimensional sensor array.

13. The semiconductor device of claim 12 , wherein the one or more crossbar arrays comprise a plurality of crossbar arrays arranged in a plurality of different planes.

14. A semiconductor device comprising: a sensing module configured to generate a plurality of analog sensing signals; a machine learning processor configured to process the analog sensing signal using one or more machine learning models to generate a predicted output; Equipped with The machine learning processor: a plurality of crossbar arrays configured to generate a plurality of analog outputs representative of the predicted outputs; an analog-to-digital conversion unit configured to convert the plurality of analog outputs representing the predicted outputs into digital signals representing the predicted outputs; 10. A semiconductor device comprising:

15. It also has a transceiver, 15. The semiconductor device of claim 14, wherein the transceiver is configured to transmit a signal representing a predicted output generated by the machine learning processor to a computing device.

16. the sensing module is fabricated on a sensor wafer; the machine learning processor is fabricated on a processor wafer; 15. The semiconductor device of claim 14, wherein the sensor wafer is connected to the processor wafer via a first interconnect layer.

17. Further comprising a package substrate; 17. The semiconductor device of claim 16, wherein the processor wafer is connected to the package substrate via a second interconnect layer.

18. the sensing module includes an image sensor array; The semiconductor device of claim 14 , wherein the plurality of analog sensing signals comprises a plurality of analog image signals.

19. the sensing module includes a two-dimensional sensor array; 15. The semiconductor device of claim 14, wherein a plurality of crosspoint devices of the plurality of crossbar arrays are configured to receive as inputs analog sensing signals generated by the two-dimensional sensor array.

20. 20. The semiconductor device of claim 19, wherein the multiple crossbar arrays are arranged in different planes.

Images