Integrated sensing and machine learning processing device

Abstract

The invention provides a semiconductor device integrating sensing and processing functions. The semiconductor device includes a sensing module configured to generate a plurality of analog sensing signals, one or more crossover arrays configured to process the analog sensing signals to generate analog pre-processed sensing data, an analog-to-digital converter (ADC) configured to convert the analog pre-processed sensing data to digital pre-processed sensing data and a machine learning processing unit configured to process the digitally pre-processed sensing data using one or more machine learning models. The machine learning processing unit, the crossover array, and the ADC are integrated on a processor wafer of the semiconductor device. The sensing module is integrated on a sensor wafer stacked with the processor wafer.

Description

Cross-references

[0001] This application claims priority to U.S. patent application No. 17 / 932,432, filed on September 15, 2022, entitled “INTEGRATED SENSING AND MACHINE LEARNING PROCESSING DEVICE,” each of which is incorporated in its entirety into this application. Technical Field

[0002] Embodiments of the present disclosure generally relate to a computing device, and more particularly, to an integrated sensing and machine learning processing device. Background Art

[0003] Machine learning (ML) is currently widely used in fields such as face recognition, speech recognition, natural language processing, and image processing. ML typically involves analyzing large amounts of sensor data based on complex machine learning models. Traditional edge devices (local devices close to sensors that collect sensor data) lack the computing power to perform such analysis. Therefore, the sensor data generated by the sensor may need to be digitized and transmitted to a remote computing device (e.g., a data center) for processing. This may require digitizing a large amount of data and may require advanced communication capabilities, a large amount of energy, and time to transmit the digitized sensor data. Transmitting raw sensor data from sensors to remote devices may raise privacy issues, and encrypting raw sensor data for secure data transmission may further increase the computing cost required for ML. In addition, some applications (e.g., medical applications using ML) may require real-time data processing. Therefore, there is a demand for running machine learning models locally on edge devices. However, traditional edge devices cannot provide integrated sensing and processing capabilities for local extraction of information and features from analog sensor data provided by local sensors and ML processing. Summary of the invention

[0004] The following is a brief summary of the present disclosure, which is used to provide a basic understanding of some aspects of the present disclosure. The summary is not an extensive overview of the present disclosure. The summary is not intended to identify the key or important elements of the present disclosure, nor is it intended to illustrate any scope of a particular implementation of the present disclosure or any scope of the claims. The sole purpose of the summary is to present some concepts of the present disclosure as a language simplification of a more detailed description presented subsequently.

[0005] According to one or more aspects of the present disclosure, a semiconductor device that can be used as an integrated sensing and machine learning processing device is provided. The semiconductor device may include: a sensing module configured to generate a plurality of analog sensing signals; one or more crossbar arrays configured to process the analog sensing signals to generate analog pre-processed sensing data; a digital-to-analog converter (ADC) configured to convert the analog pre-processed sensing data into digital pre-processed sensing data; and a machine learning processing unit configured to process the digital pre-processed sensing data using one or more machine learning models, wherein the machine learning processing unit is manufactured on a processor chip of the semiconductor device.

[0006] In some embodiments, the sensing module is fabricated on a sensor wafer, wherein the sensor wafer is connected to the processor wafer via a first interconnect layer.

[0007] In some embodiments, the one or more crossbar arrays are fabricated on the processor wafer.

[0008] In some embodiments, the ADC is fabricated on the processor die.

[0009] In some embodiments, the sensing module includes an image sensor array, wherein the plurality of analog sensing signals include a plurality of analog image signals.

[0010] In some embodiments, the simulated pre-processed sensor data corresponds to a plurality of features extracted from the simulated sensor data, wherein the machine learning processing unit performs machine learning using the features extracted from the simulated sensor signals.

[0011] In some embodiments, the semiconductor device further includes a packaging substrate, wherein the processor die is connected to the packaging substrate through a second interconnect layer.

[0012] In some embodiments, the machine learning processing unit is powered by the analog sensing signal.

[0013] In some embodiments, the semiconductor device further includes a transceiver configured to transmit a prediction output generated by the machine learning processing unit based on one or more machine learning models.

[0014] In some embodiments, the analog pre-processed sensor data represents a convolution of the analog sensor signal and a convolution kernel.

[0015] In some embodiments, conductance values ​​of a plurality of cross-point devices in the one or more crossbar arrays are programmed to represent values ​​of a convolution kernel.

[0016] In some embodiments, the sensing module includes a two-dimensional sensor array, wherein a plurality of cross-point devices in the one or more cross-point arrays are configured to receive as input an analog sensing signal generated by the two-dimensional sensor array.

[0017] In some embodiments, the one or more crossbar arrays include multiple crossbar arrays located in multiple different planes.

[0018] According to one or more aspects of the present disclosure, a semiconductor device that can be used as an integrated sensing and machine learning processing device is provided. The semiconductor device may include: a sensing module configured to generate a plurality of analog sensing signals; a machine learning processor configured to generate a prediction output by processing the analog sensing signals using one or more machine learning models, wherein the machine learning processor includes: a plurality of crossbar arrays configured to generate a plurality of analog outputs representing the prediction outputs; and a digital-to-analog converter unit configured to convert the plurality of analog outputs representing the prediction outputs into digital signals representing the prediction outputs.

[0019] In some embodiments, the semiconductor device further includes: a transceiver configured to transmit a signal representing the prediction output generated by the machine learning processor to a computing device.

[0020] In some embodiments, the transceiver may receive instructions from a computing device for performing an operation based on the predicted output.

[0021] In some embodiments, the sensing module is manufactured on a sensor wafer. The machine learning processor is manufactured on a processor wafer. The sensor wafer is connected to the processor wafer via a first interconnect layer.

[0022] In some embodiments, the semiconductor device further includes a packaging substrate, wherein the processor die is connected to the packaging substrate through a second interconnect layer.

[0023] In some embodiments, the sensing module includes an image sensor array, wherein the plurality of analog sensing signals include a plurality of analog image signals.

[0024] In some embodiments, the sensing module includes a two-dimensional sensor array. In some embodiments, a plurality of cross-point devices in the plurality of cross-point arrays are configured to receive as input an analog sensing signal generated by the two-dimensional sensor array.

[0025] In some embodiments, the plurality of crossbar arrays are located in different planes. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The present disclosure will be more fully understood from the detailed description given below and the accompanying drawings of various embodiments of the present disclosure. However, the accompanying drawings should not be used to limit the present disclosure to specific embodiments, but are only for explanation and understanding.

[0027] Figure 1A and 1B is a schematic diagram showing an example of a processing device with integrated sensing and processing capabilities according to some embodiments of the present disclosure.

[0028] Figure 2 is a schematic diagram showing an example of a crossbar array according to some embodiments of the present disclosure.

[0029] Figure 3 is a schematic diagram showing an example of a three-dimensional cross array according to some embodiments of the present disclosure.

[0030] Figure 4A and 4B is a schematic diagram showing an example of a semiconductor device that can be used as a machine learning processor in some embodiments of the present disclosure.

[0031] Figure 5A and 5B is a schematic diagram showing a cross-section of an example image sensor wafer according to some embodiments of the present disclosure.

[0032] Fig. 6A and 6B is a schematic diagram showing examples of functional components in a processor die according to some embodiments of the present disclosure.

[0033] Fig. 7A , 7B 7C are schematic diagrams showing cross sections of examples of semiconductor devices according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0034] Various aspects of the present disclosure provide a method for manufacturing a processing device machine with integrated sensing and machine learning capabilities. A processing device according to the present disclosure may include a sensing module and a machine learning (ML) processor integrated on the same semiconductor device using three-dimensional (3D) core integration or monolithic 3D integration. The sensing module may include a sensor array that can generate analog sensing data (e.g., an analog image signal generated by an image sensor). The ML processor may process the analog sensing data using one or more machine learning models.

[0035] In one embodiment, the ML processor may include a preprocessing unit that can preprocess the analog sensor data for ML processing, for example, by performing feature extraction, dimensionality reduction, image processing, etc. The preprocessing unit may include one or more crossbar arrays that can be used to preprocess the analog sensor data in the analog domain. Each crossbar array can be a circuit structure having interconnected conductive lines with a resistance conversion material sandwiched at the intersection. The resistance conversion material may include, for example, a memristor (also known as a resistance random access memory (RRAM or ReRAM)). The analog sensor data can be provided to the crossbar array as an input signal. The crossbar array can generate an analog output signal representing the preprocessed sensor data. Then, the analog output signal can be converted into a digital signal representing the preprocessed sensor data, and the ML processor performs subsequent machine learning processing. By preprocessing the analog sensing data in the analog domain and digitizing the preprocessed sensing data instead of the raw sensing data, the ML sensor described in the present disclosure can achieve significant data reduction because only a small amount of information (e.g., the preprocessed sensing data) can be digitized and transmitted from the sensing module at the edge to the next layer of the network.

[0036] In another embodiment, the ML processor can run a machine learning model on the analog sensor data and can generate an analog signal representing the predicted output of the ML processing (e.g., classification results, labels assigned to the analog sensor data, outputs of a neural network layer, decisions made based on the ML model, etc.). For example, the ML processor can implement a multi-layer neural network using a crossbar array. The analog output signal generated by the crossbar array can represent the predicted output, can be converted to a digital output, and can be transmitted to other computing devices.

[0037] In some embodiments, the processing device can be implemented using 3D core particle integration. For example, the core particle implementing the processing device may include a processor wafer and a sensor array stacked on the processor wafer. The sensor wafer may include a sensing module. The sensor wafer may include an ML processor. The sensor wafer may be connected to the processor wafer by using a stacked interconnect layer such as through silicon vias (TSV), hybrid metal bonding, and hybrid bonding within a pixel. The sensing signal generated by the sensor array can be provided to the ML processor through the interconnect layer. Therefore, all required CMOS (complementary metal oxide semiconductor) components and integrated circuits for machine learning can be integrated into one wafer to process the analog sensing signals generated by the sensing modules embedded in the sensor wafer.

[0038] The core particles described in the present disclosure can provide 3D heterogeneous integration of sensing and processing functions, and can achieve the desired hardware processing capabilities, such as near-sensing processing, memory computing, analog computing, and parallel computing. The core particles can be used to implement 3D neural network hardware, thereby generating higher device density, complex connections, and reducing communication losses. In some embodiments, the processing device described in the present disclosure can also provide a two-dimensional interface (a cross-section of a 3D neural network) to communicate with a 2D sensor array (e.g., an image sensor array), which can achieve that the sensing data generated by the 2D sensor array can directly enter the 3D neural network for processing without the need for signal storage or reconfiguration as one-dimensional data (e.g., a vector that can be used as an input to a traditional 2D neural network).

[0039] In addition, most sensor signals can be regarded as specific forms of energy (e.g., temperature, mechanical force, photons, vibrations, chemicals, electromagnetic waves, etc.), and can be easily converted into electrical signals (e.g., 0.5V) using novel devices. These electrical signals are not only analog data to be processed, but can also be used as a potential energy source to self-power the sensor module and other components of the processing device. In some embodiments, the processing device can therefore only be awakened in the presence of a sensor signal, and event-driven applications can be implemented, further reducing the amount of data collected and the energy consumed for ML processing.

[0040] Figure 1A and 1B is a schematic diagram showing examples of processing devices 100a and 100b having integrated sensing and processing capabilities according to some embodiments of the present disclosure.

[0041] like Figure 1A As shown, the processing device 100a may include a sensing module 110, a machine learning (ML) processor 120, and a communication module 130. The ML processor 120 may further include a pre-processing unit 121, an analog-to-digital conversion (ADC) unit 123, and a machine learning (ML) processing unit 125. The sensing module 110, the ML processor 120, and the communication module 130 may be integrated in a core structure, for example, in combination with Figure 4A-4B The core particle structure.

[0042] The sensing module 110 may include one or more sensor arrays. Each sensor array may include one or more sensors that can detect and / or measure physical properties and generate electrical signals representing the physical properties. Examples of the sensors include image sensors, audio sensors, chemical sensors, pressure sensors, thermal sensors, temperature sensors, vibration sensors, microbial fuel cells, electromagnetic sensors, etc. In some embodiments, the multiple sensor arrays in the sensing module 110 may include different types of sensors. In some embodiments, the sensing module 110 may include a combination of Figure 5A-5B In some embodiments, the sensing module 110 may generate analog sensing data in the form of analog sensing signals (e.g., voltage signals, current signals, etc.), such as analog image signals generated by an array of image sensors (e.g., CMOS image sensors) that may detect light and generate analog image signals representing the detected light.

[0043] In some embodiments, the sensors in the sensing module 110 can collect enough energy to enable the ML processor 120 and / or the processing device 100a to operate without external power. For example, the electrical signal generated by the sensing module 110 can be used to power the ML processor 120 and / or the processing device 100a.

[0044] The ML processor 120 may process the analog sensor data generated by the sensor module 110 using one or more machine learning models. For example, the preprocessing unit 121 may process the analog sensor data and generate analog preprocessed sensor data. The preprocessing unit 121 may perform any suitable operation on the analog sensor signal to prepare the analog sensor data for subsequent operations of the ML processing unit 125. For example, the preprocessing unit 121 may perform feature extraction on the analog sensor data and extract features of the analog sensor data for subsequent ML processing. As another example, the preprocessing unit 121 may perform dimensionality reduction processing on the analog sensor data to reduce the amount of data to be processed in subsequent ML processing. As another example, the preprocessing unit 121 may perform one or more convolution operations (e.g., two-dimensional convolution operations, depth convolution operations, etc.) on the analog sensor data. As another example, the preprocessing unit 121 may normalize, scale and / or resize, denoise, etc. the analog sensor data. In some embodiments where the analog sensor data includes an analog image signal, the pre-processing unit 121 may apply a suitable image processing technique to process the analog sensor data.

[0045] The pre-processing unit 121 may include one or more crossbar arrays that can process analog sensor signals in the analog domain. Each crossbar array may include a plurality of interconnected conductive lines (e.g., row lines, column lines, etc.) and crosspoint devices fabricated at the intersections of the conductive lines. The crosspoint devices may include, for example, memristors, phase change memory devices, floating gates, spintronic devices, and / or any other suitable devices with programmable resistance. In some embodiments, the crossbar array may include, for example, a combination of Figure 2-3 The one or more crossbar arrays.

[0046] As an example, the cross array can receive an input voltage signal V and can generate an output current signal I. The relationship between the input voltage signal and the output current signal can be expressed as I=VG, where G represents the conductance value of the cross-point device. Therefore, the input signal of each cross-point device can be weighted by its conductance according to Ohm's law. The weighted current can be output through each bit line and can be accumulated according to Kirchhoff's current law. The conductance value of the cross-point device can be programmed to represent the values ​​and / or weights of one or more matrices used to perform preprocessing of the analog sensing signal as described above (e.g., feature extraction, dimensionality reduction, convolution, image processing, etc.). The cross array can receive as input an analog sensing signal generated by a sensor in the sensing module 110, and can generate an analog output signal (e.g., a current signal) representing the preprocessed analog sensing data.

[0047] In some embodiments, the sensing module 110 may include sensors arranged as a two-dimensional (2D) sensor array. Since each sensor can generate an analog sensing signal, the output of the two-dimensional sensor array can be regarded as a two-dimensional output including the analog sensing signal generated by the sensor (for example, m*n analog sensing signals generated by m*n sensors). The pre-processing unit 121 may include a three-dimensional (3D) cross array, which includes multiple two-dimensional cross arrays arranged in a three-dimensional manner. For example, the two-dimensional cross arrays can be located in different planes (for example, parallel planes perpendicular to the substrate on which the three-dimensional cross array is manufactured). The cross-section of the three-dimensional cross array is two-dimensional, and can receive and / or process the two-dimensional output (for example, m*n analog sensing signals) generated by the sensing module 110 without converting the two-dimensional output into one-dimensional data (for example, a vector representing the sensing signal generated by the sensor). The three-dimensional cross array circuit can be and / or include a combination of Figure 3 In some embodiments, the three-dimensional crossbar array circuit can be manufactured using the technology described in US Patent Application 16 / 521,975, entitled "Crossbar Array Circuit with 3D Vertical RRAM", which is incorporated by reference in its entirety into the present disclosure.

[0048] Analog-to-digital converter (ADC) 123 may include any suitable circuitry for converting analog pre-processed sensor data into digital pre-processed sensor data. In some embodiments, ADC unit 123 may include a combination of Figure 2 The one or more ADCs 250.

[0049] The ML processing unit 125 may include circuits for processing digital preprocessed sensor data using one or more machine learning models. In some embodiments, the ML processing unit 125 may include a digital signal processor. The ML processing unit 125 may obtain a predicted output by running a trained machine learning model using the digital preprocessed sensor data. The predicted output may represent, for example, a classification result (e.g., a category label assigned to the sensor data), a decision made based on a machine learning model, and the like. The machine learning model may refer to a model product created by a processing device using training data, the training data including known training inputs and corresponding known outputs (the correct answer for each training input). The processing device may find patterns in the training data that map known inputs to known outputs (predicted outputs), and provide a machine learning model that can obtain these patterns.

[0050] The machine learning model may include a machine learning model composed of a single-stage linear or nonlinear operation (e.g., a support vector machine), a neural network composed of a multi-stage nonlinear operation, etc. The neural network may include an input layer, one or more hidden layers, and an output layer. The neural network may be trained by, for example, adjusting the weights of the neural network according to a back propagation learning algorithm, etc. In some embodiments, the crossbar array in the preprocessing unit 121 may implement one or more layers of the neural network. For example, the analog domain processed sensor data generated by the preprocessing unit 121 may represent the output of an input layer or a hidden layer in the neural network.

[0051] The communication module 130 may include any suitable hardware and / or software to facilitate communication between the processing unit 100a and one or more other computing devices. For example, the communication module 130 may include one or more transceivers that can transmit and / or receive RF (radio frequency) signals. The communication module 130 may include components for implementing one or more other wireless transmission protocols (e.g., Wi-Fi, Bluetooth, ZIGBEE, cellular light). In some embodiments, the communication module 130 may include one or more antennas that may be integrated in the Figure 4A-4BThe processor chip 420 or the package substrate 430 in the ML processor 120 may be on the processor chip 420 or the package substrate 430 in the ML processor 120. The communication module 130 may transmit the output of the ML processor 120 to other computing devices (e.g., cloud computing devices) for subsequent processing. In some embodiments, the communication module 130 may further receive instructions from the computing device for performing operations based on the predicted output (e.g., turning on a display based on the face recognition result, sending data to one or more other processing devices, presenting media content, etc.).

[0052] refer to Figure 1B , the processing device 100b may include a sensing module 110, a machine learning (ML) processor 140, and a communication module 130. The sensing module 110 and the communication module 130 may be combined with Figure 1A The corresponding parts described are the same.

[0053] The ML processor 140 may process the analog sensing data generated by the sensing module 110 using one or more machine learning models. The ML processor 140 may include a machine learning (ML) processing unit 141 and an ADC 143. The ML processing unit 141 may process the analog sensing data generated by the sensing module 110 using one or more machine learning models to generate a simulated prediction output. The simulated prediction output may include one or more analog signals.

[0054] In some embodiments, the ML processing unit 141 may include one or more crossbar arrays, each of which may include a combination of the following: Figure 2 In some embodiments, the ML processing unit 141 may include a combination of the following Figure 3The 3D cross array. In some embodiments, the cross array can implement a neural network that executes a machine learning algorithm. The output signal of the cross array can represent the output of the neural network. The neural network can include multiple convolution layers, each of which can perform a specific convolution operation (e.g., 2D convolution, deep convolution, etc.). Each layer of the neural network can be implemented using one or more cross arrays. For example, one or more first cross arrays can implement the first layer (e.g., input layer) of the neural network. The first cross array can receive the analog sensor data generated by the sensor module 110 as input and perform one or more convolution operations on the analog sensor data. Performing a convolution operation on the analog sensor data can involve convolving different parts of the sensor data using one or more kernels. For example, a 2D convolution can be performed by applying a single convolution kernel to the analog sensor data. More specifically, the convolution kernel can be used to scan each part of the sensor data that has the same size as the convolution kernel to produce a convolution result. As another example, performing a deep convolution on the sensor data can involve convolving each channel of the sensor data using a corresponding kernel and superimposing the convolution outputs together. For example, the conductance values ​​of the multiple crosspoint devices of the first cross array can be programmed to represent the value of the 2D convolution kernel. The analog sensor signal can be provided to the first cross array as an input signal. The first cross array can output a current signal representing the convolution of the analog sensor signal and the 2D convolution kernel. In some embodiments, the convolution array can store multiple 2D convolution kernels by mapping each 2D convolution kernel to the multiple crosspoint devices of the first cross array. The first cross array can output multiple output signals (e.g., current signals) representing the convolution results. The output of the first cross array can be provided to one or more second cross arrays that implement the second layer of the neural network for processing. The output of the second cross array (e.g., the analog current signal) can represent the output of the second layer of the neural network. The output of the second cross array can provide one or more cross arrays that implement the subsequent layer of the neural network (e.g., the second hidden layer) for processing. One or more third cross arrays can implement the output layer of the neural network. The output of the third cross array (e.g., the analog current signal) can represent the output of the neural network. In some embodiments, the neural network may be implemented using a crossbar array as disclosed in U.S. Patent Application 16 / 125,454, entitled "Implementing a Multilayer Neural Network Using a Crossbar Array," the entire contents of which are incorporated herein by reference.

[0055] ADC 143 may include any suitable circuit for converting the analog output of ML processing unit 141 into a digital output. The digital output may represent the predicted output. In some embodiments, ADC 143 may include Figure 2 ADC250 in.

[0056] The communication module 130 may transmit the output of the ML processor 140 to another computing device (e.g., a cloud computing device) for further processing. In some embodiments, the communication module 130 may further receive instructions from the computing device for performing operations based on the predicted output (e.g., turning on a display based on the face recognition result, transmitting data to other processing devices, presenting media content, etc.).

[0057] In some embodiments, the processing devices 100a-b may be self-powered and may operate without an external power source. For example, the sensing module 110 may power the components of the processing devices 100a-b. The ML processors 120 and 140 and their components may be powered by the analog outputs generated by the sensing module 110 to implement operations as described herein.

[0058] Figure 2 2 is a schematic diagram showing an example 200 of a cross array according to some embodiments of the present disclosure. As shown in the figure, the cross array 200 may include a plurality of interconnected conductive lines, for example, one or more row lines 211a, 211b, ..., 211i, ..., 211n and column lines 213a, 213b, ..., 213j, ..., 213m, for an n-row by m-column cross array. The cross array 200 may further include cross point devices 220a, 220b, ..., 220z, etc. Each cross point device may connect a row line and a column line. For example, the cross point device 220ij may connect the row line 211i and the column line 213j. The number of column lines 213a-m and the number of row lines 211a-n may be the same or different. The cross array 200 may further include a word line (WL) logic 205 connected to the cross point device via the row lines 211a-n. The WL logic 205 may include any suitable components for applying input signals to the selected cross-point devices via the row lines 211a-n, such as one or more digital-to-analog converters (DACs), amplifiers, etc. Each input signal may be a voltage signal, a current signal, etc. The input signal may correspond to Figure 1A-1B The analog sensing signal generated by the sensing module 110.

[0059] The row lines 211 may include a first row line 211a, a second row line 211b, ..., 211i, ..., and an nth row line 211n. Each of the row lines 211a, ..., 211n may be and / or include any suitable conductive material. In some embodiments, each row line 211a-n may be a metal line.

[0060] The column lines 213 may include a first column line 213a, a second column line 213b, ..., and an mth column line 213m. Each column line 213a-m may be and / or include any suitable conductive material. In some embodiments, each column line 213a-m may be a metal line.

[0061] Each cross-point device 220a-z may be and / or include any suitable device having adjustable resistance, such as a memristor, a phase change memory (PCM) device, a floating gate, a spintronic device, a ferroelectric device, a RRAM device, or the like.

[0062] Each row line 211a-n can be connected to one or more row switches 231 (e.g., row switches 231a, 231b, ..., 231n). Each row switch 231 can include any suitable circuit structure that can control the current flowing through the row line 211a-n. For example, the row switch 231 can be and / or include a CMOS switch circuit.

[0063] Each column line 231a-231m can be connected to one or more column switches 233 (e.g., switches 233a-233m). Each column switch 233a-233m can include any suitable circuit structure that can control the current flowing through the column line 213a-213m. For example, the column switches 233a-233m can be and / or include CMOS switch circuits. In some embodiments, one or more of the switches 231a-231n and 233a-233m can further provide fault protection, electrostatic discharge (ESD) protection, noise reduction, and / or any other suitable function for one or more portions of the crossbar array 200.

[0064] The output sensor 240 may include any suitable components that convert the current flowing through the column lines 213a-n into an output signal, such as one or more TIAs (transimpedance amplifiers) 240a-n. Each TIA 240a-n may convert the current flowing through the corresponding column line into a corresponding voltage signal. Each ADC 250a-250m may convert the voltage signal generated by the corresponding TIA into a digital output. In some embodiments, the output sensor 240 may further include one or more multiplexers (not shown).

[0065] The programming circuit 260 can program the cross-point devices 220a-z selected by switches 231 and / or 233 to a suitable conductance value. For example, programming the cross-point device can involve applying a suitable voltage signal or current signal to the cross-point device. The resistance of each cross-point device can be electrically switched between a high-resistance state and a low-resistance state. Setting the cross-point device can involve switching the resistance of the cross-point device from a high-resistance state to a low-resistance state. Resetting the cross-point device can involve switching the resistance of the cross-point device from a low-resistance state to a high-resistance state.

[0066] The crossbar array 200 can perform parallel weighted voltage multiplication and current summation. For example, an input voltage signal can be applied to one or more rows (e.g., one or more selected rows) of the crossbar array 200. The input signal can flow through the crosspoint devices of the rows of the crossbar array 200. The conductance of the crosspoint device can be adjusted to a specific value (also referred to as a "weighted value"). According to Ohm's law, the input voltage is multiplied by the crosspoint conductance and generates a current flowing through the crosspoint device. According to Kirchhoff's law, the sum of the currents through the devices on each column generates a current as an output signal, which can be read from the column (e.g., the output of an ADC). According to Ohm's law and Kirchhoff's current law, the input-output relationship of the crossbar array can be expressed as I=VG, where I is the output signal matrix, expressed as current; V is the input signal matrix, expressed as voltage; and G is the conductance matrix of the crosspoint device. Therefore, the input signal is weighted by its conductance at each crosspoint device according to Ohm's law. The weighted current is output through each column line and accumulated according to Kirchhoff's current law. This can be achieved by implementing parallel multiplications and summations in a crossbar array to achieve in-memory computing (IMC).

[0067] The crossbar array 200 may be configured to perform vector matrix multiplication (VMM). A VMM operation may be represented as Y=XA, where Y, X, and A represent matrices, respectively. More specifically, for example, the input vector X may be mapped to an input voltage V of the crossbar array 200. The matrix A may be mapped to a conductance value G. The output current I may be read and mapped back to an output result Y. In some embodiments, the crossbar array 200 may be configured to implement a portion of a neural network by executing a VMM.

[0068] In some embodiments, the cross array 200 can perform a convolution operation. For example, performing a 2D convolution on the input data can involve applying a single convolution kernel to the input signal. Performing a depth convolution on the input signal can involve convolving each channel of the input data with a corresponding kernel corresponding to the channel and stacking the convolution outputs together. The convolution kernel can have a specific size defined by multiple dimensions (e.g., width, height, channel, etc.). The convolution kernel can be applied to a portion of the input data of the same size to produce an output. The output can be mapped to an element of the convolution result that is located at a position corresponding to the position of the input data portion.

[0069] The programming circuit 260 can program the cross array 200 to store a convolution kernel for performing a 2D convolution operation. For example, the convolution kernel can be converted into a vector and mapped to a plurality of cross-point devices in the cross array connected to a specified bit line. Specifically, the conductance value of the cross-point device can be programmed to represent the value of the convolution kernel. In response to an input signal, the cross array 200 can output a current signal representing the convolution of the input signal and the 2D convolution kernel through a specified bit line. In some embodiments, the cross array 200 can store multiple 2D convolution kernels by mapping each 2D convolution kernel to a cross-point device connected to a corresponding bit line. The cross array 200 can output multiple output signals (e.g., current signals) representing the convolution results through the column line 213.

[0070] Figure 3 FIG. 4 is a schematic diagram showing an example 300 of a 3D crossbar array circuit according to some embodiments of the present disclosure.

[0071] As shown in the figure, the 3D cross array circuit 300 may include a first cross array 310, a second cross array 320, and a third cross array 330 located in different planes. In some embodiments, the first cross array 310, the second cross array 320, and the third cross array 330 may be located on a first plane, a second plane, and a third plane, respectively. The first plane, the second plane, and the third plane are parallel to each other. In some embodiments, the first plane, the second plane, and the third plane may be perpendicular to or parallel to the substrate formed by the first cross array 310, the second cross array 320, and the third cross array 330. Each of the first cross array 310, the second cross array 320, and the third cross array 330 may include one or more combinations of Figure 2 The 2D crossbar array. Figure 3 While three crossbar arrays are shown, the 3D crossbar array circuit 300 may include any suitable number of 2D crossbar arrays integrated into a 3D crossbar array circuit.

[0072] The first crossbar array 310 may include a crosspoint device 315 connecting a first plurality of word lines (WL1_1, WL2_1, WL3_1, etc.) and a first plurality of bit lines (BL1_1, BL2_1, BL3_1, etc.). The second crossbar array 320 may include a crosspoint device 325 connecting a second plurality of word lines (WL1_2, WL2_2, WL3_2, etc.) and a second plurality of bit lines (BL1_2, BL2_2, BL3_2, etc.). The third crossbar array 330 may include a crosspoint device 335 connecting a third plurality of word lines (WL1_3, WL2_3, WL3_3, etc.) and a third plurality of bit lines (BL1_3, BL2_3, BL3_3, etc.).

[0073] The 3D crossbar array circuit 300 may further include transistors 340. Each transistor 340 may be connected to a corresponding gate line (GL1, GL2, GL3, etc.) through its gate region. For example, the gate line GL1 may be connected to the gate region of the first transistor in the first crossbar array 310, the gate region of the second transistor in the second crossbar array 320, and the gate region of the third transistor in the third crossbar array 330. The source region of the corresponding transistor 340 may be connected to a word line. It should be noted that Figure 3 The components of the 3D crossbar array circuit 300 and their connections are schematically shown. Figure 3 The schematic diagram shown does not represent the physical layout of the components of the 3D crossbar array circuit 300. The components of the 3D crossbar array circuit 300 can be physically laid out and positioned in any suitable manner to implement the 3D crossbar array shown in the present disclosure. For example, in one physical layout of the 3D crossbar array circuit 300, the transistors 340 can be located in the same layer of the substrate and then connected to the corresponding gate lines, bit lines, and word lines through vertical vias.

[0074] To select the cross-point device at the intersection of WL3_3 and BL3_3, apply a voltage V to GL3. G , while the other GLs can be grounded, making the transistor channel on GL3 open. Voltage V D It can be applied to the drain region of the transistor connected to WL3_3, while the drain regions of other transistors on the same horizontal layer are grounded, so that current can only flow through WL3_3. Voltage V ground can be applied to BL3_3, while maintaining the voltage V on other BLs crossing WL3_3 S . V S Can be equal to V d -V ds , where V S Represents the voltage of the transistor source region, V ds represents the voltage drop between the drain and source regions of the transistor. Therefore, only one device on WL can be S and V ground Other devices on the same WL (WL3_3) cannot be programmed because there is no voltage difference across these devices.

[0075] Since the crossbar arrays are arranged in a 3D manner in the 3D crossbar array circuit 300, the cross section of the 3D crossbar array circuit 300 can be regarded as a 2D crossbar array. Therefore, the 3D crossbar array circuit 300 can receive and process 2D inputs (e.g., m*n analog input signals) without storing the 2D inputs or converting the 2D inputs into one-dimensional data (e.g., vectors representing the 2D inputs). For example, the crosspoint devices located on WL1_1, WL1_2, WL1_3, etc. can be selected as described above to receive and process analog sensing signals generated by the 2D sensor array.

[0076] Figure 4A and 4B 2 is a schematic diagram showing examples of semiconductor devices 400a and 400b that can be used as machine learning processors in some embodiments of the present disclosure. The semiconductor devices 400a-b can also be considered as core particles.

[0077] As shown, semiconductor device 400a may include sensor wafer 410, processor wafer 420 and package substrate 430. In some embodiments, each of sensor wafer 410 and processor wafer 420 may be implemented as multiple wafers. For example, processor wafer 420 may include multiple wafers stacked in a 3D manner as described in the present disclosure.

[0078] Sensor wafer 410 may include a sensor module including one or more sensor arrays (e.g., a combination of Figure 1A-1B In some embodiments, the sensor wafer 410 may include a Figure 5A-5B The one or more image sensor wafers.

[0079] Processor die 420 may include one or more dies in which ADCs, crossbar arrays, driver ICs (integrated circuits), CMOS elements for implementing transceivers, and / or any other suitable components for implementing machine learning processing are embedded. Figure 1A ML processor 120 and / or Figure 1B The ML processor 140 in the embodiment may be fabricated on a processor wafer 420. The processor wafer 420 may be and / or include a combination of the following: Figure 6A-6B The processor chips 600a-b.

[0080] The processor wafer 420 and the sensor wafer 410 may be connected via a first interconnect layer 440. The interconnect layer 440 may include one or more metal interconnects (e.g., metal vias, metal pads, etc.). In some embodiments, a portion of the interconnect layer 440 may be considered as a portion of the processor wafer 420. The processor wafer 420 and the sensor wafer 410 may be interconnected using through silicon via (TSV) packaging technology, hybrid bond metal (HBM) packaging technology, intra-pixel hybrid bonding (IPHB) technology, and / or any other suitable chip packaging technology to package and / or stack multiple wafers.

[0081] The package substrate 430 may include an antenna, a connector, a power supply, etc. In some embodiments, the package substrate 430 does not include a CMOS component. The processor die 420 may be connected to the package substrate 430 via a second interconnect layer 450. For example, the interconnect layer 450 may include a ball grid array (BGA) bump. In some embodiments, the package substrate 430 may be connected to a PCB (printed circuit board) substrate (not shown).

[0082] refer to Figure 4B In some embodiments, a plurality of processor wafers and sensor wafers may be stacked on a packaging substrate 430. As shown, a first sensor wafer 410a may be stacked on a first processor wafer 420a and connected to the first processor wafer 420a via an interconnect layer 440-1. A second processor wafer 410b may be stacked on a second processor wafer 420b and connected to the second processor wafer 420b via an interconnect layer 440-2. Similarly, a third processor wafer 410c may be stacked on a third processor wafer 420c and connected to the third processor wafer 420c via an interconnect layer 440-3. Processor wafers 420a, 420b, and 420c may be connected to the packaging substrate 430 via interconnect layers 450a, 450b, and 450c, respectively. In some embodiments, sensor wafers 410a-c may include different types of sensors and may sense different signals. In these embodiments, processor wafers 420a-c may process different sensing signals generated by different sensor wafers 410a-c. Each of the sensor wafers 410a-c, the processor wafers 420a-c, the interconnect layers 440-1, 440-2, ..., 440-3, and the interconnect layers 450a-c may be and / or include a combination of Figure 4A Its counterparts (ie, sensor wafer 410, processor wafer 420, interconnect layer 440, and interconnect layer 450) are described above. Although Figure 4A and 4B A specific number of dies is shown in FIG. 4 , but this is merely illustrative. Any suitable number of sensor dies and processor dies may be stacked on packaging substrate 430 as described in the present disclosure.

[0083] Figure 5A and 5B is a diagram showing a cross section of example image sensor wafers 500a and 500b according to some embodiments of the present disclosure. Each of the image sensor wafers 500a and 500b may also be considered a CMOS image sensor (CIS) wafer.

[0084] like Figure 5A As shown, image sensor wafer 500a may include an image sensor including microlenses 511, color filters 513, and photodiodes 515a. Photodiodes 515a may be fabricated on substrate 505 (eg, a silicon substrate). Metal lines 520a may be located between color filters 513 and photodiodes 515a.

[0085] Incident light may be focused by the microlens 511 and may be separated into a plurality of color components by the color filter 513. For example, the red color filter 513a, the green color filter 513b, and the blue color filter 513c may separate a red light component, a green light component, and a yellow light component, respectively. The photodiode 515a may accumulate photon charges when exposed to light and convert the charges into electrical signals (voltage signals).

[0086] refer to Figure 5B , the image sensor wafer 500b includes a back-illuminated structure, in which the photodiode 515b is arranged behind the color filter 513, and the metal line 520b is arranged behind the photodiode 515b. The image sensor wafer 500b can be manufactured by manufacturing the photodiode 515b and the metal line 520b on the front silicon substrate 505, then flipping the substrate 505, thinning the back substrate, and manufacturing the color filter 513 and the microlens 511 on the back. As shown in the figure, the metal line 520b is located behind the photodiode 515b, and the metal line 520a is located in front of the photodiode. Light can be incident on the interface 531a of the image sensor wafer 500a and the interface 531b of the image sensor wafer 500b respectively. Therefore, since the light can reach the photodiode 515b without passing through the metal line 520b, the photodiode 515b can obtain more light signals than the photodiode 515a.

[0087] Fig. 6A and 6B is a schematic diagram showing an example of functional components of a processor chip according to some embodiments of the present disclosure. Fig. 6A As shown, the processor chip 600a may include a combination of Figure 1A The ML processor 120. Figure 6B As shown, the processor chip 600b may include a combination of Figure 1BThe processor wafers 600a and 600b may further include CMOS components, ICs, etc. (not shown) for implementing other functions of the processing devices described herein. For example, the processor wafers 600a and 600b may include components for implementing Figure 1A-1B CMOS components of one or more functions of the communication module 130.

[0088] Fig. 7A , 7B 7 and 7C are schematic diagrams showing examples of semiconductor devices 700a, 700b, and 700c integrating a sensor wafer and a processor wafer according to some implementations of the disclosure.

[0089] refer to Fig. 7A , the sensor wafer 410 and the processor wafer 420 may be connected by a TSV (Through Silicon Via) stack 440a. The TSV stack 440a may connect a portion of the metal interconnect 411 in the sensor wafer 410 and a portion of the metal interconnect 421 in the processor wafer 420. Figure 7B , the sensor wafer 410 and the processor wafer 420 may be connected via a hybrid bonding metal (HBM) connection 440b. Figure 7C , the sensor wafer 410 and the processor wafer 420 can be connected via an intra-pixel hybrid bonding (IPHB) connection 440c. The sensor wafer 410 and the processor wafer 420 can be connected at the pixel level to further facilitate fast data processing.

[0090] For the sake of brevity of explanation, the method of the present disclosure is depicted and described as a series of actions. However, actions according to the present disclosure can occur in various orders and / or simultaneously, and occur together with other actions that are not proposed and described in the present disclosure. In addition, not all actions of explanation can be required to realize the method according to the disclosed subject matter. In addition, those skilled in the art will understand and recognize that the method can be represented as a series of mutual states via state diagrams or events alternatively.

[0091] As used herein, the terms "approximately," "about," and "substantially" may refer to within a normal tolerance range in the art, such as within 2 standard deviations of the mean, within ±20% of a target size in some embodiments, within ±10% of a target size in some embodiments, within ±5% of a target size in some embodiments, within ±2% of a target size in some embodiments, within ±1% of a target size in some embodiments, and within ±0.1% of a target size in some embodiments. The terms "approximately" and "about" may include a target size. Unless otherwise specified or apparent from the context, all numerical values ​​described herein are modified by the term "about."

[0092] As used herein, a range includes all values ​​within the range. For example, a range of 1 to 10 may include any number, combination of numbers, sub-ranges of numbers from 1, 2, 3, 4, 5, 6, 7, 8, 9 and 10, and fractions thereof.

[0093] The present disclosure has been described in detail in the above description. However, it is obvious that the present disclosure can be implemented without these specific details. In some examples, in order to highlight the content of the present invention, well-known structures and devices are shown in the form of block diagrams rather than specific details.

[0094] The terms "first", "second", "third", "fourth", etc. used in this document are marks used to distinguish different components and may not necessarily have the ordinal meaning of the numerical numbers used.

[0095] The word "example" or "exemplary" as used herein means serving as an example, instance or illustration. Any aspect or design described herein as "example" or "exemplary" is not necessarily to be understood as being more preferred or advantageous than other aspects or designs. On the contrary, the purpose of using the words "example" or "exemplary" is to present the concept in a concrete way. In this application, the term "or" means including "or" rather than excluding "or". That is, unless otherwise specified or as can be seen from the context, "X includes A or B" means any natural inclusive permutation and combination. That is, if X includes A; X includes B; or X includes both A and B, then in any of the above cases, "X includes A or B" is satisfied. In addition, "a" and "an" used in this application and the appended claims should generally be understood as "one or more", unless otherwise specified or as can be clearly indicated from the context as being directed to a singular form. "One embodiment" or "an embodiment" mentioned in this specification means that a specific feature, structure or characteristic associated with the embodiment is included in at least one embodiment. Therefore, the phrases "one embodiment" or "an embodiment" appearing in different places in this specification do not necessarily all refer to the same embodiment.

[0096] As used herein, when an element or layer is referred to as being “on” another element or layer, the element or layer may be directly on the other element or layer, or intervening elements or layers may be present. In contrast, when an element or layer is referred to as being “directly on” another element or layer, there are no intervening elements or layers present.

[0097] Although it is obvious to those skilled in the art that additional changes and modifications may be made to the present disclosure after understanding the above description, it should be understood that any specific embodiments shown and described by way of illustration should not be considered limiting. Therefore, the details of various embodiments are not intended to limit the scope of the claims, which themselves merely describe the disclosed technical features.

Claims

1. A semiconductor device, comprising: A sensing module configured to generate a plurality of analog sensing signals; one or more crossbar arrays configured to process the analog sensor signals to generate analog pre-processed sensor data; an analog-to-digital converter (ADC) configured to convert the analog pre-processed sensor data into digital pre-processed sensor data; and A machine learning processing unit is configured to process the digital pre-processed sensor data using one or more machine learning models, wherein the machine learning processing unit is manufactured on a processor chip of the semiconductor device. 2 . The semiconductor device of claim 1 , wherein the sensing module is fabricated on a sensor wafer, wherein the sensor wafer is connected to the processor wafer through a first interconnect layer. 3 . The semiconductor device of claim 2 , wherein the one or more crossbar arrays are fabricated on the processor wafer. The semiconductor device of claim 3 , wherein the ADC is fabricated on the processor wafer. 5 . The semiconductor device according to claim 2 , wherein the sensing module comprises an image sensor array, and wherein the plurality of analog sensing signals comprises a plurality of analog image signals. 6 . The semiconductor device according to claim 2 , wherein the analog pre-processed sensing data corresponds to a plurality of features extracted from the analog sensing signal, and wherein the machine learning processing unit performs machine learning using the extracted features.

7. The semiconductor device of claim 2, further comprising a packaging substrate, wherein the processor die is connected to the packaging substrate through a second interconnect layer. 8 . The semiconductor device according to claim 2 , wherein the machine learning processing unit is powered by the analog sensing signal.

9. The semiconductor device according to claim 1, further comprising a transceiver configured to: Sending a prediction output generated by the machine learning processing unit based on one or more machine learning models to a computing device; and receiving from the computing device an instruction for executing an operation based on the prediction output. 10 . The semiconductor device of claim 1 , wherein the analog pre-processed sensing data represents a convolution of an analog sensing signal and a convolution kernel. 11 . The semiconductor device of claim 10 , wherein conductance values ​​of a plurality of cross-point devices in the one or more crossbar arrays are programmed to represent values ​​of a convolution kernel.

12. The semiconductor device of claim 1, wherein the sensing module comprises a two-dimensional sensor array, wherein a plurality of cross-point devices in the one or more cross-point arrays are configured to receive as input analog sensing signals generated by the two-dimensional sensor array.

13. The semiconductor device of claim 12, wherein the one or more crossbar arrays include a plurality of crossbar arrays located in a plurality of different planes.

14. A semiconductor device comprising: A sensing module configured to generate a plurality of analog sensing signals; and A machine learning processor configured to generate a prediction output by processing the analog sensor signal using one or more machine learning models, wherein the machine learning processor comprises: a plurality of crossbar arrays configured to generate a plurality of simulated outputs representing the predicted outputs; The analog-to-digital converter unit is configured to convert a plurality of analog outputs representing the predicted outputs into digital signals representing the predicted outputs.

15. The semiconductor device according to claim 14, further comprising: A transceiver configured to transmit a signal generated by the machine learning processor and representing the predicted output to a computing device.

16. The semiconductor device of claim 14, wherein the sensing module is fabricated on a sensor wafer, wherein the machine learning processor is fabricated on a processor wafer, wherein the sensor wafer is connected to the processor wafer via a first interconnect layer. 17 . The semiconductor device of claim 16 , further comprising a packaging substrate, wherein the processor die is connected to the packaging substrate through a second interconnect layer. 18 . The semiconductor device of claim 14 , wherein the sensing module comprises an image sensor array, wherein the plurality of analog sensing signals comprises a plurality of analog image signals.

19. The semiconductor device of claim 14, wherein the sensing module comprises a two-dimensional sensor array, wherein a plurality of cross-point devices in the plurality of cross-point arrays are configured to receive as input an analog sensing signal generated by the two-dimensional sensor array.

20. The semiconductor device according to claim 19, wherein the plurality of crossbar arrays are located in different planes.

Images