Wearable electronic device with built-in intelligent monitoring implemented using a deep learning accelerator and random access memory

Abstract

Systems, apparatuses, and methods related to deep learning accelerators and memory are described. For example, a wearable electronic device can be configured to execute instructions having matrix operands and configured with: a housing to be worn on a person; a sensor having one or more sensor elements to generate measurements associated with the person; a random access memory to store instructions executable by the deep learning accelerator and to store matrices of an artificial neural network; a transceiver; and a controller to monitor an output of the artificial neural network generated using the measurements as input to the artificial neural network. Based on the output, the controller can control selective storage of measurement data from the sensor and / or selective transfer of data from the wearable electronic device to a separate computer system.

Description

[0001] Related applications

[0002] This application claims priority to U.S. Patent Application No. 16 / 906,241, filed June 19, 2020, entitled “Wearable Electronic Device with Built-in Intelligent Monitoring Implemented Using Deep Learning Accelerator and Random Access Memory,” the entire disclosure of which is incorporated herein by reference. Technical Field

[0003] At least some of the embodiments disclosed herein generally relate to wearable electronic devices, and more specifically, but not limited to, wearable devices having integrated accelerators for artificial neural networks (ANNs), such as ANNs configured via machine learning and / or deep learning. Background Technology

[0004] Artificial neural networks (ANNs) use a network of neurons to process the inputs to the network and generate the outputs from the network.

[0005] For example, each neuron in a network receives a set of inputs. Some inputs to a neuron may be the outputs of other neurons in the network; and some inputs to a neuron may be inputs provided to the neural network. The input / output relationships between neurons in the network represent the neuronal connectivity in the network.

[0006] For example, each neuron can have a set of biases, activation functions, and synaptic weights for its inputs. Activation functions can take the form of step functions, linear functions, log-sigmoid functions, etc. Different neurons in the network can have different activation functions.

[0007] For example, each neuron can generate a weighted sum of its input and its bias, and then produce an output as a function of the weighted sum, which is calculated using the neuron's activation function.

[0008] The relationship between the inputs and outputs of an ANN is generally defined by the ANN model, which contains data representing the connectivity of neurons in the network, as well as the biases, activation functions, and synaptic weights of each neuron. Based on a given ANN model, a computing device can be configured to compute the network's output from a given set of inputs to the network.

[0009] For example, input to an ANN network can be generated based on camera input; and the output from the ANN network can be the recognition of items such as events or objects.

[0010] Generally, supervised methods can be used to train ANNs, where the parameters in the ANN are tuned to minimize or reduce the error between a known output associated with or produced by a given input and a computed output generated by applying the input to the ANN. Examples of supervised learning / training methods include reinforcement learning and learning with error correction.

[0011] Alternatively or in combination, unsupervised methods can be used to train an ANN, where the exact output produced by a given set of inputs is unknown until training is complete. An ANN can be trained to classify items into multiple categories or data points into clusters.

[0012] Multiple training algorithms can be used for complex machine learning / training paradigms.

[0013] Deep learning uses multiple layers of machine learning to progressively extract features from input data. For example, lower layers can be configured to identify edges in an image; and higher layers can be configured to identify captured items in the image, such as faces, objects, events, etc., based on the edges detected by the lower layers. Deep learning can be implemented via artificial neural networks (ANNs), such as deep neural networks, deep belief networks, recurrent neural networks, and / or convolutional neural networks.

[0014] Deep learning has been applied in many fields, such as computer vision, speech / audio recognition, natural language processing, machine translation, bioinformatics, drug design, medical image processing, and games. Attached Figure Description

[0015] Embodiments are shown in the figures by way of example rather than limitation, and similar reference numerals indicate similar elements in the figures.

[0016] Figure 1 An integrated circuit device having a configured deep learning accelerator and random access memory is shown according to one embodiment.

[0017] Figure 2 A processing unit configured to perform matrix-matrix operations is shown according to one embodiment.

[0018] Figure 3 A processing unit configured to perform matrix-vector operations is shown according to one embodiment.

[0019] Figure 4 A processing unit configured to perform vector-to-vector operations is shown according to one embodiment.

[0020] Figure 5 The illustration shows a deep learning accelerator and random access memory configured to autonomously apply inputs to a trained artificial neural network, according to one embodiment.

[0021] Figure 6 This invention illustrates a wearable electronic device configured with intelligent monitoring, according to one embodiment, using an integrated circuit device having configured sensors, a deep learning accelerator, and random access memory.

[0022] Figure 7 A wearable electronic device configured with a deep learning accelerator and random access memory is shown according to one embodiment.

[0023] Figure 8 Another wearable electronic device with a configured deep learning accelerator and random access memory is shown according to one embodiment.

[0024] Figure 9 A method implemented in a wearable electronic device according to one embodiment is shown. Detailed Implementation

[0025] At least some embodiments disclosed herein provide a wearable electronic device having a general-purpose integrated circuit configured to perform computations of an artificial neural network (ANN) with reduced power consumption and computation time. The integrated circuit includes a deep learning accelerator (DLA) and random access memory (RAM). Sensor data generated in the wearable device is stored in the RAM and processed by the artificial neural network (ANN) implemented using the deep learning accelerator (DLA) for intelligent monitoring of a person wearing the device. The output of the artificial neural network (ANN) can be trained to identify events, patterns, features, or classifications of interest in a particular application. The output can be stored in the RAM for generating alerts emitted from the wearable electronic device to a computer system, and / or for selectively retaining and / or emitting sensor data generated in the wearable device.

[0026] A deep learning accelerator (DLA) comprises a collection of general-purpose programmable hardware computational logic that is specialized and / or optimized to perform parallel vector and / or matrix computations, including but not limited to vector and / or matrix multiplication and accumulation.

[0027] In addition, a deep learning accelerator (DLA) may contain one or more arithmetic logic units (ALUs) for performing arithmetic and bitwise operations on integer binary numbers.

[0028] Deep learning accelerators (DLAs) can be programmed via instruction sets to perform computations of artificial neural networks (ANNs).

[0029] The granularity of a deep learning accelerator (DLA) operating on vectors and matrices corresponds to the largest unit of vector / matrix that can be operated on during the execution of an instruction by the DLA. During instructions used to perform predefined operations on vector / matrix operands, the DLA can operate on elements of the vector / matrix operands in parallel to reduce execution time and / or energy consumption associated with memory / data access. Operations on vector / matrix operands at the granularity of the DLA can be used as building blocks to perform computations on larger vectors / matrices.

[0030] Typical / practical implementations of artificial neural networks (ANNs) involve vector / matrix operands with a size larger than the operational granularity of a deep learning accelerator (DLA). To implement this ANN using a DLA, computations involving large-sized vector / matrix operands can be decomposed into computations of vector / matrix operands at the granularity of the DLA. The DLA can be programmed via instructions to perform computations involving large vector / matrix operands. For example, the atomic computational capabilities of the DLA can be programmed to perform computations within the ANN when manipulating vectors and matrices at the granularity of the DLA in response to instructions.

[0031] In some implementations, a deep learning accelerator (DLA) does not possess some of the logical computational capabilities of a typical central processing unit (CPU). However, a deep learning accelerator (DLA) can be configured with sufficient logic units to process the input data provided to the artificial neural network (ANN) and generate the output of the ANN according to an instruction set generated for the deep learning accelerator (DLA). Therefore, a deep learning accelerator (DLA) can perform ANN computations with minimal or no assistance from a central processing unit (CPU) or another processor. Optionally, a conventional general-purpose processor can also be configured as part of the deep learning accelerator (DLA) to perform operations that cannot be efficiently implemented using the vector / matrix processing unit of the deep learning accelerator (DLA), and / or operations that cannot be performed by the vector / matrix processing unit of the deep learning accelerator (DLA).

[0032] A typical artificial neural network (ANN) can be described / specified in a standard format (e.g., Open Neural Network Exchange (ONNX)). A compiler can be used to translate the description of an ANN into an instruction set for deep learning accelerators (DLAs) to perform computations on the ANN. The compiler can optimize the instruction set to improve the performance of the DLA when implementing the ANN.

[0033] Deep learning accelerators (DLAs) may have local memory, such as registers, buffers, and / or caches, configured to store vector / matrix operands and the results of vector / matrix operations. Intermediate results in registers can be pipelined / shifted within the DLA as operands for subsequent vector / matrix operations, reducing time and energy consumption during memory / data access and thus accelerating typical vector / matrix operations when implementing typical artificial neural networks (ANNs). The capacity of registers, buffers, and / or caches in a DLA is often insufficient to hold the entire dataset used for computations required to implement a typical artificial neural network (ANN). Therefore, random access memory coupled to the DLA is configured to provide improved data storage capabilities for implementing typical artificial neural networks (ANNs). For example, the DLA loads data and instructions from random access memory and stores the results back into random access memory.

[0034] The communication bandwidth between the Deep Learning Accelerator (DLA) and random access memory (RAM) is configured to optimize or maximize the utilization of the DLA's computational power. For example, high communication bandwidth between the DLA and RAM can be provided to allow vector / matrix operands to be loaded from RAM into the DLA and stored back into RAM within a time period approximately equal to the time it takes for the DLA to perform computations on the vector / matrix operands. The granularity of the DLA can be configured to increase the ratio between the amount of computation performed by the DLA and the size of the vector / matrix operands, thereby reducing data access traffic between the DLA and RAM, which lowers the bandwidth requirements for communication between the DLA and RAM. Therefore, bottlenecks in data / memory access can be reduced or eliminated.

[0035] In at least some embodiments, the wearable electronic device is configured to have sensors, a deep learning accelerator (DLA), and random access memory, which has an artificial neural network (ANN) for intelligent monitoring of the condition of the person wearing the device and / or the surrounding environment.

[0036] For example, wearable electronic devices can be configured in the form of glasses, contact lenses, watches, wristbands, armbands, rings, headbands, head-mounted displays, gloves, clothing, etc.

[0037] For example, sensors configured in wearable electronic devices may include one or more sensor elements that are configured to measure various quantitative data such as sound, light, vibration, motion, orientation, physical activity, and health parameters.

[0038] Instead of transmitting sensor data to a separate computer system for processing, wearable electronics use artificial neural networks (ANNs) to process sensor data to identify events, features, classifications, and / or patterns of interest in the specific application of the wearable electronics. Sensor data that is not of interest can be discarded to enhance the privacy of the person wearing the sensor device and / or reduce the data traffic and / or energy consumption of the wearable electronics.

[0039] For example, wearable electronic devices are equipped with sensors to monitor a user's health. Integrated circuit devices with deep learning accelerators and random access memory are configured within the wearable electronic device to process sensor data locally, reducing the need to store large amounts of raw data and protecting user privacy. Deep learning accelerators can be implemented, at least in part, via neuromorphic memory that supports multiplication and summation during memory read processes. Using neuromorphic memory to implement deep learning accelerators can further reduce the power consumption of the wearable device when processing sensor data.

[0040] Some sensor elements of wearable electronic devices can be configured within integrated circuit devices that have deep learning accelerators (DLAs) and random access memory. Other sensor elements can be connected to the integrated circuit devices with deep learning accelerators (DLAs) and random access memory in wearable electronic devices via printed circuit boards.

[0041] Wearable electronics can be customized for specific applications of intelligent monitoring by storing models of artificial neural networks (ANNs) that can be executed by a deep learning accelerator (DLA). For example, a compiler can be used to transform the model based on a description of the artificial neural network (ANN); and the model contains the weight / kernel matrix of the artificial neural network (ANN) and instructions with matrix operands, which can be executed by a deep learning accelerator (DLA) to perform computations of the artificial neural network (ANN) based on the weight / kernel matrix.

[0042] After the wearable electronics are customized to perform computations of an artificial neural network (ANN) by storing the model in random access memory, raw sensor data generated by the at least one sensor in the wearable electronics is provided as input to the artificial neural network (ANN); and the output of the artificial neural network (ANN) can be used to generate warnings, selectively retain and / or report sensor data, and / or provide it as the main output of the wearable electronics.

[0043] For example, wearable electronic devices may include wireless transceivers (e.g., communication devices for wireless personal area networks, such as Bluetooth transceivers). The output of an artificial neural network (ANN) can be reported to a separate computer system, such as a smartphone, personal media player, mobile computer, personal computer, Internet of Things (IoT) hub, and / or server computer, via a wireless connection.

[0044] Alternatively, wearable electronics may have a port for a wired connection to a separate computer system to report the output of an artificial neural network (ANN) or download the output stored in the wearable electronics over a period of time.

[0045] For example, wearable electronic devices can provide intelligent outputs generated locally within the wearable device using artificial neural networks (ANNs) to a computer system without assistance from a computer system. For example, wearable electronic devices can be used to monitor health-related events and generate warnings when such events are detected. For example, wearable electronic devices can be used to monitor indications related to the diagnosis of health problems and record the occurrence of such indications and / or associated data for further analysis. For example, wearable electronic devices can be used to detect user falls and generate a call for help when a fall is detected. For example, wearable electronic devices can be used to detect the presence of objects in the user's surrounding environment and provide object identification for further processing. For example, wearable electronic devices can be used to detect user gestures and provide the recognition of the gestures to a separate computer (e.g., a smartphone, game console, personal media player, personal computer, set-top box) to control operations on that separate computer.

[0046] Random access memory (RAM) in wearable electronics may contain a portion configured to store inputs to an artificial neural network (ANN) and another portion configured to store outputs from the ANN. Data generated by sensors in the wearable electronics can be stored cyclically in the input portion of the RAM. Therefore, the latest cycle of raw sensor data used for the operation of the wearable electronics can be found in the input portion of the RAM. A deep learning accelerator (DLA) can transform the data in the input portion into inference results stored in the output portion of the RAM in real time.

[0047] For example, the sensor input data stream to an artificial neural network (ANN) can be configured as a sequence of input datasets. Each input dataset is a set of inputs to the ANN during a given time slot. While the deep learning accelerator (DLA) is computing the output from the current input set, the controller can store the next input set in random access memory; and the controller can simultaneously retrieve the output generated for the previous input set from the random access memory.

[0048] Therefore, the task of preparing and processing input data from sensors to artificial neural networks (ANNs) can be performed locally on the wearable electronics to reduce the amount of data transmitted to a separate computer system. This arrangement reduces the power consumption associated with transmitting large amounts of sensor data from wearable electronics and improves privacy for users of the wearable electronics. Furthermore, neuromorphic memories can be used to perform matrix / vector multiplication and summation computations to reduce the power consumption of deep learning accelerators (DLAs).

[0049] For example, a neuromorphic memory can be implemented using a cross-array of memristors configured to perform multiply-accumulate (MAC) operations via an analog circuit system. The current flowing through a set of memristors in the cross-array across the word line to the bit line is summed in the bit line, corresponding to an accumulation operation. The current corresponds to the multiplication of the voltage applied to the word line and a parameter associated with the resistance of the memristor, corresponding to a multiplication operation. The current in the bit line can be compared to a threshold to determine whether the neuron represented by the bit line is activated under current input. The memristor array can be individually connected to the bit line and programmed to have thresholds corresponding to the activation level threshold of the neuron. A current detector can be configured for each memristor connected to the output of the bit line to determine whether the level of the current in the bit line corresponds to a level exceeding the memristor's threshold.

[0050] Figure 1 An integrated circuit device (101) with a configured deep learning accelerator (103) and random access memory (105) is shown according to one embodiment.

[0051] Figure 1 The deep learning accelerator (103) includes a processing unit (111), a control unit (113), and local memory (115). When vector and matrix operands are in local memory (115), the control unit (113) can use the processing unit (111) to perform vector and matrix operations according to instructions. In addition, the control unit (113) can load instructions and operands from random access memory (105) via a memory interface (117) and a high-speed / bandwidth connection (119).

[0052] The integrated circuit device (101) is configured to be enclosed within an integrated circuit package using pins or contacts of a memory controller interface (107).

[0053] The memory controller interface (107) is configured to support standard memory access protocols, such that the integrated circuit device (101) behaves as a typical memory controller in the same way as a conventional random access memory device without a deep learning accelerator (DLA) (103). For example, a memory controller external to the integrated circuit device (101) can access the random access memory (105) in the integrated circuit device (101) via the memory controller interface (107) using standard memory access protocols.

[0054] The integrated circuit device (101) is configured with a high-bandwidth connection (119) between a random access memory (105) enclosed within the integrated circuit device (101) and a deep learning accelerator (DLA) (103). The bandwidth of the connection (119) is higher than the bandwidth of the connection (109) between the random access memory (105) and the memory controller interface (107).

[0055] In one embodiment, both the memory controller interface (107) and the memory interface (117) are configured to access the random access memory (105) via the same set of buses or wires. Therefore, the bandwidth used to access the random access memory (105) is shared between the memory interface (117) and the memory controller interface (107). Alternatively, the memory controller interface (107) and the memory interface (117) are configured to access the random access memory (105) via separate array buses or wires. Optionally, the random access memory (105) may contain multiple segments that can be accessed simultaneously via a connection (119). For example, while the memory interface (117) is accessing a segment of the random access memory (105), the memory controller interface (107) may simultaneously access another segment of the random access memory (105). For example, different segments can be configured on different planes / rows of different integrated circuit dies and / or memory cells; and different segments can be accessed in parallel to increase the throughput when accessing random access memory (105). For example, the memory controller interface (107) is configured to access a data cell of a predetermined size at a time; and the memory interface (117) is configured to access multiple data cells at a time, each data cell being the same predetermined size.

[0056] In one embodiment, the random access memory (105) and the integrated circuit device (101) are disposed on different integrated circuit dies disposed within the same integrated circuit package. Furthermore, the random access memory (105) may be disposed on one or more integrated circuit dies that allow simultaneous parallel access to multiple data elements.

[0057] In some implementations, the number of data elements of a vector or matrix that can be accessed in parallel via connection (119) corresponds to the granularity of the deep learning accelerator (DLA) operating on the vector or matrix. For example, when the processing unit (111) can operate on several vector / matrix elements in parallel, connection (119) is configured to load or store the same number or a multiple thereof of elements in parallel via connection (119).

[0058] Optionally, the data access speed of the connection (119) can be configured based on the processing speed of the deep learning accelerator (DLA) (103). For example, after a certain amount of data and instructions have been loaded into the local memory (115), the control unit (113) can use the processing unit (111) to execute instructions to manipulate the data to generate output. During the processing time used to generate the output, the access bandwidth of the connection (119) allows the same amount of data and instructions to be loaded into the local memory (115) for the next operation, and the same amount of output to be stored back into the random access memory (105). For example, while the control unit (113) is processing data and generating output using a portion of the local memory (115), the memory interface (117) can unload the output of the previous operation from another portion of the local memory (115) into the random access memory (105), and load operand data and instructions into the other portion of the local memory. Therefore, the utilization and performance of the deep learning accelerator (DLA) are not limited or reduced by the bandwidth of the connection (119).

[0059] Random access memory (105) can be used to store model data of the artificial neural network (ANN) and to buffer input data of the artificial neural network (ANN). The model data does not change frequently. The model data may contain output generated by a compiler for a deep learning accelerator (DLA) to implement the artificial neural network (ANN). The model data typically contains matrices used in the description of the artificial neural network (ANN) and instructions generated for the deep learning accelerator (DLA) (103) to perform vector / matrix operations on the artificial neural network (ANN) at a granularity based on the deep learning accelerator (DLA) (103). The instructions operate not only on the vector / matrix operations of the artificial neural network (ANN) but also on the input data of the artificial neural network (ANN).

[0060] In one embodiment, when input data is loaded or updated in random access memory (105), the control unit (113) of the deep learning accelerator (DLA) (103) can automatically execute the instructions of the artificial neural network (ANN) to generate the output of the ANN. The output is stored in a predefined area in the random access memory (105). The deep learning accelerator (DLA) (103) can execute instructions without the assistance of a central processing unit (CPU). Therefore, communication for coordination between the deep learning accelerator (DLA) (103) and a processor (e.g., a central processing unit (CPU)) outside the integrated circuit device (101) can be reduced or eliminated.

[0061] Optionally, the logic circuitry of the deep learning accelerator (DLA) (103) can be implemented via complementary metal-oxide-semiconductor (CMOS). For example, under-array CMOS (CUA) technology of the memory cells of random access memory (105) can be used to implement the logic circuitry of the deep learning accelerator (DLA) (103), including a processing unit (111) and a control unit (113). Alternatively, CMOS technology in the memory cell array of random access memory (105) can be used to implement the logic circuitry of the deep learning accelerator (DLA) (103).

[0062] In some implementations, the deep learning accelerator (DLA) (103) and random access memory (105) may be implemented on separate integrated circuit dies and connected using through-silicon vias (TSVs) to increase the data bandwidth between the deep learning accelerator (DLA) (103) and the random access memory (105). For example, the deep learning accelerator (DLA) (103) may be formed on an integrated circuit die of a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC).

[0063] Alternatively, the deep learning accelerator (DLA) (103) and random access memory (105) can be configured in a separate integrated circuit package and connected via multiple point-to-point connections on a printed circuit board (PCB) for parallel communication, thereby increasing data transfer bandwidth.

[0064] Random access memory (105) can be volatile memory, non-volatile memory, or a combination of volatile and non-volatile memory. Examples of non-volatile memory include flash memory, memory cells formed based on NAND logic gates, NOR logic gates, phase-change memory (PCM), magnetic RAM (MRAM), resistive random access memory, cross-point memory devices, and memory devices. Cross-point memory devices can use transistorless memory elements, each of which has memory cells and selectors stacked together in a column. The column of memory elements is connected via two layers of lines extending in a vertical direction, wherein the lines of one layer extending in one direction are above the column of memory elements, and the lines of the other layer extending in the other direction are below the column of memory elements. Each memory element can be selected individually at the intersection of a line on each of the two layers. Cross-point memory devices are fast and non-volatile and can be used as a unified memory pool for processing and storage. Other examples of non-volatile memory include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), and electronically erasable programmable read-only memory (EEPROM). Examples of volatile memory include dynamic random access memory (DRAM) and static random access memory (SRAM).

[0065] For example, non-volatile memory may be configured to implement at least a portion of random access memory (105). The non-volatile memory within random access memory (105) can be used to store model data of an artificial neural network (ANN). Therefore, after the integrated circuit device (101) is powered off and restarted, it is not necessary to reload the model data of the artificial neural network (ANN) into the integrated circuit device (101). Furthermore, the non-volatile memory may be programmable / rewritable. Therefore, the model data of the artificial neural network (ANN) in the integrated circuit device (101) can be updated or replaced to implement an updated artificial neural network (ANN) or another artificial neural network (ANN).

[0066] The processing unit (111) of the deep learning accelerator (DLA) (103) may include vector-vector units, matrix-vector units, and / or matrix-matrix units. The following is in conjunction with... Figure 2-4 Discuss instances of units configured to perform vector-vector operations, matrix-vector operations, and matrix-matrix operations.

[0067] Figure 2 A processing unit (121) configured to perform matrix-matrix operations is shown according to one embodiment. For example, Figure 2 The matrix-matrix unit (121) can be used as Figure 1One of the processing units (111) of the Deep Learning Accelerator (DLA) (103).

[0068] exist Figure 2 In this configuration, the matrix-matrix unit (121) comprises multiple kernel buffers (131 to 133) and multiple map groups (151 to 153). Each of the map groups (151 to 153) stores a vector of a matrix operand, which has multiple vectors stored in the map groups (151 to 153); and each of the kernel buffers (131 to 133) stores a vector of another matrix operand, which has multiple vectors stored in the kernel buffers (131 to 133). The matrix-matrix unit (121) is configured to perform multiplication and accumulation operations on the elements of the two matrix operands using multiple matrix-vector units (141 to 143) operating in parallel.

[0069] A crossbar switch (123) connects the mapping groups (151 to 153) to the matrix-vector units (141 to 143). Identical matrix operands stored in the mapping groups (151 to 153) are provided to each of the matrix-vector units (141 to 143) via the crossbar switch (123); and the matrix-vector units (141 to 143) receive data elements from the mapping groups (151 to 153) in parallel. Each of the kernel buffers (131 to 133) is connected to a corresponding matrix-vector unit (141 to 143) and provides vector operands to the corresponding matrix-vector unit. The matrix-vector units (141 to 143) simultaneously operate to compute the operation of multiplying the identical matrix operands stored in the mapping groups (151 to 153) by the corresponding vector stored in the kernel buffers (131 to 133). For example, the matrix-vector unit (141) performs multiplication on matrix operands stored in the map array (151 to 153) and vector operands stored in the kernel buffer (131), while the matrix-vector unit (143) simultaneously performs multiplication on matrix operands stored in the map array (151 to 153) and vector operands stored in the kernel buffer (133).

[0070] Figure 2 Each of the matrix-vector units (141 to 143) in the matrix can be like... Figure 3 Implemented as shown.

[0071] Figure 3 A processing unit (141) configured to perform matrix-vector operations is shown according to one embodiment. For example, Figure 3 The matrix-vector unit (141) can be used as Figure 2Any one of the matrix-vector units in the matrix-matrix unit (121).

[0072] exist Figure 3 In the diagram, each of the mapping groups (151 to 153) is arranged in a manner similar to... Figure 2 A vector of matrix operands is stored in a mapping group (151 to 153), wherein the matrix operands have multiple vectors stored in the mapping group (151 to 153). Figure 3 The crossbar switch (123) in the middle provides vectors from the mapping group (151) to the vector-vector units (161 to 163). The same vectors stored in the kernel buffer (131) are provided to the vector-vector units (161 to 163).

[0073] Vector-vector units (161 to 163) operate simultaneously to compute the operation of multiplying the corresponding vector operands stored in the map arrays (151 to 153) by the same vector operands stored in the kernel buffer (131). For example, vector-vector unit (161) performs a multiplication operation on the vector operands stored in the map array (151) and the vector operands stored in the kernel buffer (131), while vector-vector unit (163) simultaneously performs a multiplication operation on the vector operands stored in the map array (153) and the vector operands stored in the kernel buffer (131).

[0074] when Figure 3 The matrix-vector unit (141) is implemented in Figure 2 When in the matrix-matrix unit (121), the matrix-vector unit (141) can use the mapping group (151 to 153), the cross switch (123) and the kernel buffer (131) of the matrix-matrix unit (121).

[0075] Figure 3 Each of the vector-vector units (161 to 163) in the vector array can be as follows: Figure 4 Implemented as shown.

[0076] Figure 4 A processing unit (161) configured to perform vector-to-vector operations is shown according to one embodiment. For example, Figure 4 The vector-vector unit (161) can be used as Figure 3 Any one of the vector-vector units in the matrix-vector unit (141).

[0077] exist Figure 4In this, the vector-vector unit (161) has multiple multiply-accumulate (MAC) units (171 to 173). Each of the multiply-accumulate (MAC) units (171 to 173) can receive two numbers as operands, perform the multiplication of the two numbers, and add the result of the multiplication to the sum maintained in the multiply-accumulate (MAC) unit.

[0078] Each of the vector buffers (181 and 183) stores a list of numbers. A pair of numbers, each from one of the vector buffers (181 and 183), can be provided as input to each of the multiply-accumulate (MAC) units (171 to 173). The multiply-accumulate (MAC) units (171 to 173) can receive multiple pairs of numbers from the vector buffers (181 and 183) in parallel and perform the multiply-accumulate (MAC) operation in parallel. The outputs from the multiply-accumulate (MAC) units (171 to 173) are stored in a shift register (175); and the accumulator (177) calculates the sum of the results in the shift register (175).

[0079] when Figure 4 The vector-vector unit (161) is implemented in Figure 3 When in the matrix-vector unit (141), the vector-vector unit (161) can use a mapping group (e.g., 151 or 153) as a vector buffer (181) and use the kernel buffer (131) of the matrix-vector unit (141) as another vector buffer (183).

[0080] Vector buffers (181 and 183) may have the same length to store the same number / count of data elements. The length may be equal to or a multiple of the count of the multiply-accumulate (MAC) units (171 to 173) in the vector-vector unit (161). When the length of the vector buffers (181 and 183) is a multiple of the count of the multiply-accumulate (MAC) units (171 to 173), input pairs equal to the number of counts of the multiply-accumulate (MAC) units (171 to 173) may be provided as input from the vector buffers (181 and 183) to the multiply-accumulate (MAC) units (171 to 173) in each iteration; and the vector buffers (181 and 183) feed their elements into the multiply-accumulate (MAC) units (171 to 173) over multiple iterations.

[0081] In one embodiment, the communication bandwidth of the connection (119) between the deep learning accelerator (DLA) (103) and the random access memory (105) is sufficient for the matrix-matrix unit (121) to use a portion of the random access memory (105) as a map group (151 to 153) and a kernel buffer (131 to 133).

[0082] In another embodiment, the mapping groups (151 to 153) and kernel buffers (131 to 133) are implemented in a portion of the local memory (115) of the deep learning accelerator (DLA) (103). While the matrix-matrix unit (121) is performing computations using the mapping groups (151 to 153) and kernel buffers (131 to 133) implemented in different portions of the local memory (115) of the deep learning accelerator (DLA) (103), the communication bandwidth of the connection (119) between the deep learning accelerator (DLA) (103) and random access memory (105) is sufficient to load the matrix operands of the next operating cycle of the matrix-matrix unit (121) into another portion of the local memory (115).

[0083] Figure 5 The illustration shows a deep learning accelerator and random access memory configured to autonomously apply inputs to a trained artificial neural network, according to one embodiment.

[0084] Artificial neural networks (ANNs) (201) that have been trained by machine learning (e.g., deep learning) can be described using standard formats (e.g., Open Neural Network Exchange (ONNX)). Describing a trained ANN (201) using standard formats identifies the properties of the artificial neurons and their connectivity.

[0085] exist Figure 5 In this process, the deep learning accelerator (DLA) compiler (203) transforms the trained ANN (201) by generating instructions (205) for the deep learning accelerator (DLA) (103) and a matrix (207) corresponding to the properties and connectivity of the artificial neuron. The instructions (205) and matrix (207) generated by the DLA compiler (203) from the trained ANN (201) can be stored in the random access memory (105) for the deep learning accelerator (DLA) (103).

[0086] For example, random access memory (105) and deep learning accelerator (DLA) (103) can be used as follows: Figure 1 The integrated circuit device (101) is connected via a high-bandwidth connection (119). Figure 5 Autonomous computation based on instructions (205) and matrices (207) can be implemented in Figure 1 In an integrated circuit device (101). Alternatively, random access memory (105) and deep learning accelerator (DLA) (103) may be configured on a printed circuit board having multiple point-to-point serial buses with parallel extensions for connection (119).

[0087] exist Figure 5In the process, after the results of the DLA compiler (203) are stored in the random access memory (105), the application of the trained ANN (201) to process the input (211) to the trained ANN (201) to generate the corresponding output (213) of the trained ANN (213) can be triggered by the presence of the input (211) in the random access memory (105) or by another instruction provided in the random access memory (105).

[0088] In response, the deep learning accelerator (DLA) (103) executes instructions (205) to combine the input (211) and the matrix (207). The execution of instructions (205) may involve generating a mapping matrix by a grouping (151 to 153) of one or more matrix-matrix units (e.g., 121) of the deep learning accelerator (DLA) (103).

[0089] In some embodiments, the input to the ANN (211) is in the form of an initial mapping matrix. A portion of the initial mapping matrix can be retrieved from random access memory (105) as a matrix operand stored in a mapping array (151 to 153) of matrix-matrix units (121). Alternatively, the DLA instructions (205) may also include instructions for the deep learning accelerator (DLA) (103) to generate the initial mapping matrix from the input (211).

[0090] According to the DLA instruction (205), the deep learning accelerator (DLA) (103) loads matrix operands into the kernel buffers (131 to 133) and map groups (151 to 153) of its matrix-matrix unit (121). The matrix-matrix unit (121) performs matrix computations on the matrix operands. For example, the DLA instruction (205) decomposes the matrix computations of the trained ANN (201) according to the computational granularity of the deep learning accelerator (DLA) (103) (e.g., the size / dimension of the matrix loaded in the matrix-matrix unit (121) as a matrix operand), and applies the input feature maps to the kernel of one layer of artificial neurons to generate outputs as inputs for the next layer of artificial neurons.

[0091] After the computation of the trained ANN (201) executed according to the instruction (205) is completed, the deep learning accelerator (DLA) (103) immediately stores the output (213) of the ANN (201) at a predefined location in the random access memory (105) or at a location specified in the instruction to trigger computation provided in the random access memory (105).

[0092] when Figure 5 The technology implemented in Figure 1When the input (211) is in the integrated circuit device (101), an external device connected to the memory controller interface (107) can write the input (211) into the random access memory (105) and trigger the deep learning accelerator (DLA) (103) to apply the input (211) to the autonomous computation of the trained ANN (201). After a certain time period, the output (213) is available in the random access memory (105); and the external device can read the output (213) via the memory controller interface (107) of the integrated circuit device (101).

[0093] For example, a predefined location in random access memory (105) can be configured to store an indication that triggers the autonomous execution of instructions (205) by a deep learning accelerator (DLA) (103). The indication may optionally include the location of inputs (211) within random access memory (105). Thus, during the autonomous execution of instructions (205) that process inputs (211), an external device can retrieve outputs generated during previous runs of instructions (205) and / or store another set of inputs for the next run of instructions (205).

[0094] Optionally, another predefined location in the random access memory (105) may be configured to store an indication of the current progress status of the instruction (205). Furthermore, the indication may include a prediction of the completion time of the current execution of the instruction (205) (e.g., estimated based on previous executions of the instruction (205)). Thus, an external device can check the completion status at an appropriate time window to retrieve the output (213).

[0095] In some embodiments, the random access memory (105) is configured with sufficient capacity to store multiple sets of inputs (e.g., 211) and outputs (e.g., 213). Each set may be configured in a predetermined time slot / region within the random access memory (105).

[0096] The deep learning accelerator (DLA) (103) can autonomously execute instructions (205) to generate an output (213) from an input (211) based on a matrix (207) stored in random access memory (105) without the assistance of a processor or device located outside the integrated circuit device (101).

[0097] In one embodiment of the method, the computing device (e.g., 101) may access the random access memory (105) of the computing device (e.g., 101) via an interface (107) to a memory controller. The computing device (e.g., 101) may have a processing unit (e.g., 111) configured to perform computations on at least matrix operands, such as matrix operands stored in a map array (151 to 153) and matrix operands stored in a kernel buffer (131 to 133).

[0098] For example, a computing device (e.g., 101) may be enclosed within an integrated circuit package; and a set of connections may connect an interface (107) to a memory controller located outside the integrated circuit package.

[0099] Instructions (205) executable by a processing unit (e.g., 111) can be written into random access memory (105) via an interface (107).

[0100] The matrix (207) of the artificial neural network (201) can be written into the random access memory (105) via the interface (107). The matrix (207) identifies the properties and / or state of the artificial neural network (201).

[0101] Optionally, at least a portion of the random access memory (105) is non-volatile and configured to store a matrix (207) of instructions (205) and an artificial neural network (201).

[0102] The first input (211) to the artificial neural network can be written into the random access memory (105) via the interface (107).

[0103] An instruction is provided in the random access memory (105) to cause the processing unit (111) to begin execution of the instruction (205). In response to the instruction, the processing unit (111) executes an instruction to combine the first input (211) with the matrix (207) of the artificial neural network (201) to generate a first output (213) from the artificial neural network (201) and stores the first output (213) in the random access memory (105).

[0104] For example, the indication may be the address of a first input (211) in random access memory (105); and the indication may be stored at a predetermined location in random access memory (105) such that execution of an instruction (205) for the input (211) identified by the address is initiated. Optionally, the indication may also include an address for storing an output (213).

[0105] The first output (213) can be read from the random access memory (105) via the interface (107).

[0106] For example, a computing device (e.g., 101) may have a deep learning accelerator (103) formed on a first integrated circuit die and random access memory (105) formed on one or more second integrated circuit dies. The connection (119) between the first integrated circuit die and the one or more second integrated circuit dies may include through-silicon vias (TSVs) to provide high bandwidth for memory access.

[0107] For example, a compiler (203) can be used to convert the description of the artificial neural network (201) into instructions (205) and matrices (207). The combination of the instructions (205) and matrices (207) stored in random access memory (105) and the deep learning accelerator (103) provides an autonomous implementation of the artificial neural network (201) that can automatically convert the input (211) to the artificial neural network (201) into its output (213).

[0108] For example, during the time period in which the deep learning accelerator (103) executes instructions (205) to generate a first output (213) from a first input (211) according to the matrix (207) of the artificial neural network (201), a second input to the artificial neural network (201) can be written to the random access memory (105) at an alternative location via the interface (107). After the first output (213) is stored in the random access memory (105), an instruction can be provided in the random access memory to cause the deep learning accelerator (103) to resume execution of instructions and generate a second output from the second input.

[0109] During the time period in which the deep learning accelerator (103) executes instructions (205) to generate a second output from a second input according to the matrix (207) of the artificial neural network (201), the first output (213) can be read from the random access memory (105) via the interface (107); and another input can be written into the random access memory in place of the first input (211), or written at a different location. The process can be repeated for a sequence of inputs.

[0110] The deep learning accelerator (103) may include at least one matrix-matrix unit (121) capable of executing instructions on two matrix operands. The two matrix operands may be a first matrix and a second matrix. Each of the two matrices has multiple vectors. The matrix-matrix unit (121) may include multiple matrix-vector units (141 to 143) configured to operate in parallel. Each of the matrix-vector units (141 to 143) is configured to operate on the first matrix and a vector from the second matrix in parallel with other matrix-vector units. Furthermore, each of the matrix-vector units (141 to 143) may have multiple vector-vector units (161 to 163) configured to operate in parallel. Each of the vector-vector units (161 to 163) is configured to operate on a vector from the first matrix and a common vector operand of the corresponding matrix-vector unit in parallel with other vector-vector units. Furthermore, each of the vector-vector units (161 to 163) may have multiple multiply-accumulate units (171 to 173) configured to operate in parallel.

[0111] In addition to the processing unit (111), the deep learning accelerator (103) may also have local memory (115) and a control unit (113). The control unit (113) may load instructions (205) and matrix operands (e.g., 207) from random access memory (105) for execution by the processing unit (111). The local memory may cache matrix operands used by the matrix-matrix unit. The connection (119) may be configured with bandwidth sufficient to load a set of matrix operands from random access memory (105) into local memory (115) during a time period in which the matrix-matrix unit performs operations on two other matrix operands. Furthermore, during the time period, the bandwidth is sufficient to store the results generated by the matrix-matrix unit (121) in a previous instruction execution from local memory (115) into random access memory (105).

[0112] The deep learning accelerator (103) and random access memory (105) can be combined with a sensor having one or more sensor elements to form a wearable electronic device. The wearable electronic device can be configured to monitor the condition of the user wearing the device and / or the surrounding environment to generate warnings and / or selectively store sensor data.

[0113] For example, a sensor may be part of its sensor element built into an integrated circuit device (101) having a deep learning accelerator (103) and random access memory (105). Other sensor elements may be connected to the integrated circuit device (101) on a printed circuit board in the wearable electronic device. The wearable electronic device may have a transceiver for wireless communication connectivity to a separate computer system, such as a mobile device, smartphone, personal media player, personal computer, set-top box, Internet of Things (IoT) hub, server computer, etc. The wearable electronic device may provide data generated by and / or selected based on the output of the artificial neural network (201) for further processing by the separate computer.

[0114] Wearable electronic devices can be customized, updated, and / or upgraded by receiving matrices (207) and instructions (205) of an artificial neural network (201) via a transceiver and storing them in random access memory (105).

[0115] An artificial neural network (201), implemented by executing instructions (205) via a deep learning accelerator (103), converts measurements from sensors into inferences. This conversion improves the quality of the wearable electronics' output, reduces the communication bandwidth requirements for connections to computer systems, and / or reduces the computational workload of computer systems.

[0116] For example, wearable electronic devices can convert audio signals into recognized text and / or images into recognized objects, events, categories, features, etc. For example, wearable electronic devices can convert measured motion parameter data into recognized directional actions. For example, wearable electronic devices can convert measured biometric data into health status diagnoses. For example, wearable electronic devices can convert measured biometric data and motion data into athletic performance levels.

[0117] Figure 6 A wearable electronic device (191) with intelligent monitoring is shown according to one embodiment, using an integrated circuit device (101) having a configured sensor (102), a deep learning accelerator (103), and a random access memory (105).

[0118] For example, a wearable electronic device (191) can be configured as a pair of glasses, contact lenses, a watch, a wristband, an armband, a ring, a headband, a head-mounted display, gloves, clothing, etc. Therefore, the casing of the wearable electronic device (191) can be configured to be attached to a person, such as a pair of glasses, contact lenses, a watch, a strap suitable for attachment to the wrist, arm, or head, a head-mounted display, gloves, adhesive pads, etc.

[0119] The sensor (102) can be used to measure parameters of the user of the wearable device (191) and / or environmental parameters of the user's surrounding environment. For example, the sensor (102) may include sensor elements for measuring temperature, biocurrent or voltage, light intensity, pressure, mechanical stress or strain, touch, acceleration, rotation, infrared radiation, vibration, etc., or any combination thereof.

[0120] Figure 6 The sensor (102) may be formed on or disposed on the substrate of the integrated circuit device (101). In some embodiments, the sensor (102) may be enclosed within an integrated circuit package. In other embodiments, the sensor (102) may be positioned on the surface of the integrated circuit package. For example, the sensor (102) may be disposed on the top surface of the integrated circuit package when the pins or contacts of the integrated circuit device (101) are disposed on the side or bottom surface of the integrated circuit device (101).

[0121] Figure 6 The integrated circuit device (101) has a controller (217) configured to control the operation of the sensor (102) via a connection (104) in the integrated circuit device (101). The controller (217) may be implemented, for example, using a microcontroller or sequencer that controls the timing of the operation of the sensor (102) and loads sensor data / measurements into random access memory (105). Alternatively, the controller (217) may be implemented using a microprocessor that runs an application stored as firmware in the random access memory (105) to coordinate the operation between the sensor (102), the random access memory (105), the deep learning accelerator (103), and / or the transceiver (106).

[0122] After the collection of sensor data is stored in random access memory (105) as input (211) to artificial neural network (201), the controller (217) can cause the deep learning accelerator (103) to execute instructions (205) and generate the output (213) of artificial neural network (201).

[0123] For example, the controller (217) can instruct the deep learning accelerator (103) to execute a start instruction (205) by writing the address of the input (211) to a predefined location in random access memory (105). When the deep learning accelerator (103) is idle, it can periodically read the address stored in the predefined location in random access memory (105). When a new and / or valid address is retrieved from the predefined location, the deep learning accelerator (103) starts the execution of the start instruction (205). Optionally, after the execution of the start instruction (205), the deep learning accelerator (103) can optionally clear, erase, or invalidate the address previously stored in the predefined location in random access memory (105).

[0124] Alternatively, the controller (217) is configured to send signals or messages to the deep learning accelerator (103) to instruct the deep learning accelerator (103) to execute instructions (205). Signals or messages can be transmitted from the controller (217) to the deep learning accelerator (103) using a direct connection to a memory cell that does not pass through the random access memory (105).

[0125] In some implementations, the controller (217) and the deep learning accelerator (103) have separate connections (109 and 119) to the random access memory (105). When the controller (217) and the deep learning accelerator (103) are not accessing the same block or address of the random access memory (105), the connections (109 and 119) can be used in parallel by the controller (217) and the deep learning accelerator (103) to access different portions of the random access memory (105) simultaneously.

[0126] In other embodiments, the control unit (113) and the controller (217) may share at least a portion of their circuitry in the deep learning accelerator (103) and use the same memory interface (117) to access the random access memory (105).

[0127] A portion of the processing unit (111) may be implemented using a neuromorphic memory (225). For example, the neuromorphic memory (225) may comprise a cross array of memristors configured to perform multiply-accumulate (MAC) operations via an analog circuit system. For example, the multiply-accumulate unit (e.g., 171 or 173) in the vector-vector unit (e.g., 161) of a deep learning accelerator (103) may be implemented using a cross array of memristors. The memristors may be connected in an array having word lines and bit lines configured to address the memristors as memory cells. Typically, a memristor is connected to one word line and one bit line in the array. The current flowing through the word lines and bit lines via a set of memristors in the cross array is summed in the bit lines, corresponding to an accumulation operation. The current corresponds to the multiplication of a voltage applied to the word line and a parameter associated with the resistance of the memristor, corresponding to a multiplication operation. The current in the bit lines may be compared to a threshold to determine whether the neuron represented by the bit line is activated under current input. The memristor array can be individually connected to bit lines and programmed to have thresholds corresponding to the activation level thresholds of neurons. A current detector can be configured for each memristor connected to the output of the bit line to determine whether the level of current in the bit line corresponds to a level exceeding the memristor's threshold. The neuromorphic memory (225) can perform multiply-accumulate (MAC) operations in a manner similar to how a memory device reads an array of memory cells, thus offering low energy cost and high computational speed.

[0128] The controller (217) operates via connection (108). Figure 6 The transceiver (106) of the integrated circuit device (101) communicates with a separate computer system (223) via a wireless connection (228). Alternatively or in combination, the wearable electronic device (191) includes a port (or cable) for a wired connection to the computer system (223).

[0129] For example, the transceiver (106) can be configured to communicate according to a wireless communication protocol for wireless personal area networks or wireless local area networks or a communication protocol for the Internet of Things (IoT). For example, the transceiver (106) can be formed on a radio frequency (RF) CMOS (complementary metal-oxide-semiconductor) integrated circuit chip.

[0130] For example, a wearable electronic device (191) can use a transceiver (106) to transmit the output (213) of an artificial neural network (201) to a computer system (223).

[0131] For example, a wearable electronic device (191) can use a transceiver (106) to send an alert to a computer system (223) based on the output (213) of an artificial neural network (201).

[0132] The transceiver (106) can be used by the wearable electronic device (191) to receive data and / or instructions, such as the matrix (207) and instructions (205) of the artificial neural network (201), from the computer system (223). The transceiver (106) can also be used by the wearable electronic device (191) to report to the computer system (223) and retrieve the output (213) of the artificial neural network (201) computed by the deep learning accelerator (103).

[0133] Optionally, the computer system (223) may communicate with the wearable electronic device (191) to request the wearable electronic device (191) to transmit an input (211) associated with the output (213). In response, the transceiver (106) transmits the input (211) to the computer system (223), which allows the computer system (223) to selectively analyze the input (211) to the artificial neural network (201).

[0134] Alternatively, the wearable electronic device (191) automatically selects the input (211) for transmission to the computer system (223) based on the output (213) of the artificial neural network (201).

[0135] In some implementations, the wearable electronic device (191) is configured to report outputs (213) to a computer system (223). For example, when the deep learning accelerator (103) completes computation of a set of outputs (213), the controller (217) generates a message reporting the availability of outputs (213). The transceiver (106) transmits the message to the computer system (223). In response, the computer system (223) may optionally accept the transmission of outputs (213) immediately, request a delay in the transmission of outputs (213) for a certain time period, or request a delay in the generation of the next set of outputs (213).

[0136] In some implementations, the control unit (113) of the deep learning accelerator (103) may include a controller (217); and the logic circuitry of the transceiver (106) may be implemented on the integrated circuit die of the deep learning accelerator (103), such as Figure 7 As shown in the image.

[0137] Figure 7 A wearable electronic device (191) configured with a deep learning accelerator (103) and random access memory (105) according to one embodiment is shown.

[0138] exist Figure 7In this configuration, a deep learning accelerator (103) is mounted on an integrated circuit die; and a random access memory (105) is mounted on one or more integrated circuit dies. The control unit (113) not only controls the execution of instructions (205) of the artificial neural network (201), but also controls the communication between the transceiver (106) and the computer system (223) as well as the operation of the sensor (102).

[0139] For example, the control unit (113) periodically retrieves measurement values ​​from the sensor (102) and stores the measurement values ​​in random access memory (105) via a high-bandwidth connection (119).

[0140] Some sensors (102) can be formed on an integrated circuit substrate or an integrated circuit die, and therefore can be enclosed in an integrated circuit package of the integrated circuit device (101) (e.g., such as...). Figure 6 (as shown in the image).

[0141] Alternatively, the sensor (102) may be a discrete component outside the integrated circuit package enclosing the deep learning accelerator (103) and random access memory (105).

[0142] For example, the sensor (102) and the integrated circuit device (101) having a deep learning accelerator (103) and random access memory (105) can be mounted on a printed circuit board configured in a wearable electronic device (191).

[0143] Figure 8 Another wearable electronic device (191) with a configured deep learning accelerator (103) and random access memory (105) is shown according to one embodiment.

[0144] Figure 8 The wearable electronic device (191) has a substrate (229) that provides connections between its components, such as a deep learning accelerator (103), random access memory (105), a sensor (102), a controller (217), and a transceiver (106).

[0145] In some embodiments, the substrate (229) includes an integrated circuit die with wires for connecting components. Some components (e.g., integrated circuit dies of random access memory (105), deep learning accelerator (103), controller (217), and / or transceiver (106)) may be connected to the integrated circuit die of the substrate (229) via through-silicon vias (TSVs). Other components may be connected to the substrate (229) via wire bonding, die attachment, or another technology.

[0146] In some embodiments, the substrate (229) further includes a printed circuit board having wires for connecting components and other components, such as power supplies (e.g., batteries), displays, light-emitting diode (LED) indicators, etc.

[0147] In some implementations, the logic circuitry of the transceiver (106) and / or controller (217) is configured on the integrated circuit die of the deep learning accelerator (103) or another integrated circuit die.

[0148] Figure 9 A method implemented in a wearable electronic device according to one embodiment is illustrated. For example, Figure 9 The method can be implemented in Figure 6 , Figure 7 or Figure 8 In the wearable electronic device (191).

[0149] At frame 301, the transceiver (106) of the wearable electronic device (191) receives the matrix (207) of the artificial neural network (201) and instructions (205) that can be executed by at least one processing unit (111) enclosed within the wearable electronic device (191) to perform calculations of the artificial neural network (201) using the matrix (207).

[0150] For example, the at least one processing unit (111) may be formed on an integrated circuit die of a field-programmable gate array (FPGA) or application-specific integrated circuit (ASIC) implementing the deep learning accelerator (103). The deep learning accelerator (103) may include the at least one processing unit (111) for matrix instruction execution, a local memory (115) for buffering matrix operands and results, a control unit (113) for loading instructions (205) from random access memory (105) for execution, and a memory interface (117) for accessing random access memory (105).

[0151] For example, an integrated circuit package is configured to enclose at least one or more integrated circuit dies of an FPGA or ASIC and a random access memory.

[0152] For example, the random access memory (105) and the deep learning accelerator (103) are formed on separate integrated circuit dies and connected by through silicon vias (TSVs).

[0153] For example, the wearable electronic device (191) may have a controller (217) to operate the transceiver (106). The controller (217) may be separate from the control unit (113) or integrated into the control unit of the deep learning accelerator (103).

[0154] At box 303, the wearable electronic device (191) stores a matrix (207) and instructions (205) in its random access memory (105).

[0155] At frame 305, a sensor (102) having one or more sensor elements, configured in the wearable electronic device (191), generates a measurement value related to the person wearing the wearable electronic device (191).

[0156] For example, a wearable electronic device (191) may have a housing suitable for wearing on a person and / or attaching to (e.g., contacting) a part of a person.

[0157] For example, the casing may include straps suitable for wearing on or attached to a person's wrist, arm, or head.

[0158] For example, the casing can take the form of a watch, a pair of glasses, a head-mounted display, gloves, a ring, or an adhesive patch.

[0159] For example, the one or more sensor elements may be configured to measure temperature, biocurrent, biovoltage, light intensity, pressure, mechanical stress, mechanical strain, touch, acceleration, rotation, infrared radiation, or vibration, or any combination thereof.

[0160] At block 307, the at least one processing unit (111) generates an output (213) from the artificial neural network (201) by executing instructions (205) based on the measurement value that serves as the input (211) to the artificial neural network (201).

[0161] For example, the output (213) may contain the identification of events, features, objects, categories, patterns, or diagnoses, or any combination thereof.

[0162] At box 309, the wearable electronic device (191) monitors the status based on the output (213) from the artificial neural network (201).

[0163] At frame 311, the wearable electronic device (191) communicates with the computer system (223) via transceiver (106) in response to status recognition. For example, the communication may be based on the communication protocols of wireless personal area networks (e.g., Bluetooth) or wireless local area networks (e.g., WiFi).

[0164] For example, the controller (217) and / or control unit (113) may be configured to generate a warning to the computer system (223) based on the output (213) of the artificial neural network (201).

[0165] Based on the output (213) of the artificial neural network (201), the wearable electronic device (191) can control the selective storage of data in the wearable electronic device (191) and / or the selective transmission of data from the wearable electronic device (191) to the computer system (223).

[0166] For example, a wearable electronic device (191) can selectively discard measurements from sensors based on the output (213) from an artificial neural network (201).

[0167] For example, a wearable electronic device (191) can transmit the output (213) of an artificial neural network (201) to a computer system (223) without transmitting the measurement (e.g., 211) from which it generates the output (213) to the computer system (223) (e.g., to reduce the amount of data transmitted).

[0168] For example, a wearable electronic device (191) may initially communicate (311) with a computer system (223) in response to the identification of a condition and then store the measurement value therein for a certain period of time. Subsequently, if a request for the measurement value is received from the computer system (223) within the time period, the wearable electronic device (191) may transmit the measurement value from the wearable electronic device (191) to the computer system (223) as a response to the request. Otherwise, the wearable electronic device (191) may delete the measurement value from the wearable electronic device (191).

[0169] This disclosure includes methods and apparatus for performing the methods described above, a data processing system for performing these methods, and a computer-readable medium containing, when executed on the data processing system, causing the system to perform these methods.

[0170] A typical data processing system may contain interconnects (such as buses and system core logic) that interconnect microprocessors and memory. Microprocessors are typically coupled to cache memory.

[0171] Interconnectors link the microprocessor and memory together and also connect them to input / output (I / O) devices via I / O controllers. I / O devices may include display devices and / or peripheral devices such as mice, keyboards, modems, network interfaces, printers, scanners, cameras, and other devices known in the art. In one embodiment, when the data processing system is a server system, some of the I / O devices (e.g., printers, scanners, mice, and / or keyboards) are optional.

[0172] Interconnectors may include one or more buses interconnected with each other via various bridges, controllers, and / or adapters. In one embodiment, the I / O controller includes a USB adapter for controlling USB (Universal Serial Bus) peripherals and / or an IEEE-1394 bus adapter for controlling IEEE-1394 peripherals.

[0173] The memory may include one or more of the following: ROM (Read-Only Memory), volatile RAM (Random Access Memory), and non-volatile memory, such as hard disk drives, flash memory, etc.

[0174] Volatile RAM is typically implemented as dynamic RAM (DRAM), which requires constant power to refresh or retain data in memory. Non-volatile memory is typically magnetic hard disk drives, magnetic optical disk drives, optical disk drives (e.g., DVD RAM), or other types of memory systems that retain data even after power is removed from the system. Non-volatile memory can also be random access memory.

[0175] Non-volatile memory can be a local device directly coupled to the rest of the data processing system. Alternatively, it can be a non-volatile memory located off-system, such as a network storage device coupled to the data processing system via a network interface, such as a modem or Ethernet interface.

[0176] In this disclosure, some functions and operations are described as being executed or caused by software code for the sake of simplicity. However, such expressions are also used to specify that the functions are generated by a processor, such as a microprocessor, executing code / instructions.

[0177] Alternatively or in combination, the functions and operations described herein can be implemented using dedicated circuit systems, with or without software instructions, such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs). Embodiments can be implemented using hard-wired circuit systems, either without or in combination with software instructions. Therefore, the techniques described are neither limited to any particular combination of hardware circuit systems and software, nor to any particular source of instructions executed by a data processing system.

[0178] While one embodiment may be implemented in a full-featured computer and computer system, various embodiments can be distributed as a variety of computing products and are applicable regardless of the specific type of machine or computer-readable medium actually used to implement the distribution.

[0179] At least some of the disclosed aspects may be embodied, at least in part, in software. That is, the technology may be implemented in a computer system or other data processing system in response to its processor (e.g., a microprocessor) executing a sequence of instructions contained in memory (e.g., ROM, volatile RAM, non-volatile memory, cache, or remote storage device).

[0180] The routines executed to implement the embodiments may be implemented as part of an operating system or a particular application, component, program, object, module, or a sequence of instructions referred to as a "computer program." A computer program typically contains one or more sets of instructions at various times in various memories and storage devices in a computer, and when read and executed by one or more processors in the computer, the sets of instructions cause the computer to perform the necessary operations to perform elements relating to various aspects.

[0181] Machine-readable media can be used to store software and data that, when executed by a data processing system, causes the system to perform various methods. Executable software and data can be stored in various locations, including, for example, ROM, volatile RAM, non-volatile memory, and / or cache. Parts of this software and / or data can be stored in any of these storage devices. Additionally, data and instructions can be obtained from a centralized server or a peer-to-peer network. Different portions of the data and instructions can be obtained at different times and in different communication sessions or within the same communication session from different centralized servers and / or peer-to-peer networks. Data and instructions can be obtained completely before application execution. Alternatively, portions of data and instructions can be obtained dynamically and as needed for execution. Therefore, it is not required that data and instructions be completely present on the machine-readable medium at a specific moment.

[0182] Examples of computer-readable media include (but are not limited to) media of non-transitory recordable and non-recordable types, such as volatile and non-volatile memory devices, read-only memory (ROM), random access memory (RAM), flash memory devices, floppy disks and other removable disks, magnetic disk storage media, optical storage media (e.g., optical disc read-only memory (CD-ROM), digital versatile optical disc (DVD), etc.), and so on. Computer-readable media can store instructions.

[0183] Instructions can also be embodied in digital and analog communication links that use electrical, optical, acoustic, or other forms of propagated signals, such as carrier waves, infrared signals, digital signals, etc. However, propagated signals such as carrier waves, infrared signals, digital signals, etc., are not tangible machine-readable media and are not configured to store instructions.

[0184] Generally, a machine-readable medium contains any mechanism that provides (i.e., stores and / or transmits) information in a form accessible by a machine (e.g., a computer, a network device, a personal digital assistant, a manufacturing tool, any device having one or more processor sets, etc.).

[0185] In various embodiments, hardwired circuitry systems can be used in combination with software instructions to implement the technology. Therefore, the technology is neither limited to any particular combination of hardware circuitry systems and software, nor to any particular source of instructions executed by a data processing system.

[0186] The above descriptions and figures are illustrative and should not be construed as limiting. Many specific details have been described to provide a thorough understanding. However, in some cases, well-known or conventional details have not been described to avoid obscuring the description. References to one or more embodiments in this disclosure do not necessarily refer to the same embodiment; and such references imply at least one.

[0187] In the foregoing description, this disclosure has been described with reference to specific exemplary embodiments thereof. It will be apparent that various modifications may be made thereto without departing from the broader spirit and scope set forth in the appended claims. Therefore, the description and drawings should be viewed in an illustrative rather than restrictive sense.

Claims

1. A wearable device comprising: The outer shell, which is suitable for wearing on a person; At least one processing unit, disposed within the housing and configured to execute instructions having matrix operands; A random access memory disposed within the housing and configured to store first data representing the weights of an artificial neural network and second data representing instructions executable by the at least one processing unit to perform matrix calculations of the artificial neural network using the first data representing the weights of the artificial neural network; A transceiver, disposed within the housing and configured to communicate with a computer system separate from the wearable device; A sensor having one or more sensor elements configured to generate third data representing measurements related to the person; as well as A controller, disposed within the housing and coupled to the transceiver, the sensor, and the random access memory, wherein the controller is configured to write third data representing the measurement value into the random access memory as input to the artificial neural network; The at least one processing unit is further configured to execute the instructions to generate the output of the artificial neural network based at least in part on the first data and the third data stored in the random access memory; and The controller is further configured to monitor conditions associated with the person based on the output of the artificial neural network, and to control the transceiver to communicate with the computer system in response to recognizing the conditions from the output of the artificial neural network.

2. The wearable device of claim 1, wherein the housing includes a strap adapted to be worn on the wrist, arm, or head of the person.

3. The wearable device of claim 1, wherein the housing comprises a watch, a pair of glasses, a head-mounted display, a glove, a ring, or an adhesive patch.

4. The wearable device of claim 1, wherein the one or more sensor elements are configured to measure temperature, biocurrent, biovoltage, light intensity, pressure, mechanical stress, mechanical strain, touch, acceleration, rotation, infrared radiation, or vibration, or any combination thereof.

5. The wearable device of claim 4, wherein the output comprises the identification of events, features, objects, categories, patterns, or diagnoses, or any combination thereof.

6. The wearable device of claim 5, wherein the controller is configured to selectively store data based on the output of the artificial neural network for transmission to the computer system.

7. The wearable device of claim 5, wherein the controller is configured to generate a warning to the computer system based on the output of the artificial neural network.

8. The wearable device according to claim 6, further comprising: An integrated circuit die of a Field Programmable Gate Array (FPGA) or Application-Specific Integrated Circuit (ASIC) implementing a deep learning accelerator, the deep learning accelerator including the at least one processing unit and a control unit configured to load the instructions from the random access memory for execution.

9. The wearable device of claim 8, wherein the control unit comprises the controller.

10. The wearable device of claim 8, further comprising: An integrated circuit package configured to at least enclose the integrated circuit die of the FPGA or ASIC and one or more integrated circuit dies of the random access memory.

11. The wearable device of claim 10, wherein the at least one processing unit comprises a matrix-matrix unit configured to operate on two matrix operands of an instruction; The matrix-matrix unit comprises multiple matrix-vector units configured to operate in parallel; Each of the plurality of matrix-vector units contains a plurality of vector-vector units configured to operate in parallel; Each of the plurality of vector-vector units comprises a plurality of multiply-accumulate units configured to operate in parallel; and Each of the plurality of multiply-accumulate units includes a neuromorphic memory configured to perform multiply-accumulate operations via an analog circuit system.

12. The wearable device of claim 11, wherein the random access memory and the deep learning accelerator are formed on separate integrated circuit dies and connected by through-silicon vias (TSVs).

13. The wearable device of claim 12, wherein the transceiver is configured to communicate according to a wireless personal area network or wireless local area network communication protocol.

14. A method implemented in a wearable electronic device, comprising: The transceiver of the wearable electronic device receives first data and second data representing the weights of the artificial neural network, the second data representing instructions having matrix operands and executable by at least one processing unit enclosed within the wearable electronic device to perform matrix calculations of the artificial neural network using the first data representing the weights of the artificial neural network; The first data representing the weight and the second data representing the instruction are stored in the random access memory of the wearable electronic device; The wearable electronic device generates third data representing measurements related to the person wearing the wearable electronic device, which is generated by sensors having one or more sensor elements. The instructions represented by the second data stored in the random access memory are executed by the at least one processing unit, generating an output from the artificial neural network based at least in part on the first data and the third data stored in the random access memory; The wearable electronic device monitors the status based on the output from the artificial neural network; as well as The transceiver communicates with the computer system in response to the recognition of the condition.

15. The method of claim 14, further comprising: The measurements from the sensor are discarded based on the output from the artificial neural network.

16. The method of claim 14, further comprising: The output from the artificial neural network is transmitted from the wearable electronic device to the computer system, instead of transmitting the measurement value from which the output is generated to the computer system.

17. The method of claim 14, further comprising: After communicating with the computer system, the third data representing the measured value is stored in the wearable electronic device for a certain period of time; During the time period, a request for third data representing the measured value is received from the computer system; and In response to the request, the third data representing the measurement value is transmitted from the wearable electronic device to the computer system.

18. The method of claim 14, further comprising: After communicating with the computer system, the third data representing the measured value is stored in the wearable electronic device for a predetermined time period. as well as In response to determining that no request for the third data representing the measurement value has been received from the computer system within the predetermined time period, the third data representing the measurement value is deleted from the wearable electronic device.

19. A wearable device comprising: An outer casing, adapted to be attached to a part of a person; A sensor having one or more sensor elements, disposed on or in the housing and configured to generate measurements of the person; Random access memory configured to store models of artificial neural networks; Field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs) have the following characteristics: A memory interface for accessing the random access memory; Control unit; as well as At least one processing unit is configured to execute instructions having matrix operands to perform computations of the artificial neural network according to the model; and A transceiver configured to communicate with a computer system using a wireless communication connection; The sensor is configured to store the measured values ​​in the random access memory as input to the artificial neural network; The FPGA or ASIC is configured to perform the computations of the artificial neural network according to the model, to convert the input into an output from the artificial neural network; and The wearable device is configured to monitor the output from the artificial neural network to control the storage of the measurements in the wearable device and the transmission of data to the computer system.

20. The wearable device of claim 19, wherein the random access memory includes non-volatile memory configured to store the model of the artificial neural network; the model includes instructions executable by the FPGA or ASIC; and the at least one processing unit includes a matrix-matrix unit configured to operate on two matrix operands of the instructions.

Images