Smart digital camera with deep learning accelerator and random access memory

Abstract

Systems, apparatuses, and methods related to deep learning accelerators and memory are described. For example, a digital camera can be configured to execute instructions having matrix operands and configured with: a housing; a lens; an image sensor positioned behind the lens to produce image data of a field of view of the digital camera; a random access memory to store instructions executable by the deep learning accelerator and to store matrices of an artificial neural network; a transceiver; and a controller configured to produce a description of an item or event in the field of view captured in the image data based on an output of the artificial neural network receiving the image data as input and to transmit it to a separate computer using the transceiver. The separate computer can selectively request a portion of image data from the digital camera based on processing of the description.

Description

[0001] Related applications

[0002] This application claims U.S. Patent Application No. 16 / 906,224, filed June 19, 2020, entitled “Intelligent Digital Camera Having Deep Learning Accelerator and Random Access Memory,” the entire disclosure of which is hereby incorporated by reference. Technical Field

[0003] At least some of the embodiments disclosed herein relate to digital cameras, and more specifically, but not limited to, intelligent digital cameras powered by an integrated accelerator of an artificial neural network (ANN), such as an ANN configured through machine learning and / or deep learning. Background Technology

[0004] Artificial neural networks (ANNs) use a network of neurons to process the inputs to the network and produce the outputs from the network.

[0005] For example, each neuron in a network receives a set of inputs. Some inputs to a neuron may be the outputs of other neurons in the network; and some inputs to a neuron may be inputs provided to the neural network. The input / output relationships between neurons in the network represent the neuronal connectivity in the network.

[0006] For example, each neuron can have a bias, an activation function, and a set of synaptic weights for its input. The activation function can be a step function, a linear function, a log-sigmoid function, etc. Different neurons in the network can have different activation functions.

[0007] For example, each neuron can generate a weighted sum of its input and its bias, and then produce an output as a function of the weighted sum, which is calculated using the neuron's activation function.

[0008] The relationship between the inputs and outputs of an ANN is generally defined by the ANN model, which contains data representing the connectivity of neurons in the network, as well as the biases, activation functions, and synaptic weights of each neuron. Based on a given ANN model, a computing device can be configured to compute the network's outputs given a set of inputs to the network.

[0009] For example, the input to the ANN network can be generated based on the camera input; and the output from the ANN network can be the recognition of items such as events or objects.

[0010] Generally, supervised methods can be used to train ANNs, where the parameters in the ANN are tuned to minimize or reduce the error between a known output associated with or produced by a given input and a computed output generated by applying the input to the ANN. Examples of supervised learning / training methods include reinforcement learning and learning with error correction.

[0011] Alternatively or in combination, unsupervised methods can be used to train an ANN, where the exact output produced by a given set of inputs is unknown until training is complete. An ANN can be trained to classify items into multiple categories or data points into clusters.

[0012] Multiple training algorithms can be used for complex machine learning / training paradigms.

[0013] Deep learning uses multiple layers of machine learning to progressively extract features from input data. For example, lower layers can be configured to identify edges in an image; and higher layers can be configured to identify captured items in the image, such as faces, objects, events, etc., based on the edges detected by the lower layers. Deep learning can be implemented via artificial neural networks (ANNs), such as deep neural networks, deep belief networks, recurrent neural networks, and / or convolutional neural networks.

[0014] Deep learning has been applied in many fields, such as computer vision, speech / audio recognition, natural language processing, machine translation, bioinformatics, drug design, medical image processing, and games. Attached Figure Description

[0015] The embodiments are shown in the accompanying drawings by way of example and not limitation, and the same reference numerals indicate similar elements in the drawings.

[0016] Figure 1 An integrated circuit device having a configured deep learning accelerator and random access memory is shown according to one embodiment.

[0017] Figure 2 A processing unit configured to perform matrix-matrix operations according to one embodiment is shown.

[0018] Figure 3 A processing unit configured to perform matrix-vector operations according to one embodiment is shown.

[0019] Figure 4 A processing unit configured to perform vector-to-vector operations according to one embodiment is shown.

[0020] Figure 5A deep learning accelerator and random access memory, configured according to one embodiment, are shown to be autonomously applied to the input of a trained artificial neural network.

[0021] Figures 6 to 8 A digital camera with a configured deep learning accelerator and random access memory is shown according to some embodiments.

[0022] Figure 9 A method implemented in a digital camera according to one embodiment is shown. Detailed Implementation

[0023] At least some embodiments disclosed herein provide a digital camera having a general-purpose integrated circuit configured to perform computations of an artificial neural network (ANN) with less power consumption and computation time. The integrated circuit includes a deep learning accelerator (DLA) and random access memory. Image data generated in the digital camera is largely consumed by the artificial neural network implemented in the digital camera to produce intelligent output smaller than the image data. Therefore, the requirements for the capacity to store the image data generated by the digital camera and / or the communication bandwidth requirements between the digital camera and the computing system using the digital camera can be reduced. For example, the artificial neural network (ANN) can be trained to output identified events, patterns, features, or classifications of interest in a particular application. The output can be stored in the random access memory of the digital camera, used to generate warnings transmitted from the digital camera to a standalone computing system, and / or used to selectively retain and / or transmit image data generated in the digital camera.

[0024] A deep learning accelerator (DLA) contains a set of general-purpose programmable hardware computation logic that is specialized and / or optimized to perform parallel vector and / or matrix computations, including but not limited to vector and / or matrix multiplication and accumulation.

[0025] In addition, a deep learning accelerator (DLA) may contain one or more arithmetic logic units (ALUs) to perform arithmetic and bitwise operations on integer binary numbers.

[0026] Deep learning accelerators (DLAs) can be programmed via a set of instructions to perform computations on artificial neural networks (ANNs).

[0027] The granularity of a Deep Learning Accelerator (DLA) for vector and matrix operations corresponds to the largest unit of a vector / matrix that can be operated on during the execution of an instruction by the DLA. During instructions performing predefined operations on vector / matrix operands, the DLA can operate on the elements of the vector / matrix operands in parallel to reduce the execution time and / or energy consumption associated with memory / data access. Operations on vector / matrix operands at the granularity of the DLA can be used as building blocks to perform computations on larger vectors / matrices.

[0028] The implementation of a typical / practical artificial neural network (ANN) involves vector / matrix operands with a size larger than the operational granularity of a deep learning accelerator (DLA). To implement such an ANN using a DLA, computations involving large-sized vector / matrix operands can be decomposed into computations of vector / matrix operands at the granularity of the DLA. A DLA can be programmed with instructions to perform computations involving large vector / matrix operands. For example, the atomic computational capabilities of a DLA in manipulating vectors and matrices at the granularity of the DLA in response to instructions can be programmed to perform computations within an artificial neural network (ANN).

[0029] In some implementations, a deep learning accelerator (DLA) lacks some of the logical computational capabilities of a typical central processing unit (CPU). However, a DLA can be configured with sufficient logic units to process the input data provided to an artificial neural network (ANN) and produce the output of the ANN according to a set of instructions generated for the DLA. Therefore, a DLA can perform ANN computations with minimal or no assistance from a CPU or another processor. Optionally, a conventional general-purpose processor can also be configured as part of the DLA to perform operations that cannot be efficiently implemented using the DLA's vector / matrix processing units, and / or operations that cannot be performed by the DLA's vector / matrix processing units.

[0030] Typical artificial neural networks (ANNs) can be described / specified in a standard format (e.g., Open Neural Network Exchange (ONNX)). A compiler can be used to translate the description of an ANN into a set of instructions for a deep learning accelerator (DLA) to perform computations on the ANN. The compiler can optimize this set of instructions to improve the performance of the DLA when implementing the ANN.

[0031] Deep learning accelerators (DLAs) may have local memory, such as registers, buffers, and / or caches, to store vector / matrix operands and the results of vector / matrix operations. Intermediate results in registers can be pipelined / shifted within the DLA as operands for subsequent vector / matrix operations, reducing time and energy consumption in accessing memory / data and thus accelerating the typical pattern of vector / matrix operations when implementing typical artificial neural networks (ANNs). The capacity of registers, buffers, and / or caches in a DLA is often insufficient to hold the entire dataset used for computations to implement a typical artificial neural network (ANN). Therefore, random access memory coupled to the DLA is configured to provide improved data storage capabilities for implementing typical artificial neural networks (ANNs). For example, the DLA loads data and instructions from random access memory and stores the results back into random access memory.

[0032] The communication bandwidth between the Deep Learning Accelerator (DLA) and Random Access Memory (RAM) is configured to optimize or maximize the utilization of the DLA's computational power. For example, high communication bandwidth between the DLA and RAM can be provided to allow vector / matrix operands to be loaded from RAM into the DLA and stored back into RAM within a time period approximately equal to the DLA's processing time, enabling computation on the vector / matrix operands to be performed. The granularity of the DLA can be configured to increase the ratio between the amount of computation performed by the DLA and the size of the vector / matrix operands, thereby reducing data access traffic between the DLA and RAM, which lowers the bandwidth requirements for communication between the DLA and RAM. Therefore, bottlenecks in data / memory access can be reduced or eliminated.

[0033] Digital cameras can generate large amounts of data during continuous operation. Storing and / or uploading the image data generated by digital cameras to servers, cloud platforms, or central processing units (CPUs) for processing requires large data storage capacity and / or high communication bandwidth.

[0034] In at least some embodiments, the digital camera is configured to have a deep learning accelerator (DLA) and random access memory having an artificial neural network (ANN) for intelligent local processing of image data captured by the digital camera.

[0035] For example, in order to reduce data storage requirements and / or data communication traffic, artificial neural networks (ANNs) implemented in digital cameras can transform image data into inference results of interest to specific applications, and thus eliminate the need to store and / or transmit most or all of the image data captured by the image sensor of the digital camera.

[0036] For example, an artificial neural network (ANN) can be trained to identify objects and / or events in images that may be of interest in an application, and to convert the image into a description or recognition of the objects and events appearing in an image captured by a digital camera. The description or recognition of objects and events can serve as a high-level overview of the content in the image, which can then be used as input to an application running on a computer system that uses a digital camera as an input device. Such a computer system can be a mobile computer, smartphone, personal media player, personal computer, server computer, or server computer cluster. Compared to graphically representing static images or video images of corresponding objects and events, this high-level overview of the content in an image identified using an artificial neural network uses a significantly smaller amount of data storage capacity and / or data transmission bandwidth.

[0037] Optionally, images of unidentified objects and / or events in abnormal / exceptional situations or scenes can be transferred to a separate computer system for further processing. For example, a human operator can inspect the images to identify objects and events captured in the images of abnormal / exceptional situations or scenes; and the results can be used to further train an artificial neural network (ANN) (e.g., using supervised machine learning techniques) to improve its ability to identify objects and events. The further trained ANN can then be loaded into the digital camera to further reduce the amount of data that needs to be stored and / or processed outside the digital camera.

[0038] Optionally, the digital camera can store an image stream converted into an overview or description for a predetermined time period. The image stream can be annotated with an overview or description and identification of portions of the images showing objects and events. Within the predetermined time period, an application receiving the overview or description as input can select objects or events of interest and request the digital camera to transmit relevant images of said objects or events.

[0039] In some applications, digital cameras are configured to monitor image streams for objects and / or events that meet predetermined selection criteria. When such objects and / or events are detected in the output of an artificial neural network (ANN), the digital camera selects a representative portion from the image stream for storage and / or for uploading to a separate computer. Through analysis performed by the ANN, information in the image stream can be compressed into an overview that includes the identification of the object of interest, a representative image of the object, and a description of the activity of the object in the scene captured in the image stream. Information about identified objects that are not of interest can be reduced and / or discarded. Such an overview of the image stream can achieve compression ratios superior to conventional lossy or lossless image / video compression techniques; and in many applications of digital cameras, such an overview can be sufficiently useful and / or more useful than the raw image stream generated by the digital camera's image sensor.

[0040] For example, instead of transmitting the image stream to a separate computer system for processing, the digital camera can provide a description of the content of the image stream to the computer system for processing. Representative images of the identified images can be transmitted along with the description, or provided when the computer system sends a request for such images. Furthermore, the computer system can optionally request the digital camera to transmit frames or segments of the image stream containing objects or events of interest.

[0041] The description of the content in an image stream can take the form of identification of objects, events, features, categories, positions in the image, sizes in the image, etc.

[0042] For example, digital cameras are equipped with image sensors to monitor the user's health. Integrated circuit devices with deep learning accelerators and random access memory are incorporated into the digital camera to process sensor data locally. Images of users determined not to be associated with health problems can be automatically discarded, reducing the need to store large amounts of raw image data and protecting user privacy.

[0043] Digital cameras can be customized for specific applications of intelligent monitoring by storing models of artificial neural networks (ANNs) that can be executed by a deep learning accelerator (DLA). For example, a compiler can be used to transform the model based on a description of the ANN; and the model contains the weight / kernel matrix of the ANN and instructions with matrix operands that can be executed by a deep learning accelerator (DLA) to perform computations on the ANN based on the weight / kernel matrix.

[0044] For example, digital cameras can be configured in medical facilities to monitor patients' conditions. Different patients may have different health problems, which require monitoring to detect the onset of different symptoms. Therefore, digital cameras can be customized for the patients being monitored by installing models of artificial neural networks (ANNs) trained to detect patient-related symptoms.

[0045] After customizing a digital camera by storing the model in random access memory to perform computations of an artificial neural network (ANN), the raw image data generated by the image sensor in the digital camera is provided as input to the artificial neural network (ANN); and the output of the artificial neural network (ANN) can be used to generate warnings, selectively retain and / or report sensor data and / or provide it as the main output of the digital camera.

[0046] For example, a digital camera may include a wireless transceiver (e.g., a communication device for a wireless personal area network, such as a Bluetooth transceiver, or a communication device for a wireless local area network, such as a WiFi transceiver). The output of an artificial neural network (ANN) can be reported to a standalone computer system, such as a smartphone, personal media player, mobile computer, personal computer, Internet of Things (IoT) hub, and / or server computer, via a wireless connection.

[0047] Alternatively, the digital camera may have a port for a wired connection to a standalone computer system to report the output of an artificial neural network (ANN) over a period of time or download the output stored in the digital camera.

[0048] For example, an artificial neural network (ANN) can be trained to classify whether an image is of interest to a specific application. The output of the ANN can be used to selectively store image data for subsequent analysis and / or downloading.

[0049] For example, a digital camera can provide intelligent output generated locally by an artificial neural network (ANN) to a computer system without requiring assistance from the computer system. For example, a digital camera can be used to monitor health-related events and generate alerts when such events are detected. For example, a digital camera can be used to monitor indications related to the diagnosis of health problems and record the occurrence and / or associated data of such indications for further analysis. For example, a digital camera can be used to detect a user's fall and generate a call for help when a fall is detected. For example, a digital camera can be used to detect the appearance of objects around the user and provide object identification for further processing. For example, a digital camera can be used to detect a user's movements and provide the identification of said movements to a standalone computer (e.g., a smartphone, game console, personal media player, personal computer, set-top box) to control operations on the standalone computer.

[0050] The random access memory (RAM) in a digital camera may include a portion configured to store inputs to an artificial neural network (ANN), and another portion configured to store the outputs from the ANN. Data generated by the sensors in the digital camera can be stored cyclically in the input portion of the RAM. Therefore, the latest cycle of raw sensor data used for the operation of the digital camera can be found in the input portion of the RAM. A deep learning accelerator (DLA) can transform the data in the input portion into inference results stored in the output portion of the RAM in real time.

[0051] For example, an image data stream to an artificial neural network (ANN) can be configured as a series of input datasets. Each input dataset is a set of inputs used to feed into the ANN during a time slot. While the deep learning accelerator (DLA) is computing the output based on the current set of inputs, the controller can store the next set of inputs in random access memory; and the controller can simultaneously retrieve the output generated for the previous set of inputs from the random access memory.

[0052] Therefore, image data preparation and processing tasks for artificial neural networks (ANNs) can be performed locally within the digital camera, reducing the amount of data transmitted to a separate computer system. This arrangement reduces power consumption associated with transmitting large amounts of sensor data from the digital camera and improves user privacy for digital camera users.

[0053] Furthermore, neuromorphic memories can be used to perform matrix / vector multiplication and summation computations to reduce the power consumption of deep learning accelerators (DLAs).

[0054] For example, a neuromorphic memory can be implemented using a lattice array of memristors configured to perform multiply-accumulate (MAC) operations via analog circuitry. In the lattice array, currents passing through a set of memristors across word lines to bit lines are added in the bit lines, corresponding to an accumulation operation. The current corresponds to the multiplication of the voltage applied to the word line and a parameter associated with the resistance of the memristor, corresponding to a multiplication operation. The current in the bit lines can be compared to a threshold to determine whether the neuron represented by the bit line is activated under the current input. The memristor array can be individually connected to the bit lines and programmed to have thresholds corresponding to the activation level threshold of the neuron. Current detectors can be configured for each memristor connected to the output of the bit lines to determine whether the current level in the bit line corresponds to a level exceeding the memristor's threshold.

[0055] Figure 1 An integrated circuit device (101) with a configured deep learning accelerator (103) and random access memory (105) is shown according to one embodiment.

[0056] exist Figure 1 The deep learning accelerator (103) includes a processing unit (111), a control unit (113), and local memory (115). When vector and matrix operands are in local memory (115), the control unit (113) can use the processing unit (111) to perform vector and matrix operations according to instructions. In addition, the control unit (113) can load instructions and operands from random access memory (105) via a memory interface (117) and a high-speed / bandwidth connection (119).

[0057] The integrated circuit device (101) is configured to be enclosed within an integrated circuit package using the pins or contacts of a memory controller interface (107).

[0058] The memory controller interface (107) is configured to support standard memory access protocols, enabling the integrated circuit device (101) to behave as a typical memory controller in the same way as a conventional random access memory device without a deep learning accelerator (DLA) (103). For example, a memory controller external to the integrated circuit device (101) can access the random access memory (105) in the integrated circuit device (101) via the memory controller interface (107) using standard memory access protocols.

[0059] The integrated circuit device (101) is configured with a high-bandwidth connection (119) between a random access memory (105) enclosed within the integrated circuit device (101) and a deep learning accelerator (DLA) (103). The bandwidth of the connection (119) is higher than the bandwidth of the connection (109) between the random access memory (105) and the memory controller interface (107).

[0060] In one embodiment, both the memory controller interface (107) and the memory interface (117) are configured to access the random access memory (105) via the same set of buses or wires. Therefore, the bandwidth used to access the random access memory (105) is shared between the memory interface (117) and the memory controller interface (107). Alternatively, the memory controller interface (107) and the memory interface (117) are configured to access the random access memory (105) via a separate set of buses or wires. Optionally, the random access memory (105) may comprise multiple segments that can be accessed simultaneously via a connection (119). For example, while the memory interface (117) is accessing a segment of the random access memory (105), the memory controller interface (107) may simultaneously access another segment of the random access memory (105). For example, different segments can be configured on different planes / rows of different integrated circuit dies and / or memory cells; and different segments can be accessed in parallel to increase the throughput when accessing random access memory (105). For example, the memory controller interface (107) is configured to access a data cell of a predetermined size at a time; and the memory interface (117) is configured to access multiple data cells at a time, each data cell being the same predetermined size.

[0061] In one embodiment, the random access memory (105) and the integrated circuit device (101) are disposed on different integrated circuit dies disposed within the same integrated circuit package. Furthermore, the random access memory (105) may be disposed on one or more integrated circuit dies that allow simultaneous parallel access to multiple data elements.

[0062] In some implementations, the number of data elements of a vector or matrix that can be accessed in parallel via connection (119) corresponds to the granularity of the deep learning accelerator (DLA) operating on the vector or matrix. For example, when the processing unit (111) can operate on several vector / matrix elements in parallel, connection (119) is configured to load or store the same number or a multiple thereof of elements in parallel via connection (119).

[0063] Optionally, the data access speed of the connection (119) can be configured based on the processing speed of the deep learning accelerator (DLA) (103). For example, after a certain amount of data and instructions have been loaded into the local memory (115), the control unit (113) can use the processing unit (111) to execute instructions to manipulate the data to produce output. During the processing time used to produce the output, the access bandwidth of the connection (119) allows the same amount of data and instructions to be loaded into the local memory (115) for the next operation, and the same amount of output to be stored back into the random access memory (105). For example, while the control unit (113) uses a portion of the local memory (115) to process data and produce output, the memory interface (117) can unload the output of the previous operation from another portion of the local memory (115) into the random access memory (105), and load operand data and instructions into the other portion of the local memory. Therefore, the utilization and performance of the deep learning accelerator (DLA) are not limited or reduced by the bandwidth of the connection (119).

[0064] Random access memory (105) can be used to store model data of the artificial neural network (ANN) and to buffer input data of the artificial neural network (ANN). The model data does not change frequently. The model data may contain output generated by a compiler for a deep learning accelerator (DLA) to implement the artificial neural network (ANN). The model data typically contains matrices used in the description of the artificial neural network (ANN) and instructions generated for the deep learning accelerator (DLA) (103) to perform vector / matrix operations on the artificial neural network (ANN) at a granularity based on the deep learning accelerator (DLA) (103). The instructions operate not only on the vector / matrix operations of the artificial neural network (ANN) but also on the input data of the artificial neural network (ANN).

[0065] In one embodiment, when input data is loaded or updated in random access memory (105), the control unit (113) of the deep learning accelerator (DLA) (103) can automatically execute the instructions of the artificial neural network (ANN) to produce the output of the artificial neural network (ANN). The output is stored in a predefined area in random access memory (105). The deep learning accelerator (DLA) (103) can execute instructions without the assistance of a central processing unit (CPU). Therefore, communication for coordination between the deep learning accelerator (DLA) (103) and a processor (e.g., a central processing unit (CPU)) outside the integrated circuit device (101) can be reduced or eliminated.

[0066] Optionally, the logic circuitry of the deep learning accelerator (DLA) (103) can be implemented via complementary metal-oxide-semiconductor (CMOS). For example, under-array CMOS (CUA) technology of the memory cells of random access memory (105) can be used to implement the logic circuitry of the deep learning accelerator (DLA) (103), including a processing unit (111) and a control unit (113). Alternatively, CMOS technology in the memory cell array of random access memory (105) can be used to implement the logic circuitry of the deep learning accelerator (DLA) (103).

[0067] In some implementations, the deep learning accelerator (DLA) (103) and random access memory (105) may be implemented on a separate integrated circuit die and connected using through-silicon vias (TSVs) to increase the data bandwidth between the deep learning accelerator (DLA) (103) and the random access memory (105). For example, the deep learning accelerator (DLA) (103) may be formed on an integrated circuit die of a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC).

[0068] Alternatively, the deep learning accelerator (DLA) (103) and random access memory (105) can be configured in a separate integrated circuit package and connected via multiple point-to-point connections on a printed circuit board (PCB) for parallel communication, thereby increasing data transmission bandwidth.

[0069] Random access memory (105) can be volatile memory, non-volatile memory, or a combination of volatile and non-volatile memory. Examples of non-volatile memory include flash memory, memory cells formed based on NAND logic gates, NOR logic gates, phase-change memory (PCM), magnetic RAM (MRAM), resistive random access memory, crosspoint memory, and memory devices. Crosspoint memory devices can use transistorless memory elements, each of which has memory cells and selectors stacked together in a column. The column of memory elements is connected via two layers of wires extending in a vertical direction, wherein the wires of one layer extending in one direction are above the column of memory elements, and the wires of the other layer extending in the other direction are below the column of memory elements. Each memory element can be selected individually at the intersection of a wire on each of the two layers. Crosspoint memory devices are fast and non-volatile and can be used as a general-purpose memory pool for processing and storage. Other examples of non-volatile memory include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), and electronically erasable programmable read-only memory (EEPROM). Examples of volatile memory include dynamic random access memory (DRAM) and static random access memory (SRAM).

[0070] For example, the non-volatile memory may be configured to implement at least a portion of the random access memory (105). The non-volatile memory in the random access memory (105) can be used to store model data of an artificial neural network (ANN). Therefore, after the integrated circuit device (101) is powered off and restarted, it is not necessary to reload the model data of the artificial neural network (ANN) into the integrated circuit device (101). Furthermore, the non-volatile memory may be programmable / rewritable. Therefore, the model data of the artificial neural network (ANN) in the integrated circuit device (101) can be updated or replaced to implement an updated artificial neural network (ANN) or another artificial neural network (ANN).

[0071] The processing unit (111) of the deep learning accelerator (DLA) (103) may include vector-vector units, matrix-vector units, and / or matrix-matrix units. The following is in conjunction with... Figures 2 to 4 Discuss instances of units configured to perform vector-vector operations, matrix-vector operations, and matrix-matrix operations.

[0072] Figure 2 A processing unit (121) configured to perform matrix-matrix operations according to one embodiment is shown. For example, Figure 2 The matrix-matrix unit (121) can be used as Figure 1One of the processing units (111) of the Deep Learning Accelerator (DLA) (103).

[0073] exist Figure 2 In this configuration, the matrix-matrix unit (121) comprises multiple kernel buffers (131 to 133) and multiple map groups (151 to 153). Each of the map groups (151 to 153) stores a vector of a matrix operand, which has multiple vectors stored in the map groups (151 to 153); and each of the kernel buffers (131 to 133) stores a vector of another matrix operand, which has multiple vectors stored in the kernel buffers (131 to 133). The matrix-matrix unit (121) is configured to perform multiplication and accumulation operations on the elements of the two matrix operands using multiple matrix-vector units (141 to 143) operating in parallel.

[0074] A crossbar switch (123) connects the mapping groups (151 to 153) to the matrix-vector units (141 to 143). Identical matrix operands stored in the mapping groups (151 to 153) are provided to each of the matrix-vector units (141 to 143) via the crossbar switch (123); and the matrix-vector units (141 to 143) receive data elements from the mapping groups (151 to 153) in parallel. Each of the kernel buffers (131 to 133) is connected to the corresponding matrix-vector unit (141 to 143) and provides a vector operand to the corresponding matrix-vector unit. The matrix-vector units (141 to 143) simultaneously operate to compute the operation of multiplying the identical matrix operands stored in the mapping groups (151 to 153) by the corresponding vector stored in the kernel buffers (131 to 133). For example, the matrix-vector unit (141) performs a multiplication operation on matrix operands stored in the map array (151 to 153) and vector operands stored in the kernel buffer (131), while the matrix-vector unit (143) simultaneously performs a multiplication operation on matrix operands stored in the map array (151 to 153) and vector operands stored in the kernel buffer (133).

[0075] exist Figure 2 Each of the matrix-vector units (141 to 143) in the matrix can be like... Figure 3 Implemented as shown.

[0076] Figure 3 A processing unit (141) configured to perform matrix-vector operations according to one embodiment is shown. For example, Figure 3 The matrix-vector unit (141) can be used as Figure 2Any one of the matrix-vector units in the matrix-matrix unit (121).

[0077] exist Figure 3 In the diagram, each of the mapping groups (151 to 153) is arranged in a manner similar to... Figure 2 A vector of matrix operands is stored in a mapping group (151 to 153), wherein the matrix operands have multiple vectors stored in the mapping group (151 to 153). Figure 3 The crossbar switch (123) in the middle provides vectors from the mapping group (151) to the vector-vector units (161 to 163). The same vectors stored in the kernel buffer (131) are provided to the vector-vector units (161 to 163).

[0078] Vector-vector units (161 to 163) operate simultaneously to compute the multiplication of corresponding vector operands stored in map groups (151 to 153) by the same vector operand stored in kernel buffer (131). For example, vector-vector unit (161) performs a multiplication operation on the vector operands stored in map group (151) and the vector operands stored in kernel buffer (131), while vector-vector unit (163) simultaneously performs a multiplication operation on the vector operands stored in map group (153) and the vector operands stored in kernel buffer (131).

[0079] when Figure 3 The matrix-vector unit (141) is implemented in Figure 2 When in the matrix-matrix unit (121), the matrix-vector unit (141) can use the mapping group (151 to 153), the cross switch (123) and the kernel buffer (131) of the matrix-matrix unit (121).

[0080] Figure 3 Each vector-vector unit (161 to 163) in the matrix can be represented as follows: Figure 4 Implemented as shown.

[0081] Figure 4 A processing unit (161) configured to perform vector-vector operations according to one embodiment is shown. For example, Figure 4 The vector-vector unit (161) can be used as Figure 3 Any one of the vector-vector units in the matrix-vector unit (141).

[0082] exist Figure 4In this, the vector-vector unit (161) has multiple multiply-accumulate (MAC) units (171 to 173). Each of the multiply-accumulate (MAC) units (171 to 173) can receive two numbers as operands, perform the multiplication of the two numbers, and add the result of the multiplication to the sum maintained in the multiply-accumulate (MAC) unit.

[0083] Each of the vector buffers (181 and 183) stores a list of numbers. A pair of numbers, each from one of the vector buffers (181 and 183), can be provided as input to each of the multiply-accumulate (MAC) units (171 to 173). The multiply-accumulate (MAC) units (171 to 173) can receive multiple pairs of numbers from the vector buffers (181 and 183) in parallel and perform the multiply-accumulate (MAC) operation in parallel. The outputs from the multiply-accumulate (MAC) units (171 to 173) are stored in a shift register (175); and the accumulator (177) calculates the sum of the results in the shift register (175).

[0084] when Figure 4 The vector-vector unit (161) is implemented in Figure 3 When in the matrix-vector unit (141), the vector-vector unit (161) can use a mapping group (e.g., 151 or 153) as a vector buffer (181) and use the kernel buffer (131) of the matrix-vector unit (141) as another vector buffer (183).

[0085] Vector buffers (181 and 183) may have the same length to store the same number / count of data elements. The length may be equal to or a multiple of the count of the multiply-accumulate (MAC) units (171 to 173) in the vector-vector unit (161). When the length of the vector buffers (181 and 183) is a multiple of the count of the multiply-accumulate (MAC) units (171 to 173), input pairs equal to the number of counts of the multiply-accumulate (MAC) units (171 to 173) may be provided as input from the vector buffers (181 and 183) to the multiply-accumulate (MAC) units (171 to 173) in each iteration; and the vector buffers (181 and 183) feed their elements into the multiply-accumulate (MAC) units (171 to 173) over multiple iterations.

[0086] In one embodiment, the communication bandwidth of the connection (119) between the deep learning accelerator (DLA) (103) and the random access memory (105) is sufficient for the matrix-matrix unit (121) to use a portion of the random access memory (105) as a map group (151 to 153) and a kernel buffer (131 to 133).

[0087] In another embodiment, the mapping groups (151 to 153) and kernel buffers (131 to 133) are implemented in a portion of the local memory (115) of the deep learning accelerator (DLA) (103). While the matrix-matrix unit (121) is performing computations using the mapping groups (151 to 153) and kernel buffers (131 to 133) implemented in different portions of the local memory (115) of the deep learning accelerator (DLA) (103), the communication bandwidth of the connection (119) between the deep learning accelerator (DLA) (103) and random access memory (105) is sufficient to load the matrix operands of the next operating cycle of the matrix-matrix unit (121) into another portion of the local memory (115).

[0088] Figure 5 A deep learning accelerator and random access memory, configured according to one embodiment, are shown to be autonomously applied to the input of a trained artificial neural network.

[0089] Artificial neural networks (ANNs) (201) trained by machine learning (e.g., deep learning) can be described using standard formats (e.g., Open Neural Network Exchange (ONNX)). Describing a trained ANN (201) using standard formats allows for the identification of artificial neurons and their connectivity properties.

[0090] exist Figure 5 In this process, the deep learning accelerator (DLA) compiler (203) transforms the trained ANN (201) by generating instructions (205) for the deep learning accelerator (DLA) (103) and a matrix (207) corresponding to the properties of the artificial neuron and its connectivity. The instructions (205) and matrix (207) generated by the DLA compiler (203) based on the trained ANN (201) can be stored in the random access memory (105) for the deep learning accelerator (DLA) (103).

[0091] For example, random access memory (105) and deep learning accelerator (DLA) (103) can be used as follows: Figure 1 The integrated circuit device (101) is connected via a high-bandwidth connection (119). Figure 5 Autonomous computation based on instructions (205) and matrices (207) can be implemented in Figure 1 In the integrated circuit device (101). Alternatively, the random access memory (105) and the deep learning accelerator (DLA) (103) may be configured on a printed circuit board having multiple point-to-point serial buses with parallel extensions for connection (119).

[0092] exist Figure 5In the process, after the results of the DLA compiler (203) are stored in random access memory (105), the application of the trained ANN (201) to process the input (211) to the trained ANN (201) to produce the corresponding output (213) of the trained ANN (213) can be triggered by the presence of the input (211) in random access memory (105) or by another indication provided in random access memory (105).

[0093] In response, the deep learning accelerator (DLA) (103) executes instructions (205) to combine the input (211) and the matrix (207). The execution of instructions (205) may involve generating a mapping matrix by a grouping (151 to 153) of one or more matrix-matrix units (e.g., 121) of the deep learning accelerator (DLA) (103).

[0094] In some embodiments, the input to the ANN (211) is in the form of an initial mapping matrix. A portion of the initial mapping matrix can be retrieved from random access memory (105) as a matrix operand stored in a mapping array (151 to 153) of matrix-matrix units (121). Alternatively, the DLA instructions (205) also include instructions for the deep learning accelerator (DLA) (103) to generate the initial mapping matrix based on the input (211).

[0095] According to the DLA instruction (205), the deep learning accelerator (DLA) (103) loads matrix operands into the kernel buffers (131 to 133) and map groups (151 to 153) of its matrix-matrix unit (121). The matrix-matrix unit (121) performs matrix computations on the matrix operands. For example, the DLA instruction (205) decomposes the matrix computations of the trained ANN (201) according to the computational granularity of the deep learning accelerator (DLA) (103) (e.g., the size / dimension of the matrix loaded in the matrix-matrix unit (121) as a matrix operand), and applies the input feature maps to the kernel of one layer of artificial neurons to produce outputs as inputs for the next layer of artificial neurons.

[0096] After the computation of the trained ANN (201) executed according to the instruction (205) is completed, the deep learning accelerator (DLA) (103) immediately stores the output (213) of the ANN (201) in a predefined location in random access memory (105) or in a location provided in random access memory (105) as specified in the instruction to trigger the computation.

[0097] when Figure 5 The technology implemented in Figure 1When the input (211) is in the integrated circuit device (101), an external device connected to the memory controller interface (107) can write the input (211) into the random access memory (105) and trigger the deep learning accelerator (DLA) (103) to apply the input (211) to the autonomous computation of the trained ANN (201). After a certain time period, the output (213) is available in the random access memory (105); and the external device can read the output (213) via the memory controller interface (107) of the integrated circuit device (101).

[0098] For example, a predefined location in random access memory (105) can be configured to store an instruction that triggers the autonomous execution of an instruction (205) by a deep learning accelerator (DLA) (103). The instruction may optionally include the location of an input (211) within random access memory (105). Thus, during the autonomous execution of an instruction (205) that processes the input (211), an external device can retrieve outputs generated during a previous run of the instruction (205) and / or store another set of inputs for the next run of the instruction (205).

[0099] Optionally, another predefined location in the random access memory (105) may be configured to store an indication of the current progress status of the instruction (205). Furthermore, the indication may include a prediction of the completion time of the current execution of the instruction (205) (e.g., estimated based on previous executions of the instruction (205)). Thus, an external device can check the completion status within a suitable time window to retrieve the output (213).

[0100] In some embodiments, the random access memory (105) is configured with sufficient capacity to store multiple sets of inputs (e.g., 211) and outputs (e.g., 213). Each set can be configured in a predetermined time slot / region in the random access memory (105).

[0101] The deep learning accelerator (DLA) (103) can autonomously execute instructions (205) to produce an output (213) from an input (211) based on a matrix (207) stored in random access memory (105) without the assistance of a processor or device located outside the integrated circuit device (101).

[0102] In one embodiment of the method, the computing device (e.g., 101) may access the random access memory (105) of the computing device (e.g., 101) via an interface (107) to a memory controller. The computing device (e.g., 101) may have a processing unit (e.g., 111) configured to perform computations on at least matrix operands, such as matrix operands stored in a map array (151 to 153) and matrix operands stored in a kernel buffer (131 to 133).

[0103] For example, a computing device (e.g., 101) may be enclosed within an integrated circuit package; and a set of connections may connect an interface (107) to a memory controller located outside the integrated circuit package.

[0104] Instructions (205) that can be executed by a processing unit (e.g., 111) can be written into random access memory (105) via an interface (107).

[0105] The matrix (207) of the artificial neural network (201) can be written into the random access memory (105) via the interface (107). The matrix (207) identifies the properties and / or state of the artificial neural network (201).

[0106] Optionally, at least a portion of the random access memory (105) is non-volatile and configured to store a matrix (207) of instructions (205) and an artificial neural network (201).

[0107] The first input (211) to the artificial neural network can be written to the random access memory (105) via the interface (107).

[0108] An instruction is provided in the random access memory (105) to cause the processing unit (111) to begin execution of the instruction (205). In response to the instruction, the processing unit (111) executes an instruction to combine the first input (211) with the matrix (207) of the artificial neural network (201) to generate a first output (213) from the artificial neural network (201) and stores the first output (213) in the random access memory (105).

[0109] For example, the indication may be the address of a first input (211) in random access memory (105); and the indication may be stored at a predetermined location in random access memory (105) to initiate the execution of an instruction (205) for the input (211) identified by the address. Optionally, the indication may also include an address for storing an output (213).

[0110] The first output (213) can be read from the random access memory (105) via the interface (107).

[0111] For example, a computing device (e.g., 101) may have a deep learning accelerator (103) formed on a first integrated circuit die and random access memory (105) formed on one or more second integrated circuit dies. The connection (119) between the first integrated circuit die and one or more second integrated circuit dies may include through-silicon vias (TSVs) to provide high bandwidth for memory access.

[0112] For example, a compiler (203) can be used to convert the description of the artificial neural network (201) into instructions (205) and matrices (207). The combination of the instructions (205) and matrices (207) stored in random access memory (105) and the deep learning accelerator (103) provides an autonomous implementation of the artificial neural network (201) that can automatically convert the input (211) to the artificial neural network (201) into its output (213).

[0113] For example, during the time period in which the deep learning accelerator (103) executes instructions (205) to generate a first output (213) from a first input (211) according to the matrix (207) of the artificial neural network (201), a second input to the artificial neural network (201) can be written to an alternative location in the random access memory (105) via the interface (107). After the first output (213) is stored in the random access memory (105), an instruction can be provided in the random access memory to cause the deep learning accelerator (103) to resume execution of instructions and generate a second output from the second input.

[0114] During the time period in which the deep learning accelerator (103) executes instructions (205) to generate a second output from a second input according to the matrix (207) of the artificial neural network (201), the first output (213) can be read from the random access memory (105) via the interface (107); and another input can be written to the random access memory in place of the first input (211) or written to a different location. The process can be repeated for a series of inputs.

[0115] A deep learning accelerator (103) may include at least one matrix-matrix unit (121) capable of executing instructions on two matrix operands. The two matrix operands may be a first matrix and a second matrix. Each of the two matrices has multiple vectors. The matrix-matrix unit (121) may include multiple matrix-vector units (141 to 143) configured to operate in parallel. Each of the matrix-vector units (141 to 143) is configured to operate on the first matrix and a vector from the second matrix in parallel with other matrix-vector units. Furthermore, each of the matrix-vector units (141 to 143) may have multiple vector-vector units (161 to 163) configured to operate in parallel. Each of the vector-vector units (161 to 163) is configured to operate on a vector from the first matrix and a common vector operand of the corresponding matrix-vector unit in parallel with other vector-vector units. Furthermore, each of the vector-vector units (161 to 163) may have multiple multiply-accumulate units (171 to 173) configured to operate in parallel.

[0116] In addition to the processing unit (111), the deep learning accelerator (103) may also have local memory (115) and a control unit (113). The control unit (113) may load instructions (205) and matrix operands (e.g., 207) from random access memory (105) for execution by the processing unit (111). The local memory may cache matrix operands used by the matrix-matrix unit. The connection (119) may be configured with bandwidth sufficient to load a set of matrix operands from random access memory (105) into local memory (115) during the time period in which the matrix-matrix unit performs operations on two other matrix operands. Furthermore, during the time period, the bandwidth is sufficient to store the results produced by the matrix-matrix unit (121) in the previous instruction execution from local memory (115) into random access memory (105).

[0117] A deep learning accelerator (103) and random access memory (105) can be configured in a digital camera to convert image data into a description of what the artificial neural network can identify from the image data. For example, the digital camera can be used to monitor the situation and / or surrounding environment of a user or location to generate warnings and / or selectively store image data.

[0118] For example, a digital camera may have a transceiver for a wired or wireless communication connection to an independent computer system, such as a mobile device, smartphone, personal media player, personal computer, set-top box, Internet of Things (IoT) hub, server computer, etc. The digital camera can use this connection to provide identification of identified items or events by an artificial neural network (201). The independent computer system further processes the identification to generate warnings, provide displays, selectively request image data from the digital camera, and / or selectively store the image data.

[0119] The digital camera can be customized, updated, and / or upgraded by receiving the matrix (207) and instructions (205) of the artificial neural network (201) via a transceiver and storing them in random access memory (105).

[0120] An artificial neural network (201), implemented via a deep learning accelerator (103) executing instructions (205), transforms the image stream from the image sensor into an inference result. This transformation improves the quality of the digital camera's output, reduces the communication bandwidth requirements of the connection to the computer system, and / or reduces the computational workload of the computer system.

[0121] Figures 6 to 8 A digital camera (191) with a configured deep learning accelerator (103) and random access memory (105) is shown according to some embodiments.

[0122] Figure 6 , 7 The digital camera (191) in 8 includes an image sensor (102). Examples of image sensors include charge-coupled device (CCD) image sensors and complementary metal-oxide-semiconductor (CMOS) image sensors. The image sensor (102) may be formed on an integrated circuit die.

[0123] exist Figure 6 In the integrated circuit device (101), not only is a deep learning accelerator (103) and a random access memory (105), but also logic circuits for an image sensor (102), a controller (107) and a transceiver (106).

[0124] For example, the image sensor (102), controller (107), and transceiver (106) may be formed on an integrated circuit die stacked on and connected to the integrated circuit die of the random access memory (105). Thus, image data generated by the image sensor (102) may be stored by the controller (107) in the random access memory (105) as input (211) to an artificial neural network (201) implemented using a deep learning accelerator (103).

[0125] Alternatively, the image sensor (102) may be formed on an integrated circuit die separate from the controller (107) and / or transceiver (106).

[0126] In some implementations, the image sensor (102) includes a state machine that cyclically writes image data generated in the image sensor (102) into a predetermined region for input (211) to an artificial neural network (201). For example, once the state machine is instructed to operate, the image sensor captures an image frame, stores the image in a time slot within the predetermined region for input (211), and captures the next frame of the image. Subsequent frames are stored in subsequent time slots. When the last time slot within the predetermined region for input (211) is used, the state machine returns to storing a subsequent frame in a first time slot, which erases / overwrites the oldest frame in the predetermined region. Another subsequent frame is then stored in a subsequent time slot containing the oldest frame.

[0127] In some implementations, the state machine of the image sensor (102) can write image data to random access memory (105) through a set of through-silicon vias (TSVs) without passing through the controller (107).

[0128] The image sensor (102) may be configured on the top portion of the integrated circuit device (101) with an opening that allows light to pass through the lens (227) of the digital camera (191) to reach the image sensor (102). The connector (e.g., pins or contacts) of the integrated circuit device (102) may be configured on the bottom or side of the integrated circuit device (101).

[0129] Figure 6 The integrated circuit device (101) has a controller (107) configured to control the operation of the image sensor (102) via a connection (104) in the integrated circuit device (101). The controller (107) may be implemented, for example, using a microcontroller or sequencer that controls the timing of the operation of the image sensor (102) and loads sensor data / measurements into random access memory (105).

[0130] Optionally, the controller (107) may use a microprocessor-implemented firmware running an application program (215) stored in random access memory (105) to coordinate operation between the image sensor (102), random access memory (105), deep learning accelerator (103) and / or transceiver (106).

[0131] After storing a set of image data in random access memory (105) as input (211) to artificial neural network (201), controller (107) can cause deep learning accelerator (103) to execute instructions (205) and produce output (213) of artificial neural network (201).

[0132] For example, the controller (107) can instruct the deep learning accelerator (103) to execute a start instruction (205) by writing the address of input (211) to a predefined location in random access memory (105). When the deep learning accelerator (103) is idle, it can periodically read the address stored in the predefined location in random access memory (105). When a new and / or valid address is retrieved from the predefined location, the deep learning accelerator (103) starts the execution of the start instruction (205). Optionally, after the execution of the start instruction (205), the deep learning accelerator (103) can optionally clear, erase, or invalidate the address previously stored in the predefined location in random access memory (105).

[0133] Alternatively, the controller (107) is configured to send signals or messages to the deep learning accelerator (103) to instruct the deep learning accelerator (103) to execute instructions (205). Signals or messages can be transmitted from the controller (107) to the deep learning accelerator (103) using a direct connection to a memory cell that does not pass through the random access memory (105).

[0134] In some implementations, the controller (107) and the deep learning accelerator (103) have independent connections (109 and 119) to the random access memory (105). When the controller (107) and the deep learning accelerator (103) do not access the same block or address of the random access memory (105), the connections (109 and 119) can be used in parallel by the controller (107) and the deep learning accelerator (103) to access different portions of the random access memory (105) simultaneously.

[0135] In other embodiments, the control unit (113) and the controller (107) may share at least a portion of their circuitry in the deep learning accelerator (103) and access the random access memory (105) using the same memory interface (117).

[0136] A portion of the processing unit (111) may be implemented using a neuromorphic memory (225). For example, the neuromorphic memory (225) may comprise a lattice array of memristors configured to perform multiply-accumulate (MAC) operations via analog circuitry. For example, the multiply-accumulate unit (e.g., 171 or 173) in the vector-vector unit (e.g., 161) of a deep learning accelerator (103) may be implemented using a lattice array of memristors. The memristors may be connected in an array having word lines and bit lines configured to address the memristors as memory cells. Typically, a memristor is connected to one word line and one bit line in the array. Currents passing through a set of memristors through the word lines and bit lines in the lattice array are added in the bit lines, corresponding to an accumulation operation. The current corresponds to the multiplication of a voltage applied to the word line and a parameter associated with the resistance of the memristor, corresponding to a multiplication operation. The current in the bit line may be compared to a threshold to determine whether the neuron represented by the bit line is activated under a current input. The memristor array can be individually connected to the bit line and programmed to have thresholds corresponding to the activation level thresholds of neurons. Current detectors can be configured for each memristor connected to the output of the bit line to determine whether the current level in the bit line corresponds to a level exceeding the memristor's threshold. The neuromorphic memory (225) can perform multiply-accumulate (MAC) operations in a manner similar to how a memory device reads an array of memory cells, thus offering low energy cost and high computational speed.

[0137] Through connection (108), controller (107) operates. Figure 6 The transceiver (106) of the integrated circuit device (101) is used to communicate with a standalone computer system (223) via a wired connection (228). Alternatively or in combination, the digital camera (191) includes a transceiver for wireless connection (228).

[0138] For example, the transceiver (106) can be configured to communicate according to a communication protocol for a local area network, peripheral bus, mobile industry processor interface, wireless personal area network or wireless local area network or Internet of Things (IoT). For example, the transceiver (106) can be formed on a radio frequency (RF) complementary metal-oxide-semiconductor (CMOS) integrated circuit chip.

[0139] For example, a digital camera (191) can use a transceiver (106) to transmit the output (213) of an artificial neural network (201) to a computer system (223).

[0140] For example, a digital camera (191) can use a transceiver (106) to transmit an alert to a computer system (223) based on the output (213) of an artificial neural network (201).

[0141] The transceiver (106) can be used by the digital camera (191) to receive data and / or instructions, such as the matrix (207) and instructions (205) of the artificial neural network (201), from the computer system (223). The transceiver (106) can also be used by the digital camera (191) to report to and retrieve the output (213) of the artificial neural network (201) computed by the deep learning accelerator (103) to the computer system (223).

[0142] Optionally, the computer system (223) may communicate with the digital camera (191) to request the digital camera (191) to transmit input (211) associated with the output (213). In response, the transceiver (106) transmits the input (211) to the computer system (223), which allows the computer system (223) to selectively analyze the input (211) to the artificial neural network (201).

[0143] Alternatively, the digital camera (191) automatically selects the input (211) for transmission to the computer system (223) based on the output (213) of the artificial neural network (201).

[0144] In some implementations, the digital camera (191) is configured to report outputs (213) to the computer system (223). For example, when the deep learning accelerator (103) completes computation of a set of outputs (213), the controller (107) generates a message reporting the availability of the outputs (213). The transceiver (106) transmits the message to the computer system (223). In response, the computer system (223) may optionally accept the transmission of the outputs (213) immediately, request a delay in the transmission of the outputs (213) for a period of time, or request a delay in the generation of the next set of outputs (213).

[0145] In some implementations, the control unit (113) of the deep learning accelerator (103) may include a controller (107); and the logic circuitry of the transceiver (106) may be implemented on the integrated circuit die of the deep learning accelerator (103), such as... Figure 7 As shown in the image.

[0146] exist Figure 7 In this configuration, a deep learning accelerator (103) is mounted on an integrated circuit die; and a random access memory (105) is mounted on one or more integrated circuit dies. The control unit (113) controls not only the execution of instructions (205) of the artificial neural network (201), but also the communication between the transceiver (106) and the computer system (223) and the operation of the image sensor (102).

[0147] For example, the control unit (113) periodically retrieves image data from the sensor (102) and stores the image data in the random access memory (105) via a high-bandwidth connection (119).

[0148] In some implementations, the image sensor (102) is enclosed in an integrated circuit package of the integrated circuit device (101) (e.g., as shown in the image sensor package). Figure 6 (as shown in the image).

[0149] Alternatively, the image sensor (102) may be a separate component outside the integrated circuit package enclosing the deep learning accelerator (103) and random access memory (105).

[0150] For example, an image sensor (102) and an integrated circuit device (101) having a deep learning accelerator (103) and random access memory (105) can be mounted on a printed circuit board configured in a digital camera (191).

[0151] Figure 8 The digital camera (191) has a substrate (229) that provides connections between its components, such as a deep learning accelerator (103), random access memory (105), an image sensor (102), a controller (107), and a transceiver (106).

[0152] In some embodiments, the substrate (229) includes an integrated circuit die with wires for connecting components. Some components (e.g., integrated circuit dies of random access memory (105), deep learning accelerator (103), controller (107), and / or transceiver (106)) may be connected to the integrated circuit die of the substrate (229) via through-silicon vias (TSVs). Other components may be connected to the substrate (229) via wire bonding, die attachment, or another technology.

[0153] In some embodiments, the substrate (229) further includes a printed circuit board having wires for connecting components and other components, such as power supplies (e.g., batteries), displays, light-emitting diode (LED) indicators, etc.

[0154] In some implementations, the logic circuitry of the transceiver (106) and / or controller (107) is configured on the integrated circuit die of the deep learning accelerator (103) or another integrated circuit die.

[0155] Figure 9 A method implemented in a digital camera according to one embodiment is shown. For example, Figure 9 The method can be found Figure 6 , Figure 7 or Figure 8Implemented in a digital camera (191).

[0156] At box 301, the digital camera (191) stores the matrix (207) and instructions (205) of the artificial neural network (201) in its random access memory (105), the instructions being executable by at least one processing unit (111) enclosed within the digital camera (191) to perform computations of the artificial neural network (201) using the matrix (207).

[0157] For example, at least one processing unit (111) may be formed on an integrated circuit die of a field-programmable gate array (FPGA) or application-specific integrated circuit (ASIC) implementing the deep learning accelerator (103). The deep learning accelerator (103) may include at least one processing unit (111) for matrix instruction execution, local memory (115) for buffering matrix operands and results, a control unit (113) for loading instructions (205) from random access memory (105) for execution, and a memory interface (117) for accessing random access memory (105).

[0158] For example, an integrated circuit package is configured to enclose at least one or more integrated circuit dies containing an integrated circuit die of an FPGA or ASIC and random access memory.

[0159] For example, the random access memory (105) and the deep learning accelerator (103) are formed on a separate integrated circuit die and connected via through-silicon vias (TSVs).

[0160] For example, a digital camera (191) may have a controller (107) to operate a transceiver (106). The controller (107) may be separate from the control unit (113) or integrated into the control unit of the deep learning accelerator (103).

[0161] At frame 303, the image sensor (102) of the digital camera (191) generates image data that captures the field of view of the digital camera (191).

[0162] For example, an image sensor (102) may be formed on an integrated circuit die stacked on one or more integrated circuit dies of a random access memory (105) or an integrated circuit die of a deep learning accelerator (103); and the circuitry in the image sensor may be connected from the integrated circuit die of the image sensor (102) to the integrated circuit die of the random access memory (105) or the integrated circuit die of the deep learning accelerator (103) using through-silicon vias (TSVs).

[0163] At box 305, the image sensor (102) stores image data in the random access memory (105) integrated in the digital camera (191) as input (211) to the artificial neural network (201).

[0164] At box 307, at least one processing unit (111) executes instructions to compute output (213) from an artificial neural network (201) that takes image data as input (211).

[0165] At box 309, the digital camera (191) generates a description of items or events in the field of view captured in the image data based on the output (213) of the artificial neural network (201).

[0166] At box 311, the digital camera (191) uses its transceiver (106) to transmit the description to the computer system (223).

[0167] For example, the output (213) of the artificial neural network (201) may include the recognition, classification, or category of an object, person, or feature. Furthermore, the output (213) of the artificial neural network (201) may include the location and size of the object, person, or feature. A description is generated based on the recognition, classification, or category and / or based on location and size and provided to an independent computer system (223).

[0168] For example, the output (213) of the artificial neural network (201) may include the recognition of events associated with objects, people or features; and the description may include the recognition of events.

[0169] Optionally, the controller (107) of the digital camera (191) may be configured (e.g., via instructions from an application (215) that runs the digital camera as firmware) to control the transceiver (106) to provide the computer system (223) with representative images of objects, people or features extracted from the image data based on the output (213) of the artificial neural network (201).

[0170] For example, the transceiver (106) can be configured to communicate according to a communication protocol of a wireless personal area network, a wireless local area network, a wired local area network, or a serial peripheral bus.

[0171] Representative images may be transmitted to a standalone computer system (223) along with the description as input to the standalone computer system (223), or transmitted in response to a request from the computer system (223) after the computer system (223) has processed the description.

[0172] Optionally, the controller (107) of the digital camera (191) can be configured to selectively store image data based on the output (213) of an artificial neural network (201) for transmission to a computer system (223).

[0173] For example, the application (215) can be configured for the digital camera (191) to determine whether to discard image data based on the processing results described in the computer system (223) and / or the output (213) of the artificial neural network (201).

[0174] For example, the application (215) can be configured to determine whether to transfer a portion of image data from the digital camera (191) to the computer system (223) based on the processing results described in the computer system (223) and / or based on the output (213) of the artificial neural network (201).

[0175] For example, said portion of the image data may be extracted based on the identification of items or events in the output (213) of an artificial neural network (201); and the identification of items or events may include the size and location of the items and / or their activity in the scene captured in the field of view of the digital camera.

[0176] This disclosure includes the methods described above and the apparatus for performing the methods, a data processing system for performing the methods, and a computer-readable medium containing instructions that, when executed on the data processing system, cause the system to perform the methods.

[0177] A typical data processing system may contain interconnects (such as buses and system core logic) that interconnect microprocessors and memory. Microprocessors are typically coupled to cache memory.

[0178] Interconnectors link the microprocessor and memory together and also connect them to input / output (I / O) devices via I / O controllers. I / O devices may include display devices and / or peripheral devices such as mice, keyboards, modems, network interfaces, printers, scanners, cameras, and other devices known in the art. In one embodiment, when the data processing system is a server system, some of the I / O devices (e.g., printers, scanners, mice, and / or keyboards) are optional.

[0179] Interconnectors may include one or more buses that are interconnected to each other via various bridges, controllers, and / or adapters. In one embodiment, the I / O controller includes a USB adapter for controlling Universal Serial Bus (USB) peripherals and / or an IEEE-1394 bus adapter for controlling IEEE-1394 peripherals.

[0180] Memory may include one or more of the following: read-only memory (ROM), volatile random access memory (RAM), and non-volatile memory, such as hard disk drives, flash memory, etc.

[0181] Volatile RAM is typically implemented as dynamic RAM (DRAM), which requires continuous power to refresh or retain data in memory. Non-volatile memory is typically magnetic hard disk drives, magnetic optical disk drives, optical disk drives (e.g., DVD RAM), or other types of memory systems that retain data even after power is removed from the system. Non-volatile memory can also be random access memory.

[0182] Non-volatile memory can be a local device directly coupled to other components in the data processing system. Alternatively, it can be a non-volatile memory located off-system, such as a network storage device coupled to the data processing system via a network interface, such as a modem or Ethernet interface.

[0183] In this disclosure, some functions and operations are described as being executed by or caused by software code for the sake of simplicity. However, such expressions are also used to specify that the functions are generated by a processor, such as a microprocessor, executing code / instructions.

[0184] Alternatively or in combination, the functions and operations described herein may be implemented using a special-purpose circuit system with or without software instructions, such as using an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA). Embodiments may also be implemented using hardwired circuitry without or in combination with software instructions. Therefore, the technology is not limited to any specific combination of hardware circuitry and software, nor to any particular source of instructions executed by the data processing system.

[0185] While one embodiment may be implemented in a full-featured computer and computer system, various embodiments can be distributed as a variety of computing products and can be applied regardless of the specific type of machine or computer-readable medium actually used to implement the distribution.

[0186] At least some of the disclosed aspects may be embodied, at least in part, in software. That is, the technology can be implemented in computer systems or other data processing systems in response to its processor (e.g., a microprocessor) executing a sequence of instructions contained in memory (e.g., ROM, volatile RAM, non-volatile memory, cache, or remote storage device).

[0187] The routines executed to implement the embodiments can be implemented as an operating system or a specific application, component, program, object, module, or part of a sequence of instructions referred to as a "computer program." A computer program typically contains one or more sets of instructions at various times in various memories and storage devices in a computer, and when read and executed by one or more processors in the computer, the sets of instructions cause the computer to perform necessary operations to perform elements involving various aspects.

[0188] Machine-readable media can be used to store software and data that enable a data processing system to perform various methods when executed. Executable software and data can be stored in various locations, including, for example, ROM, volatile RAM, non-volatile memory, and / or caches. Parts of this software and / or data can be stored in any of these storage devices. Additionally, data and instructions can be obtained from a centralized server or a peer-to-peer network. Different portions of the data and instructions can be obtained at different times and in different communication sessions or within the same communication session from different centralized servers and / or peer-to-peer networks. Data and instructions can be obtained in their entirety before the application is executed. Alternatively, portions of data and instructions can be obtained dynamically and as needed for execution. Therefore, it is not required that all data and instructions be on the machine-readable media at any given time.

[0189] Examples of computer-readable media include, but are not limited to, media of non-transitory recordable and non-recordable types, such as volatile and non-volatile memory devices, read-only memory (ROM), random access memory (RAM), flash memory devices, floppy and other removable disks, disk storage media, optical storage media (e.g., optical disc read-only memory (CDROM), digital versatile disc (DVD), etc.), and others. Computer-readable media can store instructions.

[0190] Instructions can also be embodied in digital and analog communication links for the propagation of electrical, optical, acoustic, or other forms of signals, such as carrier waves, infrared signals, digital signals, etc. However, propagation signals such as carrier waves, infrared signals, digital signals, etc., are not tangible machine-readable media and are not configured to store instructions.

[0191] Generally, machine-readable media includes any mechanism that provides (i.e., stores and / or transmits) information in a form accessible by a machine (e.g., a computer, a network device, a personal digital assistant, a manufacturing tool, any device having one or more sets of processors, etc.).

[0192] In various embodiments, hard-wired circuitry can be used in combination with software instructions to implement the techniques. Therefore, the techniques are not limited to any specific combination of hardware circuitry and software, nor to any particular source of instructions executed by the data processing system.

[0193] The above descriptions and figures are illustrative and should not be construed as limiting. Numerous specific details have been described to provide a thorough understanding. However, in some instances, well-known or conventional details have been omitted to avoid obscuring the description. References to one or more embodiments in this disclosure do not necessarily refer to the same embodiment; and such references imply at least one.

[0194] In the foregoing description, this disclosure has been described with reference to specific exemplary embodiments. It will be apparent that various modifications may be made without departing from the broader spirit and scope set forth in the following claims. Therefore, the description and drawings should be regarded as illustrative rather than restrictive.

Claims

1. An integrated circuit device, comprising: An image sensor configured to generate image data of the field of view of the integrated circuit device; At least one processing unit configured to execute instructions having matrix operands; A random access memory configured to store first data representing the weights of an artificial neural network and second data representing instructions that can be executed by the at least one processing unit to perform matrix calculations of the artificial neural network using the first data representing the weights of the artificial neural network; A transceiver configured to communicate with a computer system, the computer system being separate from the integrated circuit device; as well as A controller coupled to the transceiver, the sensor, and the random access memory, wherein the controller is configured to write the image data to the random access memory as input to the artificial neural network; The at least one processing unit is further configured to execute the instructions represented by the second data stored in the random access memory to generate the output of the artificial neural network based at least in part on the first data stored in the random access memory and the image data; and The controller is further configured to provide third data representing a description of an item or event captured in the image data based on the output of the artificial neural network, and to control the transceiver to provide the third data representing the description to the computer system.

2. The integrated circuit device of claim 1, wherein the output of the artificial neural network includes the identification, classification or category of an object, person or feature, and the position and size of the object, person or feature; and the description is based on the identification, classification or category, and based on the position and size.

3. The integrated circuit device of claim 2, wherein the output of the artificial neural network includes the recognition of an event associated with the object, person, or feature; and the description includes the recognition of the event.

4. The integrated circuit device of claim 3, wherein the controller is further configured to control the transceiver to provide the computer system with a representative image of the object, person, or feature extracted based on the output of the artificial neural network.

5. The integrated circuit device of claim 4, wherein the controller is configured to cause the transceiver to transmit the description and the representative image together to the computer system.

6. The integrated circuit device of claim 4, wherein the controller is configured to cause the transceiver to transmit the representative image to the computer system in response to a request from the computer system regarding the description.

7. The integrated circuit device of claim 4, wherein the controller is configured to selectively store image data based on the output of the artificial neural network for transmission to the computer system.

8. The integrated circuit device according to claim 7, further comprising: An integrated circuit die of a field-programmable gate array (FPGA) or application-specific integrated circuit (ASIC) implementing a deep learning accelerator, the deep learning accelerator including the at least one processing unit and a control unit configured to load the instructions from the random access memory for execution.

9. The integrated circuit device of claim 8, wherein the control unit comprises the controller.

10. The integrated circuit device according to claim 8, further comprising: An integrated circuit package configured to at least enclose one or more integrated circuit dies of the FPGA or ASIC and the random access memory; and The circuitry in the image sensor is connected to one or more integrated circuit dies of the random access memory using through-silicon vias (TSVs).

11. The integrated circuit device of claim 10, wherein the at least one processing unit comprises a matrix-matrix unit configured to operate on two matrix operands of an instruction; The matrix-matrix unit comprises multiple matrix-vector units configured to operate in parallel; Each of the plurality of matrix-vector units comprises a plurality of vector-vector units configured to operate in parallel; Each of the plurality of vector-vector units comprises a plurality of multiply-accumulate units configured to operate in parallel; and Each of the plurality of multiply-accumulate units includes a neuromorphic memory configured to perform multiply-accumulate operations via analog circuitry.

12. The integrated circuit device of claim 11, wherein the random access memory and the deep learning accelerator are formed on a separate integrated circuit die and are connected by through-silicon vias (TSVs).

13. The integrated circuit device of claim 12, wherein the transceiver is configured to communicate according to a wireless personal area network or wireless local area network communication protocol.

14. A method implemented in a digital camera, the method comprising: First data representing the weights of the artificial neural network and second data representing the instructions are stored in the random access memory of the digital camera. The instructions can be executed by at least one processing unit of the digital camera to perform matrix calculations of the artificial neural network using the first data representing the weights. Image data capturing the field of view of the digital camera is generated by the image sensor of the digital camera; The image data is stored in the random access memory of the digital camera as input to the artificial neural network; The at least one processing unit executes the instructions represented by the second data stored in the random access memory of the digital camera to calculate the output from the artificial neural network based at least in part on the first data stored in the random access memory of the digital camera and the image data; The digital camera generates third data, based on the output of the artificial neural network, describing items or events within the field of view of the image data; and The third data representing the description is transmitted to the computer system via the transceiver of the digital camera.

15. The method of claim 14, further comprising: The digital camera determines whether to discard the image data based on the processing results described in the computer system.

16. The method of claim 14, further comprising: The digital camera determines whether to transmit a portion of the image data based on the processing results described in the computer system.

17. The method of claim 16, further comprising: The portion of the image data is extracted based on the identification of the item or event, wherein the output of the artificial neural network contains the identification of the item or event.

18. The method of claim 17, wherein the identification of the item or event includes the size and location of the item.

19. A digital camera comprising: shell; Lens; An image sensor is positioned behind the lens to generate image data that captures the field of view of the digital camera through the lens; Random access memory configured to store models of artificial neural networks; Field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs) have the following characteristics: A memory interface for accessing the random access memory; as well as At least one processing unit configured to execute instructions having matrix operands to perform computations of the artificial neural network according to the model; and A transceiver configured to communicate with a computer system using a wired or wireless connection; The image sensor is configured to store the image data in the random access memory as input to the artificial neural network; The FPGA or ASIC is configured to perform the computations of the artificial neural network according to the model, to convert the input into an output from the artificial neural network; and The digital camera is configured to generate a description of items or events captured in the field of view in the image data and transmit it to the computer system.

20. The digital camera of claim 19, wherein the random access memory includes non-volatile memory configured to store the model of the artificial neural network; the model includes instructions executable by the FPGA or ASIC; and the at least one processing unit includes a matrix-matrix unit configured to operate on two matrix operands of the instructions.

Images