Surveillance camera upgrade via removable media with deep learning accelerator and random access memory

Abstract

Systems, devices, and methods related to deep learning accelerators and memory are described. For example, a removable media (e.g., a memory card or USB drive) can be configured to execute instructions with matrix operands and configured with an interface to receive a video stream and a random access memory to buffer a portion of the video stream as input to an artificial neural network and store matrices of the artificial neural network and instructions executable by the deep learning accelerator. Such removable media can be used in place of existing removable media used in a surveillance camera to record video or images. The deep learning accelerator can execute instructions to generate an analysis of the buffered portion using the artificial neural network, enabling the surveillance camera upgraded via use of the removable media to provide intelligent services based on the analysis.

Description

[0001] Related applications

[0002] This application claims priority to U.S. Patent Application No. 16 / 906,253, filed June 19, 2020, entitled "Surveillance Camera Upgrade via Removable Media having Deep Learning Accelerator and Random Access Memory," the entire disclosure of which is hereby incorporated by reference. Technical Field

[0003] At least some of the embodiments disclosed herein generally relate to surveillance cameras, and more specifically, but not limited to, intelligent surveillance cameras powered by artificial neural networks (ANNs), such as ANNs configured through machine learning and / or deep learning. Background Technology

[0004] Artificial neural networks (ANNs) use a network of neurons to process the inputs to the network and produce the outputs from the network.

[0005] For example, each neuron in a network receives a set of inputs. Some inputs to a neuron can be the outputs of other neurons in the network; and some inputs to a neuron can be inputs provided to the neural network. The input / output relationships among neurons in a network represent the neuronal connectivity within the network.

[0006] For example, each neuron can have a bias, an activation function, and a set of synaptic weights for its input. The activation function can be a step function, a linear function, a log-sigmoid function, etc. Different neurons in the network can have different activation functions.

[0007] For example, each neuron can produce a weighted sum of its input and its bias, and then produce an output as a function of the weighted sum, which is computed using the neuron's activation function.

[0008] The relationship between the inputs and outputs of an ANN is generally defined by the ANN model, which contains data representing the connectivity of neurons in the network, as well as the biases, activation functions, and synaptic weights of each neuron. Based on a given ANN model, a computing device can be configured to compute the network's outputs based on a given set of inputs to the network.

[0009] For example, the input to the ANN network can be generated based on camera input; and the output from the ANN network can be the recognition of items such as events or objects.

[0010] Generally, supervised methods can be used to train ANNs, where the parameters in the ANN are tuned to minimize or reduce the error between a known output associated with or produced by a given input and a computed output produced by applying the input to the ANN. Examples of supervised learning / training methods include reinforcement learning and learning with error correction.

[0011] Alternatively or in combination, unsupervised methods can be used to train an ANN, where the exact output produced by a given set of inputs is unknown until training is complete. An ANN can be trained to classify items into multiple categories or data points into clusters.

[0012] Multiple training algorithms can be used for complex machine learning / training paradigms.

[0013] Deep learning uses multiple layers of machine learning to progressively extract features from input data. For example, lower layers can be configured to identify edges in an image; and higher layers can be configured to identify items captured in the image, such as faces, objects, events, etc., based on the edges detected by the lower layers. Deep learning can be implemented via artificial neural networks (ANNs), such as deep neural networks, deep belief networks, recurrent neural networks, and / or convolutional neural networks.

[0014] Deep learning has been applied in many fields, such as computer vision, speech / audio recognition, natural language processing, machine translation, bioinformatics, drug design, medical image processing, and games. Attached Figure Description

[0015] The embodiments are shown in the accompanying drawings by way of example and not limitation, and the same reference numerals indicate similar elements in the drawings.

[0016] Figure 1 An integrated circuit device having a configured deep learning accelerator and random access memory is shown according to one embodiment.

[0017] Figure 2 A processing unit configured to perform matrix-matrix operations according to one embodiment is shown.

[0018] Figure 3 A processing unit configured to perform matrix-vector operations according to one embodiment is shown.

[0019] Figure 4 A processing unit configured to perform vector-vector operations according to one embodiment is shown.

[0020] Figure 5 A deep learning accelerator and random access memory, configured according to one embodiment, are shown to be autonomously applied to the input of a trained artificial neural network.

[0021] Figure 6 This illustrates a technique for upgrading surveillance cameras using removable media, according to one embodiment.

[0022] Figures 7 to 9 A removable medium having a configured deep learning accelerator and random access memory is shown according to some embodiments.

[0023] Figure 10 A method implemented in a surveillance camera according to one embodiment is shown. Detailed Implementation

[0024] At least some embodiments disclosed herein provide removable media having a general-purpose integrated circuit configured to perform computations of an artificial neural network (ANN) with less power consumption and computation time. The integrated circuit includes a deep learning accelerator (DLA) and random access memory. Such removable media can be inserted into a surveillance camera as an alternative storage device. Inserting removable media into a surveillance camera further upgrades the camera to include the ability to generate video analytics based on the artificial neural network (ANN) using the integrated circuit.

[0025] For example, input video data stored in removable media can be analyzed using artificial neural networks to produce analysis and / or inference results. These analysis and / or inference results can be used for video compression performed by a video encoder. The compressed video, smaller than the input video data, can be stored in the removable media as a replacement for the input video data and can be retrieved for playback demonstrations that are substantially identical to the playback of the input video data.

[0026] Alternatively or in combination, the analysis and / or inference results can be used to generate intelligent outputs smaller in volume than the video data. For example, an artificial neural network (ANN) can be trained to output identified events, patterns, features, or classifications of interest in a specific surveillance application. The output can be stored in removable media, used to generate alerts, transmitted from surveillance cameras to stand-alone computing systems, and / or used to selectively retain and / or transmit video data generated from surveillance cameras.

[0027] A deep learning accelerator (DLA) contains a set of general-purpose programmable hardware computation logic that is specialized and / or optimized to perform parallel vector and / or matrix computations, including but not limited to vector and / or matrix multiplication and accumulation.

[0028] In addition, a deep learning accelerator (DLA) may contain one or more arithmetic logic units (ALUs) to perform arithmetic and bitwise operations on integer binary numbers.

[0029] Deep learning accelerators (DLAs) can be programmed via a set of instructions to perform computations on artificial neural networks (ANNs).

[0030] The granularity of a Deep Learning Accelerator (DLA) for vector and matrix operations corresponds to the largest unit of a vector / matrix that can be operated on during the execution of an instruction by the DLA. During instructions performing predefined operations on vector / matrix operands, the DLA can operate on the elements of the vector / matrix operands in parallel to reduce the execution time and / or energy consumption associated with memory / data access. Operations on vector / matrix operands at the granularity of the DLA can be used as building blocks to perform computations on larger vectors / matrices.

[0031] The implementation of a typical / practical artificial neural network (ANN) involves vector / matrix operands with a size larger than the operational granularity of a deep learning accelerator (DLA). To implement such an ANN using a DLA, computations involving large-sized vector / matrix operands can be decomposed into computations of vector / matrix operands at the granularity of the DLA. A DLA can be programmed with instructions to perform computations involving large vector / matrix operands. For example, the atomic computational capabilities of a DLA in manipulating vectors and matrices at the granularity of the DLA in response to instructions can be programmed to perform computations within an artificial neural network (ANN).

[0032] In some implementations, a deep learning accelerator (DLA) lacks some of the logical computational capabilities of a typical central processing unit (CPU). However, a DLA can be configured with sufficient logic units to process the input data provided to an artificial neural network (ANN) and produce the output of the ANN according to a set of instructions generated for the DLA. Therefore, a DLA can perform ANN computations with minimal or no assistance from a CPU or another processor. Optionally, a conventional general-purpose processor can also be configured as part of the DLA to perform operations that cannot be efficiently implemented using the DLA's vector / matrix processing units, and / or operations that cannot be performed by the DLA's vector / matrix processing units.

[0033] Typical artificial neural networks (ANNs) can be described / specified in a standard format (e.g., Open Neural Network Exchange (ONNX)). A compiler can be used to translate the description of an ANN into a set of instructions for a deep learning accelerator (DLA) to perform computations on the ANN. The compiler can optimize this set of instructions to improve the performance of the DLA when implementing the ANN.

[0034] Deep learning accelerators (DLAs) may have local memory, such as registers, buffers, and / or caches, to store vector / matrix operands and the results of vector / matrix operations. Intermediate results in registers can be pipelined / shifted within the DLA as operands for subsequent vector / matrix operations, reducing time and energy consumption in accessing memory / data and thus accelerating the typical pattern of vector / matrix operations when implementing typical artificial neural networks (ANNs). The capacity of registers, buffers, and / or caches in a DLA is often insufficient to hold the entire dataset used for computations to implement a typical artificial neural network (ANN). Therefore, random access memory coupled to the DLA is configured to provide improved data storage capabilities for implementing typical artificial neural networks (ANNs). For example, the DLA loads data and instructions from random access memory and stores the results back into random access memory.

[0035] The communication bandwidth between the Deep Learning Accelerator (DLA) and Random Access Memory (RAM) is configured to optimize or maximize the utilization of the DLA's computational power. For example, high communication bandwidth between the DLA and RAM can be provided to allow vector / matrix operands to be loaded from RAM into the DLA and stored back into RAM within a time period approximately equal to the DLA's processing time, enabling computation on the vector / matrix operands to be performed. The granularity of the DLA can be configured to increase the ratio between the amount of computation performed by the DLA and the size of the vector / matrix operands, thereby reducing data access traffic between the DLA and RAM, which lowers the bandwidth requirements for communication between the DLA and RAM. Therefore, bottlenecks in data / memory access can be reduced or eliminated.

[0036] Some video decoding technologies are based on the analysis of videos using deep learning techniques. Deep learning-based video decoding technologies may include decoding schemes primarily built on deep neural networks (DNNs) and / or decoding tools that use deep neural networks (DNNs) to generate models, predictions, parameters, and / or options for use with conventional decoding schemes and / or conventional decoding tools.

[0037] For example, deep neural networks (DNNs) can be used to compute / estimate pixel probability models to drive deep learning-based decoding schemes. DNNs can be used to perform or assist in intra-image prediction, inter-image prediction, cross-channel prediction, probability distribution prediction, transformation, post-loop filtering, intra-loop filtering, downsampling, upsampling, encoding optimization, etc. Prediction / optimization can be used in conjunction with conventional decoding schemes and / or conventional decoding tools.

[0038] At least one embodiment disclosed herein includes a surveillance camera configured with a deep learning accelerator (DLA) and random access memory to facilitate intelligent video / image processing using artificial neural networks (ANNs). The deep learning accelerator (DLA) can be provided via removable media.

[0039] For example, existing surveillance cameras may not have a deep learning accelerator (DLA), but they do have slots configured to receive removable media to record video generated by the surveillance camera. Such slots can be configured as part of the surveillance camera's memory card reader; and such removable media can be in the form of a memory card. The memory card can have the physical size and interface of a Secure Digital (SD) card, MicroSD card, Flash Memory (CF) card, Memory Stick card, etc.

[0040] The alternative removable media can be configured with a deep learning accelerator (DLA) and random access memory to autonomously perform artificial neural network (ANN) computations using a model of the artificial neural network (ANN) stored in the removable media. After the alternative removable media is inserted into the slot in the reader of the surveillance camera, the surveillance camera can store video / image data in the alternative removable media in the same way as storing video / image data in regular removable media.

[0041] In response to the video / image data being stored in the alternative removable media disclosed herein, the Deep Learning Accelerator (DLA) automatically analyzes the video / image data to produce video analysis and / or inference results.

[0042] For example, video analysis and / or inference results can be used to compress video / image data using deep learning-based video decoding techniques. Deep learning-based video decoding computations can be performed within removable media. Therefore, removable media can autonomously convert / compress input video without assistance from surveillance cameras and / or host systems to which the cameras can be connected.

[0043] For example, after a removable media device receives high-resolution video, the removable media's deep learning accelerator (DLA) performs video analysis to produce highly accurate video analysis. Video analysis may include pixel probability models, intra-image prediction, inter-image prediction, cross-channel prediction, probability distribution prediction, etc. Alternatively or in combination, video analysis may identify transforms, post-loop filtering, intra-loop filtering, downsampling, upsampling, encoding optimization, etc. High-resolution video can be compressed through decoding based on video analysis to produce compressed video as an alternative to high-resolution video for storage in the device. Optionally, a set of configuration settings may be stored in the removable media to control various aspects of the creation and storage of the compressed video. Such configuration settings may include input resolution, storage resolution, storage location, etc.

[0044] Removable media can be memory cards that can be inserted into the slots of a memory card reader in a host computer system. Memory cards can be configured with the physical size and interface of Security Digital (SD) cards, MicroSD cards, Flash Memory (CF) cards, Memory Stick cards, etc.

[0045] Alternatively, removable media can be a network storage device that can be connected to a surveillance camera and / or a host computer system via a wired or wireless computer network connection.

[0046] Alternatively, removable media can be storage devices that can be connected to surveillance cameras and / or host computer systems via standardized ports, such as Universal Serial Bus (USB) ports.

[0047] Therefore, in general, removable media can be solid-state drives, which can be connected to a host computer system via a port, such as a port for receiving Universal Serial Bus (USB) devices, network storage devices, or memory cards or other types of removable data storage devices.

[0048] For example, removable media can be plug-and-play devices that can be inserted into a port or slot of a running computer system and instantly recognized and configured by the running computer system to operate normally as a storage device without requiring the computer system to be restarted or rebooted.

[0049] For example, removable media can be dynamically added to a running computer system via a wired or wireless computer connection without requiring the computer system to be restarted or rebooted.

[0050] In some simplified embodiments, the removable media may be a data storage device that is non-removable and / or non-hot-pluggable into an operating computer system for configuration for normal operation; and after the storage device is connected to the computer system (e.g., using the computer system's peripheral bus or memory bus), it is necessary to reboot the computer system to configure the storage device for normal operation within the computer system. In some embodiments, the computer system is powered off to allow the storage device to be connected to the computer system.

[0051] For example, removable media may include a host interface for wired or wireless connection to a host computer system using a wireless personal area network (WLAN), wireless local area network (WLAN), wired local area network (LAN), universal serial bus (USB), or similar means. The host computer system may be a surveillance camera, personal computer, mobile computer, smartphone, personal media player, set-top box, Internet of Things (IoT) hub, or server computer.

[0052] After the removable media is connected to the host computer system, the host computer system can send commands to the removable media to store data such as video files. In some cases, video can be streamed to the removable media in either uncompressed or compressed format. A deep learning accelerator in the storage device analyzes the video file / stream to generate video analytics, such as pixel probability models, intra-image prediction, inter-image prediction, cross-channel prediction, probability distribution prediction, transforms, post-loop filtering, intra-loop filtering, downsampling, upsampling, and encoding optimization. The removable media's video encoder decodes / encodes / compresses the video file based on the video analytics, producing a compressed video file stored on the removable media as a replacement for the video file / stream received from the host computer system.

[0053] Optionally, when the host computer system streams video to the removable media, the removable media can perform compression in real time, on the fly. As subsequent portions of the video are streamed to the removable media, the removable media provides previously buffered portions of the video as input to an artificial neural network; and the deep learning accelerator of the removable media performs computations on the artificial neural network to generate video analysis of the buffered portions; and while streaming is in progress, the video encoder uses the video analysis to decode / encode / compress the buffered portions of the video stream on the fly.

[0054] As an alternative to deep learning-based compression or combination of video data, deep learning accelerators (DLAs) with artificial neural networks (ANNs) and random access memory provided in surveillance cameras (e.g., via removable media) can be used to intelligently process the image data captured by the surveillance cameras locally.

[0055] For example, in order to reduce data storage requirements and / or data communication traffic, artificial neural networks (ANNs) implemented in surveillance cameras can transform video / image data into inference results of interest to specific applications, and thus eliminate the need to store and / or transmit most or all of the video / image data captured by the image sensors of the surveillance camera.

[0056] For example, an artificial neural network (ANN) can be trained to identify objects and / or events in images that may be of interest in a surveillance application, and convert the image into a description or recognition of the objects and events appearing in the image captured by a surveillance camera. The description or recognition of objects and events can serve as a high-level overview of the content in the image, which can then be used as input to an application running on a computer system that uses the surveillance camera as an input device. Such a computer system can be a mobile computer, smartphone, personal media player, personal computer, server computer, or server computer cluster. Compared to presenting static images or video images of corresponding objects and events graphically, this high-level overview of the content in an image identified using an artificial neural network uses a significantly smaller amount of data storage capacity and / or data transmission bandwidth.

[0057] Optionally, images of unidentified objects and / or events in abnormal / exceptional situations or scenarios can be transferred to a separate computer system for further processing. For example, a human operator can inspect the images to identify objects and events captured in the images of abnormal / exceptional situations or scenarios; and the results can be used to further train an artificial neural network (ANN) (e.g., using supervised machine learning techniques) to improve its ability to identify objects and events. The further trained ANN can then be loaded into the surveillance camera to further reduce the amount of data that needs to be stored and / or processed outside the surveillance camera.

[0058] Optionally, the surveillance camera can store an image stream converted into an overview or description for a predetermined time period. The image stream can be annotated with an overview or description and identification of portions of the images showing objects and events. Within the predetermined time period, an application receiving the overview or description as input can select objects or events of interest and request the surveillance camera to transmit relevant images of said objects or events.

[0059] In some applications, surveillance cameras are configured (e.g., via removable media) to monitor image streams for objects and / or events that meet predetermined selection criteria. When such objects and / or events are detected in the output of an artificial neural network (ANN), the surveillance camera selects a representative portion from the image stream for storage and / or uploading to a separate computer. Through analysis performed by the ANN, information in the image stream can be compressed into an overview that includes the identification of the object of interest, a representative image of the object, and a description of the activity of the object in the scene captured in the image stream. Information about identified objects that are not of interest can be reduced and / or discarded. Such an overview of the image stream can achieve compression ratios superior to conventional lossy or lossless image / video compression techniques; and in many applications of surveillance cameras, such an overview can be sufficiently useful and / or more useful than the raw image stream generated by the surveillance camera's image sensor.

[0060] For example, instead of transmitting the image stream to a separate computer system for processing, the surveillance camera can provide a description of the content in the image stream to the computer system for processing. Representative images of the identified images can be transmitted along with the description, or provided when the computer system sends a request for such images. Furthermore, the computer system can optionally request the surveillance camera to transmit frames or segments of the image stream containing objects or events of interest.

[0061] The description of the content in an image stream can take the form of identification of objects, events, features, categories, positions in the image, sizes in the image, etc.

[0062] For example, surveillance cameras are equipped with image sensors to monitor a user's health. Integrated circuit devices with deep learning accelerators and random access memory are incorporated into the surveillance camera to process sensor data locally. Images of users determined not to be associated with health problems can be automatically discarded, reducing the need to store large amounts of raw image data and protecting user privacy.

[0063] Surveillance cameras can be customized for specific applications of intelligent monitoring by storing models of artificial neural networks (ANNs) that can be executed by a deep learning accelerator (DLA). For example, a compiler can be used to transform the model based on a description of the ANN; and the model contains the weight / kernel matrix of the ANN and instructions with matrix operands that can be executed by a deep learning accelerator (DLA) to perform computations of the ANN based on the weight / kernel matrix.

[0064] For example, surveillance cameras can be configured in medical facilities to monitor patients' conditions. Different patients may have different health problems, which require monitoring to detect the onset of different symptoms. Therefore, surveillance cameras can be customized for the patients being monitored by installing models of artificial neural networks (ANNs) trained to detect patient-related symptoms.

[0065] After customizing a surveillance camera by storing the model in random access memory to perform computations of an artificial neural network (ANN), raw image data generated by the image sensor in the surveillance camera is provided as input to the artificial neural network (ANN); and the output of the artificial neural network (ANN) can be used to generate warnings, selectively retain and / or report sensor data and / or provide it as the main output of the surveillance camera.

[0066] For example, a surveillance camera may include a wireless transceiver (e.g., a communication device for a wireless personal area network, such as a Bluetooth transceiver, or a communication device for a wireless local area network, such as a Wi-Fi transceiver). The output of an artificial neural network (ANN) can be reported to a standalone computer system, such as a smartphone, personal media player, mobile computer, personal computer, Internet of Things (IoT) hub, and / or server computer, via a wireless connection.

[0067] Alternatively, the surveillance camera may have a port for a wired connection to a standalone computer system to report the output of an artificial neural network (ANN) over a period of time or download the output stored in the surveillance camera.

[0068] For example, an artificial neural network (ANN) can be trained to classify whether an image is of interest to a specific application. The output of the ANN can be used to selectively store image data for subsequent analysis and / or downloading.

[0069] For example, a surveillance camera can provide intelligent output generated locally by an artificial neural network (ANN) to a computer system without requiring assistance from the computer system. For example, a surveillance camera can be used to monitor health-related events and generate alerts when such events are detected. For example, a surveillance camera can be used to monitor indications related to the diagnosis of health problems and record the occurrence and / or associated data of such indications for further analysis. For example, a surveillance camera can be used to detect a user's fall and generate a call for help when a fall is detected. For example, a surveillance camera can be used to detect the appearance of objects around the user and provide object identification for further processing. For example, a surveillance camera can be used to detect a user's movements and provide the identification of said movements to a standalone computer (e.g., a smartphone, game console, personal media player, personal computer, set-top box) to control operations on the standalone computer.

[0070] A random access memory (RAM) coupled to a deep learning accelerator may include a portion configured to store the input to an artificial neural network (ANN) and another portion configured to store the output from the ANN. Input video data received from a host computer system may be stored cyclically in the input portion of the RAM. Therefore, input video data from the most recent period of the buffered video stream in the storage device can be found in the input portion of the RAM for analysis by the deep learning accelerator according to the ANN and for encoding by a video encoder. The deep learning accelerator (DLA) can convert the input video and audio data in the input portion into video analysis and / or inference results stored in the output portion of the RAM in real time. The video encoder may be configured to compress the input video data stored in the input portion of the RAM using the video analysis and / or inference results stored in the output portion of the RAM. The compressed video may contain a description of the content in the video as identified by the ANN and may be stored in the RAM or another storage medium according to a set of configuration parameters.

[0071] For example, the input video data stream to an artificial neural network (ANN) can be analyzed by the ANN to identify segments associated with different scenes depicted in the video stream. Each video segment can be configured to be compressed as a unit. The length of a video segment can be limited by a predetermined number of frames. Furthermore, each video segment can be provided to the ANN as a set of inputs during a time slot. While the deep learning accelerator (DLA) computes video analysis based on a set of current inputs, the next video segment can be stored in random access memory as the next input to the ANN; and simultaneously, the output generated for the previous video segment can be retrieved from the random access memory for decoding / encoding / compressing the previous video segment. The input area of ​​the random access memory can accommodate the storage of multiple video segments simultaneously; and after the previous video segment has been encoded and stored, a new video segment can be stored / streamed to the area storing the previous video segment.

[0072] Therefore, according to the embodiments disclosed herein, the task of compressing video streams using artificial neural networks (ANNs) can be added to a surveillance camera by inserting removable media. The computation of the artificial neural network (ANN) can be performed locally on the removable media to reduce data storage requirements and / or reduce the computational workload on the surveillance camera and / or the host computer system using the surveillance camera.

[0073] Furthermore, neuromorphic memories can be used to perform matrix / vector multiplication and summation computations to reduce the power consumption of deep learning accelerators (DLAs).

[0074] For example, a neuromorphic memory can be implemented using a lattice array of memristors configured to perform multiply-accumulate (MAC) operations via analog circuitry. In the lattice array, currents passing through a set of memristors across word lines to bit lines are added in the bit lines, corresponding to an accumulation operation. The current corresponds to the multiplication of the voltage applied to the word line and a parameter associated with the resistance of the memristor, corresponding to a multiplication operation. The current in the bit lines can be compared to a threshold to determine whether the neuron represented by the bit line is activated under the current input. The memristor array can be individually connected to the bit lines and programmed to have thresholds corresponding to the activation level threshold of the neuron. Current detectors can be configured for each memristor connected to the output of the bit lines to determine whether the current level in the bit line corresponds to a level exceeding the memristor's threshold.

[0075] Figure 1 An integrated circuit device (101) with a configured deep learning accelerator (103) and random access memory (105) is shown according to one embodiment.

[0076] exist Figure 1The deep learning accelerator (103) includes a processing unit (111), a control unit (113), and local memory (115). When vector and matrix operands are in local memory (115), the control unit (113) can use the processing unit (111) to perform vector and matrix operations according to instructions. In addition, the control unit (113) can load instructions and operands from random access memory (105) via a memory interface (117) and a high-speed / bandwidth connection (119).

[0077] The integrated circuit device (101) is configured to be enclosed within an integrated circuit package using the pins or contacts of a memory controller interface (107).

[0078] The memory controller interface (107) is configured to support standard memory access protocols, enabling the integrated circuit device (101) to behave as a typical memory controller in the same way as a conventional random access memory device without a deep learning accelerator (DLA) (103). For example, a memory controller external to the integrated circuit device (101) can access the random access memory (105) in the integrated circuit device (101) via the memory controller interface (107) using standard memory access protocols.

[0079] The integrated circuit device (101) is configured with a high-bandwidth connection (119) between a random access memory (105) enclosed within the integrated circuit device (101) and a deep learning accelerator (DLA) (103). The bandwidth of the connection (119) is higher than the bandwidth of the connection (109) between the random access memory (105) and the memory controller interface (107).

[0080] In one embodiment, both the memory controller interface (107) and the memory interface (117) are configured to access the random access memory (105) via the same set of buses or wires. Therefore, the bandwidth used to access the random access memory (105) is shared between the memory interface (117) and the memory controller interface (107). Alternatively, the memory controller interface (107) and the memory interface (117) are configured to access the random access memory (105) via a separate set of buses or wires. Optionally, the random access memory (105) may comprise multiple segments that can be accessed simultaneously via a connection (119). For example, while the memory interface (117) is accessing a segment of the random access memory (105), the memory controller interface (107) may simultaneously access another segment of the random access memory (105). For example, different segments can be configured on different planes / rows of different integrated circuit dies and / or memory cells; and different segments can be accessed in parallel to increase the throughput when accessing random access memory (105). For example, the memory controller interface (107) is configured to access a data cell of a predetermined size at a time; and the memory interface (117) is configured to access multiple data cells at a time, each data cell being the same predetermined size.

[0081] In one embodiment, the random access memory (105) and the integrated circuit device (101) are disposed on different integrated circuit dies disposed within the same integrated circuit package. Furthermore, the random access memory (105) may be disposed on one or more integrated circuit dies that allow simultaneous parallel access to multiple data elements.

[0082] In some implementations, the number of data elements of a vector or matrix that can be accessed in parallel via connection (119) corresponds to the granularity of the deep learning accelerator (DLA) operating on the vector or matrix. For example, when the processing unit (111) can operate on several vector / matrix elements in parallel, connection (119) is configured to load or store the same number or a multiple thereof of elements in parallel via connection (119).

[0083] Optionally, the data access speed of the connection (119) can be configured based on the processing speed of the deep learning accelerator (DLA) (103). For example, after a certain amount of data and instructions have been loaded into the local memory (115), the control unit (113) can use the processing unit (111) to execute instructions to manipulate the data to produce output. During the processing time used to produce the output, the access bandwidth of the connection (119) allows the same amount of data and instructions to be loaded into the local memory (115) for the next operation, and the same amount of output to be stored back into the random access memory (105). For example, while the control unit (113) uses a portion of the local memory (115) to process data and produce output, the memory interface (117) can unload the output of the previous operation from another portion of the local memory (115) into the random access memory (105), and load operand data and instructions into the other portion of the local memory. Therefore, the utilization and performance of the deep learning accelerator (DLA) are not limited or reduced by the bandwidth of the connection (119).

[0084] Random access memory (105) can be used to store model data of the artificial neural network (ANN) and to buffer input data of the artificial neural network (ANN). The model data does not change frequently. The model data may contain output generated by a compiler for a deep learning accelerator (DLA) to implement the artificial neural network (ANN). The model data typically contains matrices used in the description of the artificial neural network (ANN) and instructions generated for the deep learning accelerator (DLA) (103) to perform vector / matrix operations on the artificial neural network (ANN) at a granularity based on the deep learning accelerator (DLA) (103). The instructions operate not only on the vector / matrix operations of the artificial neural network (ANN) but also on the input data of the artificial neural network (ANN).

[0085] In one embodiment, when input data is loaded or updated in random access memory (105), the control unit (113) of the deep learning accelerator (DLA) (103) can automatically execute the instructions of the artificial neural network (ANN) to produce the output of the artificial neural network (ANN). The output is stored in a predefined area in random access memory (105). The deep learning accelerator (DLA) (103) can execute instructions without the assistance of a central processing unit (CPU). Therefore, communication for coordination between the deep learning accelerator (DLA) (103) and a processor (e.g., a central processing unit (CPU)) outside the integrated circuit device (101) can be reduced or eliminated.

[0086] Optionally, the logic circuitry of the deep learning accelerator (DLA) (103) can be implemented via complementary metal-oxide-semiconductor (CMOS). For example, under-array CMOS (CUA) technology of the memory cells of random access memory (105) can be used to implement the logic circuitry of the deep learning accelerator (DLA) (103), including a processing unit (111) and a control unit (113). Alternatively, CMOS technology in the memory cell array of random access memory (105) can be used to implement the logic circuitry of the deep learning accelerator (DLA) (103).

[0087] In some implementations, the deep learning accelerator (DLA) (103) and random access memory (105) may be implemented on a separate integrated circuit die and connected using through-silicon vias (TSVs) to increase the data bandwidth between the deep learning accelerator (DLA) (103) and the random access memory (105). For example, the deep learning accelerator (DLA) (103) may be formed on an integrated circuit die of a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC).

[0088] Alternatively, the deep learning accelerator (DLA) (103) and random access memory (105) can be configured in a separate integrated circuit package and connected via multiple point-to-point connections on a printed circuit board (PCB) for parallel communication, thereby increasing data transmission bandwidth.

[0089] Random access memory (105) can be volatile memory, non-volatile memory, or a combination of volatile and non-volatile memory. Examples of non-volatile memory include flash memory, memory cells formed based on NAND logic gates, NOR logic gates, phase-change memory (PCM), magnetic RAM (MRAM), resistive random access memory, crosspoint memory, and memory devices. Crosspoint memory devices can use transistorless memory elements, each of which has memory cells and selectors stacked together in a column. The column of memory elements is connected via two layers of wires extending in a vertical direction, wherein one layer of wires extending in one direction is above the column of memory elements, and the other layer of wires extends in the other direction and is below the column of memory elements. Each memory element can be selected individually at the intersection of a wire on each of the two layers. Crosspoint memory devices are fast and non-volatile and can be used as a general-purpose memory pool for processing and storage. Other examples of non-volatile memory include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), and electronically erasable programmable read-only memory (EEPROM). Examples of volatile memory include dynamic random access memory (DRAM) and static random access memory (SRAM).

[0090] For example, the non-volatile memory may be configured to implement at least a portion of the random access memory (105). The non-volatile memory in the random access memory (105) can be used to store model data of an artificial neural network (ANN). Therefore, after the integrated circuit device (101) is powered off and restarted, it is not necessary to reload the model data of the artificial neural network (ANN) into the integrated circuit device (101). Furthermore, the non-volatile memory may be programmable / rewritable. Therefore, the model data of the artificial neural network (ANN) in the integrated circuit device (101) can be updated or replaced to implement an updated artificial neural network (ANN) or another artificial neural network (ANN).

[0091] The processing unit (111) of the deep learning accelerator (DLA) (103) may include vector-vector units, matrix-vector units, and / or matrix-matrix units. The following is in conjunction with... Figures 2 to 4 Discuss instances of units configured to perform vector-vector operations, matrix-vector operations, and matrix-matrix operations.

[0092] Figure 2 A processing unit (121) configured to perform matrix-matrix operations according to one embodiment is shown. For example, Figure 2 The matrix-matrix unit (121) can be used as Figure 1One of the processing units (111) of the Deep Learning Accelerator (DLA) (103).

[0093] exist Figure 2 In this configuration, the matrix-matrix unit (121) comprises multiple kernel buffers (131 to 133) and multiple map groups (151 to 153). Each of the map groups (151 to 153) stores a vector of a matrix operand, which has multiple vectors stored in the map groups (151 to 153); and each of the kernel buffers (131 to 133) stores a vector of another matrix operand, which has multiple vectors stored in the kernel buffers (131 to 133). The matrix-matrix unit (121) is configured to perform multiplication and accumulation operations on the elements of the two matrix operands using multiple matrix-vector units (141 to 143) operating in parallel.

[0094] A crossbar switch (123) connects the mapping groups (151 to 153) to the matrix-vector units (141 to 143). Identical matrix operands stored in the mapping groups (151 to 153) are provided to each of the matrix-vector units (141 to 143) via the crossbar switch (123); and the matrix-vector units (141 to 143) receive data elements from the mapping groups (151 to 153) in parallel. Each of the kernel buffers (131 to 133) is connected to the corresponding matrix-vector unit (141 to 143) and provides a vector operand to the corresponding matrix-vector unit. The matrix-vector units (141 to 143) simultaneously operate to compute the operation of multiplying the identical matrix operands stored in the mapping groups (151 to 153) by the corresponding vector stored in the kernel buffers (131 to 133). For example, the matrix-vector unit (141) performs a multiplication operation on matrix operands stored in the map array (151 to 153) and vector operands stored in the kernel buffer (131), while the matrix-vector unit (143) simultaneously performs a multiplication operation on matrix operands stored in the map array (151 to 153) and vector operands stored in the kernel buffer (133).

[0095] exist Figure 2 Each of the matrix-vector units (141 to 143) in the matrix can be like... Figure 3 Implemented as shown.

[0096] Figure 3 A processing unit (141) configured to perform matrix-vector operations according to one embodiment is shown. For example, Figure 3 The matrix-vector unit (141) can be used as Figure 2Any one of the matrix-vector units in the matrix-matrix unit (121).

[0097] exist Figure 3 In the diagram, each of the mapping groups (151 to 153) is arranged in a manner similar to... Figure 2 A vector of matrix operands is stored in a mapping group (151 to 153), wherein the matrix operands have multiple vectors stored in the mapping group (151 to 153). Figure 3 The crossbar switch (123) in the middle provides vectors from the mapping group (151) to the vector-vector units (161 to 163). The same vectors stored in the kernel buffer (131) are provided to the vector-vector units (161 to 163).

[0098] Vector-vector units (161 to 163) operate simultaneously to compute the multiplication of corresponding vector operands stored in map groups (151 to 153) by the same vector operand stored in kernel buffer (131). For example, vector-vector unit (161) performs a multiplication operation on the vector operands stored in map group (151) and the vector operands stored in kernel buffer (131), while vector-vector unit (163) simultaneously performs a multiplication operation on the vector operands stored in map group (153) and the vector operands stored in kernel buffer (131).

[0099] when Figure 3 The matrix-vector unit (141) is implemented in Figure 2 When in the matrix-matrix unit (121), the matrix-vector unit (141) can use the mapping group (151 to 153), the cross switch (123) and the kernel buffer (131) of the matrix-matrix unit (121).

[0100] Figure 3 Each vector-vector unit (161 to 163) in the matrix can be represented as follows: Figure 4 Implemented as shown.

[0101] Figure 4 A processing unit (161) configured to perform vector-vector operations according to one embodiment is shown. For example, Figure 4 The vector-vector unit (161) can be used as Figure 3 Any one of the vector-vector units in the matrix-vector unit (141).

[0102] exist Figure 4In this, the vector-vector unit (161) has multiple multiply-accumulate (MAC) units (171 to 173). Each of the multiply-accumulate (MAC) units (171 to 173) can receive two numbers as operands, perform the multiplication of the two numbers, and add the result of the multiplication to the sum maintained in the multiply-accumulate (MAC) unit.

[0103] Each of the vector buffers (181 and 183) stores a list of numbers. A pair of numbers, each from one of the vector buffers (181 and 183), can be provided as input to each of the multiply-accumulate (MAC) units (171 to 173). The multiply-accumulate (MAC) units (171 to 173) can receive multiple pairs of numbers from the vector buffers (181 and 183) in parallel and perform the multiply-accumulate (MAC) operation in parallel. The outputs from the multiply-accumulate (MAC) units (171 to 173) are stored in a shift register (175); and the accumulator (177) calculates the sum of the results in the shift register (175).

[0104] when Figure 4 The vector-vector unit (161) is implemented in Figure 3 When in the matrix-vector unit (141), the vector-vector unit (161) can use a mapping group (e.g., 151 or 153) as a vector buffer (181) and use the kernel buffer (131) of the matrix-vector unit (141) as another vector buffer (183).

[0105] Vector buffers (181 and 183) may have the same length to store the same number / count of data elements. The length may be equal to or a multiple of the count of the multiply-accumulate (MAC) units (171 to 173) in the vector-vector unit (161). When the length of the vector buffers (181 and 183) is a multiple of the count of the multiply-accumulate (MAC) units (171 to 173), input pairs equal to the number of counts of the multiply-accumulate (MAC) units (171 to 173) may be provided as input from the vector buffers (181 and 183) to the multiply-accumulate (MAC) units (171 to 173) in each iteration; and the vector buffers (181 and 183) feed their elements into the multiply-accumulate (MAC) units (171 to 173) over multiple iterations.

[0106] In one embodiment, the communication bandwidth of the connection (119) between the deep learning accelerator (DLA) (103) and the random access memory (105) is sufficient for the matrix-matrix unit (121) to use a portion of the random access memory (105) as a map group (151 to 153) and a kernel buffer (131 to 133).

[0107] In another embodiment, the mapping groups (151 to 153) and kernel buffers (131 to 133) are implemented in a portion of the local memory (115) of the deep learning accelerator (DLA) (103). While the matrix-matrix unit (121) is performing computations using the mapping groups (151 to 153) and kernel buffers (131 to 133) implemented in different portions of the local memory (115) of the deep learning accelerator (DLA) (103), the communication bandwidth of the connection (119) between the deep learning accelerator (DLA) (103) and random access memory (105) is sufficient to load the matrix operands of the next operating cycle of the matrix-matrix unit (121) into another portion of the local memory (115).

[0108] Figure 5 A deep learning accelerator and random access memory, configured according to one embodiment, are shown to be autonomously applied to the input of a trained artificial neural network.

[0109] Artificial neural networks (ANNs) (201) trained by machine learning (e.g., deep learning) can be described using standard formats (e.g., Open Neural Network Exchange (ONNX)). Describing a trained ANN (201) using standard formats allows for the identification of artificial neurons and their connectivity properties.

[0110] exist Figure 5 In this process, the deep learning accelerator (DLA) compiler (203) transforms the trained ANN (201) by generating instructions (205) for the deep learning accelerator (DLA) (103) and a matrix (207) corresponding to the properties of the artificial neuron and its connectivity. The instructions (205) and matrix (207) generated by the DLA compiler (203) based on the trained ANN (201) can be stored in the random access memory (105) for the deep learning accelerator (DLA) (103).

[0111] For example, random access memory (105) and deep learning accelerator (DLA) (103) can be used as follows: Figure 1 The integrated circuit device (101) is connected via a high-bandwidth connection (119). Figure 5 Autonomous computation based on instructions (205) and matrices (207) can be implemented in Figure 1 In the integrated circuit device (101). Alternatively, the random access memory (105) and the deep learning accelerator (DLA) (103) may be configured on a printed circuit board having multiple point-to-point serial buses with parallel extensions for connection (119).

[0112] exist Figure 5In the process, after the results of the DLA compiler (203) are stored in random access memory (105), the application of a trained ANN (201) to process the input (211) to the trained ANN (201) to produce the corresponding output (213) of the trained ANN (213) can be triggered by the presence of the input (211) in random access memory (105) or by another indication provided in random access memory (105).

[0113] In response, the deep learning accelerator (DLA) (103) executes instructions (205) to combine the input (211) and the matrix (207). The execution of instructions (205) may involve generating a mapping matrix by a grouping (151 to 153) of one or more matrix-matrix units (e.g., 121) of the deep learning accelerator (DLA) (103).

[0114] In some embodiments, the input to the ANN (211) is in the form of an initial mapping matrix. A portion of the initial mapping matrix can be retrieved from random access memory (105) as a matrix operand stored in a mapping array (151 to 153) of matrix-matrix units (121). Alternatively, the DLA instructions (205) also include instructions for the deep learning accelerator (DLA) (103) to generate the initial mapping matrix based on the input (211).

[0115] According to the DLA instruction (205), the deep learning accelerator (DLA) (103) loads matrix operands into the kernel buffers (131 to 133) and map groups (151 to 153) of its matrix-matrix unit (121). The matrix-matrix unit (121) performs matrix computations on the matrix operands. For example, the DLA instruction (205) decomposes the matrix computations of the trained ANN (201) according to the computational granularity of the deep learning accelerator (DLA) (103) (e.g., the size / dimension of the matrix loaded in the matrix-matrix unit (121) as a matrix operand), and applies the input feature maps to the kernel of one layer of artificial neurons to produce outputs as inputs for the next layer of artificial neurons.

[0116] After the computation of the trained ANN (201) executed according to the instruction (205) is completed, the deep learning accelerator (DLA) (103) immediately stores the output (213) of the ANN (201) in a predefined location in random access memory (105) or in a location provided in random access memory (105) as specified in the instruction to trigger the computation.

[0117] when Figure 5 The technology implemented in Figure 1When the input (211) is in the integrated circuit device (101), an external device connected to the memory controller interface (107) can write the input (211) into the random access memory (105) and trigger the deep learning accelerator (DLA) (103) to apply the input (211) to the autonomous computation of the trained ANN (201). After a certain time period, the output (213) is available in the random access memory (105); and the external device can read the output (213) via the memory controller interface (107) of the integrated circuit device (101).

[0118] For example, a predefined location in random access memory (105) can be configured to store an instruction that triggers the autonomous execution of an instruction (205) by a deep learning accelerator (DLA) (103). The instruction may optionally include the location of an input (211) within random access memory (105). Thus, during the autonomous execution of an instruction (205) that processes the input (211), an external device can retrieve outputs generated during a previous run of the instruction (205) and / or store another set of inputs for the next run of the instruction (205).

[0119] Optionally, another predefined location in the random access memory (105) may be configured to store an indication of the current progress status of the instruction (205). Furthermore, the indication may include a prediction of the completion time of the current execution of the instruction (205) (e.g., estimated based on previous executions of the instruction (205)). Thus, an external device can check the completion status within a suitable time window to retrieve the output (213).

[0120] In some embodiments, the random access memory (105) is configured with sufficient capacity to store multiple sets of inputs (e.g., 211) and outputs (e.g., 213). Each set can be configured in a predetermined time slot / region in the random access memory (105).

[0121] The deep learning accelerator (DLA) (103) can autonomously execute instructions (205) to produce an output (213) from an input (211) based on a matrix (207) stored in random access memory (105) without the assistance of a processor or device located outside the integrated circuit device (101).

[0122] In one embodiment of the method, the computing device (e.g., 101) may access the random access memory (105) of the computing device (e.g., 101) through an interface (107) to a memory controller. The computing device (e.g., 101) may have a processing unit (e.g., 111) configured to perform calculations on at least matrix operands, such as matrix operands stored in a map array (151 to 153) and matrix operands stored in a kernel buffer (131 to 133).

[0123] For example, a computing device (e.g., 101) may be enclosed within an integrated circuit package; and a set of connections may connect an interface (107) to a memory controller located outside the integrated circuit package.

[0124] Instructions (205) that can be executed by a processing unit (e.g., 111) can be written into random access memory (105) via an interface (107).

[0125] The matrix (207) of the artificial neural network (201) can be written into the random access memory (105) via the interface (107). The matrix (207) identifies the properties and / or state of the artificial neural network (201).

[0126] Optionally, at least a portion of the random access memory (105) is non-volatile and configured to store a matrix (207) of instructions (205) and an artificial neural network (201).

[0127] The first input (211) to the artificial neural network can be written to the random access memory (105) via the interface (107).

[0128] An instruction is provided in the random access memory (105) to cause the processing unit (111) to begin execution of the instruction (205). In response to the instruction, the processing unit (111) executes an instruction to combine the first input (211) with the matrix (207) of the artificial neural network (201) to generate a first output (213) from the artificial neural network (201) and stores the first output (213) in the random access memory (105).

[0129] For example, the indication may be the address of a first input (211) in random access memory (105); and the indication may be stored at a predetermined location in random access memory (105) to initiate the execution of an instruction (205) for the input (211) identified by the address. Optionally, the indication may also include an address for storing an output (213).

[0130] The first output (213) can be read from the random access memory (105) via the interface (107).

[0131] For example, a computing device (e.g., 101) may have a deep learning accelerator (103) formed on a first integrated circuit die and random access memory (105) formed on one or more second integrated circuit dies. The connection (119) between the first integrated circuit die and one or more second integrated circuit dies may include through-silicon vias (TSVs) to provide high bandwidth for memory access.

[0132] For example, a compiler (203) can be used to convert the description of the artificial neural network (201) into instructions (205) and matrices (207). The combination of the instructions (205) and matrices (207) stored in random access memory (105) and the deep learning accelerator (103) provides an autonomous implementation of the artificial neural network (201) that can automatically convert the input (211) to the artificial neural network (201) into its output (213).

[0133] For example, during the time period in which the deep learning accelerator (103) executes instructions (205) to generate a first output (213) from a first input (211) according to the matrix (207) of the artificial neural network (201), a second input to the artificial neural network (201) can be written to an alternative location in the random access memory (105) via the interface (107). After the first output (213) is stored in the random access memory (105), an instruction can be provided in the random access memory to cause the deep learning accelerator (103) to resume execution of instructions and generate a second output from the second input.

[0134] During the time period in which the deep learning accelerator (103) executes instructions (205) to generate a second output from a second input according to the matrix (207) of the artificial neural network (201), the first output (213) can be read from the random access memory (105) via the interface (107); and another input can be written to the random access memory in place of the first input (211) or written to a different location. The process can be repeated for a series of inputs.

[0135] A deep learning accelerator (103) may include at least one matrix-matrix unit (121) capable of executing instructions on two matrix operands. The two matrix operands may be a first matrix and a second matrix. Each of the two matrices has multiple vectors. The matrix-matrix unit (121) may include multiple matrix-vector units (141 to 143) configured to operate in parallel. Each of the matrix-vector units (141 to 143) is configured to operate on the first matrix and a vector from the second matrix in parallel with other matrix-vector units. Furthermore, each of the matrix-vector units (141 to 143) may have multiple vector-vector units (161 to 163) configured to operate in parallel. Each of the vector-vector units (161 to 163) is configured to operate on a vector from the first matrix and a common vector operand of the corresponding matrix-vector unit in parallel with other vector-vector units. Furthermore, each of the vector-vector units (161 to 163) may have multiple multiply-accumulate units (171 to 173) configured to operate in parallel.

[0136] In addition to the processing unit (111), the deep learning accelerator (103) may also have local memory (115) and a control unit (113). The control unit (113) may load instructions (205) and matrix operands (e.g., 207) from random access memory (105) for execution by the processing unit (111). The local memory may cache matrix operands used by the matrix-matrix unit. The connection (119) may be configured with bandwidth sufficient to load a set of matrix operands from random access memory (105) into local memory (115) during the time period in which the matrix-matrix unit performs operations on two other matrix operands. Furthermore, during the time period, the bandwidth is sufficient to store the results produced by the matrix-matrix unit (121) in the previous instruction execution from local memory (115) into random access memory (105).

[0137] A deep learning accelerator (103) and random access memory (105) may be configured in a data storage device, such as removable media, to compute video analytics for compressing video data. For example, removable media configured in a surveillance camera may use deep learning-based decoding techniques to compute and use video analytics to compress video captured by its image sensors. Alternatively or in combination, the video analytics may include inferences about the identification of items and / or events of interest, such that the input video is compressed into a description of the content identified in the video by an artificial neural network and, optionally, representative images / fragments of the identified items and / or events.

[0138] For example, a data storage device (e.g., in the form of removable media) can receive video streams from the image sensor of a surveillance camera for recording. In response, the data storage device uses deep learning-based video decoding technology to automatically generate compressed video files that store the video stream and / or describe the identified content in the video stream.

[0139] For example, a set of configuration parameters can be stored in a data storage device to control various aspects of the compression of video files / streams provided to the data storage device. The configuration parameters may include identification of the attributes of the video files / streams to be compressed for storage in the data storage device, such as the format and resolution of the input video file / stream, the resolution of the compressed video file, and the location in the data storage device where the compressed video file is stored.

[0140] After the configuration parameters are stored in the data storage device, the data storage device can use deep learning-based video decoding technology to automatically convert the video files / streams received from the image sensors and / or controllers of the surveillance cameras into compressed video files.

[0141] For example, a deep learning accelerator (103) executes instructions (205) to perform computations on an artificial neural network (201) that receives a video file / stream as input (211) and produces video analysis as output (213); and a video encoder uses the video analysis to compress the video file / stream and produce a compressed video file stored in a data storage device. Subsequently, the computer system can retrieve the compressed video file for playback of the video content, or request the data storage device to decompress the video file for streaming from the data storage device in another format for playback.

[0142] For example, a data storage device in the form of removable media may have an interface to a reader (e.g., a memory card reader) or a port (e.g., a USB port). Alternatively, the data storage device may have a transceiver or host interface for wired or wireless communication connections to a standalone computer system, such as a surveillance camera, personal computer, set-top box, Internet of Things (IoT) hub, server computer, smartphone, personal media player, mobile computer, tablet computer, etc. The computer system may store video in the data storage device, which uses deep learning-based video decoding technology to compress the video for storage on the device.

[0143] The video compression capability of the data storage device can be customized, updated, and / or upgraded by receiving the matrix (207) and instructions (205) of the artificial neural network (201) through a transceiver or host interface and storing them in random access memory (105).

[0144] An artificial neural network (201), implemented via a deep learning accelerator (103) executing instructions (205), provides analysis of the compression of video received in a transceiver or host interface for storage in a data storage device. This compression improves the data storage device's ability to store video content, reduces the communication bandwidth for transmitting video content from the data storage device, and / or reduces the computational workload of the computer system in which the data storage device is installed.

[0145] Alternatively or in combination, the deep learning accelerator (103) and random access memory (105) configured in the data storage device can be configured to convert video / image data into a description of the content identified from the video / image data by an artificial neural network. For example, a surveillance camera with a data storage device can be used to monitor the situation of a user or location and / or the surrounding environment to generate warnings based on the description and / or selectively store image data.

[0146] For example, a data storage device may store a description of representative images / fragments of the identified items and / or events as a video stream version from the image sensor of a surveillance camera. The surveillance camera may provide the video stream version to a computer system to implement a specific monitoring application. The description may provide identification of the identified items or events by an artificial neural network (201). A separate computer system further processes the identification to generate warnings, provide displays, selectively request image data from the surveillance camera, and / or selectively store the image data.

[0147] The surveillance camera can be customized, updated, and / or upgraded by receiving the matrix (207) and instructions (205) of the artificial neural network (201) and storing them in the random access memory (105).

[0148] An artificial neural network (201), implemented via a deep learning accelerator (103) executing instructions (205), transforms the image stream from the image sensor into inference results. This transformation improves the quality of the surveillance camera's output, reduces the communication bandwidth requirements of the connection to the computer system, and / or reduces the computational workload of the computer system.

[0149] Figure 6 This illustrates a technique for upgrading a surveillance camera (231) using removable media (191) according to one embodiment.

[0150] When the removable media A (192) is in the surveillance camera (231), the surveillance camera (231) may not have the ability to perform calculations of the artificial neural network (201).

[0151] However, the surveillance camera (231) is configured to record video in removable media A (192).

[0152] For example, a surveillance camera (231) has an image sensor (233) that can generate a video stream of the scene as seen through the lens (239) of the surveillance camera (231).

[0153] Examples of image sensors include charge-coupled device (CCD) image sensors and complementary metal-oxide-semiconductor (CMOS) image sensors. The image sensor (233) can be formed on an integrated circuit die.

[0154] The controller (235) of the surveillance camera (231) can control the operation of the surveillance camera (231) in response to commands received in the communication interface (237) via a wired or wireless connection (228) to the computer system (223).

[0155] For example, the controller (235) may record a video stream in a removable media (192) in response to a command from the computer system (223), and provide a video stream from the removable media A (192) to the computer system (223) in response to another command from the computer system (223).

[0156] exist Figure 6 In this configuration, removable media B (191) is equipped with a deep learning accelerator (DLA) (103) and random access memory (105). When removable media A (192) can be replaced by removable media B (191), the surveillance camera (231) is upgraded to have the ability to perform computations of an artificial neural network (201).

[0157] Once the removable media B (191) is in the surveillance camera (231), the controller (235) can record a video / image stream generated by the image sensor (233) onto the removable media B (191). In response to recording the video / image stream onto the removable media (191), the deep learning accelerator (103) executes instructions (205) stored in random access memory (105) to perform computations of the artificial neural network (201) using a matrix (207) stored in random access memory (105). The output of the artificial neural network (201) can be used to generate intelligent outputs that can be used by the computer system (223). The outputs may include compressed video decoded using deep learning-based decoding techniques, and / or a description of content identified from the video by the artificial neural network (201).

[0158] For example, Figures 7 to 9 The removable media (191) shown can be used as removable media B (191) to upgrade the surveillance camera (231).

[0159] Figures 7 to 9 A removable medium (191) with a configured deep learning accelerator (103) and random access memory (105) is shown according to some embodiments.

[0160] Figure 7 , 8 Alternatively, the removable media (191) in 9 may include a video encoder (215). The video encoder (215) may be implemented using software executed by a general-purpose processor or using encoder logic circuitry. In some embodiments, encoding-specific logic circuitry is used to accelerate a portion of the encoding operation; and the remainder of the encoding operation is performed by executing a set of software instructions. Therefore, Figure 7 , 8 The video encoder (215) in 9 is not limited to a specific hardware / software implementation.

[0161] The operation of the video encoder (215) is based at least in part on video analysis (102) in the output (213) of the artificial neural network (201) that receives video (104) as input (211).

[0162] Input video (104) can be initially received from the image sensor (233) of the surveillance camera (231) in an uncompressed format, a format compressed using lossless compression technology, or a format compressed using lossy compression technology. Configuration parameters (221) are stored in random access memory (105) to identify the format of the input video (104) and / or the compression operation to be performed on the input video (104).

[0163] When the input video (104) is initially in a compressed format, the removable media (191) uses a corresponding decoder to generate video content in an uncompressed format, where the image attributes of individual pixels in individual frames are explicitly specified. When the video is in a compressed format, at least some pixels of some frames in the video are implicitly specified using data associated with other pixels and / or other frames. The decoder can be implemented in the removable media (191) using hardware and / or software.

[0164] In some implementations, the decoder decompresses the input video (104) on the fly when the input video (104) is stored in random access memory as input (211) to the artificial neural network (201), or when the input video (104) is retrieved from random access memory as input (211) to the artificial neural network (201).

[0165] The deep learning accelerator (103) executes instructions (205) to generate video analysis (102) of the input video (104).

[0166] For example, depending on the deep learning-based decoding technique used with the video encoder (215), video analysis (102) may include pixel probability models, intra-image prediction, inter-image prediction, cross-channel prediction, probability distribution prediction, transformation, post-loop filtering, intra-loop filtering, downsampling, upsampling, coding optimization, etc.

[0167] For example, video analysis (102) may include the identification, classification, or categorization of items, objects, people, features, and / or events displayed in the images of the input video (104). Furthermore, the video analysis (102) in the output (213) of the artificial neural network (201) may include the location and size of objects, people, or features identified from the input video (104). The video encoder (215) may annotate the compressed video (227) using information / descriptions of items, objects, people, features, and / or events displayed in the images of the input video (104).

[0168] For example, video analysis (102) may include representative images / segments of the identified items, objects, people, features and / or events shown in the images of the input video (104).

[0169] In some implementations, the video encoder (215) constructs and produces a compressed video (227) using representative images / segments, and annotates the compressed video (227) with descriptions of identified items, objects, people, features and / or events shown in the images in the input video (104).

[0170] For example, the output (213) of the artificial neural network (201) may contain the recognition of events associated with objects, people or features; and the description decoded into the compressed video (227) may contain the recognition of events.

[0171] Therefore, using video analysis (102) and / or the input video (104), the video encoder (215) produces a compressed video (227). The removable media (191) stores the compressed video (227) as a replacement for the input video (104).

[0172] In one embodiment, the input video (104) is part of a video stream. After the input video (104) is compressed and stored, another portion of the video stream can be stored in the random access memory (105) previously occupied by the input video (104). Therefore, it is not necessary to store the entire uncompressed version of the compressed video (227) in the random access memory (105) simultaneously.

[0173] In some implementations, the removable media (191) is configured to generate compressed video (227) instantaneously and in real time, wherein the video is streamed to the removable media (191). For example, while analyzing the input video (104) to generate video analysis (102), subsequent portions of the video stream are stored in random access memory (105) as the next input to the artificial neural network (201). After receiving the subsequent portions of the video stream in the removable media (191), the video encoder (215) completes the generation of compressed video (227) of the input video (104) using video analysis (102). Thus, the deep learning accelerator (103) can re-execute instructions to analyze the subsequent portions of the video stream for use by the video encoder (215) to compress the subsequent portions, while another portion is received in the removable media (191) and / or random access memory (105).

[0174] exist Figure 7 In this integrated circuit device (101), not only is a deep learning accelerator (103) and random access memory (105), but also a controller (107) and logic circuitry for a host interface (106) for wired or wireless connection to a surveillance camera (231) and / or a computer system (223).

[0175] In some embodiments, the host interface (106) includes an interface to a memory card reader. The removable media (191) can be configured as a memory card.

[0176] In some embodiments, the host interface (106) includes an interface to a plug-and-play port, such as a Universal Serial Bus (USB) port. The removable media (191) can be configured as a USB drive.

[0177] In some embodiments, the host interface (106) includes a transceiver for wired or wireless communication, such as a LAN transceiver, a wireless personal area network transceiver (e.g., a Bluetooth transceiver), or a wireless LAN transceiver (e.g., a Wi-Fi transceiver). The removable media (191) can be configured as a network storage drive.

[0178] For example, the controller (107) and the host interface (106) may be formed on an integrated circuit die stacked on and connected to the integrated circuit die of the random access memory (105). Thus, video data received in the host interface (106) may be stored by the controller (107) in the random access memory (105) as input (211) to an artificial neural network (201) implemented using a deep learning accelerator (103).

[0179] Figure 7The integrated circuit device (101) has a controller (107) configured to control the operation of a host interface (106), a video encoder (215), and / or a deep learning accelerator (103).

[0180] The controller (107) can be implemented, for example, using a microcontroller or sequencer that controls the timing of loading the input video (104) into the random access memory (105) and generating the compressed video (227) from the input video (104) and video analysis (102).

[0181] Optionally, the controller (107) may use a microprocessor-implemented firmware running an application stored in random access memory (105) to coordinate the operation between the video encoder (215), random access memory (105), deep learning accelerator (103), and / or host interface (106).

[0182] After storing a set of frames of the input video (104) into random access memory (105) as input (211) to the artificial neural network (201), the controller (107) can cause the deep learning accelerator (103) to execute instructions (205) and produce video analysis (102) as output (213) of the artificial neural network (201).

[0183] For example, the controller (107) can instruct the deep learning accelerator (103) to execute a start instruction (205) by writing the address of input (211) to a predefined location in random access memory (105). When the deep learning accelerator (103) is idle, it can periodically read the address stored in the predefined location in random access memory (105). When a new and / or valid address is retrieved from the predefined location, the deep learning accelerator (103) starts the execution of the start instruction (205). Optionally, after the execution of the start instruction (205), the deep learning accelerator (103) can optionally clear, erase, or invalidate the address previously stored in the predefined location in random access memory (105).

[0184] Alternatively, the controller (107) is configured to send signals or messages to the deep learning accelerator (103) to instruct the deep learning accelerator (103) to execute instructions (205). Signals or messages can be transmitted from the controller (107) to the deep learning accelerator (103) using a direct connection to a memory cell that does not pass through the random access memory (105).

[0185] In some implementations, the controller (107) and the deep learning accelerator (103) have independent connections (109 and 119) to the random access memory (105). When the controller (107) and the deep learning accelerator (103) do not access the same block or address of the random access memory (105), the connections (109 and 119) can be used in parallel by the controller (107) and the deep learning accelerator (103) to access different portions of the random access memory (105) simultaneously.

[0186] In other embodiments, the control unit (113) and the controller (107) may share at least a portion of their circuitry in the deep learning accelerator (103) and access the random access memory (105) using the same memory interface (117).

[0187] A portion of the processing unit (111) may be implemented using a neuromorphic memory (225). For example, the neuromorphic memory (225) may comprise a lattice array of memristors configured to perform multiply-accumulate (MAC) operations via analog circuitry. For example, the multiply-accumulate unit (e.g., 171 or 173) in the vector-vector unit (e.g., 161) of a deep learning accelerator (103) may be implemented using a lattice array of memristors. The memristors may be connected in an array having word lines and bit lines configured to address the memristors as memory cells. Typically, a memristor is connected to one word line and one bit line in the array. Currents passing through a set of memristors through the word lines and bit lines in the lattice array are added in the bit lines, corresponding to an accumulation operation. The current corresponds to the multiplication of a voltage applied to the word line and a parameter associated with the resistance of the memristor, corresponding to a multiplication operation. The current in the bit line may be compared to a threshold to determine whether the neuron represented by the bit line is activated under a current input. The memristor array can be individually connected to the bit line and programmed to have thresholds corresponding to the activation level thresholds of neurons. Current detectors can be configured for each memristor connected to the output of the bit line to determine whether the current level in the bit line corresponds to a level exceeding the memristor's threshold. The neuromorphic memory (225) can perform multiply-accumulate (MAC) operations in a manner similar to how a memory device reads an array of memory cells, thus offering low energy cost and high computational speed.

[0188] Through connection (108), controller (107) operates. Figure 7 The integrated circuit device (101) has a host interface (106) for communicating with a surveillance camera (231) and / or a standalone computer system (223) via a wired or wireless connection.

[0189] For example, the host interface (106) can be configured to communicate according to the communication protocols of a memory card interface, Universal Serial Bus (USB), Peripheral Component Interconnect (PCI) bus, PCIe bus, local area network, peripheral bus, Mobile Industry Processor Interface (MIPI), wireless personal area network or wireless local area network, or Internet of Things (IoT). For example, the host interface (106) can be formed on a radio frequency (RF) complementary metal-oxide-semiconductor (CMOS) integrated circuit chip.

[0190] The host interface (106) may be enabled by removable media (191) to receive data and / or instructions, such as configuration parameters (221), matrices (207), and instructions (205) of an artificial neural network (201), from a surveillance camera (231) and / or a computer system (223). The host interface (106) may also be enabled by removable media (191) to provide compressed video (227) to the computer system (223) or another device.

[0191] In some implementations, the host interface (106) includes an interface for receiving video streams from the surveillance camera (231) for recording, and another interface for communicating directly with the computer system (223) without going through the communication interface (237). Therefore, the computer system (223) can access the random access memory (105) using the additional interface in a manner unsupported by the communication interface (237) of the surveillance camera (231). For example, the computer system (223) can use the additional interface to update the matrix (207), DLA instructions (205), configuration parameters (221), and / or video encoder (215) in the random access memory (105) without going through the communication interface (237) of the surveillance camera (231). For example, the computer system (223) can use the additional interface to access a portion of the compressed video (227), input video (104), and / or video analytics (102).

[0192] For example, the computer system (223) may request removable media (191) and / or a surveillance camera (231) to stream compressed video (227) to the computer system (223) or another device for playback. Optionally, the controller (107) may decompress the compressed video (227) and stream the video at the resolution and / or format identified in the configuration parameters (221).

[0193] In some implementations, the control unit (113) of the deep learning accelerator (103) may include a controller (107); and the logic circuitry of the host interface (106) may be implemented on the integrated circuit die of the deep learning accelerator (103), such as Figure 8 As shown in the image.

[0194] exist Figure 8 In this configuration, a deep learning accelerator (103) is mounted on an integrated circuit die; and a random access memory (105) is mounted on one or more integrated circuit dies. The control unit (113) controls not only the execution of instructions (205) of the artificial neural network (201), but also the communication between the host interface (106) and the surveillance camera (231) and / or the computer system (223), as well as the operation of the video encoder (215).

[0195] For example, the control unit (113) controls the host interface (106) to receive video streams from the surveillance camera (231) and store the input video (104) into the random access memory (105) via a high-bandwidth connection (119).

[0196] In some implementations, a portion of the video encoder (215) is implemented using logic circuitry enclosed in an integrated circuit package of the integrated circuit device (101). Figure 7 (as shown in the image).

[0197] Alternatively, the logic circuitry of the video encoder (215) may be a separate component (e.g., an integrated circuit chip) outside the integrated circuit package enclosing the deep learning accelerator (103) and the random access memory (105).

[0198] For example, the integrated circuit chip of the video encoder (215) and the integrated circuit device (101) having a deep learning accelerator (103) and random access memory (105) can be mounted on a printed circuit board configured in a removable medium (191).

[0199] Figure 9 The removable media (191) has a substrate (229) that provides connections between its components, such as a deep learning accelerator (103), random access memory (105), a video encoder (215), a controller (107), and a host interface (106).

[0200] In some embodiments, the substrate (229) includes an integrated circuit die with wires for connecting components. Some components (e.g., integrated circuit dies for random access memory (105), deep learning accelerator (103), controller (107), video encoder (215), and / or host interface (106)) may be connected to the integrated circuit die of the substrate (229) via through-silicon vias (TSVs). Other components may be connected to the substrate (229) via wire bonding, die attachment, or another technology.

[0201] In some embodiments, the substrate (229) further includes a printed circuit board having wires for connecting components and other components, such as power supplies (e.g., batteries), displays, light-emitting diode (LED) indicators, etc.

[0202] In some implementations, the circuitry for the host interface (106), controller (107), and video encoder (215) is integrated into the same integrated circuit chip; and the output of the integrated circuit chip includes a video stream from the surveillance camera (231) to the random access memory (105). In other implementations, the host interface (106) is separate from the integrated circuit chips for the controller (107) and / or the video encoder (215).

[0203] In some implementations, the logic circuitry of the host interface (106) and / or controller (107) is configured on the integrated circuit die of the deep learning accelerator (103) or another integrated circuit die.

[0204] Figure 10 A method implemented in a surveillance camera according to one embodiment is shown. For example, Figure 9 The method can be used in having Figure 7 , Figure 8 or Figure 9 Removable media (191) Figure 6 Implemented in the surveillance cameras.

[0205] At block 301, a removable data storage device (191) is provided having random access memory (105) and at least one processing unit (111) configured to perform matrix operations.

[0206] For example, removable data storage devices can be memory cards, Universal Serial Bus (USB) drives, solid-state drives (SSDs), network storage devices, etc.

[0207] For example, the removable storage media may be a plug-and-play device that can be connected to a computer system (223) for normal operation and can be disconnected from the computer system (223) without restarting the computer system (223).

[0208] For example, at least one processing unit (111) may be formed on an integrated circuit die of a field-programmable gate array (FPGA) or application-specific integrated circuit (ASIC) implementing the deep learning accelerator (103). The deep learning accelerator (103) may include at least one processing unit (111) for matrix instruction execution, local memory (115) for buffering matrix operands and results, a control unit (113) for loading instructions (205) from random access memory (105) for execution, and a memory interface (117) for accessing random access memory (105).

[0209] For example, the integrated circuit package is configured to enclose at least one or more integrated circuit dies of an FPGA or ASIC and random access memory. In some embodiments, the integrated circuit package further encloses a controller and / or host interface (106).

[0210] For example, the random access memory (105) and the deep learning accelerator (103) are formed on a separate integrated circuit die and connected via through-silicon vias (TSVs).

[0211] At box 303, the matrix (207) of the artificial neural network (201) and the instructions (205) executable to perform the computation of the artificial neural network (201) using the matrix (207) are stored in the random access memory (105) of the removable data storage device (191).

[0212] For example, the matrix (207) and instructions (205) may be stored in the removable data storage device (191) before being installed in the surveillance camera (e.g., 231).

[0213] For example, a removable data storage device (191) can be connected to a computer system (223) to store the matrix (207) and instructions (205) when preparing to upgrade the surveillance camera (e.g., 231). To upgrade the surveillance camera (231), existing removable media (e.g., 192) can be removed from the surveillance camera (231); and then, the removable data storage device (191) with a deep learning accelerator (103) and random access memory (105) storing the matrix (207) and instructions (205) can be connected to the surveillance camera (231) as a replacement for the previously installed removable media (192).

[0214] At box 305, in response to the detection of a removable data storage device (191) in the surveillance camera (231), the surveillance camera (231) configures the removable data storage device (191) as part of the storage capacity of the surveillance camera (231).

[0215] For example, a surveillance camera (231) can store a video stream in a removable data storage device (191) in the same way as recording a video stream into a previously installed removable media (192) that does not have a deep learning accelerator (103).

[0216] At box 307, the host interface (106) of the removable data storage device (191) receives a portion of the video stream from the surveillance camera (231) (e.g., 104). The removable media (191) can store the video stream in its random access memory (105) in a cyclic manner, wherein the oldest portion is erased and replaced with the newest portion of the video stream.

[0217] For example, the host interface (106) can be configured to communicate with a host (e.g., a surveillance camera (231), a computer system (223)) according to a protocol of a memory card interface, a universal serial bus (USB), a peripheral component interconnect (PCI) bus, a PCI high speed (PCIe) bus, a peripheral bus, a local area network, a mobile industry processor interface (MIPI), a wireless personal area network, a wireless local area network, or an Internet of Things (IoT) or a combination thereof.

[0218] At box 309, at least one processing unit (111) of the removable data storage device (191) executes instructions (205) to calculate an output (213) from an artificial neural network (201) having a portion of a video stream (e.g., 104) as input (211). The output (213) may contain video analysis (102) determined by the artificial neural network (201) based on the portion of the video stream.

[0219] At box 311, a removable data storage device (191) and / or a surveillance camera (231) may provide input to a computer system (223) connected to the surveillance camera (231) based on video analytics.

[0220] For example, the input provided to the computer system (223) may include compressed video (227) generated based on video analysis (102) and / or portions of the video stream (e.g., 104).

[0221] For example, compressed video (227) can be provided to a computer system (223) via a communication interface (237) of a surveillance camera (231) in a manner similar to how the computer system (223) retrieves video from a previously installed removable media (192) that does not have a deep learning accelerator (103).

[0222] Alternatively, the removable data storage device (191) may include an additional interface for communicating directly with the computer system (223) without going through the existing communication interface (237) of the surveillance camera (231). The additional interface may facilitate access to data (e.g., video analytics (102) and / or input video (104)) in the random access memory (105) that is not supported by the communication interface (237) of the surveillance camera (231).

[0223] The content of the compressed video (227) can be substantially the same as the video stream from the image sensor (233) of the surveillance camera (231). For example, deep learning-based video decoding techniques implemented based on video analytics (102) can be used to compress the video stream and produce the compressed video (227).

[0224] For example, the removable data storage device (191) may include a video encoder configured to decode and generate compressed video (227) based on video analytics (102), the video analytics identifying pixel probability models, intra-picture prediction, inter-picture prediction, cross-channel prediction, probability distribution prediction, transform, post-loop filtering options, intra-loop filtering options, downsampling options, upsampling options, or coding optimization options or combinations thereof.

[0225] Alternatively, the content of compressed video (227) can be selectively generated from the video stream captured by the image sensor (233) of the surveillance camera (231) based on the content identified from the video stream by the artificial neural network (201).

[0226] For example, a video encoder (215) may extract images or segments from a portion (e.g., 104) of a video stream based on video analysis (102), which identifies objects, people, or features identified from the portion (e.g., 104) of the video stream, as well as the location and size of the objects, people, or features. The video encoder (215) may use the images or segments extracted from the portion (e.g., 104) of the video stream to construct a compressed video (227) and thus discard the remaining video content from the video stream from the image sensor (233).

[0227] Optionally, the video encoder (215) may generate a description of the content in a portion (e.g., 104) of the video stream based on video analysis (102), which identifies, classifies, or categorizes objects, people, or features as well as their location and size, by an artificial neural network (201) from the portion (e.g., 104) of the video stream. As described, the video encoder (215) may annotate the compressed video (227).

[0228] In some implementations, the video encoder (215) uses a description of the content of a portion of the video stream (e.g., 104) as a substitute for the compressed video (227) and the portion of the video stream (e.g., 104). For example, removable media (191) may provide a description to the computer system (223) without providing images or clips extracted from the portion of the video stream (e.g., 104) to display objects, people, or features identified by the artificial neural network (201). Optionally, the computer system (223) may optionally request the removable media (191) to provide images or clips extracted from the portion of the video stream (e.g., 104). In some implementations, configuration parameters (221) determine whether the removable data storage device (191) provides compressed video (227) decoded using deep learning-based video decoding technology (with or without a description of content), or compressed video (227) reconstructed from images / fragments of identified objects, people, or features of interest to a particular surveillance application (with or without a description of content).

[0229] In some implementations, the video encoder (215) is implemented at least in part by a controller (107) that executes software instructions.

[0230] Video analysis (102) can be generated for a portion of the video stream (e.g., input video (104)) and used to compress said portion (104) during a time period in which another portion of the video stream is received in the host interface (106). Thus, compression of the video stream is performed in real time during the streaming of the video stream from the image sensor (233) of the surveillance camera (231) computer system (223) to the host interface (106).

[0231] For example, while an artificial neural network (201) analyzes a first portion of a video stream to generate video analysis (102) subsequently used to compress the first portion, a second portion of the video stream is received from the image sensor (233) of the surveillance camera (231) at the host interface (106) and buffered into random access memory (105). Compression of the first portion is completed during the streaming of the second portion. After the second portion is buffered in random access memory (105), the deep learning accelerator (103) executes instructions (205) again to generate video analysis (102) for the second portion, while a third portion is received from the image sensor (233) of the surveillance camera (231) at the host interface (106). Since the first portion has been stored as compressed video, the storage capacity in random access memory (105) previously used to store / buffer the first portion can be released to store / buffer the third portion.

[0232] This disclosure includes the methods described above and the apparatus for performing the methods, a data processing system for performing the methods, and a computer-readable medium containing instructions that, when executed on the data processing system, cause the system to perform the methods.

[0233] A typical data processing system may contain interconnects (such as buses and system core logic) that interconnect microprocessors and memory. Microprocessors are typically coupled to cache memory.

[0234] Interconnectors link the microprocessor and memory together and also connect them to input / output (I / O) devices via I / O controllers. I / O devices may include display devices and / or peripheral devices such as mice, keyboards, modems, network interfaces, printers, scanners, cameras, and other devices known in the art. In one embodiment, when the data processing system is a server system, some of the I / O devices (e.g., printers, scanners, mice, and / or keyboards) are optional.

[0235] Interconnectors may include one or more buses that are interconnected to each other via various bridges, controllers, and / or adapters. In one embodiment, the I / O controller includes a USB adapter for controlling Universal Serial Bus (USB) peripherals and / or an IEEE-1394 bus adapter for controlling IEEE-1394 peripherals.

[0236] Memory may include one or more of the following: read-only memory (ROM), volatile random access memory (RAM), and non-volatile memory, such as hard disk drives, flash memory, etc.

[0237] Volatile RAM is typically implemented as dynamic RAM (DRAM), which requires continuous power to refresh or retain data in memory. Non-volatile memory is typically magnetic hard disk drives, magnetic optical disk drives, optical disk drives (e.g., DVD RAM), or other types of memory systems that retain data even after power is removed from the system. Non-volatile memory can also be random access memory.

[0238] Non-volatile memory can be a local device directly coupled to other components in the data processing system. Alternatively, it can be a non-volatile memory located off-system, such as a network storage device coupled to the data processing system via a network interface, such as a modem or Ethernet interface.

[0239] In this disclosure, some functions and operations are described as being executed by or caused by software code for the sake of simplicity. However, such expressions are also used to specify that the functions are generated by a processor, such as a microprocessor, executing code / instructions.

[0240] Alternatively or in combination, the functions and operations described herein may be implemented using a special-purpose circuit system with or without software instructions, such as using an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA). Embodiments may also be implemented using hardwired circuitry without or in combination with software instructions. Therefore, the technology is not limited to any specific combination of hardware circuitry and software, nor to any particular source of instructions executed by the data processing system.

[0241] While one embodiment may be implemented in a full-featured computer and computer system, various embodiments can be distributed as a variety of computing products and can be applied regardless of the specific type of machine or computer-readable medium actually used to implement the distribution.

[0242] At least some of the disclosed aspects may be embodied, at least in part, in software. That is, the technology can be implemented in computer systems or other data processing systems in response to its processor (e.g., a microprocessor) executing a sequence of instructions contained in memory (e.g., ROM, volatile RAM, non-volatile memory, cache, or remote storage device).

[0243] The routines executed to implement the embodiments can be implemented as an operating system or a specific application, component, program, object, module, or part of a sequence of instructions referred to as a "computer program." A computer program typically contains one or more sets of instructions at various times in various memories and storage devices in a computer, and when read and executed by one or more processors in the computer, the sets of instructions cause the computer to perform necessary operations to perform elements involving various aspects.

[0244] Machine-readable media can be used to store software and data that enable a data processing system to perform various methods when executed. Executable software and data can be stored in various locations, including, for example, ROM, volatile RAM, non-volatile memory, and / or caches. Parts of this software and / or data can be stored in any of these storage devices. Additionally, data and instructions can be obtained from a centralized server or a peer-to-peer network. Different portions of the data and instructions can be obtained at different times and in different communication sessions or within the same communication session from different centralized servers and / or peer-to-peer networks. Data and instructions can be obtained in their entirety before the application is executed. Alternatively, portions of data and instructions can be obtained dynamically and as needed for execution. Therefore, it is not required that all data and instructions be on the machine-readable media at any given time.

[0245] Examples of computer-readable media include, but are not limited to, media of non-transitory recordable and non-recordable types, such as volatile and non-volatile memory devices, read-only memory (ROM), random access memory (RAM), flash memory devices, floppy and other removable disks, disk storage media, optical storage media (e.g., optical disc read-only memory (CDROM), digital versatile disc (DVD), etc.), and others. Computer-readable media can store instructions.

[0246] Instructions can also be embodied in digital and analog communication links for the propagation of electrical, optical, acoustic, or other forms of signals, such as carrier waves, infrared signals, digital signals, etc. However, propagation signals such as carrier waves, infrared signals, digital signals, etc., are not tangible machine-readable media and are not configured to store instructions.

[0247] Generally, machine-readable media includes any mechanism that provides (i.e., stores and / or transmits) information in a form accessible by a machine (e.g., a computer, a network device, a personal digital assistant, a manufacturing tool, any device having one or more sets of processors, etc.).

[0248] In various embodiments, hardwired circuitry can be used in combination with software instructions to implement the technology. Therefore, the technology is not limited to any specific combination of hardware circuitry and software, nor to any particular source of instructions executed by the data processing system.

[0249] The above descriptions and figures are illustrative and should not be construed as limiting. Numerous specific details have been described to provide a thorough understanding. However, in some instances, well-known or conventional details have been omitted to avoid obscuring the description. References to one or more embodiments in this disclosure do not necessarily refer to the same embodiment; and such references imply at least one.

[0250] In the foregoing description, this disclosure has been described with reference to specific exemplary embodiments. It will be apparent that various modifications may be made without departing from the broader spirit and scope set forth in the following claims. Therefore, the description and drawings should be regarded as illustrative rather than restrictive.

Claims

1. An apparatus comprising: A random access memory configured to store first data representing the weights of an artificial neural network and second data representing instructions having matrix operands, the instructions being executable to perform matrix computations of the artificial neural network using the first data representing the weights of the artificial neural network; A host interface configured to present the device as a removable storage medium to a surveillance camera, wherein the surveillance camera is configured to write third data representing a portion of a video stream into the random access memory via the host interface; as well as At least one processing unit coupled to the random access memory and configured to execute the instructions to generate video analysis as the output of the artificial neural network, based at least in part on the first data and the third data stored in the random access memory; The device is configured to provide input data to a computer system connected to the surveillance camera based on the video analysis.

2. The apparatus according to claim 1, further comprising: A video encoder coupled to the random access memory and configured to generate compressed video based on the portion of the video stream and the video analysis; The device is further configured to store the compressed video as a replacement for the portion of the video stream.

3. The apparatus of claim 2, wherein the video analysis comprises the identification of pixel probability models, intra-image prediction, inter-image prediction, cross-channel prediction or probability distribution prediction, transformation, post-loop filtering, intra-loop filtering, downsampling, upsampling or coding optimization or any combination thereof.

4. The apparatus of claim 3, wherein the video analysis further comprises the identification, classification, or categorization of objects, people, or features, and the location and size of the objects, people, or features; and the video encoder is configured to annotate the compressed video based on the identification, classification, or categorization and based on the location and size using a description of the content in the compressed video.

5. The apparatus of claim 4, wherein the video encoder is configured to extract images or segments representing the object, person, or feature from the portion of the video stream based on the video analysis, and to construct the compressed video based on the images or segments extracted from the portion of the video stream.

6. The apparatus of claim 4, wherein the input data to the computer system includes the description of the content in the compressed video.

7. The apparatus of claim 4, wherein the apparatus is configured as a removable storage medium in the form of a solid-state drive, a memory card, a network storage device, or a universal serial bus (USB) drive.

8. The apparatus of claim 7, wherein the host interface is configured to communicate according to a protocol of a memory card interface, a universal serial bus (USB), a peripheral component interconnect (PCI) bus, a PCIe bus, a peripheral bus, a local area network, a mobile industry processor interface (MIPI), a wireless personal area network, a wireless local area network, or the Internet of Things (IoT), or any combination thereof.

9. The apparatus of claim 7, wherein the video analysis is generated for the portion of the video stream and is used to compress the portion during a time period in which another portion of the video stream is received in the host interface; and the compression of the video stream is performed instantaneously during the streaming of the video stream from the surveillance camera to the host interface.

10. The apparatus of claim 9, further comprising: A controller coupled to the random access memory and the host interface, wherein at least a portion of the video encoder is implemented via the controller that executes instructions.

11. The apparatus of claim 9, further comprising: An integrated circuit die implementing a field-programmable gate array (FPGA) or application-specific integrated circuit (ASIC), the deep learning accelerator including the at least one processing unit and a control unit configured to load the instructions from the random access memory for execution.

12. The apparatus of claim 11, wherein the at least one processing unit comprises a matrix-matrix unit configured to operate on two matrix operands of an instruction; The matrix-matrix unit comprises multiple matrix-vector units configured to operate in parallel; Each of the plurality of matrix-vector units comprises a plurality of vector-vector units configured to operate in parallel; Each of the plurality of vector-vector units comprises a plurality of multiply-accumulate units configured to operate in parallel; and Each of the plurality of multiply-accumulate units includes a neuromorphic memory configured to perform multiply-accumulate operations via analog circuitry.

13. The apparatus of claim 12, wherein the random access memory and the deep learning accelerator are formed on a separate integrated circuit die and are connected by through-silicon vias (TSVs); and the apparatus further comprises: An integrated circuit package configured to at least enclose the random access memory and the deep learning accelerator.

14. A method comprising: First data representing the weights of the artificial neural network and second data representing instructions are stored in the random access memory of a removable data storage device, the instructions being executable to perform matrix calculations of the artificial neural network using the first data representing the weights of the artificial neural network; The removable data storage device is configured in the surveillance camera in response to detection of the removable data storage device in the surveillance camera; Third data representing a portion of the video stream is received from the surveillance camera via the host interface of the removable data storage device; The instructions are executed by at least one processing unit of the removable data storage device to calculate an output from the artificial neural network based at least in part on the first data and the third data stored in the random access memory, the output including video analysis determined by the artificial neural network based on the portion of the video stream; as well as The input data is provided to the computer system connected to the surveillance camera based on the video analysis.

15. The method of claim 14, wherein the input data to the computer system comprises compressed video generated based on the video analysis and the portion of the video stream.

16. The method of claim 15, further comprising: The compressed video is decoded based on the video analysis, which identifies pixel probability models, intra-image prediction, inter-image prediction, cross-channel prediction or probability distribution prediction, transformation, post-loop filtering, intra-loop filtering, downsampling, upsampling or coding optimization or any combination thereof.

17. The method of claim 15, further comprising: Based on the video analysis, images or segments are extracted from the portion of the video stream, and the video analysis identifies objects, people or features identified from the portion of the video stream, as well as the position and size of the objects, people or features; The compressed video is constructed using the images or segments extracted from the portion of the video stream.

18. The method of claim 15, further comprising: Based on the video analysis, a description of the content in the portion of the video stream is generated, wherein the video analysis identifies, classifies, or categorizes objects, people, or features identified by the artificial neural network based on the portion of the video stream, as well as the position and size of the objects, people, or features; and The compressed video is annotated according to the description.

19. An apparatus comprising: A memory configured to store matrices of an artificial neural network and to store instructions having matrix operands, the instructions being executable to perform computations of the artificial neural network using the matrices; An interface that can be connected to a surveillance camera to add the memory as at least a portion of the surveillance camera's storage capacity, the surveillance camera being configured to write a portion of a video stream into the memory via the interface; Field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs) have the following characteristics: A memory interface for accessing the memory; as well as At least one processing unit is operable to execute the instructions and generate, using the matrix stored in random access memory, the output of the artificial neural network that receives the portion of the video stream as input, the output including analysis of the portion of the video stream; as well as A video encoder that is operable to use the analysis to produce compressed video as a replacement for the portion of the video stream.

20. The device of claim 19, wherein the device is configured as a removable storage medium for a computer system; and the compressed video includes a description of the content of the portion of the video stream identified in the analysis.

Images