Distributed camera system

Abstract

The present disclosure relates to a distributed camera system. A system having a central station and a plurality of cameras installed at different locations. To search for and locate an item of interest, the central station generates an item model and sends the item model to the cameras. When stored in a camera, the item model causes a logic circuit (e.g., a deep learning accelerator) of the camera to use image data received from an image sensor for an image stored in a memory device of the camera as input to an artificial neural network. The logic circuit performs matrix calculations of the artificial neural network to generate a classification of whether the image is related to the item of interest characterized by the item model. If related, the camera transmits the related image to the central station for further processing to determine a real-time location of the item of interest.

Description

Technical Field

[0001] At least some of the embodiments disclosed herein relate generally to camera systems, and more precisely, but not limited to, techniques for searching and locating items of interest using distributed camera systems. Background Technology

[0002] The surveillance system consists of a group of security cameras installed in various locations. These security cameras can be connected to a storage subsystem to store video images captured by the cameras. The stored video images can be viewed during the investigation of an incident.

[0003] The memory subsystem may include one or more memory devices for storing data. Memory devices may be, for example, non-volatile memory devices and volatile memory devices. Generally, a host system may utilize the memory subsystem to store data at the memory devices and retrieve data from the memory devices. Summary of the Invention

[0004] In one aspect, this disclosure provides an apparatus comprising: an image sensor; a memory unit configured to store first data representing an image captured by the image sensor; logic circuitry configured to perform matrix calculations; a processor; and a network interface, wherein the processor is configured to receive second data representing an item model via the network interface and from a central station, and to store the second data in the memory unit such that the logic circuitry uses the first data as input to perform calculations on an artificial neural network; wherein the output of the artificial neural network is configured to indicate whether the image is associated with a category of interest represented by the item model; and wherein, in response to the output having a first category of interest associated with the image, the processor is configured to transmit a third data, including a report containing the image, to the central station via the network interface.

[0005] In another aspect, this disclosure further provides a system comprising: a central station coupled to a communication network connected to a plurality of cameras respectively mounted at a plurality of locations, wherein each of the plurality of cameras has: a non-volatile memory for storing first data representing an image captured by the respective camera; and logic circuitry configured to perform matrix calculations; wherein the central station is configured to transmit second data representing an item model to the plurality of cameras via the communication network; wherein when the item model is stored in the respective camera, the logic circuitry is configured to perform calculations on a first artificial neural network using the first data as input and to produce an output of the first artificial neural network, the output having a classification of whether the image is related to an item of interest represented by the item model; and wherein in response to the output having a first classification of the image being related to the item of interest, the respective camera is configured to transmit a report containing the image to the central station via the communication network.

[0006] In another aspect, this disclosure further provides a method comprising: communicating via a communication network with a plurality of cameras respectively configured at a plurality of locations, wherein each of the plurality of cameras has: a non-volatile memory for storing first data representing an image captured by the respective camera; and logic circuitry configured to perform matrix calculations; generating an item model by the central station, the item model being configured, when stored in the random access memory of the respective camera, such that the logic circuitry uses the first data as input to perform calculations on a first artificial neural network and produces an output of the first artificial neural network, the output having a classification of whether the image is related to an item of interest represented by the item model; transmitting a second data representing the item model from the central station to the plurality of cameras via the communication network, such that the item model is stored in the random access memory of the respective cameras; and receiving, at the central station and in response to the output having a first classification of the image being related to the item of interest, a report containing the image from the respective camera via the communication network. Attached Figure Description

[0007] The embodiments are illustrated by way of example rather than limitation in the accompanying drawings, in which similar reference numerals indicate similar elements.

[0008] Figure 1 A camera system configured to facilitate real-time search of items of interest, according to one embodiment, is demonstrated.

[0009] Figure 2 This describes a digital camera with a deep learning accelerator according to one embodiment.

[0010] Figure 3 An integrated circuit device is shown, which has a deep learning accelerator and random access memory configured according to one embodiment.

[0011] Figure 4 A processing unit configured to perform matrix-matrix operations according to one embodiment is shown.

[0012] Figure 5 A processing unit configured to perform matrix-vector operations according to one embodiment is shown.

[0013] Figure 6 A processing unit configured to perform vector-vector operations according to one embodiment is shown.

[0014] Figure 7 This demonstrates a technique for controlling image processing in a camera using a project model, according to one embodiment.

[0015] Figure 8 This demonstrates a method for searching and locating items of interest according to one embodiment.

[0016] Figure 9 This describes an example computing system having a memory subsystem according to some embodiments of the present disclosure.

[0017] Figure 10 This describes an integrated circuit memory device configured according to one embodiment.

[0018] Figure 11 This is a block diagram of an example computer system in which embodiments of the present disclosure may be operated. Detailed Implementation

[0019] At least some aspects of this disclosure relate to a camera system for searching and locating items of interest using the computational power of a distributed camera system. Typical cameras in the system are configured with non-volatile memory to record recent images of the surrounding environment and a deep learning accelerator for processing the images according to an item model received from a central station. To search and locate the item of interest, the central station broadcasts the item model to the cameras in the system. Cameras receiving the item model use their deep learning accelerators to process the stored images according to the item model. When a matching classification is determined based on the item model processed against the recent images, the cameras are configured to report the matching and / or corresponding images to the central station.

[0020] Video images from surveillance cameras contain large amounts of data, requiring significant communication bandwidth for transmission and computational power for processing. Searching for items (e.g., vehicles) to determine their location within a city or region can be challenging.

[0021] At least some aspects of this disclosure address the above and other deficiencies and / or challenges by distributing computational tasks across surveillance cameras. To search for the location of an item of interest (e.g., a vehicle), a central station can generate or compile an artificial neural network model of the item. After the central station sends the model to the network of security cameras, each security camera can use its deep learning accelerator (DLA) to process its recorded video images. The item model is configured to instruct the camera's DLA to determine if the most recent clip matches the item model. The matching results can be sent back to the central station for further analysis and / or verification by authorized personnel. This reduces the communication bandwidth requirements for transmitting data from security cameras to the central station; and also reduces the computational power requirements of the central station.

[0022] For example, an integrated circuit device configured in a camera can perform computations on an artificial neural network (ANN) with reduced power consumption and computation time. The integrated circuit device includes a deep learning accelerator (DLA) and random access memory. The deep learning accelerator has a unique read-only and read-write data access pattern with multiple parallel large data block transfers. Therefore, the integrated circuit device can use a heterogeneous memory system architecture to optimize its memory configuration to support the deep learning accelerator, thereby improving performance and power usage.

[0023] Figure 1 A camera system configured to facilitate real-time search of items of interest, according to one embodiment, is demonstrated.

[0024] exist Figure 1 In the system, a network of smart cameras (e.g., 111, ..., 121) is configured to process video images based on an item model 103 received from a central station 101. The imaging processing performed by the cameras (e.g., 111, ..., 121) can facilitate real-time or near real-time searching for the location of the item of interest, as represented by the item model 103.

[0025] Cameras (e.g., 111, ..., 121) can be configured for general monitoring of a geographic area. For example, urban infrastructure may have cameras (e.g., 131, ..., 133) located at various locations. Each camera (e.g., 111 or 121) may have non-volatile memory (e.g., 113 or 123) to store recently captured video. Recent video may be recorded cyclically in the non-volatile memory (e.g., 113 or 123), such that the oldest video image is erased to store the latest video image. Cameras (e.g., 111 or 121) may have a deep learning accelerator (DLA) (e.g., 115 or 125).

[0026] A deep learning accelerator (e.g., 115 or 125) is configured to execute instructions to perform matrix computations of an artificial neural network (ANN). The processing power of the deep learning accelerator (e.g., 115 or 125) can be used to process recent video streams in the non-volatile memory (e.g., 113 or 123) of cameras (e.g., 111 or 121) to at least determine whether the video images are relevant to the item of interest. Therefore, processing at the cameras (e.g., 111 or 121) reduces the transmission of images irrelevant to the item of interest to the central station 101 via the communication network. Distributed processing at cameras 111, ..., 121 also reduces the computational burden at the central station 101 to facilitate real-time or near real-time localization of the item of interest (e.g., vehicles, missing persons, objects, etc.).

[0027] To search for and locate items of interest, central station 101 can generate an item model 103, which is configured to be executed by deep learning accelerators 115, ..., 125 of cameras 111, ..., 121. The item model 103 is transmitted from central station 101 to digital cameras 111, ..., 121 via communication network 105.

[0028] A deep learning accelerator (e.g., 115 or 125) executes item model 103 to process recent video clips stored in non-volatile memory (e.g., 113 or 123) of a corresponding camera (e.g., 111 or 121). For example, when a new video image is stored in non-volatile memory 113, the image is used as input to item model 103 executed by deep learning accelerator 115 to produce a classification output. For example, the execution of item model 103 in deep learning accelerator (e.g., 115) uses an artificial neural network to classify whether the input image in non-volatile memory (e.g., 113) is relevant to an item of interest. Furthermore, if the image is classified as relevant, the execution of item model 103 can further extract a portion of the image representing the item of interest.

[0029] When the execution of item model 103 in the camera (e.g., 111) determines that a video clip stored in non-volatile memory 113 is relevant to the item of interest, the digital camera 111 transmits the video clip to central station 101 for further processing. Central station 101 processes the video clip to improve the confidence level of classifying the video clip as relevant to the item of interest.

[0030] When central station 101 determines that a video clip has captured an item of interest, the location of the item of interest can be estimated based on the location (e.g., 131) of the camera (e.g., 111) that submitted the video clip. Furthermore, central station 101 can analyze the movement of the item captured by the video clip and / or other information to improve the estimation of the real-time location of the item of interest.

[0031] Optionally, when there is a match between a video clip and a project model 103 representing the project of interest, the camera (e.g., 111) may select a portion of the video clip with the best match to reduce the amount of data to be transmitted to the central station 101 for further processing.

[0032] When no match is found, the camera (e.g., 111 or 121) does not transmit its most recent video clip to the central station 101.

[0033] After further processing at central station 101 determines that the image or video initially submitted from camera (e.g., 111) has captured the item of interest, central station 101 may optionally request camera 111 to submit additional images or videos captured during the time period containing the initially submitted image or video. Analysis of the additional images or videos may improve the confidence level for identifying the item of interest and / or determine the movement of the item to improve the estimation of the item's current location.

[0034] Optionally, central station 101 may present video clips or best-matching images from a graphical user interface to authorized personnel for identification. In some implementations, central station 101 includes a more complex artificial neural network than project model 103 to further analyze video or images from a digital camera to eliminate false positives, and then present the results to authorized personnel.

[0035] The item model 103 configured for cameras 111, ..., 121 may be based on an artificial neural network reduced to simplify computation for deep learning accelerators 115, ..., 125. This simplification lowers the confidence level of classifications regarding whether a video or image is related to or matches an item of interest. The central station 101 may have a more accurate model that can definitively identify items from images with a higher confidence level than that produced by cameras 111, ..., 121 based on item model 103. Following definitive identification using the more accurate model, the central station 101 may request additional video images from the camera (e.g., 111) that initially reported a matching image or video. These additional video images can be processed to identify item movement as captured by the camera and to estimate the current location of the item.

[0036] In some cases, during processing time, the item of interest may move out of the field of view of one camera (e.g., 111) and into the field of view of another camera (e.g., 121). Central station 101 can use patterns of the time and location data of the item of interest, such as those recently captured by the camera, to predict or estimate the current location of the item of interest and identify one or more cameras optimally positioned to capture the current view of the item of interest. Therefore, the attention of authorized personnel can be directed to the live view of the selected camera. For example, to reduce processing latency, the live image captured by the selected camera can be streamed to the authorized personnel's monitor for close monitoring and identification.

[0037] Figure 2 This describes a digital camera with a deep learning accelerator according to one embodiment.

[0038] For example, Figure 1 Cameras 111, ..., 121 in the system can be used Figure 2 The digital camera 135 described herein is implemented.

[0039] exist Figure 2 In this device, camera 135 includes lens 151, image sensor 153, microprocessor 155, network interface 157, and memory device 137. Image sensor 153 and / or microprocessor 155 can record images and / or video to memory device 137 via host interface 145.

[0040] The memory device 137 has a host interface 145, a controller 143, a deep learning accelerator 141, and a memory unit 147 for storing the project model 103 and the image 139.

[0041] Camera 135 can be configured to receive project model 103 from central station 101 using network interface 157. After project model 103 is stored in memory device 137, deep learning accelerator 141 can process image 139 received from image sensor 153 as an image received via host interface 145 for storage in memory unit 147. In some embodiments, at least a portion of memory unit 147 is non-volatile memory (e.g., 113 or 123). Therefore, image 139 stored in memory unit 147 can be examined for subsequent investigation of events near camera 135. For example, memory unit 147 may have storage capacity for recording video clips of three minutes, one hour, or longer, such that after an event near the camera is reported, the stored image 139 can be retrieved by central station 101 for preservation and to facilitate investigation of the event.

[0042] In addition, the central station 101 can send the project model 103 to the camera 135 to facilitate the search and location of the project of interest, such as in Figure 1 In the system.

[0043] When project model 103 is stored in memory unit 147, memory device 137 automatically uses image 139 received via host interface 145 to store as input to the artificial neural network in project model 103. The output of the artificial neural network in project model 103 contains an indication of whether the input to the artificial neural network is relevant to the project of interest.

[0044] In response to one or more images received as input to an artificial neural network in project model 103 and their classification as related to a project of interest as represented by project model 103, controller 143 reports a match with microprocessor 155, which is configured to transmit the matched image 139 to central station 101 via network interface 157.

[0045] Deep learning accelerators (e.g., 141, 115, 125) contain a set of programmable hardware computation logic that is specialized and / or optimized to perform parallel vector and / or matrix computations (including, but not limited to, multiplication and summation of vectors and / or matrices).

[0046] In addition, a deep learning accelerator (DLA) may contain one or more arithmetic logic units (ALUs) to perform arithmetic and bitwise operations on integer binary numbers.

[0047] Deep learning accelerators (DLAs) can be programmed via a set of instructions to perform computations on artificial neural networks (ANNs).

[0048] For example, each neuron in an ANN receives a set of inputs. Some of the neuron's inputs can be the outputs of some neurons in the ANN; and some of the neuron's inputs can be the inputs provided to the ANN. The input / output relationship among the neurons in an ANN represents the neuronal connectivity within the ANN.

[0049] For example, each neuron can have a bias, an activation function, and a set of synaptic weights for its input. The activation function can be a step function, a linear function, a log-sigmoid function, etc. Different neurons in an ANN can have different activation functions.

[0050] For example, each neuron can produce a weighted sum of its input and its bias, and then produce an output as a function of the weighted sum, which is calculated using the neuron's activation function.

[0051] The relationship between the inputs and outputs of an ANN is generally defined by the ANN model, which contains data representing the connectivity of neurons in the ANN, as well as the biases, activation functions, and synaptic weights of each neuron. Based on a given ANN model, a computing device can be configured to compute the output of the ANN from a given set of inputs to the ANN.

[0052] For example, the input to an ANN can be generated based on camera input; and the output of the ANN can be the identification of items (e.g., events or objects).

[0053] Generally, supervised methods can be used to train ANNs, where the parameters in the ANN are tuned to minimize or reduce the error between a known output associated with or produced by a given input and a computed output produced by applying the input to the ANN. Examples of supervised learning / training methods include reinforcement learning and learning with error correction.

[0054] Alternatively or in combination, unsupervised methods can be used to train an ANN, where the exact output produced by a given set of inputs is unknown until training is complete. An ANN can be trained to classify items into multiple categories or data points into clusters.

[0055] Multiple training algorithms can be used for complex machine learning / training paradigms.

[0056] Deep learning uses multiple layers of machine learning to progressively extract features from input data. For example, lower layers can be configured to identify edges in an image; and higher layers can be configured to identify captured items in the image, such as faces, objects, events, etc., based on the edges detected by the lower layers. Deep learning can be implemented via artificial neural networks (ANNs), such as deep neural networks, deep belief networks, recurrent neural networks, and / or convolutional neural networks.

[0057] The granularity of a Deep Learning Accelerator (DLA) for vector and matrix operations corresponds to the largest unit of vector / matrix that can be operated on during the execution of an instruction by the DLA. During instructions performing predefined operations on vector / matrix operands, the DLA can operate on the elements of the vector / matrix operands in parallel to reduce execution time and / or energy consumption associated with memory / data access. Operations on vector / matrix operands at the granularity of the DLA can be used as building blocks to perform computations on larger vectors / matrixes.

[0058] The implementation of a typical / practical artificial neural network (ANN) involves vector / matrix operands with a size larger than the operational granularity of a deep learning accelerator (DLA). To implement such an ANN using a DLA, computations involving large-sized vector / matrix operands can be decomposed into computations of vector / matrix operands at the granularity of the DLA. A DLA can be programmed with instructions to perform computations involving large vector / matrix operands. For example, the atomic computational capabilities of a DLA in manipulating vectors and matrices at the granularity of the DLA in response to instructions can be programmed to perform computations within an artificial neural network (ANN).

[0059] In some implementations, a deep learning accelerator (DLA) does not possess some of the logical computational capabilities of a typical central processing unit (CPU). However, a deep learning accelerator (DLA) can be configured with sufficient logic units to process the input data provided to an artificial neural network (ANN) and produce the output of the ANN according to a set of instructions generated for the deep learning accelerator (DLA). Therefore, a deep learning accelerator (DLA) can perform ANN computations with minimal or no assistance from a central processing unit (CPU) or another processor. Optionally, a conventional general-purpose processor can also be configured as a portion of the deep learning accelerator (DLA) to perform operations that cannot be efficiently implemented using the vector / matrix processing unit of the deep learning accelerator (DLA), and / or operations that cannot be performed by the vector / matrix processing unit of the deep learning accelerator (DLA).

[0060] A typical artificial neural network (ANN) can be described / specified in a standard format (e.g., Open Neural Network Exchange (ONNX)). A compiler can be used to translate the description of an ANN into a set of instructions for a deep learning accelerator (DLA) to perform computations on the ANN. The compiler can optimize this set of instructions to improve the performance of the DLA when implementing the ANN.

[0061] Deep learning accelerators (DLAs) may have local memory, such as registers, buffers, and / or caches, configured to store vector / matrix operands and the results of vector / matrix operations. Intermediate results in registers can be pipelined / shifted within the DLA as operands for subsequent vector / matrix operations, reducing time and energy consumption in accessing memory / data and thus accelerating the typical pattern of vector / matrix operations when implementing typical artificial neural networks (ANNs). The capacity of registers, buffers, and / or caches in a DLA is typically insufficient to hold the entire dataset used for computations implementing typical ANNs. Therefore, random access memory coupled to the DLA is configured to provide improved data storage capabilities for implementing typical ANNs. For example, the DLA loads data and instructions from random access memory and stores the results back into random access memory.

[0062] The communication bandwidth between the Deep Learning Accelerator (DLA) and random access memory (RAM) is configured to optimize or maximize the utilization of the DLA's computational power. For example, high communication bandwidth between the DLA and RAM can be provided to allow vector / matrix operands to be loaded from RAM into the DLA and stored back into RAM within a timeframe approximately equal to the DLA's processing time, enabling computation on the vector / matrix operands to be performed. The granularity of the DLA can be configured to increase the ratio between the amount of computation performed by the DLA and the size of the vector / matrix operands, thereby reducing data access traffic between the DLA and RAM. This reduces the bandwidth requirements for communication between the DLA and RAM. Therefore, bottlenecks in data / memory access can be reduced or eliminated.

[0063] Figure 3 An integrated circuit device 201 is shown, which has a deep learning accelerator 203 and a random access memory 205 configured according to one embodiment.

[0064] For example, Figure 3 The integrated circuit device 201 can be used to implement Figure 2 The memory device 137 of the camera 135 described herein, and / or Figure 1 Cameras 111, ..., 121 in the system.

[0065] Figure 3The deep learning accelerator 203 includes a processing unit 211, a control unit 213, and local memory 215. When vector and matrix operands are in local memory 215, the control unit 213 can use the processing unit 211 to perform vector and matrix operations according to instructions. Furthermore, the control unit 213 can load instructions and operands from random access memory 205 via memory interface 217 and high-speed / bandwidth connection 219.

[0066] The integrated circuit device 201 is configured to be enclosed within an integrated circuit package having pins or contacts for a memory controller interface 207.

[0067] The memory controller interface 207 is configured to support standard memory access protocols, such that the integrated circuit device 201 behaves as a typical memory controller in the same way as a conventional random access memory device without the deep learning accelerator 203. For example, a memory controller external to the integrated circuit device 201 can access the random access memory 205 in the integrated circuit device 201 through the memory controller interface 207 using standard memory access protocols.

[0068] The integrated circuit device 201 is connected to a deep learning accelerator 203 via a high-bandwidth connection 219 configured with a random access memory 205 enclosed within the integrated circuit device 201. The bandwidth of connection 219 is higher than the bandwidth of connection 209 between the random access memory 205 and the memory controller interface 207.

[0069] In one embodiment, both memory controller interface 207 and memory interface 217 are configured to access random access memory 205 via the same set of buses or wires. Therefore, the bandwidth for accessing random access memory 205 is shared between memory interface 217 and memory controller interface 207. Alternatively, memory controller interface 207 and memory interface 217 are configured to access random access memory 205 via a separate set of buses or wires. Optionally, random access memory 205 may comprise multiple segments that can be accessed simultaneously via connection 219. For example, while memory interface 217 is accessing a segment of random access memory 205, memory controller interface 207 may simultaneously access another segment of random access memory 205. For example, different segments may be configured on different planes / groups of different integrated circuit dies and / or memory cells; and different segments may be accessed in parallel to increase throughput when accessing random access memory 205. For example, memory controller interface 207 is configured to access a data unit of a predetermined size at a time; and memory interface 217 is configured to access multiple data units at a time, each data unit being the same predetermined size.

[0070] In one embodiment, the random access memory 205 and the integrated circuit device 201 are disposed on different integrated circuit dies, which are disposed within the same integrated circuit package. Furthermore, the random access memory 205 may be disposed on one or more integrated circuit dies that allow simultaneous parallel access to multiple data elements.

[0071] In some implementations, the number of data elements of a vector or matrix that can be accessed in parallel via connection 219 corresponds to the granularity of the deep learning accelerator (DLA) operating on the vector or matrix. For example, when processing unit 211 can operate on several vector / matrix elements in parallel, connection 219 is configured to load or store the same number or a multiple thereof of elements in parallel via connection 219.

[0072] Optionally, the data access speed of connection 219 can be configured based on the processing speed of deep learning accelerator 203. For example, after a certain amount of data and instructions have been loaded into local memory 215, control unit 213 can use processing unit 211 to execute instructions to manipulate the data to produce output. During the processing time used to produce output, the access bandwidth of connection 219 allows the same amount of data and instructions to be loaded into local memory 215 for the next operation, and the same amount of output to be stored back into random access memory 205. For example, while control unit 213 uses a portion of local memory 215 to process data and produce output, memory interface 217 can offload the output of the previous operation from another portion of local memory 215 into random access memory 205, and load operand data and instructions into the other portion of local memory. Therefore, the utilization and performance of the deep learning accelerator (DLA) are not limited or reduced by the bandwidth of connection 219.

[0073] Random access memory 205 can be used to store model data for an artificial neural network (ANN) and to buffer input data for the ANN. The model data is not frequently changed. The model data may contain output generated by a compiler for a deep learning accelerator (DLA) to implement the ANN. The model data typically includes matrices used in the description of the ANN and instructions generated for the deep learning accelerator 203 to perform vector / matrix operations on the ANN at a granularity based on the deep learning accelerator 203. The instructions operate not only on the vector / matrix operations of the ANN but also on the input data of the ANN.

[0074] In one embodiment, when input data is loaded or updated in random access memory 205, the control unit 213 of the deep learning accelerator 203 can automatically execute instructions of the artificial neural network (ANN) to produce the output of the ANN. The output is stored in a predefined area in random access memory 205. The deep learning accelerator 203 can execute instructions without assistance from the central processing unit (CPU). Therefore, communication for coordination between the deep learning accelerator 203 and a processor (e.g., a central processing unit (CPU)) outside the integrated circuit device 201 can be reduced or eliminated.

[0075] Optionally, the logic circuitry of the deep learning accelerator 203 can be implemented via complementary metal-oxide-semiconductor (CMOS). For example, CMOS technology under the array of memory cells (CUA) of random access memory 205 can be used to implement the logic circuitry of the deep learning accelerator 203, including processing unit 211 and control unit 213. Alternatively, CMOS technology in the memory cell array of random access memory 205 can be used to implement the logic circuitry of the deep learning accelerator 203.

[0076] In some implementations, the deep learning accelerator 203 and random access memory 205 may be implemented on a separate integrated circuit die and connected using through-silicon vias (TSVs) to increase the data bandwidth between the deep learning accelerator 203 and the random access memory 205. For example, the deep learning accelerator 203 may be formed on an integrated circuit die of a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC).

[0077] Alternatively, the deep learning accelerator 203 and random access memory 205 can be configured in a separate integrated circuit package and connected via multiple point-to-point connections on a printed circuit board (PCB) for parallel communication, thereby increasing data transmission bandwidth.

[0078] Random access memory 205 can be volatile memory, non-volatile memory, or a combination of volatile and non-volatile memory. Examples of non-volatile memory include flash memory, memory cells formed based on NAND logic gates, NOR logic gates, phase-change memory (PCM), magnetic RAM (MRAM), resistive random access memory, cross-point memory devices, and memory devices. Cross-point memory devices can use transistorless memory elements, each of which has memory cells and selectors stacked together in a column. The column of memory elements is connected via two wiring arrangements laid in a vertical direction, wherein one wiring arrangement in one direction of the layer is above the column of memory elements, and the wiring of the other layer is laid in the other direction and below the column of memory elements. Each memory element can be individually selected at the intersection of a wire on each of the two layers. Cross-point memory devices are fast and non-volatile and can be used as a general-purpose memory pool for processing and storage. Other examples of non-volatile memory include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), and electronically erasable programmable read-only memory (EEPROM). Examples of volatile memory include dynamic random access memory (DRAM) and static random access memory (SRAM).

[0079] For example, non-volatile memory may be configured to implement at least a portion of random access memory 205. The non-volatile memory within random access memory 205 can be used to store model data of an artificial neural network (ANN). Therefore, after the integrated circuit device 201 is powered off and restarted, it is not necessary to reload the model data of the artificial neural network (ANN) into the integrated circuit device 201. Furthermore, the non-volatile memory may be programmable / rewritable. Therefore, the model data of the artificial neural network (ANN) in the integrated circuit device 201 can be updated or replaced to implement an updated artificial neural network (ANN) or another artificial neural network (ANN).

[0080] The processing unit 211 of the deep learning accelerator 203 may include vector-vector units, matrix-vector units, and / or matrix-matrix units. The following section combines... Figures 4 to 6 Discuss instances of units configured to perform vector-vector operations, matrix-vector operations, and matrix-matrix operations.

[0081] Figure 4 A processing unit configured to perform matrix-matrix operations according to one embodiment is shown. For example, Figure 4 The matrix-matrix unit 221 can be used as Figure 3 One of the processing units 211 of the deep learning accelerator 203.

[0082] exist Figure 4 In this configuration, matrix-matrix unit 221 includes multiple kernel buffers 231 to 233 and multiple map groups 251 to 253. Each of map groups 251 to 253 stores a vector containing matrix operands that are stored in multiple map groups 251 to 253 respectively; and each of kernel buffers 231 to 233 stores a vector containing another matrix operand that is stored in multiple kernel buffers 231 to 233 respectively. Matrix-matrix unit 221 is configured to perform multiplication and accumulation operations on the elements of two matrix operands using multiple matrix-vector units 241 to 243 operating in parallel.

[0083] A crossbar 223 connects mapping groups 251 to 253 to matrix-vector units 241 to 243. Via crossbar 223, identical matrix operands stored in mapping groups 251 to 253 are provided to each of matrix-vector units 241 to 243; and matrix-vector units 241 to 243 receive data elements from mapping groups 251 to 253 in parallel. Each kernel buffer 231 to 233 is connected to a corresponding matrix-vector unit 241 to 243 and provides vector operands to the corresponding matrix-vector unit. Matrix-vector units 241 to 243 operate simultaneously to compute the operation of multiplying identical matrix operands stored in mapping groups 251 to 253 by the corresponding vectors stored in kernel buffers 231 to 233. For example, matrix-vector unit 241 performs multiplication on matrix operands stored in mapping groups 251 to 253 and vector operands stored in kernel buffer 231, while matrix-vector unit 243 performs multiplication on matrix operands stored in mapping groups 251 to 253 and vector operands stored in kernel buffer 233 simultaneously.

[0084] Figure 4 Each of the matrix-vector units 241 to 243 in the matrix can be like... Figure 5 The method described herein shall be implemented.

[0085] Figure 5 A processing unit configured to perform matrix-vector operations according to one embodiment is shown. For example, Figure 5 The matrix-vector unit 241 can be used as Figure 4 Any of the matrix-vector units in matrix-matrix unit 221.

[0086] exist Figure 5 In the diagram, each of the mapping groups 251 to 253 is similar to... Figure 4The mapping groups 251 to 253 are used to store a vector with matrix operands that are stored in the mapping groups 251 to 253 respectively. Figure 5 The horizontal bar 223 provides the vectors from the mapping group 251 to the vector-vector units 261 to 263 respectively. The same vectors stored in the kernel buffer 231 are provided to the vector-vector units 261 to 263.

[0087] Vector-vector units 261 to 263 operate simultaneously to compute the multiplication of corresponding vector operands stored in map groups 251 to 253 by the same vector operand stored in kernel buffer 231. For example, vector-vector unit 261 performs multiplication on vector operands stored in map group 251 and vector operands stored in kernel buffer 231, while vector-vector unit 263 performs multiplication on vector operands stored in map group 253 and vector operands stored in kernel buffer 231 simultaneously.

[0088] when Figure 5 The matrix-vector unit 241 in Figure 4 When implemented in the matrix-matrix unit 221, the matrix-vector unit 241 may use the mapping groups 251 to 253 of the matrix-matrix unit 221, the bar 223 and the kernel buffer 231.

[0089] Figure 5 Each of the vector-vector units 261 to 263 in the vector-vector unit can be like... Figure 6 The method described herein shall be implemented.

[0090] Figure 6 A processing unit configured to perform vector-vector operations according to one embodiment is shown. For example, Figure 6 The vector-vector unit 261 can be used as Figure 5 Any vector-vector unit in the matrix-vector unit 241.

[0091] exist Figure 6 In this embodiment, the vector-vector unit 261 has multiple multiply-accumulate units 271 to 273. Each of the multiply-accumulate units 271 to 273 can receive two numbers as operands, perform the multiplication of the two numbers, and add the result of the multiplication to the sum maintained in the multiply-accumulate (MAC) unit.

[0092] Each of vector buffers 281 and 283 stores a list of numbers. A pair of numbers (each number from one of vector buffers 281 and 283) can be provided as input to each of the multiply-accumulate units 271 to 273. The multiply-accumulate units 271 to 273 can receive multiple pairs of numbers from vector buffers 281 and 283 in parallel and perform multiply-accumulate (MAC) operations in parallel. The outputs of the multiply-accumulate units 271 to 273 are stored in shift register 275; and accumulator 277 calculates the sum of the results in shift register 275.

[0093] when Figure 6 Vector-vector unit 261 in Figure 5 When implemented in the matrix-vector unit 241, the vector-vector unit 261 may use a mapping group (e.g., 251 or 253) as a vector buffer 281, and the kernel buffer 231 of the matrix-vector unit 241 as another vector buffer 283.

[0094] Vector buffers 281 and 283 may have the same length to store the same number / count of data elements. The length may be equal to or a multiple of the count of multiply-accumulate units 271 to 273 in vector-vector unit 261. When the lengths of vector buffers 281 and 283 are multiples of the counts of multiply-accumulate units 271 to 273, an input pair equal to the number of multiply-accumulate units 271 to 273 may be provided from vector buffers 281 and 283 in each iteration as input to multiply-accumulate units 271 to 273; and vector buffers 281 and 283 feed their elements to multiply-accumulate units 271 to 273 through multiple iterations.

[0095] In one embodiment, the communication bandwidth of the connection 219 between the deep learning accelerator 203 and the random access memory 205 is sufficient for the matrix-matrix unit 221 to use a portion of the random access memory 205 as mapping groups 251 to 253 and kernel buffers 231 to 233.

[0096] In another embodiment, mapping groups 251 to 253 and kernel buffers 231 to 233 are implemented in a portion of the local memory 215 of the deep learning accelerator 203. The communication bandwidth of the connection 219 between the deep learning accelerator 203 and the random access memory 205 is sufficient to load the matrix operands of the next operation cycle of the matrix-matrix unit 221 into another portion of the local memory 215, while the matrix-matrix unit 221 performs computations in the current operation cycle using the mapping groups 251 to 253 and kernel buffers 231 to 233 implemented in different portions of the local memory 215 of the deep learning accelerator 203.

[0097] Figure 7This demonstrates a technique for controlling image processing in a camera using a project model, according to one embodiment.

[0098] For example, Figure 7 The technology can Figure 1 Implemented in the system for searching and locating items of interest.

[0099] exist Figure 7 In this context, the description 301 of the artificial neural network 309 is provided as input to the compiler 303 of the deep learning accelerator 203, for example... Figure 1 and 2 The deep learning accelerators 141, 115 and / or 125 in the camera.

[0100] Artificial neural network 309 is configured to classify whether an image (e.g., 139) received as input to artificial neural network 309 is relevant to an item of interest. Therefore, artificial neural network 309 contains a representation of the item of interest.

[0101] The compiler 303 generates a project model 103, which includes instructions 305 for the deep learning accelerator 203 to perform matrix calculations, and a matrix 307 representing the kernel and mapping of the artificial neural network 309.

[0102] When instruction 305 and matrix 307 are connected via link 219 (e.g., Figure 3 When the high-bandwidth connection 219 in the deep learning accelerator 203 is stored in the random access memory 205 coupled to the deep learning accelerator 203, the deep learning accelerator 203 can automatically execute instructions 305 to process the input 311 of the artificial neural network 309 and generate the output 313 from the artificial neural network 309.

[0103] When configured in a camera (e.g., 135, 111, or 121), an image 139 or video received from an image sensor 153 used for recording is automatically identified as an input 311 that produces an output 313. The output 313 contains indications of whether the image 139 is relevant to the item of interest, whether the confidence level of the image 139 is relevant to the item, and / or whether the confidence level is higher than a threshold identified in the item model 103.

[0104] Optionally, output 313 may include a segment of image 139, wherein the segment is selected to represent the item of interest.

[0105] Figure 8This document illustrates a method for searching and locating items of interest according to one embodiment. The method may be executed by processing logic, which may include hardware (e.g., processing device, circuitry, dedicated logic, programmable logic, microcode, device hardware, integrated circuits, etc.), software / firmware (e.g., instructions that run or execute on the processing device), or a combination thereof. In some embodiments, the method is at least partially comprised of a central station 101, a controller 143, a microprocessor 155, and / or a deep learning accelerator 203 (e.g., 141, 115, or 125). Figure 9 Controller 409 Figure 10 The processing logic and / or memory device 419 Figure 9 The host system 401's processing device 403 executes the process. Although shown in a specific order or sequence, the order of the processes may be modified unless otherwise specified. Therefore, the illustrated embodiments should be understood as examples only, and the illustrated processes may be executed in different orders, and some processes may be executed in parallel. In addition, one or more processes may be omitted in various embodiments. Therefore, not all processes are required in every embodiment. Other process flows are possible.

[0106] At frame 331, central station 101 communicates via communication network 105 with multiple cameras 111, ..., 121 respectively configured at multiple locations 131, ..., 133. Each corresponding camera (e.g., 135) has first data representing an image 139 captured by camera 135.

[0107] For example, camera 111 has non-volatile memory 113 with storage capacity for recording video of a predetermined length (e.g., 30 minutes, one hour, or longer). When there is a post-event video recording duration, central station 101 may request camera 111 to upload its stored video to the central station for implementation and review.

[0108] At box 333, in order to search for and locate the items of interest, central station 101 generates item model 103.

[0109] For example, project model 103 may include a first artificial neural network 309 configured to classify images in relation to their relevance to the project of interest.

[0110] For example, central station 101 has a second artificial neural network to identify items of interest from images. The second artificial neural network can be scaled down to produce a first artificial neural network 309 for deep learning accelerator 141 to determine the confidence level of an image that captures the item of interest. If the confidence level is above a threshold, then the image can be considered relevant to the item of interest. The second artificial neural network is more accurate than the first artificial neural network 309 in classifying images when affirmatively identifying or recognizing the item of interest; however, the first artificial neural network 309 is sufficient to reject images that are irrelevant to the item of interest, and thus reduces communication with central station 101 and the computational burden on central station 101.

[0111] For example, the central station 101 can use compiler 303 to convert the description 301 of the first artificial neural network 309 into a project model 103, which has instructions 305 for the deep learning accelerator 203 in the camera, as well as the kernel and mapping matrix 307 of the first artificial neural network 309.

[0112] At frame 335, central station 101 transmits a second data representing project model 103 to multiple cameras 111, ..., 121 via communication network 105, so that project model 103 is stored in the random access memory 205 of the respective camera (e.g., 135, 111 or 121).

[0113] For example, project model 103 may include fourth data representing instructions 305 that can be executed by the logic circuitry of deep learning accelerator 203, and fifth data representing matrix 307 of first artificial neural network 309.

[0114] At box 337, in response to the item model 103 being stored in the random access memory 205 of the corresponding camera (e.g., 135, 111, or 121), the deep learning accelerator 203 (e.g., 141, 115, or 125) of the camera (e.g., 135, 111, or 121) performs computation on the first artificial neural network 309. The computation is performed using the first data as input 311 to produce the output 313 of the first artificial neural network 309. The output 313 has a classification of whether the image 139 is related to the item of interest represented by the item model 103.

[0115] For example, the instructions 305 of the item model 103 stored in the memory unit 147 of the memory device 137 of the camera 135 are configured to be automatically executed by the logic circuitry of the deep learning accelerator 141 and / or the controller 143 in response to the host interface 145 receiving first data representing the image 139; and the first data is automatically used as the input 311 of the first artificial neural network to produce its output 313.

[0116] At box 339, in response to output 313 having a first classification of image 139 as being related to the item of interest, third data of the report is transmitted from the camera (e.g., 135, 111, or 121) to the central station 101. The report includes image 139 transmitted from the corresponding camera (e.g., 135, 111, or 121) to the central station via communication network 105.

[0117] For example, if there is a second category output 313 for image 139 that is related to the item of interest, then the camera (e.g., 135, 111, or 121) can skip transmitting image 139 to the central station 101.

[0118] Optionally, when output 313 has a first classification, output 313 further includes segments identified by an artificial neural network for image extraction from image 139. Instead of transmitting image 139 in a report to central station 101, the image segments extracted from image 139 can be provided in the report to central station 101. Image 139 can then be submitted upon request from central station 101.

[0119] At frame 341, central station 101 determines the current location of the item of interest based on image 139.

[0120] For example, central station 101 can use its second artificial neural network, which is more accurate than the first artificial neural network 309, to independently generate a classification of whether the received image 135 is related to the item of interest.

[0121] For example, in response to the second artificial neural network recognizing an image as relevant to the project of interest, the image on the display device of the central station 101 can be presented to authorized personnel.

[0122] For example, central station 101 may request reporting cameras (e.g., 135, 111, or 121) to transmit additional images adjacent to the initial image provided in the report during the capture timing; and central station 101 may analyze the received images to determine the movement of the item of interest, as seen by the reporting cameras (e.g., 135, 111, or 121).

[0123] For example, central station 101 can analyze or detect patterns of movement, time, and location of the item of interest, such as those captured or seen by a first subset of cameras, to obtain an estimate of the current location of the item of interest.

[0124] Based on an estimate of the current location of the project of interest, the central station 101 selects or identifies a second subset of cameras and instructs the second subset of cameras to stream images from the image sensors of the cameras in the second subset to the display device of the central station 101 in real time for presentation to authorized personnel.

[0125] Figure 1 The non-volatile memory (e.g., 113, 123) of the camera (e.g., 111 or 121) Figure 2 The memory device 137 of the digital camera 135 and / or Figure 3 The integrated circuit device 201 may be implemented as a memory subsystem. The memory subsystem may have a programming manager 413 configured to perform operations such as storing the project model 103 in a configuration that causes the deep learning accelerator 141 to perform matrix calculations of the artificial neural network 309 in response to a request received in the host interface 145 to store an image 139 from an image sensor. The programming manager 413 may be further configured to generate reports to the central station 101.

[0126] The following text combines Figure 9 This describes examples of storage devices and memory modules as memory subsystems. Generally, a host system may utilize a memory subsystem that includes one or more components, such as a memory device for storing data. The host system can provide data to be stored in the memory subsystem and can request data to be retrieved from the memory subsystem.

[0127] Figure 9 This description describes an example computing system 400 including a memory subsystem 407 according to some embodiments of the present disclosure. The memory subsystem 407 may include media, such as one or more volatile memory devices (e.g., memory device 417), one or more non-volatile memory devices (e.g., memory device 419), or a combination of such media.

[0128] The memory subsystem 407 may be a storage device, a memory module, or a combination of both. Examples of storage devices include solid-state drives (SSDs), flash drives, universal serial bus (USB) flash drives, embedded multimedia controller (eMMC) drives, universal flash memory (UFS) drives, secure digital cards (SD cards), and hard disk drives (HDDs). Examples of memory modules include dual in-line memory modules (DIMMs), small outline DIMMs (SO-DIMMs), and various types of non-volatile dual in-line memory modules (NVDIMMs).

[0129] The computing system 400 may be a computing device, such as a desktop computer, a laptop computer, a web server, a mobile device, a vehicle (e.g., an airplane, a drone, a train, a car or other means of transport), a device with Internet of Things (IoT) capabilities, an embedded computer (e.g., an embedded computer contained in a vehicle, industrial equipment or a networked business device), or such a computing device containing memory and processing devices.

[0130] The computing system 400 may include a host system 401 coupled to one or more memory subsystems 407. Figure 9 This describes an example of a host system 401 coupled to a memory subsystem 407. As used herein, “coupled to” or “coupled with” generally refers to a connection between components, which can be an indirect or direct communication connection (e.g., without an intermediate component), whether wired or wireless, including connections such as electrical, optical, magnetic, etc.

[0131] Host system 401 may include a processor chipset (e.g., processing device 403) and a software stack executed by the processor chipset. The processor chipset may include one or more cores, one or more cache memories, a memory controller (e.g., controller 405) (e.g., NVDIMM controller), and a storage protocol controller (e.g., PCIe controller, SATA controller). Host system 401 uses, for example, memory subsystem 407 to write data to and read data from memory subsystem 407.

[0132] Host system 401 can be coupled to memory subsystem 407 via a physical host interface. Examples of physical host interfaces include, but are not limited to, Serial Advanced Technology Attachment (SATA) interfaces, Peripheral Component Interconnect High Speed ​​(PCIe) interfaces, Universal Serial Bus (USB) interfaces, Fibre Channel, Serial Attached SCSI (SAS) interfaces, Dual Data Rate (DDR) memory bus interfaces, Small Computer System Interface (SCSI), Dual In-line Memory Module (DIMM) interfaces (e.g., DIMM socket interfaces supporting Dual Data Rate (DDR)), Open NAND Flash Interface (ONFI), Dual Data Rate (DDR) interfaces, Low Power Dual Data Rate (LPDDR) interfaces, or any other interfaces. The physical host interface can be used to transfer data between host system 401 and memory subsystem 407. When memory subsystem 407 is coupled to host system 401 via a PCIe interface, host system 401 can further utilize an NVM High Speed ​​(NVMe) interface to access components (e.g., memory device 419). The physical host interface provides an interface for transmitting control, address, data and other signals between the memory subsystem 407 and the host system 401. Figure 9 The memory subsystem 407 is illustrated as an example. Generally, the host system 401 can access multiple memory subsystems via the same communication connection, multiple separate communication connections, and / or combinations of communication connections.

[0133] The processing device 403 of the host system 401 may be, for example, a microprocessor, a central processing unit (CPU), a processor core, an execution unit, a system-on-a-chip (SoC), etc. In some cases, the controller 405 may be referred to as a memory controller, a memory management unit, and / or an initiator. In one example, the controller 405 controls communication via a bus coupled between the host system 401 and the memory subsystem 407. Generally, the controller 405 may send commands or requests for access to memory devices 419, 417 to the memory subsystem 407. The controller 405 may further include an interface circuitry for communicating with the memory subsystem 407. The interface circuitry may convert responses received from the memory subsystem 407 into information for the host system 401.

[0134] The controller 405 of the host system 401 can communicate with the controller 409 of the memory subsystem 407 to perform operations, such as reading, writing, or erasing data at memory devices 419, 417, and other such operations. In some cases, the controller 405 is integrated within the same package as the processing device 403. In other cases, the controller 405 is packaged separately from the processing device 403. The controller 405 and / or the processing device 403 may include hardware such as one or more integrated circuits (ICs) and / or discrete components, buffer memory, cache memory, or combinations thereof. The controller 405 and / or the processing device 403 may be a microcontroller, a special-purpose logic circuit (e.g., a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), etc.), or another suitable processor.

[0135] Memory devices 419 and 417 may include different types of non-volatile memory components and / or any combination of volatile memory components. Volatile memory devices (e.g., memory device 417) may be, but are not limited to, random access memory (RAM), such as dynamic random access memory (DRAM) and synchronous dynamic random access memory (SDRAM).

[0136] Some examples of non-volatile memory components include NAND flash memory and in-place write memory, such as three-dimensional crosspoint (“3D crosspoint”) memory. Crosspoint arrays of non-volatile memory can be combined with stackable cross-grid data access arrays to perform bit storage based on changes in volume resistance. Furthermore, in contrast to many flash-based memories, crosspoint non-volatile memory can perform in-place write operations, where non-volatile memory cells can be programmed without pre-erasing them. NAND flash memory includes, for example, two-dimensional NAND (2D NAND) and three-dimensional NAND (3D NAND).

[0137] Each of the memory devices 419 may include one or more arrays of memory cells. One type of memory cell, such as a single-level cell (SLC), may store one bit per cell. Other types of memory cells, such as multi-level cell (MLC), three-level cell (TLC), four-level cell (QLC), and five-level cell (PLC), may store multiple bits per cell. In some embodiments, each of the memory devices 419 may include one or more arrays of memory cells, such as SLC, MLC, TLC, QLC, PLC, or any combination of such arrays. In some embodiments, a particular memory device may include an SLC portion, an MLC portion, a TLC portion, a QLC portion, and / or a PLC portion of memory cells. The memory cells of the memory device 419 may be grouped into pages, which may refer to logical cells of the memory device used for storing data. For some types of memory (e.g., NAND), pages may be grouped to form blocks.

[0138] Although non-volatile memory devices, such as 3D crosspoint type and NAND type memory (e.g., 2D NAND, 3D NAND), are described, memory device 419 may be based on any other type of non-volatile memory, such as read-only memory (ROM), phase-change memory (PCM), auto-select memory, other chalcogenide-based memory, ferroelectric transistor random access memory (FeTRAM), ferroelectric random access memory (FeRAM), magnetic random access memory (MRAM), spin-transfer torque (STT)-MRAM, conductive bridged RAM (CBRAM), resistive random access memory (RRAM), oxide-based RRAM (OxRAM), NOR flash memory, and electrically erasable programmable read-only memory (EEPROM).

[0139] The memory subsystem controller 409 (or, for simplicity, controller 409) can communicate with the memory device 419 to perform operations, such as reading, writing, or erasing data at the memory device 419, and other such operations (e.g., in response to commands scheduled on the command bus by controller 405). Controller 409 may include hardware, such as one or more integrated circuits (ICs) and / or discrete components, buffer memories, or combinations thereof. The hardware may include digital circuitry with dedicated (e.g., hard-decoded) logic to perform the operations described herein. Controller 409 may be a microcontroller, dedicated logic circuitry (e.g., field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), etc.), or another suitable processor.

[0140] Controller 409 may include processing device 415 (e.g., processor) configured to execute instructions stored in local memory 411. In the illustrated example, the local memory 411 of controller 409 includes embedded memory configured to store instructions for performing various processes, operations, logic flows, and routines for controlling the operation of memory subsystem 407 (including processing communication between memory subsystem 407 and host system 401).

[0141] In some embodiments, local memory 411 may include memory registers storing memory pointers, retrieved data, etc. Local memory 411 may also include read-only memory (ROM) for storing microcode. Although Figure 9 The instance memory subsystem 407 in the present disclosure is described as including controller 409, but in another embodiment of the present disclosure, memory subsystem 407 does not include controller 409 and may instead rely on external control (e.g., provided by an external host or by a processor or controller separate from the memory subsystem).

[0142] Generally, controller 409 can receive commands or operations from host system 401 and can translate these commands or operations into instructions or appropriate commands to perform the desired access to memory device 419. Controller 409 may handle other operations such as wear leveling, garbage collection, error detection and error correction code (ECC) operations, encryption, caching, and address translation between logical addresses (e.g., logical block addresses (LBAs), namespaces) and physical addresses (e.g., physical block addresses) associated with memory device 419. Controller 409 may further include a host interface circuitry for communicating with host system 401 via a physical host interface. The host interface circuitry can translate commands received from the host system into instructions for accessing memory device 419 and responses associated with memory device 419 into information for host system 401.

[0143] The memory subsystem 407 may also include additional circuitry or components not described. In some embodiments, the memory subsystem 407 may include a cache memory or buffer (e.g., DRAM) and an address circuitry (e.g., row decoder and column decoder) that can receive and decode addresses from the controller 409 to access the memory device 419.

[0144] In some embodiments, memory device 419 includes a local media controller 421 that operates in conjunction with memory subsystem controller 409 to perform operations on one or more memory cells of memory device 419. An external controller (e.g., memory subsystem controller 409) may externally manage memory device 419 (e.g., perform media management operations on memory device 419). In some embodiments, memory device 419 is a managed memory device, which is a native memory device combined with a local controller (e.g., local media controller 421) for media management within the same memory device package. An example of a managed memory device is a managed NAND (MNAND) device.

[0145] Controller 409 and / or memory device 419 may include a programming manager 413 configured to search for and locate items of interest. In some embodiments, controller 409 in memory subsystem 407 and / or controller 421 in memory device 419 may include at least a portion of programming manager 413. In other embodiments, or in combination, controller 405 in host system 401 and / or processing device 403 include at least a portion of programming manager 413. For example, controller 409, controller 405, and / or processing device 403 may include a logic circuitry system implementing programming manager 413. For example, controller 409 or processing device 403 of host system 401 (e.g., a processor) may be configured to execute instructions stored in memory for performing the operations of programming manager 413 described herein. In some embodiments, programming manager 413 is implemented in an integrated circuit chip disposed in memory subsystem 407. In other embodiments, the programming manager 413 may be part of the firmware of the memory subsystem 407, the operating system of the host system 401, the device driver or application, or any combination thereof.

[0146] For example, the programming manager 413 implemented in controller 409 and / or controller 421 can be configured via instructions and / or logic circuits to search for and locate items of interest.

[0147] Figure 10 This describes an integrated circuit memory device configured according to one embodiment. For example, Figure 9 The memory device 419 in the memory subsystem 407 can be used Figure 10 The integrated circuit memory device 419 is used for implementation.

[0148] The integrated circuit memory device 419 may be enclosed in a single integrated circuit package. The integrated circuit memory device 419 includes multiple groups 431, ..., 433 of memory cells that may be formed in one or more integrated circuit dies. A typical memory cell in group 431 (or group 433) may be programmed to store one or more data bits.

[0149] Some memory cells in integrated circuit memory device 419 can be configured to operate together for a specific type of operation. For example, memory cells on an integrated circuit die can be organized into planes, blocks, and pages. A plane contains multiple blocks; a block contains multiple pages; and a page can have multiple strings of memory cells. For example, an integrated circuit die can be the smallest unit capable of independently executing commands or reporting status; the same concurrent operation can be performed in parallel on multiple planes in the integrated circuit die; a block can be the smallest unit for performing an erase operation; and a page can be the smallest unit for performing a data programming operation (writing data into a memory cell). Each string of memory cells is connected to a common bit line; and the control gate of a memory cell at the same location in a string within a block or page is connected to a common word line. Control signals can be applied to the word line and bit line to address individual memory cells.

[0150] Integrated circuit memory device 419 has a communication interface 447 to receive a command with address 437 from controller 409 of memory subsystem 407, retrieve memory data 445 from memory cell identified by memory address 437, and provide at least memory data 445 as a response to the command. Optionally, memory device 419 may decode memory data 445 (e.g., using error correction code (ECC) technology) and provide decoded data as a response to the command. Address decoder 435 of integrated circuit memory device 419 translates address 437 into a control signal to select a group of memory cells in integrated circuit memory device 419; and read / write circuitry 441 of integrated circuit memory device 419 performs an operation to determine memory data 445 stored in the memory cell at address 437.

[0151] The integrated circuit memory device 419 has a set of latches 443 or buffers to temporarily hold memory data 445 when the read / write circuit 441 programs the threshold voltage of the memory cell group (e.g., 431 or 433) to store data, or evaluates the threshold voltage of the memory cell group (e.g., 431 or 433) to retrieve data.

[0152] Figure 11An example machine illustrating computer system 460 is described, within which a set of instructions is executable to cause the machine to perform any one or more of the methods discussed herein. In some embodiments, computer system 460 may correspond to including, coupled to, or utilizing a memory subsystem (e.g., Figure 9 The memory subsystem 407) or can be used to perform operations of the programming manager 413 (e.g., execute instructions to perform operations corresponding to the reference). Figures 1 to 10 The host system (e.g., the described operation of the programming manager 413) Figure 9 (Host system 401). In alternative embodiments, the machine may be connected (e.g., networked) to other machines in a LAN, intranet, extranet, and / or the Internet. The machine may operate as a peer machine in a peer-to-peer (or distributed) network environment or as a server or client machine in a cloud computing infrastructure or environment, or within the capacity of a server or client machine in a client-server network environment.

[0153] The machine may be a personal computer (PC), tablet PC, set-top box (STB), personal digital assistant (PDA), cellular telephone, network appliance, server, network router, switch, or bridge, or any machine capable of (sequentially or otherwise) executing a set of instructions that will take a specified action by said machine. Furthermore, although a single machine is described, it should be understood that the term "machine" also includes any collection of machines that individually or collectively execute a set (or more) of instructions to perform any one or more of the methods discussed herein.

[0154] Example computer system 460 includes processing device 467, main memory 465 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM), static random access memory (SRAM), etc.), and data storage system 473 communicating with each other via bus 471 (which may include multiple buses).

[0155] Processing device 467 may be one or more general-purpose processing devices, such as microprocessors, central processing units, etc. More specifically, processing device 467 may be a Complex Instruction Set Computing (CISC) microprocessor, a Reduced Instruction Set Computing (RISC) microprocessor, a Very Long Instruction Word (VLIW) microprocessor, or a processor implementing other instruction sets, or a processor implementing a combination of instruction sets. Processing device 467 may also be one or more special-purpose processing devices, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), network processors, etc. Processing device 467 is configured to execute instructions 469 for performing the operations and steps discussed herein. Computer system 460 may further include a network interface device 463 for communication via network 461.

[0156] Data storage system 473 may include machine-readable medium 475 (also referred to as computer-readable medium) storing one or more sets of instructions 469 or software embodying any one or more of the methods or functions described herein. Instructions 469 may also reside wholly or at least partially in main memory 465 and / or processing device 467 during execution by computer system 460, the main memory 465 and processing device 467 also constituting machine-readable storage media. Machine-readable medium 475, data storage system 473 and / or main memory 465 may correspond to... Figure 9 The memory subsystem 407.

[0157] In one embodiment, instruction 469 includes implementations corresponding to programming manager 413 (e.g., reference 413). Figures 1 to 10 The described programming manager 413) provides functional instructions. Although the machine-readable medium 475 is shown as a single medium in the exemplary embodiment, the term "machine-readable storage medium" should be considered to include a single medium or multiple media storing one or more sets of instructions. The term "machine-readable storage medium" should also be considered to include any medium capable of storing or encoding a set of instructions executable by a machine and causing the machine to perform any one or more of the methods of this disclosure. Therefore, the term "machine-readable storage medium" should be considered to include, but is not limited to, solid-state memory, optical media, and magnetic media.

[0158] Some parts of the previously described in detail have been presented based on the algorithms and symbolic representations of operations on data bits within computer memory. These algorithms are described and represented in a way that those skilled in the art of data processing can most effectively communicate the essence of their work to others skilled in the art. The algorithms in this paper generally refer to a self-consistent sequence of operations that produce a desired result. An operation is one that requires physical control over a physical quantity. These quantities are usually, but not necessarily, in the form of electrical or magnetic signals that can be stored, combined, compared, and otherwise manipulated. Primarily for general reasons, it has proven convenient to sometimes refer to these signals as bits, values, elements, symbols, characters, terms, numbers, etc.

[0159] However, it should be remembered that all these and similar terms should be associated with appropriate physical quantities and are merely convenient labels for application to those quantities. This disclosure may relate to the actions and processes of a computer system or similar electronic computing device that manipulate and transform data represented as physical (electronic) quantities within the registers and memories of a computer system into other data similarly represented as physical quantities within the computer system's memory or registers or other such information storage systems.

[0160] This disclosure also relates to apparatus for performing the operations described herein. Such apparatus may be specifically constructed for its intended purpose, or may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in a computer. Such computer programs may be stored in computer-readable storage media, such as, but not limited to, any type of disk, including floppy disks, optical disks, CD-ROMs and magneto-optical disks, read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic cards or optical cards, or any type of media suitable for storing electronic instructions, each coupled to a computer system bus.

[0161] The algorithms and displays presented herein are not inherently related to any particular computer or other device. Various general-purpose systems may be used with the programs taught herein, or it may be convenient to construct more specialized devices to perform the methods. Structures for various such systems will be presented as described below. Furthermore, this disclosure is described without reference to any particular programming language. It will be understood that the teachings of this disclosure as described herein can be implemented using various programming languages.

[0162] This disclosure may be provided as a computer program product or software, which may include a machine-readable medium having instructions stored thereon that can be used to program a computer system (or other electronic device) to perform processes according to this disclosure. The machine-readable medium includes any mechanism for storing information in a machine-readable (e.g., computer-readable) form. In some embodiments, the machine-readable (e.g., computer-readable) medium includes machine-readable storage media, such as read-only memory (“ROM”), random access memory (“RAM”), disk storage media, optical storage media, flash memory components, and the like.

[0163] In this description, for simplicity, various functions and operations may be described as being executed or caused by computer instructions. However, those skilled in the art will recognize that such expressions are intended to mean that the functions are produced by one or more controllers or processors (e.g., microprocessors) executing computer instructions. Alternatively or in combination, the functions and operations may be implemented using dedicated circuitry with or without software instructions, such as using application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs). Embodiments may be implemented using hardwired circuitry systems without or in combination with software instructions. Therefore, the technology is neither limited to any particular combination of hardware circuitry systems and software, nor to any particular source of instructions executed by a data processing system.

[0164] In the foregoing description, embodiments of this disclosure have been described with reference to specific example embodiments thereof. It will be apparent that various modifications may be made to this disclosure without departing from the broader spirit and scope of the embodiments set forth in the appended claims. Therefore, the description and drawings should be regarded as illustrative rather than restrictive.

Claims

1. An apparatus comprising: A camera, which includes: Image sensor; A memory device comprising: Non-volatile memory units configured to store first data representing images captured by the image sensor; and A deep learning accelerator having logic circuitry configured to perform matrix computations, wherein a portion of the non-volatile memory cells is used to implement the logic circuitry of the deep learning accelerator. Processor, which is different from the memory device; and A network interface, wherein the processor is configured to receive second data representing a project model via the network interface and from a central station remote from the device, and to store the second data in the non-volatile memory unit such that the logic circuitry of the deep learning accelerator uses the first data as input to perform computations on an artificial neural network; The output of the artificial neural network is configured to indicate whether the image is related to a category of interest represented by the project model; and In response to the output having a first classification relating the image to the item of interest, the processor is configured to transmit a third data, including a report containing the image, to the central station via the network interface.

2. The device of claim 1, wherein the memory device further comprises a controller and a host interface; and The memory device is configured to receive the first data to be stored in the memory device via the host interface.

3. The device of claim 2, wherein the project model includes fourth data representing instructions executable by the logic circuit, and fifth data representing a matrix of the artificial neural network; and The instructions of the project model stored in the non-volatile memory unit are configured to be executed by the logic circuit using the first data as input in response to the host interface receiving the first data, in order to produce the output.

4. The device of claim 3, wherein in response to a second image provided in the output of the artificial neural network when processing a second image as input not being related to a second category of the item of interest, the processor is configured to skip transmitting the second image to the central station.

5. The device of claim 3, wherein when the output has the first classification, the output further includes segments identified by the artificial neural network for image extraction from the image.

6. The device of claim 3, wherein the non-volatile memory unit stores video clips; and after an event near the device, the device is configured to transmit at least one of the video clips to the central station in response to a request from the central station.

7. A system comprising: A central station, coupled to a communication network connected to multiple cameras respectively installed at multiple locations, wherein each corresponding camera among the multiple cameras includes a memory device having: A non-volatile memory used to store first data representing an image captured by the respective camera; and A deep learning accelerator having logic circuitry configured to perform matrix computations, wherein cells of the non-volatile memory are used to implement the logic circuitry of the deep learning accelerator. The central station is configured to transmit a second data representing the project model to the plurality of cameras via the communication network; When the project model is stored in the corresponding camera, the logic circuitry of the deep learning accelerator is configured to perform computations on a first artificial neural network using the first data as input and to produce the output of the first artificial neural network, the output having a classification of whether the image is related to a project of interest represented by the project model; and In response to the output having a first classification of the image as being related to the item of interest, the corresponding camera is configured to transmit a report containing the image to the central station via the communication network and using a processor independent of the memory device.

8. The system of claim 7, wherein the central station has a second artificial neural network configured to receive the image transmitted from the respective camera as input to generate a classification of whether the image is related to the item of interest.

9. The system of claim 8, wherein the second artificial neural network is more accurate than the first artificial neural network in classifying whether the image is related to the item of interest.

10. The system of claim 8, wherein the central station includes a display device for presenting the image in response to the second artificial neural network recognizing the image as relevant to the item of interest.

11. The system of claim 10, wherein the image is a first image; and in response to the second artificial neural network identifying the image as related to the item of interest, the central station is configured to request the corresponding camera to transmit a second image, whose capture time is adjacent to that of the first image, to the central station; and The central station is configured to estimate the real-time location of the object of interest based at least in part on the movement of the object of interest in the first and second images.

12. The system of claim 10, wherein the central station is configured to analyze patterns of time and location of the items of interest captured in a first subset of cameras to identify a second subset of cameras, and instructs the second subset of cameras to stream images from the image sensors of the cameras in the second subset to a display device in the central station in real time; and The central station is configured to generate the first artificial neural network in the project model based on the second artificial neural network, and to compile the description of the first artificial neural network into the project model.

13. A method comprising: A central station communicates via a communication network with multiple cameras respectively configured at multiple locations, wherein each corresponding camera among the multiple cameras includes a memory device having: A non-volatile random access memory for storing first data representing an image captured by the respective camera; and A deep learning accelerator having logic circuitry configured to perform matrix computations, wherein cells of the non-volatile random access memory are used to implement the logic circuitry of the deep learning accelerator. The central station generates a project model, which is configured to cause the logic circuitry of the deep learning accelerator to perform computations on a first artificial neural network and generate an output of the first artificial neural network when stored in the non-volatile random access memory of the corresponding camera, the output having a classification of whether the image is related to a project of interest represented by the project model; The central station transmits a second data representing the project model to the plurality of cameras via the communication network, so that the project model is stored in the non-volatile random access memory of the respective camera; In the central station and in response to the output having a first classification of the image as being related to the item of interest, a report containing the image is received from the corresponding camera via the communication network, wherein the report is transmitted to the central station by the processor of the corresponding camera, wherein the processor of the corresponding camera is independent of the memory device of the corresponding camera.

14. The method of claim 13, further comprising: The central station uses a second artificial neural network, which is more accurate than the first artificial neural network, to generate a classification of whether the image is related to the item of interest. and In response to the second artificial neural network recognizing the image as related to the item of interest, the image is displayed on the display device of the central station.

15. The method of claim 14, further comprising: Analyze the patterns of movement, time, and location of the object of interest captured in the first subset of cameras to obtain an estimate of the current location of the object of interest; A second subset of cameras is identified based on the estimate of the current location of the project of interest; The second subset of cameras is instructed to stream images from the image sensors of the cameras in the second subset to the display device of the central station in real time; The first artificial neural network in the project model is generated based on the second artificial neural network; and The description of the first artificial neural network is compiled to generate the project model.

Images