Storage Edge Controller with Metadata Calculation Engine

Abstract

To provide an improved method and system for generating metadata of a media object as to a calculation engine within a storage edge controller, and storing and using thereof within a data processing system.SOLUTION: A storage device 120 which is a storage device controller comprises a host interface 136, a memory interface, and a first processor for managing to locally store or take out objects in a non-volatile memory. The storage device controller further comprises a calculation engine (AI engine 131, etc.) configured to: acquire a first calculation model from a volatile memory and a parameter set for implementing the first calculation model; and use the first calculation model for selectively calculating metadata defining contents features of objects received from one or more hosts in order to be taken out from the non-volatile memory or to be stored in the non-volatile memory.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] [Cross - Reference to Related Applications] This disclosure claims the benefit under 35 U.S.C. § 119(e) of the co - pending U.S. Provisional Patent Application Nos. 62 / 712,823, filed on July 31, 2018; 62 / 714,563, filed on August 3, 2018; 62 / 716,269, filed on August 8, 2018; 62 / 726,847, filed on September 4, 2018; and 62 / 726,852, filed on September 4, 2018, all by the same applicant. Each of the following U.S. Utility Patent Applications by the same applicant also claims the benefit of the above - mentioned U.S. Provisional Patent Applications and is filed on the same day as this specification: 1. U.S. Patent Application No. 16 / 264,248. 2. U.S. Patent Application No. 16 / 263,387. 3. U.S. Patent Application No. 16 / 262,975. 4. U.S. Patent Application No. 16 / 262,971. Each of the above - mentioned provisional and utility patent applications is incorporated herein by reference in its entirety.

[0002] This disclosure relates to storage control and management of non - volatile storage devices, and more particularly, to a storage controller having a computing engine.

Background Art

[0003] Existing storage systems often store data along with associated metadata that provides a compact form of the description or meaning of the data. Common forms of metadata include various labels, tags, data type indicators, objects and activities detected within the data, the location where the data was created, and so on. The metadata is often generated by a host system, such as a data center, that interacts with the storage system, such as a data storage center where the data is stored. For example, a storage system is configured to retrieve data stored in non-volatile memory and transmit the retrieved data to the host system over a computer network. Next, the host system can analyze the retrieved data and generate metadata associated with the retrieved data. Next, the generated metadata is returned to the storage system for storage via a host interface. Therefore, the amount of data exchanged between the storage system and the host system can be quite large, which can negatively impact the available bandwidth of the computer processing and networking systems. As a result, it is practically impossible to generate metadata for the quite large amount of media generated in today's world. SUMMARY OF THE INVENTION

[0004] Embodiments described herein provide a storage device controller for managing the storage and retrieval of data in one or more storage devices. The storage device controller includes a host interface configured to communicate with one or more hosts, a memory interface configured to communicate locally with the non-volatile memory of one or more storage devices, and a first processor configured to manage the local storage or retrieval of objects in the non-volatile memory. The storage device controller further includes a computing engine configured to obtain, from the volatile memory, a first computational model and a set of parameters for implementing the first computational model, and to selectively compute metadata that defines content characteristics of objects retrieved from or received from one or more hosts for storage in the non-volatile memory using the first computational model.

[0005] In some implementations, the volatile memory is a dynamic random access memory coupled to the storage device controller.

[0006] In some implementations, the volatile memory is a host memory buffer allocated to the storage device controller by the host system, and the host memory buffer is accessible by the storage device controller through a computer network connection or a bus connection (e.g., PCIe).

[0007] In some implementations, the computing engine includes a second processor that is separate from the first processor and is configured to execute computational tasks related to metadata generation including the implementation of the first computational model. The first processor is configured to send computational tasks related to metadata generation to the second processor in the computing engine without occupying the resources of the first processor for ongoing operations being executed by the storage device controller.

[0008] In some implementations, the computing engine further comprises a volatile memory coupled to a second processor. The volatile memory is a static random access memory configured to cache at least a portion of an object during the calculation of metadata defining content characteristics of a cached portion of the object.

[0009] In some implementations, the first processor is further configured to receive an object from one or more hosts via a host interface. The object is to be stored in non-volatile memory. The first processor is further configured to temporarily store the received object in a volatile memory disposed within a storage device controller for metadata calculation. After the calculation of metadata defining the content characteristics of the object is complete, the first processor is configured to transmit the received object from the volatile memory to the non-volatile memory for storage via a memory interface. The first processor is configured to perform at least one of transmitting the metadata to the host system via the host interface and transmitting the metadata to the non-volatile memory for storage via the memory interface.

[0010] In some implementations, the first processor is further configured to receive, via a host interface, a command to retrieve an object from non-volatile memory from a host system of one or more hosts. In response to this command, the first processor is configured to retrieve the object from a volatile memory disposed within a storage controller for metadata calculation via a memory interface. After the calculation of metadata defining the content characteristics of the object is complete, the first processor is configured to transmit the metadata and the object to the non-volatile memory for storage via the memory interface.

[0011] In some implementations, the first processor is further configured to receive, via a host interface, a metadata request from one or more hosts while the compute engine is computing metadata. The first processor waits until the requested metadata in the compute engine is computed and is further configured to asynchronously respond to the metadata request by transmitting the requested metadata to the host system via the host interface while new metadata different from the requested metadata in the compute engine is being computed.

[0012] In some implementations, the first processor is further configured to determine, in response to a command, whether the command from the host system requests an update to the first computational model. In response to determining that the command from the host system does not request an update to the computational model, the first processor is configured to instruct the compute engine to implement the existing computational model. In response to determining that the command from the host system requests that the first computational model be updated to a second computational model different from the first computational model, the first processor is configured to retrieve a set of updated parameters of the second computational model from volatile memory located within a storage controller or from host buffer memory located within the host system via the host interface. The first processor is further configured to transmit the set of updated parameters to the compute engine to implement the second computational model.

[0013] In some implementations, the computing engine is further configured to automatically generate metadata that defines content characteristics of an object by performing any of identification of persons of interest or other objects, customized insertion of advertisements into streamed video, cloud-based analysis of data from autonomous vehicles, analysis of call and response quality in a ChatBot Voice call database, text document and text message database analysis, mood detection, scene identification in video files or voice calls, identification of persons or objects in surveillance camera video, identification of the type of action occurring in surveillance camera video, identification of the type of voice or sound in a recording, classification of phrases and responses used during a conversation, and analysis of automotive sensor data and driving responses.

Brief Description of the Drawings

[0014] Further features, the nature and various advantages of the present disclosure will become apparent upon consideration of the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, like reference numerals refer to like parts.

[0015]

Figure 1

[0016]

Figure 2

[0017]

Figure 3

[0018]

Figure 4

[0019]

Figure 5

[0020]

Figure 6

[0021]

Figure 7

[0022]

Figure 8

DETAILED DESCRIPTION OF THE INVENTION

[0023] The embodiments described herein provide improved methods and systems for generating metadata of media objects in a computing engine (such as an artificial intelligence engine) within a storage edge controller and storing and using such metadata within a data processing system.

[0024] In some embodiments, the data processing system is used to store and analyze large amounts of media objects. Some non-limiting examples of media objects include videos, recordings, still images, text objects such as text messages and emails, data obtained from various types of sensors such as automotive sensors and Internet of Things (IoT) sensors, database objects, and / or any other suitable objects. Some non-limiting examples of object analysis applications include identifying persons of interest or other objects in surveillance camera video, customizing the insertion of advertisements ("ads") into streaming video, cloud-based analysis of data from autonomous vehicles, analyzing the quality of calls and responses in a ChatBot Voice database, analyzing text documents and / or text message databases, mood detection, scene identification in video files or voice calls, identifying persons or objects in surveillance camera video, identifying the type of action occurring in surveillance camera video, identifying the type of voice or sound in a recording, classifying phrases and / or responses used during a conversation, analyzing automotive sensor data and driving responses, and the like.

[0025] As discussed in the background art of the present disclosure, conventionally, a host system is configured to read a media object from non-volatile memory, generate metadata related to the media object, and then return the metadata to the non-volatile memory for storage. Therefore, the amount of data exchanged between the non-volatile memory and the host system can be quite large, which can adversely affect the available bandwidth of the computer processing and networking systems.

[0026] As described in the Background Art, a host system located remotely from where data is stored needs to transfer large amounts of data from a data storage location to a data calculation location through a computer network. In view of the inefficiencies in metadata calculation in those systems, the calculation of metadata at the storage edge is described. By calculating metadata at the storage edge, unnecessary transmission of excessive data through the computer network is eliminated. In particular, the embodiments described herein are located within a storage controller of a non-volatile storage device and provide a calculation engine that generates metadata for data that is in transit to be stored in or retrieved from a non-volatile storage device. In this way, the storage device can generate metadata locally, for example, via an internal calculation engine that exists within the storage controller, without passing the original data content to the host system for processing. By calculating metadata using a high-performance calculation engine local to the location where the data is stored, metadata generation is no longer limited, for example, by the capacity of the host interface of the storage device or the bandwidth limitations of the computer network that transfers data from storage to the calculation facility. By calculating the metadata of stored data, especially unstructured or partially structured media, at the storage edge, the efficiency of data storage and metadata generation that describes the stored data is improved.

[0027] As used herein, the term "storage edge" is used to mean a module or component local to a non-volatile storage device. For example, a controller that controls the operation of one or more storage devices to store or retrieve data in one or more instances of non-volatile memory is disposed at the storage edge. The storage edge is found, for example, within a dedicated storage device or in a storage network and is separate from a processor, for example, disposed remotely within a host computer or in a data center. The storage edge and the remote host communicate through a computer network connection.

[0028] As used herein, the terms "data object", "media object", or "object" are used to mean various types of data issued by an application running on a host system and storable on a storage device. Examples of "media objects" or "objects" include, but are not limited to, video, recordings, still images, text objects such as text messages and emails, data obtained from various types of sensors such as automotive sensors and Internet of Things (IoT) sensors, database objects, and / or any other suitable object. In many cases, media objects are unstructured. As used herein, the term "unstructured object" means that the media content of the object (e.g., text content, audio content, image content, or video content) is provided in an untreated form and not pre-edited according to a fixed field format. Unstructured objects are not a priori tagged with metadata that defines any aspect of the content for each frame or other content part. Unstructured data is not transactionally manipulable and its format does not easily conform to a relational database schema.

[0029] As used herein, the term "metadata" is used to refer to a high-level representation of the actual data content of a media object stored in a non-volatile storage device. "Metadata" can be an abstract layer of the actual data content that provides a description or meaning of the data content in a compact form. Metadata can be generated in various ways from media objects that are almost always unstructured. Examples of metadata include labels, tags, data types, objects / concepts / sentiments detected within the data content, spatial / temporal locations of such objects / concepts / sentiments within the data content, and the like.

[0030] As used herein, the term "in-storage computing" is used to refer to the processing of data stored on a storage device (e.g., a hard disk drive, a solid state drive, etc.) locally by a storage controller on that storage device to automatically generate the structure of the data. In other words, the data is processed within the "boundaries" of the storage device rather than being sent to a separate server or host system, for example, through a computer network. "In-storage computing" can refer to various types of computations, which in one implementation can be performed by an artificial intelligence (AI) engine within the storage device.

[0031] As used herein, the term "artificial intelligence (AI) model" is used to refer to, for example, any suitable AI algorithm implemented with a deep neural network or any regression neural network or any variation thereof. In some implementations, the AI model is preferably any other supervised learning or unsupervised learning or reinforcement learning algorithm. The AI model is trained using a "training set", i.e., the body of the media object and the corresponding metadata known to be accurate. The trained AI model is then applied to generate metadata for other media objects. A software or hardware module that receives a pre-trained AI model and uses this AI model to calculate the metadata of an object is referred to herein as an "AI engine" or an "AI interface engine". In some implementations, several different AI models are applied to unstructured or partially structured media objects.

[0032] FIG. 1 is a block diagram 100 showing a storage device computing structure with an on-storage DRAM for a storage device according to one embodiment described herein. The storage device includes any type of non-volatile memory such as, but not limited to, a solid state device (SSD), a hybrid hard drive, etc. Diagram 100 shows a storage device 120 connected to a host system 110. The host system 110 is located remotely from the storage device 120 and the storage device 120 is accessible via a computer network.

[0033] The storage device 120, for example, an SSD storage device controller, includes a plurality of non-volatile memories shown in FIG. 1 as NAND flash memories 119a - d (although other types of non-volatile memory are also applicable) connected to a storage controller 130 via a data bus.

[0034] In one implementation, the storage controller 130 is configured as a system - on - chip (SoC) that includes one or more integrated circuits combined together within a package. The storage controller 130 is configured to perform read or write operations in non - volatile memory (e.g., NAND flash memories 119a - d), for example, to read data contents stored in NAND flash memories 119a - d or to write data contents to NAND flash memories 119a - d for storage.

[0035] The storage controller 130 includes various modules such as a host interface 136, a central processing unit (CPU) 133 of the storage controller, local memory (SRAM 137 or DRAM 125 via a DRAM controller 134), a media controller 138, etc. The various modules are configured to interact with each other via a fabric of a control or data bus 135. Specifically, the CPU 133 is configured to issue instructions for the storage controller 130 to perform various tasks such as write or read operations in one or more of the NAND memories 119a - n. The SRAM 137 is configured to cache data generated or used during operations executed by the storage controller 130. The media controller 138 is configured to interface communicate with the NAND flash memories 119a - d in one implementation. The host interface 136 is configured to interface communicate with an external host system 110, for example, through a computer network connection 111.

[0036] According to one implementation, the storage controller 130 further includes a computing engine, such as an AI engine 131, that communicates with other modules within the storage controller 130 via the fabric of the control or data bus 135. The AI engine 131 is configured to process data content separately from the CPU 133 as an accelerator and generate metadata that describes data content stored in, retrieved from, or in the process of being stored in one of the NAND memories 119a - h. In one implementation, the AI engine comprises one or more of a vector processor, a DSP, and other cores suitable for analyzing media data and generating metadata. The detailed implementation of metadata generation by the AI engine 131 will be further described with respect to FIGS. 3 - 6. Alternatively, some or all of the functions provided by the dedicated AI engine may be provided by one or more of the CPUs 132 and 133 that execute suitable software or firmware.

[0037] The AI engine 131 optionally includes its own CPU 132, separate from the CPU 133 of the storage controller 130. When the AI engine 131 includes the CPU 132, the CPU 132 is preferably configured as a coprocessor that, for example, offloads various AI - related computational operations from the main CPU 133, manages AI engine interrupts and register programming, and assists in metadata generation. When the AI engine 131 does not include the CPU 132, any CPU operations required by the AI engine 131, such as any computational tasks for generating metadata, are executed by the CPU 133 of the storage controller 130. In such a case, the AI engine 131 shares CPU resources with other storage - related operations. The detailed implementation of metadata generation by the dedicated CPU in the AI engine will be described with respect to FIGS. 7 - 8.

[0038] The storage device 120 includes a local volatile memory such as DRAM 125 configured to store data parameters for an AI model, such as coefficients and weights of a deep neural network. In this way, the AI engine 131 is configured to obtain data parameters from the DRAM 125 via the DRAM controller 134 in order to suitably execute the calculations required by the AI model.

[0039] Figure 2 is a block diagram showing an alternative storage calculation structure without an on-chip DRAM for a storage device according to one alternative embodiment described herein. Diagram 200 shows a storage device 120 and a host system 110 similar to those of diagram 100 in FIG. 1. Different from the structure shown in diagram 100, the storage device 120 does not include local DRAM, and thus does not store data parameters of the AI model in local DRAM. Instead, the host system 110 is configured to allocate a memory segment as the host buffer memory 108 to the storage controller 130 for storing data parameters for the AI model. The host memory buffer 108 is located in the host 110 and is accessible by the storage controller 130 via the host interface 136, for example, through the data bus connection 112 or in one or more of the NAND flash memories 119a - d. Therefore, the data parameters 108 for the AI model are passed to the AI engine 131 from the host memory buffer 108 or from the NAND flash via the host interface 136, for example, through the data bus connection 112.

[0040] In some embodiments, the in-storage computing structure may be applicable when the host system 110 is local to the storage device 120 and connected to the storage device 120 via the data bus 112. Since the host system 110 is not remote from the storage device 120, the data exchange between the host system 110 and the storage device 120 is relatively more efficient than the scenario where the host system is remotely connected to the storage device via a network connection. Therefore, the parameters of the AI model stored on the host side (e.g., the host memory buffer 108) can be efficiently read out and transmitted to the storage side. When the data parameters of the AI model include a large amount of data, the host memory buffer 108 can function as local memory to store a large amount of data without being limited by the data capacity of SRAM or DRAM located within the storage device 120.

[0041] FIG. 3 is a schematic data flow diagram showing various modules within a non-volatile memory storage device and the data flow between those modules for generating metadata of a data stream transmitted from a host system, and FIG. 4 is a logical flow diagram providing an exemplary logical flow of the data flow shown in FIG. 3 according to one embodiment described herein. Diagram 300 shows a storage controller 130 communicatively coupled to the host system 110 and the flow of data through various components. The components of the storage controller 130 and the host system 110 are similar to the structures described in diagrams 100-200 shown in FIGS. 1-2. Process 400 is implemented in the illustrated example through the exchange of data between various modules of the storage device 120 and the host system 110 using the structure shown in diagram 300.

[0042] Specifically, process 400 starts at 402. Here, unstructured data such as media objects is received directly or indirectly from a media generator. For example, in some implementations, a media object is received from host application 105 running on host system 100. In some other implementations, an unstructured media object is received directly or indirectly from a device that generates the media object after passing through a server different from the host processor (i.e., a cloud server) through which metadata can be generated or analyzed, without passing through the host processor. The write command is sent to storage device 120 for storage along with the data stream 301 of the media object. Specifically, the data stream 301 sent to storage device 120 includes an unstructured (i.e., untreated) media object or a partially structured object (e.g., with framed or partial metadata that only describes some of the attributes of a part of the media content).

[0043] At 404, the data stream 301 of an unstructured or partially structured media object is transmitted through the host content manager 142 and temporarily stored in the local memory 125. The host content manager 142 is a software module implemented by the CPU 133 to operate with the host interface 136 of the storage controller 130. For example, the host interface 136 is configured to receive the data stream 301 of an unstructured media object from the host system 110. Next, the host data command handling module 141 is configured at 302a to transfer the unstructured media object to the host content manager 142. For example, the host data command handling module 141 and the host content manager 142 are software modules executed on the same CPU 133 and are configured to exchange data. The host content manager 142 is configured to transfer the data content to the local memory 125 via the data bus 302b. In one implementation, the local memory 125 is a DRAM as shown in FIG. 1. In another implementation, instead of transmitting the data stream 301 to be temporarily stored in the local memory 125 within the SSD storage device 130, the data stream 301 is temporarily stored in a NAND-based cache system (not shown) formed in one of the NAND memories. Further examples of NAND-based cache systems can be found in U.S. Patent No. 9,477,611 issued on October 25, 2016 and U.S. Patent No. 10,067,687 issued on September 4, 2018 by the same applicant.

[0044] At 406, unstructured media objects and / or parameters of the AI model (e.g., weights / coefficients of a deep neural network) are read from the local memory 125. For example, the parameters of the AI model are previously obtained from the host system 110 and pre-stored in the local memory 125. In some implementations, the parameters of the AI model are periodically, constantly, or intermittently updated with new parameters from the host system 110 via the data buses 302a - b. The host content manager module 142 is configured to read unstructured media objects and parameters for the AI model from the local memory 125 via the data bus 304. In another implementation, the parameters for the AI model are optionally read from the host memory buffer 108 shown in FIG. 2.

[0045] At 408, unstructured or partially structured media objects are processed by the AI engine 131 through classification, labeling, or documentation, etc. of various types of media objects such as videos, recordings, still images, text objects such as text messages and emails, data obtained from various types of sensors such as automotive sensors and Internet of Things (IoT) sensors, database objects, and / or any other suitable objects. Specifically, the host content manager is configured to send data content and data parameters for the AI model to an AI driver 144 (e.g., a HAL driver) that operates with the AI engine 131. Next, the AI driver 144 incorporates the data parameters and is configured to implement a desired AI model (with the desired coefficients / weights for a deep neural network). Next, the AI engine 131 is configured to generate metadata describing the data content by executing the AI model. Next, the generated metadata can be sent to the media controller 138 via the fabric of the data bus 135 shown in FIGS. 1 - 2.

[0046] At 410, the generated metadata is stored together with the original data content on the NAND flash memories 119a - n in one implementation. Specifically, the host content manager 142 is configured to send the generated metadata from the AI engine 131 to the internal file system 148 via the data bus 306 which is part of the fabric of the data bus 135. Next, the internal file system 148 is configured to link the metadata to the corresponding media object. A further example of the internal file system 148 that links metadata to the corresponding media object can be found in U.S. Patent Application No. 16 / 262,971, filed on the same day by the same applicant and co - pending. Next, the metadata is sent from the internal file system 148 to the media command processor 149 via the data bus 307 which is part of the fabric of the data bus 135. Further, the media command processor 149 issues a write command to write the metadata to the NAND flash memories 119a - n via the flash controller 129.

[0047] In some implementations, the generated metadata is stored together with the original data content within the NAND flash memories 119a - n. For example, each data stream 301a - n is accompanied by corresponding metadata 317a - n that describes the respective data stream. In this way, the metadata 317a - n can be separately retrieved from the original data content 301a - n by storing it in a separate physical location or by logically separating it.

[0048] In some implementations, the metadata can be stored separately from the data content, e.g., not at contiguous memory addresses, not within the same memory page, or even not on the same NAND memory die. In this case, the internal file system 148 is configured to assign pointers that link the memory address where the media object of the data content exists and the memory address where the metadata that describes the data content exists. In this way, when a portion of the original data content within the media object of interest is identified using the metadata, the corresponding segment of the media can be retrieved from the unstructured media object. Further details of the storage structure of the unstructured media object and the metadata that describes the media object can be found in U.S. Patent Application No. 16 / 263,387, filed on the same day by the same applicant and co-pending herewith.

[0049] At 412, if data processing has not ended, e.g., if there is a new data stream for a media object to be processed, a separate set of metadata corresponding to different sets of content attributes is generated using different AI models, and process 400 proceeds by repeating 406, 408, and 410. If data processing has ended, process 400 ends.

[0050] FIG. 5 is a schematic data flow diagram showing various modules within a non-volatile memory storage device and the data flow between those modules for the generation of metadata of data stored in the non-volatile memory, and FIG. 6 is a logical flow diagram providing an exemplary logical flow of the data flow shown in FIG. 5 according to another embodiment described herein. Diagram 500 shows an SSD storage device 120 communicatively coupled to a host system 110, which is similar in structure to the structures described in diagrams 100-200 shown in FIGS. 1-2 and similar to the structure shown in diagram 300 of FIG. 3. Process 600 is implemented through the exchange of data between various modules of SSD storage device 120 and host system 110 in the structure shown in diagram 500.

[0051] Specifically, unlike process 400 that receives and processes a data stream from host system 110, process 600 and the data flow in diagram 500 describe a scenario where unstructured or partially structured media objects are already stored on NAND flash memories 119a - n, but the corresponding metadata has not yet been fully generated. Thus, process 600 can be applied to an "offline" scenario where unstructured or partially structured media objects are processed (or reprocessed) at a time later than the time when the unstructured or partially structured media objects were initially received from the host system.

[0052] At 602, a data request is received from host system 110 to process unstructured media objects previously stored in non-volatile memories, such as NAND flash memories 119a - n. Specifically, host application 105 is configured to issue command 501 to storage controller 130 to retrieve a portion of an unstructured or partially structured media object of data content from non-volatile memories 119a - n. Specifically, upon receiving command 501, host data command handling module 141 is configured to transfer the command via data bus 502a to host content manager 142, and host content manager 142 further causes the media object to be retrieved from NAND memories 119a - d to local memory 125.

[0053] At 604, host content manager 142 is configured to determine whether command 501 requests data processing using an existing AI model already loaded in AI engine 131, or requests reprocessing of stored data content using a new AI model not yet loaded in AI engine 131.

[0054] If command 501 requests data processing with an existing AI model, process 600 proceeds to 608, where AI engine 131 is configured to resume with an existing AI model already loaded in AI engine 131 for metadata generation.

[0055] When command 501 requests data processing with a new AI model, process 600 proceeds to 606, where new data parameters for the AI model are loaded into AI engine 131 and local memory 125. Specifically, new data parameters (e.g., new parameters / weights, new neural network models, etc.) may be sent from host system 110 to host data command handling module 141 along with command 501, and host data command handling module 141 further sends the new data parameters to host content manager 142 via data bus 502a. Next, host content manager 142 is configured to send the new data parameters to local memory 125 for storage via data bus 502c, which is part of the fabric of data bus 135 (e.g., to update previously saved parameters for the AI model, etc.). Also, host content manager 142 sends the new data parameters to AI driver 144 via data bus 503, which is part of the fabric of data bus 135. Next, AI driver 144 is configured to incorporate the new data parameters to implement the new AI model in AI engine 131.

[0056] In some implementations, when the storage controller 130 is idle, i.e., it performs no storage or retrieval operations on the data stored on the non-volatile memory, or processes no data stream (e.g., 301 in FIG. 3) directly received from the host system 110, instead of being instructed by the host system 110, the storage controller 130 is configured to automatically start calculating metadata for the unstructured media objects stored in the non-volatile memories 119a - d. Thus, when the storage controller 130 is idle and does not perform storage operations such as reading or writing data to the non-volatile memory, the CPU resources of the storage controller 130 are used for extensive AI calculations to generate metadata for the content stored in one or more of the non-volatile memories associated with the storage controller 130. In some implementations, the storage controller 130 is an aggregator that operates with multiple storage devices such as memory arrays and is configured to control those storage devices. Further details regarding the use of the aggregator for calculating and aggregating metadata for unstructured media objects stored in the memory array can be found in U.S. Patent Application No. 16 / 264,248, filed on the same day by the same applicant and currently co-pending.

[0057] After operation 606 or 608, process 600 then proceeds to 610. Operations 610 - 615 are similar to operations 406 - 412 in FIG. 4 respectively, except that the data content is reprocessed with a new AI model in one implementation. In this case, some metadata may have been previously generated for the same data content and is updated or modified with the newly generated metadata from the new AI model.

[0058] At 610, the media object of the data content is retrieved from the NAND flash memories 119a - n to the local memory 125. Specifically, the flash controller 129 receives, via the data bus 512, the media object of the data content retrieved from the NAND memories 119a - n, and provides the data content to the host content manager 142. The host content manager 142 is configured to supply, via the data bus 502c, the data content to be processed or re - processed to the local memory 125.

[0059] At 612, the unstructured media object is processed by the AI engine 131 that executes the AI model previously loaded at 608 or generates and executes a new AI model at 606. Specifically, the host content manager 142 is configured to read the data content from the local memory 125 via the data bus 504, and then transmit the data content, via the data bus 503, to the AI driver 144 that operates with the AI engine 131. Next, the AI engine 131 is configured to generate metadata that describes the data content by executing an available (existing or new) AI model.

[0060] At 614, the generated metadata is provided to the NAND flash memories 119a - n for storage. Specifically, the host content manager 142 is configured to cause the generated metadata from the AI engine 131 to be transmitted to the internal file system 148 via the data bus 506, which is part of the fabric of the data bus 135. Next, the internal file system 148 is configured to link the metadata to the corresponding media object. A further example of the internal file system 148 that links metadata to the corresponding media object can be found in U.S. Patent Application No. 16 / 262,971, filed on the same day by the same applicant and co - pending.

[0061] The media command processor 149 further issues a write command to write metadata to the NAND flash memories 119a - n via the flash controller 129.

[0062] In some implementations, the newly generated metadata is saved and combined with the metadata previously generated for the same media object of the data content. For example, one of the unstructured media object data streams 512a - n is accompanied by a corresponding set of metadata 507a - n that describes each data stream. Among the metadata 507a - n, segments of each set of metadata 507a - n can be newly generated metadata generated via the new AI model shown at 508 in one implementation. Further details of the storage structure of the unstructured media object and the metadata describing the media object can be found in U.S. Patent Application No. 16 / 263,387, filed on the same day by the same applicant and co - pending.

[0063] At 615, if the data processing is not finished, for example, if there is additional data to be processed, a separate set of metadata corresponding to different sets of content attributes using different AI models is generated, etc., and process 600 proceeds by repeating 610, 612, and 614. If the data processing is finished, process 600 proceeds to 616, where the newly generated metadata is optionally sent back to host system 110, for example, in the form of a report including new metadata 509. In one implementation, storage controller 130 is configured to send back a report of new metadata 509 to host system 110 when the data processing is finished. Or alternatively, storage controller 130 is configured to send back a report of new metadata 509 to host system 110 asynchronously with respect to command 501. For example, the host system can request metadata at any time while the metadata update is being executed, and storage controller 130 is configured to send back a report of new metadata 509 asynchronously in response to the metadata request when the metadata update is finished.

[0064] FIG. 7 is a schematic data flow diagram showing various modules within a non-volatile memory storage device and the data flow between those modules for metadata generation using a dedicated CPU within an AI engine of a storage device, and FIG. 8 is a logical flow diagram providing an exemplary logical flow of the data flow shown in FIG. 7 according to another embodiment described herein. Diagram 700 shows a storage device 120 communicatively coupled to host system 110, which is similar in structure to diagrams 100 - 200 shown in FIGS. 1 - 2. Process 800 is implemented through the exchange of data between various modules of storage device 120 and host system 110 in the structure shown in diagram 700. Specifically, diagram 700 depicts a storage controller structure 130 in which AI engine 131 includes its own CPU 132 and SRAM 154.

[0065] In one implementation, in storage controller 130, CPU 133 is used to program the control registers in CPU 133 to execute AI-related operations. When the AI-related operations are completed, a service interrupt is sent from AI engine 131 to CPU 133, and CPU 133 can release the programmed control registers and engage in other activities. For example, CPU 133 fetches commands from host system 110 to transfer data, executes flash translation layer (FTL) operations to abstract the underlying media data from NAND flash memories 119a - d, executes media management to access the underlying media (e.g., NAND memories 119a - d, etc.) devices, and often requests resources for host interface management to perform actual data read or write operations, etc. Due to the recurrent nature of some AI models, AI operations need to be repeated multiple times at a high frequency. If CPU 133 is required to constantly allocate control registers to execute AI operations, a relatively high CPU workload will occur. This occupies the bandwidth of storage controller CPU 133, reduces its availability for executing storage-related tasks, and thereby adversely affects storage performance.

[0066] CPU 132 or the coprocessor is included in AI engine 131 to offload AI-related tasks from main CPU 133 and to provide operating system (OS) separation between the main OS that executes AI management and storage-related tasks. CPU 132 includes one or more processor units that provide high-performance general-purpose central processing capabilities, digital signal central processing capabilities, vector central processing capabilities, or other suitable central processing capabilities. In addition, dedicated SRAM 154 is disposed within AI engine 131 to cache data that does not need to be read from DRAM 125 through the fabric (or read from NAND-based cache) frequently accessed data. Thus, the bandwidth of the fabric can be applied to other non-AI-related operations.

[0067] Process 800 starts at 802, where, for example, similar to operation 402 in FIG. 4, a save request 701 and data content to be saved are received from host system 110. At 804, the data content is loaded into local memory DRAM 125 via data bus 702, similar to operation 404 in FIG. 4 for example. At 806, frequently accessed data is cached in dedicated SRAM 154 within AI engine 131. For example, in one implementation, the caching of data is gradually executed in parallel with the implementation of the AI engine while metadata is being generated. Segments of unstructured media objects of data content to be processed, data variables frequently used in regression-type AI models, etc. are each cached in SRAM 154 for access and used in subsequent iterations of the AI model. In one implementation, the data cached in SRAM 154 is overwritten constantly, periodically, or intermittently and replaced as needed while the AI model is being implemented.

[0068] At 808, AI engine 131 is configured to read the data cached from SRAM 154 and continue the implementation of the AI model. Specifically, AI hardware 153 is configured to assist with AI-related tasks, but generally these AI operations are not fully hardware-automated and still require some CPU assistance for more complex tasks. The CPU 133 of storage controller 133 is configured to allocate all or substantially all AI-related tasks 161 to dedicated CPU 132, which is local to storage controller 130, for processing.

[0069] The generated metadata is optionally stored in the local memory DRAM 125 at 810, or directly stored in the NAND flash at 812, for example, within a NAND-based cache. For example, the newly generated metadata is stored in the DRAM 125 together with the parameters for the AI model that can be used as the training data updated to revise the AI model. At 812, the data content and the metadata describing the data content are transmitted to the NAND flash memories 119a - d for storage via the data buses 703 and 705.

[0070] In some implementations, the AI operations at 808 are executed simultaneously with the storage operations at 802, 804, 810, and 812. For example, while metadata is being generated in the AI engine 131, new data can be continuously transmitted from the host system 110 to the storage controller 130. When accompanied by the dedicated CPU 132, the CPU 133 is configured to manage the storage operations (e.g., 802, 804, 810, and 812) while offloading the AI operation 161 to the dedicated CPU 132, for example, while programming the AI block registers, processing AI interrupts, etc. in the AI engine 131. Therefore, the AI-related tasks do not inhibit the operation of the storage controller CPU 133 when performing SSD-related tasks and do not effectively increase the workload of the CPU 133.

[0071] The various embodiments discussed with respect to FIGS. 1 - 8 are implemented by, but not limited to, electronic components of one or more electronic circuits such as integrated circuits, application specific integrated circuits (ASICs), etc. The various components discussed throughout the present disclosure, for example, but not limited to, the CPU 133, the AI engine 131, the host interface 136, etc. include a set of electronic circuit components and are configured to operate communicably on one or more electronic circuits.

[0072] Although various embodiments of the present disclosure have been shown and described herein, such embodiments are provided by way of example only. Numerous variations, modifications, and substitutions related to the embodiments described herein are applicable without departing from the present disclosure. It should be noted that various alternative forms of the embodiments of the present disclosure described herein can be used in the implementation of the present disclosure. It is intended that the following claims define the scope of the present disclosure and that methods and structures within the scope of these claims and their equivalents be covered by these claims.

[0073] Although the subject matter of this specification has been described with respect to specific aspects, other aspects may also be implemented and are within the scope of the following claims. For example, the acts recited in the claims may be performed in a different order and still achieve the desired results. In certain implementations, multitasking and parallel processing may be advantageous. Other variations are within the scope of the following claims.

Claims

1. A storage controller composed of a solid state device (SSD) and configured to manage data storage and retrieval in the SSD, the storage controller comprising: A host interface configured to communicate with one or more hosts; A memory interface configured to communicate locally with the non-volatile memory of the SSD; A central processing unit (CPU) configured to manage local storage or retrieval of objects in the non-volatile memory, the objects being unstructured objects of different media types including video, recordings, still images, text objects, data obtained from various types of sensors, the CPU; A computing engine and The computing engine is Obtaining, from the volatile memory via the CPU, a first computing model among a plurality of computing models and a set of parameters for implementing the first computing model; Calculating metadata that defines content characteristics of the object retrieved from the non-volatile memory using the first computing model, wherein when the storage controller is not performing a save operation, the computing engine is configured to start the calculation of the metadata, calculating; and is configured to perform The CPU is Receiving the object from the one or more hosts via the host interface, the object being to be stored in the non-volatile memory, receiving; Temporarily storing the received object in a volatile memory disposed within the storage controller for metadata calculation; After the calculation of the metadata that defines the content characteristics of the object is completed, Transmitting the received object from the volatile memory to the non-volatile memory for storage via the memory interface; Performing at least one of transmitting the metadata to a host system via the host interface and transmitting the metadata to the non-volatile memory for storage via the memory interface; A storage controller further configured to perform the above.

2. The storage controller according to claim 1, wherein the volatile memory is a dynamic random access memory connected to the storage controller.

3. The storage controller according to claim 1 or 2, wherein the volatile memory is a host memory buffer allocated to the storage controller by a host system, and the host memory buffer is accessible by the storage controller through a computer network connection or a bus connection.

4. The CPU is configured to: Receive a command from a host system of the one or more hosts via the host interface to retrieve the object from the non-volatile memory; In response to the command, retrieve the object from the volatile memory disposed within the storage controller for metadata calculation via the memory interface; After completion of calculation of metadata defining content characteristics of the object, transmit the metadata and the object to the non-volatile memory for storage via the memory interface. The storage controller according to any one of claims 1 to 3, further configured to perform the above.

5. The CPU is configured to: While the computing engine is computing the metadata, receiving a metadata request from one or more hosts via the host interface, Waiting until the requested metadata is computed in the computing engine, and While new metadata different from the requested metadata is being computed in the computing engine, transmitting the requested metadata to the host system via the host interface thereby asynchronously responding to the request for metadata and being further configured to perform, and / or The CPU In response to a command, determining whether the command from the host system requests an update to the first computational model, and In response to determining that the command from the host system does not request an update to the first computational model, instructing the computing engine to implement the existing computational model, and In response to determining that the command from the host system requests that the first computational model be updated to a second computational model different from the first computational model, Retrieving a set of updated parameters of the second computational model from the volatile memory disposed within the storage controller or from a host memory buffer disposed within the host system via the host interface, and Transmitting the set of updated parameters to the computing engine to implement the second computational model The storage controller according to any one of claims 1 to 4, being further configured to perform.

6. The calculation engine automatically generates metadata that defines the content characteristics of the object by performing any one of identification of persons of interest or other objects, customized insertion of advertisements into streaming video, cloud-based analysis of data from autonomous vehicles, analysis of the quality of calls and responses in a ChatBot Voice call database, text document and text message database analysis, mood detection, scene identification in video files or voice calls, identification of persons or objects in surveillance camera footage, identification of the type of action occurring in surveillance camera footage, identification of the type of voice or sound in a recording, classification of phrases and responses used during a conversation, and analysis of automotive sensor data and driving responses. The storage controller according to any one of claims 1 to 5 is further configured as described above.

7. A method for managing storage and retrieval of data in a Solid State Device (SSD) by a storage controller constituted by the SSD, the method comprising: communicating with one or more hosts via a host interface of the storage controller; locally communicating with the non-volatile memory of the SSD via a memory interface of the storage controller; managing local storage or retrieval of an object in the non-volatile memory via a CPU of the storage controller, the object being an unstructured object of different media types including video, recording, still image, text object, data obtained from various types of sensors; obtaining, by a calculation engine, via the CPU from volatile memory, a first calculation model among a plurality of calculation models and a set of parameters for implementing the first calculation model; A step of calculating metadata for defining content features of the object retrieved from the non-volatile memory using the first calculation model by the calculation engine, wherein the calculation engine is configured to start calculating the metadata when the storage controller is not performing a save operation. comprising, the method is Receiving the object from the one or more hosts via the host interface, wherein the object is to be stored in the non-volatile memory. Temporarily storing the received object in a volatile memory arranged in the storage controller for metadata calculation. After the calculation of the metadata defining the content features of the object is completed, Transmitting the received object from the volatile memory to the non-volatile memory for storage via the memory interface. Performing at least one of transmitting the metadata to the host system of the one or more hosts via the host interface and transmitting the metadata to the non-volatile memory for storage via the memory interface. The method further comprising.

8. Further comprising storing parameters of the first calculation model for the calculation engine in the volatile memory. The method according to claim 7, wherein the volatile memory is a dynamic random access memory connected to the storage controller.

9. Receiving a command from the host system of the one or more hosts via the host interface to retrieve the object from the non-volatile memory. In response to the command, retrieving the object from the volatile memory disposed within the storage controller for metadata calculation via the memory interface; After completion of the calculation of the metadata defining the content characteristics of the object, transmitting the metadata and the object to the non-volatile memory for storage via the memory interface; further comprising, or receiving, via the host interface, a metadata request from one or more hosts while the calculation engine is calculating the metadata; waiting until the requested metadata is calculated in the calculation engine, and transmitting the requested metadata to the host system via the host interface while new metadata different from the requested metadata is being calculated in the calculation engine; asynchronous response to the request for metadata; The method according to claim 7 or 8, further comprising.

10. In response to the command, determining whether the command requests an update of the first calculation model; In response to determining that the command from the host system does not request an update of the first calculation model, instructing the calculation engine to implement the first calculation model; In response to determining that the command from the host system requests that the first calculation model be updated to a second calculation model different from the first calculation model, retrieving a set of parameters for the second calculation model from the volatile memory disposed within the storage controller or from a host memory buffer disposed within the host system via the host interface; To implement the second computational model, sending the set of updated parameters to the computational engine; The method according to claim 9, further comprising. **Claim 11** By automatically performing any one of identification of persons of interest or other objects, customized insertion of advertisements into streamed video, cloud-based analysis of data from autonomous vehicles, analysis of the quality of calls and responses in a ChatBot Voice call database, text document and text message database analysis, mood detection, scene identification in video files or voice calls, identification of persons or objects in surveillance camera footage, identification of the type of action occurring in surveillance camera footage, identification of the type of voice or sound in recordings, classification of phrases and responses used during conversations, and analysis of automotive sensor data and driving responses, generating metadata that defines the content characteristics of the object; The method according to any one of claims 7 to 10, further comprising.

Images