Metadata Generation at the Storage Edge

Abstract

A controller (56) for use in a storage device (44, 114) of a data processing system (20, 90) includes a host interface (68), a memory interface (60), and one or more processors (64). The host interface is configured to communicate with one or more remote hosts (24, 102) of the data processing system via a computer network (28, 98). The memory interface is configured to communicate locally with a non-volatile memory (52) of the storage device. The one or more processors are configured to manage local storage or retrieval (80) of media objects at the non-volatile memory and to selectively compute metadata (84) defining content characteristics of the media objects stored or to be stored in the non-volatile memory.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure claims the benefit under 35 U.S.C. §119(e) of co-pending, commonly assigned U.S. Provisional Patent Application No. 62 / 712,823, filed on July 31, 2018; U.S. Provisional Patent Application No. 62 / 714,563, filed on August 3, 2018; U.S. Provisional Patent Application No. 62 / 716,269, filed on August 8, 2018; U.S. Provisional Patent Application No. 62 / 726,847, filed on September 4, 2018; and U.S. Provisional Patent Application No. 62 / 726,852, filed on September 4, 2018. Each of the following commonly assigned U.S. non-provisional patent applications also claims the benefit of the above-identified U.S. provisional patent applications and is filed concurrently therewith:

[0003] 1. U.S. Patent Application No. ________ entitled “SYSTEMS AND METHODS FOR GENERATING METADATA DESCRIBING UNSTRUCTURED DATA OBJECTS AT THE STORAGE EDGE” (Attorney Docket No. MP11049 / 004048 0729 101);

[0004] 2. U.S. Patent Application No. ________ entitled “STORAGE EDGE CONTROLLER WITH A METADATA COMPUTATIONAL ENGINE” (Attorney Docket No. MP11060 / 004048 0730 101);

[0005] 3. U.S. Patent Application No. ________ entitled "STORAGE AGGREGATOR CONTROLLER WITH METADATA COMPUTATION CONTROL" (Attorney Docket No. MP11065 / 004048 0731 101); and

[0006] 4. U.S. Patent Application No. ________ entitled "METADATA GENERATION FOR MULTIPLE OBJECT TYPES" (Attorney Docket No. MP11074 / 1036-1182).

[0007] Each of the foregoing provisional and non-provisional patent applications is incorporated herein by reference in its respective entirety. Technical Field

[0008] The present disclosure relates generally to processing of media objects, and in particular to methods and systems for generating metadata for media objects. Background Art

[0009] Various systems and applications involve the analysis of large amounts of content such as text items, videos, images, voice files, and sensor data, to name just a few examples. Some analysis tasks use metadata associated with the content.

[0010] The above description is presented as a general overview of the relevant art in the field and should not be construed as an admission that any of the information it contains constitutes prior art against the present patent application. Summary of the invention

[0011] One embodiment described herein provides a controller for use in a storage device of a data processing system. The controller includes a host interface, a memory interface, and one or more processors. The host interface is configured to communicate with one or more remote hosts of the data processing system via a computer network. The memory interface is configured to communicate locally with a non-volatile memory of the storage device. The one or more processors are configured to manage local storage or retrieval of media objects at the non-volatile memory, and selectively calculate metadata that defines content characteristics of media objects stored or to be stored in the non-volatile memory.

[0012] In some embodiments, one or more processors are configured to generate metadata for a plurality of unstructured media objects stored or to be stored in a non-volatile memory. In some embodiments, one or more processors are configured to receive a model specifying the extraction of metadata from a media object from one or more hosts via a computer network, generate metadata based on the received model, and store the generated metadata locally in the non-volatile memory for use by the one or more hosts. In an example embodiment, the one or more processors are configured to receive a pre-trained artificial intelligence (AI) model as a model, and generate metadata by applying an AI reasoning engine to the AI ​​model.

[0013] In another embodiment, the one or more processors are configured to assign a first priority to management of storage or retrieval of objects at a non-volatile memory, and assign a second priority, lower than the first priority, to generation of metadata. In yet another embodiment, the one or more processors are configured to generate metadata during idle periods during which at least some resources of the one or more processors do not manage storage of objects. In a disclosed embodiment, the one or more processors are configured to identify idle periods, initiate or resume generation of metadata during the identified idle periods, and suspend generation of metadata outside of the identified idle periods.

[0014] In some embodiments, the one or more processors are configured to make the metadata accessible to one or more hosts via a computer network, the one or more hosts being located remotely from the processors. In one example embodiment, the one or more processors are configured to make the metadata accessible by transmitting the metadata over the computer network, but less than all of the media objects from which the metadata was generated. In one embodiment, after sending the metadata, the one or more processors are configured to send one or more of the media objects via the computer network in response to a request from the one or more hosts.

[0015] In some embodiments, the storage device comprises a solid state drive (SSD), and the non-volatile memory comprises one or more flash memory devices. In other embodiments, the storage device comprises a hard disk drive (HDD), and the non-volatile memory comprises one or more magnetic disks.

[0016] According to one embodiment described herein, a method for generating metadata is additionally provided, the metadata defining content characteristics of a media object. The method includes: communicating with one or more remote hosts of a data processing center through a computer network via a host interface coupled to a storage device located in the data processing center; and communicating locally with a non-volatile memory of a storage device through a memory interface of the storage device. Local storage or retrieval of media objects at the non-volatile memory is managed by one or more processors of the storage device. Metadata defining content characteristics of a media object stored or to be stored in the non-volatile memory is selectively calculated by one or more processors of the storage device.

[0017] According to one embodiment described herein, a system including multiple storage devices and one or more hosts is also provided. The storage devices in the multiple storage devices respectively include non-volatile memory and a controller. The controller is configured to manage local storage or retrieval of objects at the non-volatile memory, and selectively calculate metadata, which defines the content characteristics of the media objects stored or to be stored in the non-volatile memory. One or more hosts are configured to retrieve metadata generated by the multiple storage devices and use the retrieved metadata to perform data processing operations on the media objects.

[0018] The present disclosure may be more fully understood through the following detailed description of the embodiments in conjunction with the accompanying drawings, in which: BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a block diagram schematically illustrating a data center performing metadata generation at a storage edge according to one embodiment described herein;

[0020] Figure 2 is a block diagram schematically illustrating an edge device that performs local metadata generation according to one embodiment described herein;

[0021] Figure 3 is a flow chart schematically illustrating a method for combined storage / retrieval and metadata generation in a solid state drive (SSD) controller according to one embodiment described herein; and

[0022] Figure 4 is a block diagram schematically illustrating an SSD according to one embodiment described herein. DETAILED DESCRIPTION

[0023] Embodiments described herein provide improved methods and systems for generating metadata for media objects and storing and using such metadata in a data processing system.

[0024] In some embodiments, the data processing system is used to store and analyze large amounts of content data contained in media objects. Some non-limiting examples of object analysis applications include identification of persons of interest or other objects in surveillance camera footage, custom insertion of advertisements ("ads") into streaming video, analysis of data from autonomous vehicles, analysis of call and response quality in ChatBot voice call databases, text document and / or text message database analysis, emotion detection, scene recognition in video files or voice calls, identification of persons or objects in surveillance footage, identification of types of actions occurring in surveillance footage, identification of types of voice or sound in recordings, classification of phrases and / or responses used during conversations, analysis of automotive sensor data and driving responses, etc.

[0025] Examples of media objects include videos, audio recordings, still images, text objects (such as text messages and emails), data obtained from various types of sensors (such as automotive sensors and Internet of Things (IoT) sensors), database objects, and / or any other suitable objects. For brevity, media objects are also referred to herein simply as "objects."

[0026] Typically, the system generates metadata for media objects and performs desired analysis or actions based on the metadata. For example, in a system for advertisement insertion, the system will typically analyze each streaming video, divide the video into scenes, estimate the emotions associated with each scene, characters and / or objects in the scene, identify the speech context in the scene, and store the estimated scene emotions and other information as metadata associated with the video. Based on this metadata, the system can then choose where in the video to insert a given advertisement to maximize the impact of the advertisement.

[0027] As another example in a completely different field, a data processing system is used to perform offline analysis of data acquired by cameras and other sensors of an autonomous vehicle. In this example, the system can scan the video acquired by the car's cameras and / or the output of other car sensors, identify events pre-classified by the AI ​​model as interesting, and tag them with metadata. The system can then use the metadata to investigate the identified events.

[0028] In one example embodiment, in an automotive system, very large amounts of sensor data are generated. Metadata is generated on the storage side of the vehicle that is periodically connected to a network. The metadata is used to select portions of the relevant sensor data to be at least temporarily stored at a storage device in the vehicle or discarded. The metadata along with the selected related objects are then periodically uploaded to a central processor via a network connection where the metadata objects are analyzed and applied for various purposes, such as improving the behavior of an autonomous vehicle or determining targeted advertisements to be communicated to vehicle users.

[0029] In many cases, media objects are unstructured. In this context, the term "unstructured object" refers to an object whose media content (e.g., text content, audio content, image content, or video content) is provided in raw form and is not pre-organized according to a fixed field format. Typically, unstructured objects are not a priori labeled with metadata that defines any aspect of the content of each frame or other content portion. Typically, unstructured data is non-transactional and in a format that does not easily conform to a relational database schema.

[0030] Metadata can be generated in a variety of ways from media objects, which are almost always unstructured. One possibility is to use an artificial intelligence (AI) model, such as a neural network. In a typical implementation, the AI ​​model is trained using a "training set", which is a large number of media objects and corresponding metadata that is known to be accurate. The trained model is then applied to generate metadata for other media objects. The software or hardware module that receives the pre-trained AI model and uses it to calculate the metadata of an object is referred to herein as an "AI reasoning engine". In some implementations, several different AI models will be applied to unstructured or partially structured media objects.

[0031] The embodiments described herein significantly improve the metadata generation process by performing this task at the storage edge, close to where the objects are stored, rather than at a centralized location that typically requires very large amounts of media object data to be transferred across a network between storage devices and processing devices. In this context, the term "at the storage edge" refers to one or more processors on the storage device side of the network rather than on the host side of the network. In other words, the processor at the storage edge needs to send and / or receive information across the network in order to communicate with the host, but does not need to send and / or receive information across the network in order to communicate with the storage device. Non-limiting examples of storage edge processors, as described in detail below, include processors in solid-state drive (SSD) controllers, storage array controllers (also known as aggregators), and processors in connected storage edge devices (such as, autonomous vehicles).

[0032] In a typical embodiment, a data processing system includes one or more host servers ("hosts") that analyze media objects and one or more storage devices (e.g., solid-state drives SSDs) that store media objects. The hosts and storage devices communicate via a computer network. Each storage device includes a non-volatile memory (NVM) and a controller. Similarly, devices that aggregate storage between one or more storage devices also include controllers. These controllers are configured to (i) manage local storage or retrieval of objects in the NVM, and (ii) selectively calculate metadata that defines content characteristics of media objects stored or to be stored in the NVM, thereby utilizing controllers at the edge of storage as a resource for calculating metadata about unstructured or partially structured media objects.

[0033] In some embodiments, the controller of each storage device includes its own local AI inference engine. The controller receives a pre-trained AI model from the host, compiles the AI ​​model, and generates metadata for the media object by running the compiled AI model on the local AI inference engine.

[0034] In some embodiments, a controller of each storage device maintains a local metadata database that stores metadata generated for media objects stored locally on the storage device. The local metadata databases of the various storage devices can be accessed by a host over a computer network to perform any desired object analysis tasks. In some embodiments, metadata is associated with media objects but stored separately from the media objects so that the metadata can be retrieved separately from the media objects. In some embodiments, the storage of media objects and their corresponding metadata is in the same storage device, but in other embodiments, the metadata is stored in a storage device separate from the media objects, such as at a remote location accessible over a computer network.

[0035] In some embodiments, the controller of the storage device gives a higher priority to data storage and retrieval tasks and a lower priority to metadata generation. In an example embodiment, the controller identifies idle periods during which at least some of the controller resources are free of data storage and retrieval tasks. The controller uses such idle periods to perform metadata generation. In this way, the performance of the storage device is not degraded by the additional metadata generation tasks of the controller.

[0036] By delegating metadata computation tasks from the host to the storage device, the disclosed technology eliminates the need to transfer media objects across the network in order to compute metadata. Typically, most, if not all, media objects remain within the boundaries of the storage device, and the metadata is provided to the host for analysis. Based on the metadata, the host can select which specific media object or portions thereof, if any, needs to be retrieved over the network. In this way, communication overhead on the computer network is significantly reduced.

[0037] The disclosed techniques also reduce latency, for example, because they require less data movement and because they enable multiple storage device controllers to generate distributed metadata in parallel. Because the disclosed solution is applicable to distributed implementations across multiple storage devices, the solution is highly scalable. Because the disclosed technique improves metadata generation speed, it responds quickly to updates to media objects.

[0038] Figure 1 1 is a block diagram schematically illustrating a data processing system 20, in this example a data center, that performs metadata generation at the storage edge according to one embodiment described herein. As described above, the system 20 is used to store a large number of objects, calculate metadata for the objects, and analyze the objects based on the metadata. In some embodiments, the system 20 receives, stores, and analyzes multiple different types of objects.

[0039] exist Figure 1 In an embodiment of the present invention, data processing system 20 includes one or more host servers 24 that communicate via a computer network 28. For simplicity, host servers 24 are also referred to herein as simply "hosts." Computer network 28 may include any suitable type of network, such as a local area network (LAN), a wide area network (WAN), a cloud network, a data center network, etc. In one embodiment, system 20 includes one or more media generators 30 that generate media objects that are stored and analyzed.

[0040] System 20 also includes remote storage shelves 32 that are used by hosts 24 to store objects, metadata, and other related data. In some embodiments, storage shelves 32 are part of a storage area network (SAN) that communicates with network 28. Hosts 24 communicate with storage shelves 32 over network 28 to store and retrieve data.

[0041] exist Figure 1 In the example of FIG. 1 , storage shelf 32 includes a plurality of storage units referred to as an all-flash array (AFA) 36. (In alternative embodiments, any other suitable type of storage unit, not necessarily flash-based, and any other suitable type of storage device may be used.) Top-of-rack (TOR) switches 40 manage communications between AFA 36 and hosts 24 over network 28. In this example, storage shelf 32 including its various components is considered to be located at the “storage edge” of system 20.

[0042] Figure 1 The illustration in the lower left corner shows the internal structure of the AFA 36 in one embodiment. As shown, the AFA 36 includes a plurality of solid state drives (SSDs) 44 in which data (e.g., objects and metadata) are stored. The AFA 36 includes a storage controller 50 configured to manage the storage of data (e.g., media objects). The storage controller 50 is also referred to herein as an aggregation processor or aggregator. The AFA 36 also includes a switch 48 configured to communicate with the TOR switch 40 via a suitable network cable 42.

[0043] In some embodiments, the switch 48 communicates with the SSD 44 through a common peripheral component interconnect express (PCIe) bus using, for example, a non-volatile memory express (NVMe) protocol. In other embodiments, the host 24 communicates with the SSD 44 via the switch 48 using Ethernet (e.g., using the NVMe-over-fabrics protocol). Further alternatively, other suitable protocols may be used. Communication between the switch 48 and the TOR switch 40 is typically performed using Ethernet. In one embodiment, although not required, the AFA 36 includes a central processing unit (CPU) and / or a network interface controller (NIC) not shown in the figure for communicating with the TOR switch 40.

[0044] The inset in the lower right corner of the figure shows the internal structure of the SSD 44 according to one embodiment described herein. In this embodiment, each SSD 44 includes multiple flash memory devices 52 (e.g., NAND flash memory) and an SSD controller 56. The SSD controller 56 includes a memory interface 60 for communicating with the flash memory devices 52, a host interface 68 for communicating with the host 24 (via the switch 48 and the TOR switch 40), and a processor 64. The host interface 68 can communicate with the host 24 using any suitable storage protocol, for example, non-volatile memory express (NVMe) or serial advanced technology attachment (SATA).

[0045] As will be explained in detail below, the processor 64 performs storage / retrieval tasks and metadata calculation tasks. Non-limiting examples of metadata calculation tasks include identification, description and / or labeling of objects, activities, scene characteristics, and other features of content within a media object. In other embodiments, metadata calculation tasks are performed by the storage controller (aggregator) 50.

[0046] In one embodiment, processor 64 includes flash management module 72 and artificial intelligence (AI) reasoning engine 76. Flash management module 72 is configured to store and retrieve data (e.g., objects and metadata) in flash memory device 52. The tasks of flash management module 72 are also referred to as "flash translation layer" (FTL). As described below, AI reasoning engine 76 is configured to calculate metadata for media objects. In one embodiment, SSD controller 56 stores (i) media objects 80 and (ii) metadata database 84 that stores metadata for media objects 80 in flash memory device 52.

[0047] Notice, Figure 1 The SSD configuration of is a non-limiting example configuration, and any other suitable SSD controller may be used in alternative embodiments. For example, the following Figure 4 An alternative embodiment is described in which the SSD controller includes multiple integrated circuits (ICs) that communicate with each other via a suitable structure.

[0048] Figure 2 is a block diagram schematically illustrating a data processing system 90 according to one embodiment described herein, wherein an edge device 94 performs local metadata generation. In one embodiment, the edge device 94 includes, for example, an autonomous vehicle, a surveillance box, an IoT device, or any other suitable type of edge device.

[0049] In one embodiment, edge device 94 communicates with cloud-based data center 102 via a computer network (the Internet in this example). Data center 102 includes one or more hosts (not shown). Edge device 94 includes a central processing unit (CPU) cluster 106, local memory 110 (typically random access memory (RAM) or other volatile memory), and SSD 114. The internal structure of SSD 114 is shown in the inset on the right side of the figure in one embodiment. The structure of SSD 114 is similar to Figure 1 The structure of SSD 44.

[0050] Figure 3 1 is a flow chart schematically illustrating a method for combined storage / retrieval and metadata generation performed by SSD controller 56 according to one embodiment described herein. A storage / retrieval process 120 performed by flash management module 72 in processor 64 of SSD controller 56 is depicted on the left side of the figure. A metadata generation process 124 performed by AI inference engine 76 in processor 64 of SSD controller 56 is depicted on the right side.

[0051] The object storage / retrieval process begins with processor 64 receiving one or more storage / retrieval commands from host 24 via host interface 68 at command receiving operation 128. The storage / retrieval commands may include, for example, commands to read, write, modify, or delete one or more media objects. At command execution operation 132, flash management module 72 executes the commands in flash memory device 52 via memory interface 60.

[0052] The metadata generation process begins with the processor 64 receiving a pre-trained AI model from the host 24 via the host interface 68 at a model reception operation 136. At a compilation operation 140, the processor 64 compiles the AI ​​model and configures the AI ​​inference engine 76 with the compiled model.

[0053] At object acquisition operation 144, the AI ​​inference engine selects a media object stored in the flash device 52 (i.e., from the object 80) or received via the network 28 for storage in the flash device 52. In other words, metadata can be generated for media objects that have been stored in the flash device 52 or for objects that are intercepted immediately before being stored in the flash device 52.

[0054] At idle period identification operation 148, processor 64 checks whether at least some resources of processor 64 are not processing storage / retrieval commands. If not, metadata calculation is not performed. If an idle period is identified (or, alternatively, if the analysis task is defined as a high priority task by the host), processor 64 uses AI reasoning engine 76 to calculate metadata for the selected media object at metadata calculation operation 152. AI reasoning engine 76 calculates metadata using the AI ​​model provided at operation 136 and compiled at operation 140.

[0055] At database update operation 156, processor 64 adds the metadata for the selected object to local metadata database 84. (In alternative embodiments, the metadata database need not be local. In one embodiment, processor 64 sends the metadata to a remote storage location where the object data is at least temporarily stored locally.) The method then loops back to operation 144 above for selecting the next media object for metadata calculation.

[0056] Figure 3 The method flow of is an example flow depicted for the purpose of conceptual clarity only. In alternative embodiments, any other suitable method may be used by SSD controller 56 for combined object storage / retrieval and metadata calculation.

[0057] The metadata in the local database 84 of the various SSDs 44 is accessible to the host 24 for use. In some embodiments, the host 24 reads the metadata and possibly the associated objects or portions thereof from the SSD 44. In an alternative embodiment, the processor 64 of the SSD controller 56 receives a request for certain metadata from the host 24 and, in response, sends the requested metadata to the requesting host. In some embodiments, in addition to the requested metadata, the processor 64 also sends one or more related media objects or portions thereof to the requesting host. In other embodiments, the processor 64 actively (i.e., regardless of any request from the host) sends some or even all of the metadata to the host 24. In one embodiment, the host 24 analyzes the metadata and requests selected segments of the object media based on the analysis of the metadata.

[0058] In some embodiments, the analysis task of the host 24 requires the use of metadata from multiple metadata databases 84 from different SSDs 44. In such embodiments, the host typically obtains relevant metadata from multiple SSDs and merges the metadata into the required metadata.

[0059] In some embodiments, the AI ​​model and the object for storage are received from the same host 24. In other embodiments, the AI ​​model and the object for storage are received from different hosts 24.

[0060] In some embodiments, processes 120 and 124 are performed simultaneously. In some embodiments, the processor 64 gives object storage / retrieval (process 120) a higher priority than metadata generation (process 124). As described above, in some embodiments, the AI ​​reasoning engine 76 generates metadata during idle periods during which at least some resources of the processor 64 do not manage storage / retrieval of objects. In an example embodiment, the processor 64 identifies such idle periods in real time and, during the identified idle periods, initiates or resumes metadata generation. The processor 64 suspends the generation of metadata outside of the identified idle periods. In one embodiment, when the generation of metadata is suspended (outside of the idle period), the processor 64 buffers unprocessed media objects in volatile memory until the associated metadata is generated. Alternatively, the processor 64 may use any other suitable prioritization scheme to give priority to storage / retrieval relative to metadata generation.

[0061] Figure 4 1 is a block diagram schematically illustrating an SSD 160 according to an alternative embodiment described herein. In this example, SSD 160 includes multiple flash memory devices 52 (e.g., NAND flash memory), SSD controller 164, and (optionally) dynamic RAM (DRAM) 168. SSD controller 164 includes host interface 172, media controller 176, static RAM (SRAM) 180, one or more CPUs 184, DRAM controller 188, and AI reasoning engine 76, which optionally includes CPU 192.

[0062] The host interface 172 communicates with the host 24, for example, according to the NVMe, SATA protocol, or other suitable storage protocol. The media controller 176 is a processor that manages the storage and retrieval of data in the flash memory device 52, similar to Figure 1 and Figure 2 168. The DRAM controller 188 manages the storage of data in the DRAM 168. The various elements of the SSD controller 164 communicate with each other via a fabric 196, which typically includes appropriate data and control buses.

[0063] Among other features, Figure 4 The example demonstrates that in some embodiments, an SSD controller (or other storage device controller) includes multiple processors that jointly perform data storage / retrieval (e.g., storage / retrieval of media objects) and metadata calculations. The multiple processors can reside in one or more integrated circuits (ICs), packaged as a single device in a single package, or packaged in multiple separate packages.

[0064] above Figure 1, Figure 2 and Figure 4 The system and storage device configurations depicted in are example configurations chosen for purposes of conceptual clarity only. Figure 1 , Figure 2 and Figure 4 An example configuration is shown in which a controller of a storage device (e.g., an SSD controller in an SSD) communicates with one or more remote hosts via a computer network and communicates locally with an NVM of the storage device. The controller includes one or more processors that jointly perform dual functions: (i) managing local storage or retrieval of media objects in the NVM, and (ii) computing metadata for media objects stored or to be stored in the NVM and storing the metadata in the NVM. In alternative embodiments, any other suitable system and / or storage device configuration may be used. For clarity, elements that are not necessary for understanding the disclosed technology have been omitted from the figure.

[0065] In alternative embodiments, the disclosed technology may be used with other suitable types of storage devices, such as hard disk drives (HDDs) where the storage medium is magnetic.

[0066] Various elements of data processing systems 20 and 90 and their components (such as SSDs 44 and 114) and SSD 160 and its components (e.g., AI inference engine 76) may be implemented using dedicated hardware or firmware (such as using hardwired or programmable logic in an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA), using software, or using a combination of hardware and software elements.

[0067] Typically, processor 64, CPU 184, and / or CPU 192 of SSD controller 56 comprise a programmable processor that is programmed in software to perform the functions described herein (e.g., flash memory management and metadata calculations). The software may be downloaded to the processor in electronic form, such as over a network, or may alternatively or additionally be provided and / or stored on non-transitory tangible media, such as magnetic, optical, or electronic memory.

[0068] Although the embodiments described herein are primarily directed to media processing, the methods and systems described herein may also be used in other applications, such as human behavior analysis.

[0069] Note that the above embodiments are cited only as examples, and the present invention is not limited to the embodiments specifically shown and described above. On the contrary, the scope of the present invention includes combinations and sub-combinations of the various features described above, as well as variations and modifications thereof that will occur to those skilled in the art after reading the foregoing description and that are not disclosed in the prior art. The documents incorporated by reference into this patent application shall be considered as an integral part of this application, except that any term is defined in these incorporated documents in a manner that conflicts with the explicit or implicit definition in this specification, only the definition in this specification shall be considered.

Claims

1. A controller for use in a storage device of a data processing system, the controller comprising: a host interface configured to communicate with one or more remote hosts of the data processing system via a computer network; a memory interface configured to communicate locally with a non-volatile memory of the storage device; as well as A single processor configured to perform both: managing local storage or retrieval of media objects at the non-volatile memory; identifying an idle period during which at least some resources of the single processor are free from managing storage of the object; as well as During the idle period, metadata is selectively calculated, the metadata defining content characteristics of the media object stored or to be stored in the non-volatile memory, wherein during the identified idle period, the generation of the metadata is initiated or resumed, and outside the identified idle period, the generation of the metadata is suspended. 2 . The controller of claim 1 , wherein the single processor is configured to generate the metadata for a plurality of unstructured media objects stored or to be stored in the non-volatile memory.

3. A controller according to claim 1 or 2, wherein the single processor is configured to: receive a model from the one or more hosts via the computer network, the model specifying extraction of the metadata from the media object, generate the metadata based on the received model, and store the generated metadata locally in the non-volatile memory for use by the one or more hosts.

4. The controller of claim 3, wherein the single processor is configured to: receive a pre-trained artificial intelligence (AI) model as the model, and generate the metadata by applying an AI inference engine to the AI ​​model.

5. A controller according to claim 1 or 2, wherein the single processor is configured to: assign a first priority to the management of the storage or retrieval of the object in the non-volatile memory, and assign a second priority lower than the first priority to the generation of the metadata.

6. The controller of claim 1 or 2, wherein the single processor is configured to make the metadata accessible to the one or more hosts via the computer network, the one or more hosts being located remotely from the single processor.

7. The controller of claim 6, wherein the single processor is configured to make the metadata accessible by transmitting the metadata over the computer network, but less than all of the media object from which the metadata was generated.

8. The controller of claim 7, wherein after sending the metadata, the single processor is configured to send one or more of the media objects over the computer network in response to a request from one or more of the hosts.

9. The controller of claim 1 or 2, wherein the storage device comprises a solid state drive (SSD), and wherein the non-volatile memory comprises one or more flash memory devices.

10. The controller of claim 1 or 2, wherein the storage device comprises a hard disk drive (HDD), and wherein the non-volatile memory comprises one or more magnetic disks.

11. A method for generating metadata defining content characteristics of a media object, the method comprising: Communicating with one or more remote hosts in a data processing center through a computer network via a host interface coupled to a storage device located in the data processing center; locally communicating with a non-volatile memory of the storage device via a memory interface of the storage device; managing, by a single processor of the storage device, local storage or retrieval of media objects at the non-volatile memory; identifying, by the single processor, an idle period during which at least some resources of the single processor are free from managing storage of the object; as well as During the idle period, metadata is selectively calculated by the single processor of the storage device, the metadata defining content characteristics of the media object stored or to be stored in the non-volatile memory, including initiating or resuming the generation of the metadata during the identified idle period, and pausing the generation of the metadata outside the identified idle period.

12. The method of claim 11, wherein calculating the metadata comprises: The metadata is generated for a plurality of unstructured media objects stored or to be stored in the non-volatile memory.

13. The method according to claim 11 or 12, wherein calculating the metadata comprises: A model is received from the one or more hosts over the computer network, the model specifying extraction of the metadata from the media object, the metadata is generated based on the received model, and the generated metadata is stored locally in the non-volatile memory for use by the one or more hosts.

14. The method of claim 13, wherein receiving the model comprises: A pre-trained artificial intelligence (AI) model is received, and wherein calculating the metadata includes generating the metadata by applying an AI inference engine to the AI ​​model.

15. The method according to claim 11 or 12, comprising: A first priority is assigned to management of the storage or retrieval of the object at the non-volatile memory, and a second priority lower than the first priority is assigned to generation of the metadata.

16. The method according to claim 11 or 12, comprising: The metadata is made accessible to the one or more hosts over the computer network, the one or more hosts being located remotely from the single processor.

17. The method of claim 16, wherein making the metadata accessible comprises: The metadata, but less than all of the media object from which the metadata was generated, is transmitted over the computer network.

18. The method according to claim 17, comprising: After sending the metadata, one or more of the media objects are sent over the computer network in response to a request from one or more of the hosts.

19. The method of claim 11 or 12, wherein the storage device comprises a solid state drive (SSD), and wherein the non-volatile memory comprises one or more flash memory devices.

20. The method of claim 11 or 12, wherein the storage device comprises a hard disk drive (HDD), and wherein the non-volatile memory comprises one or more magnetic disks.

Images