Dataset Signatures for Data Impact-Driven Storage Management

Abstract

A method implemented by a data management system, comprising: a data set signature mapping program of the data management system stores operational metadata describing a data set, wherein the operational metadata and the data set are stored in a memory of the data management system; the data set signature mapping program of the data management system receives business-related metadata describing the data set from a business environment, wherein the business environment is external to the data management system; the data set signature mapping program of the data management system stores a data set signature, wherein the data set signature includes a combination of the operational metadata and the business-related metadata; and a policy manager of the data management system uses the combination of the operational metadata and the business-related metadata in the data set signature to perform data manipulation operations on the data set.

Description

Technical Field

[0001] The present invention relates to the field of information processing systems and more particularly to a collection of business-related metadata and operational metadata describing a data set in an information processing system. Background Art

[0002] Business entities or companies store large amounts of data in a data management system, such as a local storage system, a cloud-based storage system, an object storage system, or any other data repository, so that business applications can access the data to perform business-related tasks or services. Large amounts of data are usually stored in the data management system in a manner that optimizes the use of storage facilities and hardware available in the data management system. For example, a data management system is used to store data in a manner that facilitates flexible and efficient use of the stored data to support processing of various tasks to be performed on the data. Alternatively, a data management system can store data based on the attributes of the data, such as image data or sensor data. Summary of the invention

[0003] A first aspect of the present invention relates to a method performed by a data management system. The method comprises: a data set signature mapping program of the data management system stores operational metadata describing a data set, wherein the operational metadata and the data set are stored in a memory of the data management system; the data set signature mapping program of the data management system receives business-related metadata describing the data set from a business environment, wherein the business environment is external to the data management system; the data set signature mapping program of the data management system stores a data set signature, wherein the data set signature includes a combination of the operational metadata and the business-related metadata; and a policy manager of the data management system uses the combination of the operational metadata and the business-related metadata in the data set signature to perform a data manipulation operation on the data set.

[0004] Optionally, according to the first aspect, in a first implementation manner, the operation metadata includes at least one of an operation name, an operation type, an operation structure, an operation size, an operation security class or tag, an operation timestamp, an operation content type, an operation content class or tag, an operation index existence, an operation source, an operation version status, an operation encryption class, an operation encoding class, an operation movement status, an operation access class, an operation access heat class, an operation access skew, an operation access blockiness, an operation local entropy, an operation global entropy, or an operation relationship.

[0005] Optionally, according to the first aspect or any other implementation of the first aspect, in a second implementation, the operation metadata includes at least one of an operation name, an operation type, an operation structure, an operation size, an operation security class or tag, an operation timestamp, an operation content type, an operation content class or tag, an operation index existence, an operation source, an operation version status, an operation encryption class, an operation encoding class, an operation movement status, an operation access class, an operation access heat class, an operation access skew, an operation access blockiness, an operation local entropy, an operation global entropy, or an operation relationship.

[0006] Optionally, according to the first aspect or any other implementation of the first aspect, in a third implementation, the method further includes: the data set signature mapping program of the data management system determines the data set from multiple data sets stored locally in the data management system based on attributes of interest associated with the data set, wherein the attributes of interest associated with the data set include at least one of an operation name, a time when the data set was last accessed, an event associated with the data set, a previous operation performed on the data set, a current operation performed on the data set, or a future operation performed on the data set.

[0007] Optionally, according to the first aspect or any other implementation of the first aspect, in a fourth implementation, the method further includes: the data set signature mapping program of the data management system determines the business environment to receive the business-related metadata from the business environment based on a business data directory of the business environment, wherein the business data directory stores information describing data belonging to the business environment.

[0008] Optionally, according to the first aspect or any other implementation of the first aspect, in a fifth implementation, the method further includes: the dataset signature mapping program of the data management system sends a query to the business environment, requesting the business-related metadata describing one or more datasets stored in the business environment; the dataset signature mapping program of the data management system matches the dataset stored locally in the data management system with the business data stored in the business environment; the dataset signature mapping program of the data management system maps the business-related metadata describing the business data stored in the business environment to the operational metadata stored locally in the data management system; the dataset signature mapping program of the data management system generates and maintains the dataset signature describing the dataset.

[0009] Optionally, according to the first aspect or any other implementation manner of the first aspect, in a sixth implementation manner, the business-related metadata is received from a single business data directory or multiple different business data directories.

[0010] Optionally, according to the first aspect or any other implementation of the first aspect, in a seventh implementation, the business-related metadata from two or more external business data directories are combined by one of the two or more external business data directories, and then combined with the operation metadata to generate the data set signature.

[0011] Optionally, according to the first aspect or any other implementation of the first aspect, in an eighth implementation, the data set signature is generated by at least one of the data management system, cloud management software, management software or user in the following manner: receiving the business-related metadata or storage-oriented operational metadata from the storage manager of the data management system, and combining the operational metadata with the business-related metadata or storage-oriented operational metadata.

[0012] Optionally, according to the first aspect or any other implementation of the first aspect, in a ninth implementation, the method further includes: a storage manager of the data management system transmitting the data set signature to another storage manager.

[0013] Optionally, according to the first aspect or any other implementation of the first aspect, in a tenth implementation, the method further includes: a storage manager of the data management system transmitting the data set signature to the business environment to store the data set signature in a business data directory associated with the business environment.

[0014] Optionally, according to the first aspect or any other implementation of the first aspect, in an eleventh implementation, the business-related metadata is received from a business data directory associated with the business environment, wherein the business data directory describes data stored in a cloud computing environment or a data virtualization environment.

[0015] Optionally, according to the first aspect or any other implementation of the first aspect, in a twelfth implementation, the method further includes: a storage manager of the data management system transmitting the data set signature to a cloud-based environment or a data virtualization environment.

[0016] Optionally, according to the first aspect or any other implementation of the first aspect, in a thirteenth implementation, the method further includes: the data set signature mapping program of the data management system uses machine learning and training data to determine other metadata describing the data set based on the business-related metadata and the operational metadata.

[0017] Optionally, according to the first aspect or any other implementation of the first aspect, in a fourteenth implementation, the method further includes: the dataset signature mapping program of the data management system determines a system profile of a workload profile based on a dataset profile; wherein the workload profile describes a time-varying workload requested by a business environment to be executed by the system; wherein the system profile describes at least one of hardware, software, services, configurations, topologies, or resource spaces to support and execute the workload described by the workload profile; wherein the dataset profile includes one or more dataset signatures, and the one or more dataset signatures are based on at least one of an assumed dataset signature or a historical dataset signature of the dataset that the workload described in the workload profile is expected to access.

[0018] Optionally, according to the first aspect or any other implementation of the first aspect, in a fifteenth implementation, the policy manager of the data management system uses the combination of the operational metadata and the business-related metadata in the data set signature to perform data manipulation operations on the data set, and also includes: the policy manager of the data management system determines a system management policy, and the system management policy indicates the data manipulation operations to be performed on one or more systems or data sets based on the metadata contained in the data set signature.

[0019] Optionally, according to the first aspect or any other implementation of the first aspect, in a sixteenth implementation, the policy manager of the data management system uses the combination of the operation metadata and the business-related metadata in the data set signature to perform data manipulation operations on the data set, and also includes: the policy manager of the data management system determines a system management policy, and the system management policy indicates the data manipulation operation to be performed on the data set based on the metadata contained in the data set signature and the current position of the data set.

[0020] Optionally, according to the first aspect or any other implementation of the first aspect, in a seventeenth implementation, the policy manager of the data management system uses the combination of the operation metadata and the business-related metadata in the data set signature to perform data manipulation operations on the data set, and also includes: the policy manager of the data management system determines a system management policy, and the system management policy indicates the data manipulation operations to be performed on one or more data sets based on the metadata contained in the data set signature and the metadata contained in a second data set signature, wherein the second data set signature describes a second data set related to the first data set.

[0021] Optionally, according to the first aspect or any other implementation of the first aspect, in an eighteenth implementation, the policy manager of the data management system uses the combination of the operation metadata and the business-related metadata in the data set signature to perform data manipulation operations on the data set, and also includes: the policy manager of the data management system determines a system management policy, and the system management policy indicates the data manipulation operation to be performed on the data set based on the metadata contained in the data set signature and the operation performed on the data set.

[0022] Optionally, according to the first aspect or any other implementation of the first aspect, in a nineteenth implementation, the policy manager of the data management system uses the combination of the operation metadata and the business-related metadata in the data set signature to perform data manipulation operations on the data set, and also includes: the policy manager of the data management system determines a system management policy, and the system management policy indicates the data manipulation operations to be performed on one or more systems or logs based on the metadata contained in the data set signature and the user requesting access to the data set.

[0023] Optionally, according to the first aspect or any other implementation of the first aspect, in a twentieth implementation, the policy manager of the data management system uses the combination of the operational metadata and the business-related metadata in the data set signature to perform data manipulation operations on the data set, and also includes: the policy manager of the data management system determines a system management policy and a workload profile, the system management policy indicating the data manipulation operations to be performed on one or more systems and the data set based on the metadata contained in the data set signature, and the workload profile indicating the workload to be performed by the data management system to access the data set.

[0024] Optionally, according to the first aspect or any other implementation of the first aspect, in a twenty-first implementation, the policy manager of the data management system uses the combination of the operational metadata and the business-related metadata in the data set signature to perform a data manipulation operation on the data set, and the policy manager of the data management system determines a system management policy, and the system management policy indicates the data manipulation operation to be performed on the data set based on the metadata contained in the data set signature, wherein the data manipulation operation includes storing the data set in a specific location or a specific type of location based on the metadata contained in the data set signature.

[0025] Optionally, according to the first aspect or any other implementation of the first aspect, in a twenty-second implementation, the policy manager of the data management system uses the combination of the operational metadata and the business-related metadata in the data set signature to perform a data manipulation operation on the data set, and the policy manager of the data management system determines a system management policy, and the system management policy indicates the data manipulation operation to be performed on the data set based on the metadata contained in the data set signature and one or more related data sets, wherein the data manipulation operation includes marking the data set based on the metadata contained in the data set signature and the one or more related data sets.

[0026] Optionally, according to the first aspect or any other implementation of the first aspect, in a twenty-third implementation, the policy manager of the data management system uses the combination of the operation metadata and the business-related metadata in the data set signature to perform a data manipulation operation on the data set, which also includes: the policy manager of the data management system determines a system management policy, and the system management policy indicates the data manipulation operation to be performed on the data set based on the metadata contained in the data set signature and the metadata contained in a second data set signature, wherein the second data set signature describes a second data set related to the first data set, and the data manipulation operation includes asserting a relationship between the data set and the second data set based on the data set signature of the data set and the second data set signature of the second data set.

[0027] Optionally, according to the first aspect or any other implementation of the first aspect, in a twenty-fourth implementation, the data management system is included in an information processing system including multiple contexts, wherein a first subordinate context includes a stored data set, a second context includes on-demand data virtualization, a third context includes a data operations (DataOps) process, and a fourth upper-level context includes Internet of Things (IoT) state representation and event identification; wherein the data set signature includes metadata from at least one of the first subordinate context, the second context, the third context or the fourth context.

[0028] Optionally, according to the first aspect or any other implementation of the first aspect, in a twenty-fifth implementation, the method further includes: splitting the metadata extension of the object based on at least one of the value representations of the object observed, inferred or captured in another context.

[0029] Optionally, according to the first aspect or any other implementation of the first aspect, in a twenty-sixth implementation, the method further includes: splitting data directory mapping and integration capabilities across layers and systems.

[0030] Optionally, according to the first aspect or any other implementation of the first aspect, in a twenty-seventh implementation, the method further includes: splitting the service of the enhanced data profile into a broader management system.

[0031] A second aspect of the present invention relates to a data management system. The data management system comprises: a memory for storing instructions; a processor coupled to the memory and for executing the instructions, so that the processor is used to: store operational metadata describing a data set, the operational metadata and the data set are stored in the memory of the data management system; receive business-related metadata describing the data set from a business environment, the business environment is external to the data management system; store a data set signature including a combination of the operational metadata and the business-related metadata; and perform operations on the data set using the combination of the operational metadata and the business-related metadata in the data set signature.

[0032] A third aspect of the present invention relates to an apparatus. The apparatus comprises: a tool for storing operational metadata describing a data set, the operational metadata and the data set being stored in the apparatus; a tool for receiving business-related metadata describing the data set from a business environment, the business environment being external to the apparatus; a tool for storing a data set signature comprising a combination of the operational metadata and the business-related metadata; and a tool for performing an operation on the data set using the combination of the operational metadata and the business-related metadata in the data set signature.

[0033] These and other features will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] For a more thorough understanding of the present invention, reference will now be made to the following brief description taken in conjunction with the accompanying drawings and detailed description, wherein like reference numerals represent like parts.

[0035] Figure 1A A diagram for an information processing system;

[0036] Figure 1B A diagram of an information processing system for implementing a data set signature for data impact driven storage management provided by various embodiments of the present invention;

[0037] Figure 2 Schematic diagram of information system elements suitable for data impact driven storage management provided for various embodiments of the present invention;

[0038] Figure 3A and Figure 3B A diagram showing various examples of metadata included in a dataset signature associated with a dataset provided by various embodiments of the present invention;

[0039] Figure 4A A flowchart showing a method for generating a data set signature according to various embodiments of the present invention;

[0040] Figure 4B A flowchart illustrating another method for generating a data set signature according to various embodiments of the present invention;

[0041] Figure 5 A diagram of another information processing system for implementing a data set signature for data impact driven storage management provided by various embodiments of the present invention;

[0042] Figure 6 A diagram of another information processing system for implementing a data set signature for data impact driven storage management provided by various embodiments of the present invention;

[0043] Figure 7A diagram of another information processing system for implementing a data set signature for data impact driven storage management provided by various embodiments of the present invention;

[0044] Figure 8 A diagram illustrating the use of historical or hypothetical data set signatures in designing and / or tuning a system for an anticipated future workload provided by various embodiments of the present invention;

[0045] Fig. 9 A flowchart of a method for determining a system profile of a workload based on one or more data set signatures provided for various embodiments of the present invention;

[0046] Fig.10 A diagram showing various examples of system management policies executed or orchestrated based on data set signatures provided by various embodiments of the present invention;

[0047] Fig.11 A flowchart of a method for implementing a data set signature for data impact driven storage management provided by various embodiments of the present invention;

[0048] Fig.12 A diagram of an apparatus for implementing a data set signature for data impact driven storage management provided by various embodiments of the present invention. DETAILED DESCRIPTION

[0049] First, it should be understood that although illustrative implementations of one or more embodiments are provided below, the disclosed systems and / or methods may be implemented using any number of techniques, whether currently known or existing. The present invention should in no way be limited to the illustrative implementations, drawings, and techniques described below, including the exemplary designs and implementations illustrated and described herein, but may be modified within the scope of the appended claims and their full scope of equivalents.

[0050] Figure 1A is a diagram of information processing system 100 . Figure 1A The information processing system 100 in the embodiment includes a business environment 103 and a data management system 106, which are interconnected by one or more links 109. The link 109 can be a wired connection, a wireless connection, an interface, or any other type of connection between one or more devices or between different software in the business environment 103 and the data management system 106.

[0051] The business environment 103 is associated with a business entity, organization, enterprise, or company that uses one or more business applications 117, etc. to perform various services or tasks for users or customers. Business applications 117 can perform transaction management, sales, marketing, accounting, procurement, enterprise resource planning (ERP), customer relationship management (CRM), human resource management, data governance, business intelligence, data visualization, analysis, data mining, business asset management, database management (using DBMS), etc., alone or in combination. The business environment 103 includes hardware resources such as servers, processors, memory, routers, switches, virtual private networks (VPNs), gateways, etc., located in or near an office associated with the business entity. The hardware resources within the business environment 103 are used to store a business data catalog 115 associated with one or more business applications 117 to perform services or processes on behalf of the business entity.

[0052] The business entity may store a data set 120 used internally or externally by the business application 117 with respect to the business entity. For example, the data set 120 may be stored in the data management system 106. The data set 120 includes one or more data items, such as objects, tables, table spaces, documents, files, directories, file systems, blocks, and / or combinations thereof. In one embodiment, the data set 120 is characterized in the business data catalog 115 by attributes, business impact attributes, and relationships with other data.

[0053] The business data catalog 115 includes information describing data (e.g., data sets 120) associated with or belonging to the business environment 103. The business data catalog 115 consolidates and organizes the information describing the data sets 120 into a catalog service. An example of a business data catalog 115 is the WATERLINE data catalog, which catalogs information describing data stored in a database management system, a big data ecosystem, a cloud, a file system, etc. The business data catalog 115 allows users and / or business applications in the business environment 103 to perform metadata-based operations, such as searches relative to the data sets 120.

[0054] In various embodiments, the business data catalog 115 stores metadata describing data sets 120 associated with the business environment 103. In some embodiments, the business data catalog 115 also stores metadata describing business applications 117, processes, or services used by the business environment 103. For example, the business data catalog 115 stores metadata about the structure and / or content of a relational database and the manner in which the business applications 117 access the content. In some cases, users within the business environment 103 can add metadata to the business data catalog 115, such as text annotations and / or tags about the data sets 120 based on business analysis or transactions.

[0055] Metadata describing a data set 120 associated with a business context 103 may include definitions, tables, synonyms, views, indexes, tags, and annotations for describing properties, history, business impact properties, and relationships with other data sets 120. Figure 1A As shown, metadata describing a data set 120 associated with a business environment 103 includes business-related metadata 123. The business-related metadata 123 describes aspects of the data set 120 that may be considered important to a business application corresponding to the business environment 103. For example, the business-related metadata 123 may include the source and derivation of a particular data set 120. For another example, the business-related metadata 123 may be a note entered by an employee of the business entity, wherein the note describes the significance or business impact that a particular data set 120 has on the business entity. For another example, the business-related metadata 123 may include an estimated monetary value associated with a data set 120 owned by the business entity. Reference will also be made below to Figure 3A and Figure 3B Other examples of business-related metadata 123 are provided.

[0056] The data management system 106 is a storage system, a data store, a cloud computing environment, a data lake, a data fabric, a data virtualization engine, or any other data repository for storing and managing data sets 120 on behalf of one or more business environments 103. In one case, the data management system 106 can be located near or within the business environment 103. Alternatively, the data management system 106 can be located in a remote storage system that is geographically remote from the business environment 103, such as a cloud computing environment.

[0057] The data management system 106 includes a data store 129 for storing data sets 120 on behalf of the business environment 103 and a storage manager 126. The storage manager 126 is a device or software process within the data management system 106 for managing the data sets 120. The data store 129 may be a storage array or any type of memory that is large enough to store the data sets 120. In one embodiment, the data store 129 stores a pointer or reference to each data set 120. The storage manager 126 maintains operational metadata 133 that describes the data sets 120 stored in the data management system 106. The operational metadata 133 describes basic characteristics of the data sets 120, such as the format of the data set 120, a timestamp indicating the date the data set 120 was created or the last time the data set 120 was accessed, the size of the data set 120, etc. In some cases, the operational metadata 133 includes the access frequency of the data set 120, indicating how often the data set 120 is accessed within a standard time period.

[0058] However, the data management system 106 does not have permission to access or store the business-related metadata 123. In this case, the business-related metadata 123 is available in a higher-level business context so that the business application 117 can be executed to perform, schedule, or orchestrate business-related tasks or directory services using the business-related metadata 123. In addition, the business-related metadata 123 is also used among individuals involved in the business entity, such as employees, analysts, and data scientists, so that these individuals can make informed business decisions using the business-related metadata 123. However, the business-related metadata 123 is not available in a lower-level storage context where the data sets 120 used by the business environment 103 are actually stored. The embodiments disclosed herein relate to an enhanced data management system 106 that stores not only the operational metadata 133, but also the business-related metadata 123.

[0059] Figure 1B A diagram of an information processing system 150 for implementing data set signatures for data impact driven storage management is provided for various embodiments of the present invention. Figure 1B The information processing system 150 shown is similar to Figure 1AThe information processing system 100 shown is different in that the data management system 106 of the information processing system 150 also stores business-related metadata 123 describing one or more data sets 120. In one embodiment, the storage manager 126 is used to execute the data set signature mapping program 136 to receive the business-related metadata 123 from one or more business data directories 115 of one or more business environments 103. For example, the data set signature mapping program 136 can send a request to the business environment 103 via the link 109 to obtain the business-related metadata 123 associated with or describing one or more data sets 120. In one embodiment, the storage manager 126 receives the business-related metadata 123 from the business data directory 115 of the business environment 103 and then performs a mapping to determine the data set 120 corresponding to the received business-related metadata 123. In another embodiment, the data set signature mapping program 136 can send a request to the business environment 103 via the link 109 to obtain the business-related metadata 123 associated with one or more business applications 117 or associated with a specified type or aspect of the data set 120.

[0060] After determining the business-related metadata 123 for a particular data set 120, the data set signature mapper 136 retrieves or otherwise references the operational metadata 133 corresponding to the data set 120. The data set signature mapper 136 combines the operational metadata 133 with the business-related metadata 123 to generate a data set signature 140 for the data set 120. The data set signature 140 for the data set 120 includes an aggregation or combination of the operational metadata 133 describing the data set 120, which has been locally stored in the data management system 106, and the business-related metadata 123 received from the business environment 103. In one embodiment, when generating the data set signature 140, the data set signature mapper 136 may first reduce or transform the business-related metadata 123 to match the operational namespace of the storage manager 126, standardize the business-related metadata, or facilitate subsequent policy-based management of the data set 120.

[0061] In various embodiments, the data set signature 140 may be transmitted back to the business environment 103, a cloud management environment, another database management system responsible for managing different portions of the data store 129, or a different data store. The data management system 106 may also transmit the data set signature 140 to one or more business information processes, such as data governance, information lifecycle management, disaster recovery, and / or data brokering.

[0062] The embodiments disclosed herein are advantageous and provide practical applications in the field of information processing for a variety of reasons. By storing and maintaining business-related metadata 123 in a data management system 106, operations can be performed on data sets 120 more efficiently, more accurately, more finely, with lower latency, more comprehensively, and / or more timely. In addition, business entities corresponding to a business environment 103 can benefit from integrating business-related metadata 123 in a business context with operational metadata 133 in a storage context. For example, the embodiments disclosed herein enable the data management system 106 to identify data sets 120 that have the greatest impact on business entities corresponding to the business environment 103. This information can be used to enhance the storage of business tools, processes, and data sets 120. Data sets 120 that have a high impact on business entities corresponding to the business environment 103 or on strategic business applications 117 can be stored in a layer or other storage location with fast access time and low latency. In addition, data set signatures 140 stored by the data management system 106 can be used to search, classify, select, manipulate, and operate data sets 120 based on combined operations and business knowledge. In embodiments where the data set signature 140 is transmitted back to the business environment, another management environment, or a business information process, these environments or business information processes can improve their operational decisions, operational efficiency, or positive impact on the business environment 103 by understanding the combined operations and business aspects of the data set signature 140.

[0063] In some embodiments, the data management system 106 performs data manipulation operations on the data set 120 based on the data set signature 140 and various system management policies associated with the data set signature 140. In one embodiment, the system management policy indicates the data manipulation operations to be performed on the data set based on the data set signature 140 describing the data set 120. For example, the system management policy may indicate that when the data set 120 is described by a specific type of metadata identified in the data set signature 140, the data management system 106 is to perform a specific data manipulation operation on the data set 120 based on the metadata identified in the data set signature 140. In this example, when the data set 120 is marked as having high business value, the data management system 106 may implement a security mechanism on the data set 120, enforce an increased cost for accessing the data set 120, or perform any other operation related to the data set 120.

[0064] For another example, the system management policy may indicate that when the data set signatures 140 of two data sets 120 indicate that the two data sets 120 are copies of each other, certain data manipulation operations should be performed. For example, the system management policy may indicate that when an operation (e.g., a write operation) is performed on one of the data sets 120, the operation should also be performed on the other copy of the data set 120. Alternatively, the system management policy may indicate that when the data set signatures 140 of the two data sets 120 indicate that the two data sets 120 are copies of each other, one of the data sets 120 should be deleted.

[0065] In this manner, data manipulation operations may be invoked on data sets 120 more efficiently, more accurately, more comprehensively, and / or more timely by using the combination of the operational metadata 133 and the business-related metadata 123 in the data set signature 140. In some cases, a business entity may save costs and resources by allowing the data set signature 140 to be created and stored in the data management system 106. For example, when the data set signatures 140 of two separate data sets 120 indicate that the two data sets 120 are copies of each other, a system management policy preconfigured in the data management system 106 may indicate that an operation performed on one of the data sets 120 should also be performed on the other copy of the data set 120. In the case where the data management system 106 is a cloud environment and the business entity is charged each time a data set 120 stored in the cloud environment is accessed, the business entity may save costs by performing an operation on only one of the data sets 120 because the data management system 106 is used to automatically perform the same operation on the other copy of the data set 120.

[0066] For another example, a specific data set 120 is marked as having important value to the business entity, which can be indicated in the data set signature 140 of the data set 120. In this case, a system management policy is pre-configured in the data management system 106 to automatically perform a data manipulation operation on the data set 120 marked as having value to the business entity. The data manipulation operation can be intended to further protect the data set 120, for example, by encrypting the data set 120 using a public key or a private key. In this example, the data management system 106 is used to automatically perform a data manipulation operation on the data set 120 based on the data set signature 140, and the data manipulation operation is intended to protect the data set 120 that is valuable to the business entity.

[0067] Furthermore, by using the described combination of operational metadata 133 and business-related metadata 123 in data set signature 140, the performance and management of data management system 106 is greatly improved. Due to the additional details stored in data set signature 140, data manipulation operations can be performed on data sets 120 more efficiently and accurately. For example, if a specific read operation is to be performed on all data sets 120 having a certain characteristic, data management system 106 can search data set signature 140 to quickly find all data sets 120 having the certain characteristic without having to analyze each data set 120 individually. In this way, data set signature 140 greatly improves the value of data management system 106 while also reducing costs and overhead from the perspective of business entities.

[0068] Figure 2 Schematic diagram of an information processing unit 200 suitable for implementing a data set signature for data impact driven storage management according to various embodiments of the present invention. In one embodiment, the information processing unit 200 may be implemented as a data management system 106, a storage manager 126, or a business environment 103.

[0069] The information processing unit 200 includes a port 220, a transceiver unit (Tx / Rx) 210, a processor 230, and a memory 233. The processor 230 includes a data set signature mapping program 136 and an optional policy manager 137. The port 220 is coupled to the Tx / Rx 210, which can be a transmitter, a receiver, or a combination thereof. The Tx / Rx 210 can send and receive data through the port 220. The processor 230 is used to process data. The memory 233 is used to store data and instructions for implementing the embodiments described herein. The information processing unit 200 may also include an electrical-to-optical (EO) component and an optical-to-electrical (OE) component coupled to the port 220 and the Tx / Rx 210 for receiving and sending electrical signals and optical signals.

[0070] The processor 230 may be implemented by hardware and software. The processor 230 may be implemented as one or more central processing units (CPU) and / or graphics processing units (GPU) chips, logic units, cores (e.g., multi-core processors), field-programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and digital signal processors (DSPs). The processor 230 communicates with the port 220, Tx / Rx 210, and memory 233. The storage manager 126, the data set signature mapping program 136, and the policy manager 137 are implemented by the processor 230 to execute instructions for implementing the various embodiments discussed herein. For example, the storage manager 126 is used to manage the data set 120 and the storage data directory 240. The data set signature mapping program 136 is used to: determine the business environment 103 from which the business-related metadata 123 is received; determine the type of business-related metadata 123 to be received from the business environment 103; and map the business-related metadata 123 received from the business environment 103 to the data set 120 stored in the data management system 106. The policy manager 137 is used to apply the system management policy 243 based on the data set signature 140. Including the storage manager 126, the data set signature mapping program 136, and the policy manager 137 improves the functionality of the information processing unit 200. The storage manager 126, the data set signature mapping program 136, and the policy manager 137 also implement the transition of the information processing unit 200 to different states. Alternatively, the storage manager 126, the data set signature mapping program 136, and the policy manager 137 are implemented as instructions stored in the memory 233. In one embodiment, the functionality of the policy manager 137 may not be a different component within the information processing unit 200. Instead, the functionality of the policy manager 137 may be outside the information processing system 200 or even the data management system 106.

[0071] Memory 233 includes one or more disks, tape drives, or solid-state drives and can be used as an overflow data storage device to store programs when they are selected for execution and to store instructions and data read during program execution. Memory 233 can be volatile and / or non-volatile and can be a read-only memory (ROM), a random access memory (RAM), a ternary content-addressable memory (TCAM), and / or a static random-access memory (SRAM).

[0072] In one embodiment, the memory 233 is used to store data sets 120 and a storage data directory 240. The storage data directory 240 includes multiple data set signatures 140 corresponding to multiple different data sets 120. Each data set signature 140 includes operational metadata 133 and business-related metadata 123 that describe the data set 120. The memory 233 also stores system management policies 243 that indicate how to manage and perform operations on data sets 120 associated with a particular data set signature 140. The system management policies 243 can also indicate how to manage and perform operations on data sets 120 with specific types of operational metadata 133 and / or specific types of business-related metadata 123. The system management policies 243 are applied to data sets 120 or various systems being managed based at least in part on the data set signatures 140, as will be referenced below. Fig.10 Further described.

[0073] It should be understood that by programming and / or loading executable instructions onto the information processing unit 200, at least one of the processor 230 and / or the memory 233 will be changed, thereby partially converting the information processing unit 200 into a specific machine or device having the novel functions described in the present invention, such as a multi-core forwarding architecture. The functions implemented by loading executable software into a computer can be converted into hardware implementations through recognized design rules, which are very basic in the fields of power engineering and software engineering. The decision to use software or hardware to implement a concept usually depends on considerations of design stability and the number of units to be produced, rather than any issues involved in converting the software domain to the hardware domain. In general, designs that change frequently are more suitable for implementation in software because rewriting the hardware implementation is more expensive than rewriting the software design. Generally, stable and mass-produced designs are more suitable for implementation in hardware such as ASICs because mass-produced hardware implementations are cheaper than software implementations. Designs can usually be developed and tested in software form, and then converted into equivalent hardware implementations in ASICs through recognized design rules, and the hardware implementations are hardwired to software instructions. A machine controlled by the new ASIC is a specific machine or device, similarly, a computer programmed and / or loaded with executable instructions can be considered a specific machine or device.

[0074] Figure 3A and Figure 3B FIG. 1 is a diagram showing various examples of metadata included in a dataset signature 140 associated with a dataset 120 provided by various embodiments of the present invention. The dataset signature 140 includes operational metadata 133 and business-related metadata 123. Examples of operational metadata 133 are as follows: Figure 3A As shown, an example of business-related metadata 123 is as follows Figure 3B shown.

[0075] Operational metadata 133 includes basic metadata describing basic characteristics of data set 120. Figure 3A As shown, for example, the operation metadata 133 may include at least one of an operation name 307, an operation type 308, an operation structure 309, an operation size 310, an operation security class or tag 311, an operation timestamp 312, an operation content type 313, an operation content class or tag 314, an operation index existence 315, an operation source 316, an operation version status 317, an operation encryption class 318, an operation encoding class 319, an operation movement status 320, an operation access class 321, an operation access heat class 322, an operation access skew 323, an operation access blockiness 324, an operation local growth rate 325, an operation global entropy 326, and / or an operation relationship 327. It should be understood that the operation metadata 133 may include Figure 3AExamples of operational metadata 133 may be stored in the data management system 106 in the form of fields, key-value pairs, RDF triples or graphs, or any other type of data structure. For example, operational metadata 133 is maintained by storage manager 126.

[0076] The operation name 307 refers to the name of the data set 120. For example, the operation name 307 may be Sensor1354_09032019_Readings, or may be a system-generated object identifier. The operation type 309 indicates the type of the data set 120, for example, whether the data set 120 is a file, an object, a storage volume, or a storage logical unit number (LUN). The operation structure 309 refers to the structure of the data set 120, for example, whether the data set 120 is unstructured, semi-structured, relational, a key-value pair, or a storage object. The operation size 310 refers to the size of the data set 120, for example, in units of physical size (e.g., megabytes (MB), gigabytes (GB), terabytes (TB), etc.). The operation security class or tag 311 refers to a security tag or class associated with the data set 120, which may be used by a data placement policy. The operation timestamp 312 indicates a timestamp of an event occurring at the data set 120, for example, a timestamp indicating the time when the data set 120 was created, last accessed, or last modified.

[0077] Operation content type 313 indicates the content type of data set 120, for example, Multipurpose Internet Mail Extensions (MIME), or whether data set 120 is an image or audio file or is related to an image or audio file. Operation content class or tag 314 refers to a content class or tag associated with data set 120. Operation content class or tag 314 can be a custom tag, such as a tag indicating that data set 120 is "sensor data" or "X-ray image". Operation index existence 315 includes a binary value indicating whether data set 120 is indexed. Operation source 316 indicates the source of data set 120, such as an identification of a public cloud.

[0078] The operational version state 317 indicates the version state of the data set 120, for example, an indication of whether the data set 120 is an active version, a deleted version, a backup copy, an archived version, etc. The operational encryption class 318 indicates whether the data set 120 is encrypted or to be encrypted. The operational encoding class 319 indicates the encoding method of the data set 120. For example, when the data set 120 is encoded as a text compressed gzip file, the operational encoding class 319 indicates that the data set 120 is encoded as a text compressed gzip file. The operational movement state 320 indicates the movement state of the data set 120. For example, the operational movement state 320 of the data set 120 can indicate whether the data set 120 is being streamed or in a static state.

[0079] The operation access class 321 may indicate the access class of the data set 120, for example, whether the data set 120 is a read-only / immutable file, a read-write file, an append-only file, or a write-once file. The operation access heat class 322 indicates the access frequency of the data set 120, for example, whether the data set 120 is cold (rarely accessed), warm (infrequently accessed), hot (frequently accessed), or very hot (very frequently accessed). In an alternative embodiment, the operation access heat class 322 may be defined using an access frequency range. The operation access skew 323 indicates where the data set 120 falls in the overall access skew of data within a certain range. A typical example of an operation access skew is "90% of reads are reading 10% of the data in the data set." The operation access blockiness 324 indicates the uniformity or non-uniformity of access to the data set 120 over time. For example, some data may only be accessed on the hour and never accessed at night or on weekends. The operation local growth rate 325 refers to the rate of change (e.g., growth or shrinkage) of the data set 120 considered in isolation.

[0080] Operational global growth rate 326 refers to the rate of change of data set 120 relative to other data within a certain range. Operational relationship 327 refers to the operational relationship between data set 120 and other data. For example, operational relationship 327 indicates whether data set 120 is a copy, version, index or subset of another data set. For another example, operational relationship 327 indicates whether data set 120 and another data set should be stored together or deleted together.

[0081] Figure 3B Business-related metadata 123 is shown, which includes metadata describing information related to business aspects or utility of the data set 120 for one or more business entities. Figure 3BAs shown, for example, the business-related metadata 123 may include a business name 330, a business type 331, a business natural language description 332, a business owner 333, a business manager 334, a business content class / tag 335, a business authority workflow 336, a business organization level 337, a business data geography 338, a business structure type 339, a business structure 340, a business language 341, a business size 342, a business security class 343, a business timestamp 344, a business source or origin 345, a business cleanup or maintenance 346, At least one of business dataset profile 347 , business dataset virtualization status 348 , business sort order 349 , business retention date 350 , business compliance obligations 351 , business relationships 352 , business usage counts 353 , business utility ratings 354 , business user authenticity ratings 355 , business data scientist utility ratings 356 , business data scientist reviews 357 , business decision impact 358 , business monetary impact 359 , business aggregate dataset impact 360 , and / or average impact of business dataset friends 361 . It should be appreciated that the business-related metadata 123 may include Figure 3B Other metadata or descriptions not shown in the example. Examples of business-related metadata 123 can be stored in the business environment 103 and / or data management system 106 in the form of fields, RDF triples or graphs, key-value pairs, or any other type of data structure. In some cases, business-related metadata 123 is stored in the business data catalog 115, one or more business applications 117, or a business information process. Business-related metadata 123 can be stored in any location and can be accessed through an API or as a service.

[0082] The business name 330 refers to the name of the data set 120, which may be the same as the operation name 307 of the data set 120, or may be different from the operation name 307 of the data set 120. The business type 331 refers to the type of the data set 120, for example, whether the data set 120 is a file, a logical object, or a table. The business natural language description 332 refers to the natural language description of the data set 120. For example, the business natural language description 332 may indicate that the data set 120 includes standardized temperature readings from a specific sensor. The business owner 333 indicates the owner of the data set, for example, the sales department of a business entity in the New York State area. The business administrator 334 refers to the business custodian or a person representing the business owner. The business content class or tag 335 indicates the content class or tag of the data set 120. For example, the business content class or tag 335 of the data set 120 may indicate that the data set 120 refers to customer purchase history, autonomous vehicle sensor readings, personally identifying information (PII), etc. Business permission workflow 336 indicates a process for allowing a user or application permission to access a data set 120, as an alternative to allowing access to the data set 120 through user roles, access control lists, and / or other mechanisms. Business organization level 337 indicates the organizational level of the business owner 333 of the data set 120. For example, business organization level 337 may indicate that the data set 120 is owned by a department, division, business line, headquarters, etc. Business data geography 338 indicates a geographic region associated with the data set 120.

[0083] Business structure type 339 indicates the type of dataset 120, for example, an architecture, encoding, markup language, or resource description framework (RDF) triple. Business structure 340 indicates the structure of dataset 120, for example, the actual schema or key space of key-value pairs. Business language 341 indicates the written language used by the business entity or user accessing dataset 120, for example, "English" or "French". Business size 342 indicates the size of dataset 120 expressed in logical units, for example, the number of relationship rows or record count. Business security class 343 indicates the security class of dataset 120 for business entities, which may be meaningful for data governance and access control. Business timestamp 344 indicates the timestamp of the event that occurred at dataset 120.

[0084] The business source or origin 345 indicates the source or origin of the data set 120. For example, the business source or origin 345 indicates whether the data set 120 comes from a data warehouse or from an operational database management system. The business source or origin 345 may also indicate actions that have been recorded in the data set 120, such as a data set merge, split, or transformation. Business cleansing or curation 346 indicates whether the data set 120 has been cleaned or repaired, such as through data normalization to add default values ​​for missing fields, etc. The business data set profile 347 may include detailed information associated with one or more fields of the data set 120, such as a value range or a minimum or maximum value. The business data set virtualization state 348 indicates whether the data set 120 is a virtual data set assembled on demand from an existing data set. The business sort order 349 includes a sort key and the direction (ascending or descending) in which the stored data set 120 is sorted relative to the sort key.

[0085] Business retention date 350 indicates how long to retain dataset 120 before it is deleted or scheduled for deletion. Business compliance obligation 351 indicates and / or references enterprise policies, professional data processing standards, or legal requirements related to operations and / or actions performed on dataset 120. Business relationships 352 include other datasets related to dataset 120, such as datasets that are part of, subsets of, reduced versions of, friends of, or derived from dataset 120. Business relationships 352 are useful because actions on dataset 120, such as move, delete, and mark as, may need to be propagated to related datasets 120. Business relationships 352 can indicate: structural composition, such as object1 is part of object2; associations, such as attribute copy, attribute version, attribute index, attribute subset, derived from, attribute reduced version; groupings, such as store together, delete together, friends; and / or type-level relationships, such as HDFS is a subtype of file_system, file marked as "contains personally identifiable information."

[0086] The business usage count 353 indicates the number of times (total or by user type) that the dataset 120 was used to provide a process or service for the business environment 103 during a given time period (e.g., year to date). The business utility rating 354 refers to a rating provided by one or more users in the business environment 103 indicating the business utility of the dataset 120. For example, a business utility rating 354 of 0 indicates that the dataset 120 is least useful to the business environment 103, and a business utility rating 354 of 10 indicates that the dataset 120 is most useful to the business environment 103. The business user authenticity rating 355 refers to a rating provided by one or more users in the business environment 103 indicating the authenticity, accuracy, and / or completeness of the dataset 120, e.g., with respect to the category or type of data described by the dataset 120. For example, a business user authenticity rating 355 of 0 indicates that the dataset 120 is least accurate or least trustworthy to the user, and a business user authenticity rating 355 of 10 indicates that the dataset 120 is most accurate or most trustworthy to the user. Business data scientist usefulness rating 356 refers to a rating provided by one or more data scientists indicating the value of data set 120 for performing data analytics on behalf of business environment 103. Business data scientist reviews 357 include one or more reviews provided by the data scientist regarding the affinity of using data set 120, for example, in the development of a particular type of analytical model.

[0087] The business decision impact 358 represents the business decision made using the data set 120. For example, a low business decision impact 358 represents a small historical impact of the data set 120 on the business decision, and a high business decision impact 358 represents a large historical impact of the data set 120 on the business decision. The business monetary impact 359 represents an estimate of the aggregate or periodic monetary result of the decision made using the data set 120. The business monetary impact 359 associated with the data set 120 may be reduced when additional data sets 120 also contribute to the aggregate or periodic monetary result. The business monetary impact 359 may also represent a monetary value associated with the data set 120, which may have been determined by a valuation professional. The business aggregate data set impact 360 represents the value or aggregate value of the data set 120 to the business environment 103 or business entity. For example, the business aggregate data set impact 360 may be based on the business monetary impact 359, the business decision impact 358, and / or reference Figure 3A and Figure 3BAn aggregated reflection of the importance of any other metadata shown and described to the dataset 120. The average impact 361 of business dataset friends refers to the business impact or value of the friends of the dataset 120. The friends of the dataset 120 refer to other datasets stored in the data management system 106 that are similar to or have a corresponding relationship with the master dataset 120. For example, for the master dataset 120, the friends of the dataset 120 may include: other datasets with similar content to the master dataset 120; other datasets describing similar categories or types of content as the master dataset 120; other datasets accessed, updated, or edited by the same user; other datasets with similar access frequencies; other datasets with comparable ratings; other datasets with similar monetary impact; or any other datasets with similarity to the master dataset 120. The friends of the dataset 120 may be manually labeled, inferred by a policy, or identified through machine learning classification.

[0088] Figure 3A and Figure 3B The data set signature 140 shown in FIG. 1 includes Figure 3A and Figure 3B All metadata shown in and described above. However, the dataset signature 140 of the dataset 120 need not include Figure 3A and Figure 3B All metadata shown in and described above. In some cases, the dataset signature 140 need only include Figure 3A and Figure 3B One or more of the metadata shown in and described above, or the dataset signature 140 includes Figure 3A and Figure 3B A subset of the metadata shown in and described above. There may be multiple instances of one or more different types of business-related metadata 123. For example, there may be multiple instances of business compliance obligations 351.

[0089] In one embodiment, the business-related metadata 123 is initially stored in the business data catalog 115 at the business environment 103. For example, the business-related metadata 123 can be stored in the business data catalog 115. For example, a portion of the business-related metadata 123 can be stored in the business data catalog 115 as annotations or inputs provided by a user of the business environment 103. In one embodiment, the data management system 106 retrieves the business-related metadata 123 from the business data catalog 115 and stores the business-related metadata 123 of the data set 120 with the operational metadata 133 of the data set 120 in the data set signature 140.

[0090] As an illustrative example, the data set 120 may include on-demand virtualized objects, data operations (DataOps) process flows, states and / or results, Internet of Things (IoT) state representations, or IoT event identification. In this case, the business-related metadata 123 may be metadata describing the DataOps objects, IoT state representations, or IoT event identifications.

[0091] Figure 4A 4 is a flowchart illustrating a method 400 for generating a data set signature 140 provided by various embodiments of the present invention. The method 400 may be implemented by the information processing unit 200 or the storage manager 126 in the data management system 106. In the storage manager 126, all or part of the method 400 may be implemented by the data set signature mapping program 136. In one embodiment, the method 400 may be implemented after the data set 120 is stored in the data storage area 129 of the data management system 106.

[0092] In one embodiment, a data set 120 for which a data set signature 140 should be created is determined. In one embodiment, the data set signature mapping program 136 determines the data set 120 from a plurality of data sets 120 stored locally in the data management system 106 based on an attribute of interest associated with the data set 120. The attribute of interest associated with the data set 120 may be at least one of an operation name 307, a time when the data set 120 was last accessed, an event associated with the data set 120, a previous operation, a current operation, or a future operation performed on the data set 120, and / or any other operation metadata associated with the data set 120.

[0093] After the data set 120 is selected, in step 403, the operational metadata 133 describing the data set 120 is stored locally in the data management system 106. In one embodiment, the data set signature mapper 136 retrieves the operational metadata 133 from an external entity and then stores the metadata locally in the storage data directory 240, which can be stored in the database management system 106. In another embodiment, the operational metadata 133 is stored in the storage data directory 240 before the data set 120 is selected. In one embodiment, the operational metadata 133 is managed elsewhere by the storage manager 126. The operational metadata 133 may include the above combined Figure 3A One or more of the metadata that describes the

[0094] In step 406, a business environment 103 is determined that stores business-related metadata 123 that may be similar, related, or interrelated to the data set 120. For example, the data set signature mapping program 136 may access multiple business environments 103, each business environment 103 storing a different business data catalog 115. The data set signature mapping program 136 selects one or more of the multiple business environments 103 that the data management system 106 can access to receive the business-related metadata 123 therefrom.

[0095] In step 409, the business data stored in the business environment 103 is matched with one or more data sets 120 stored in the data management system 106 using the corresponding metadata. In one embodiment, the business data and the data sets 120 are matched based on explicit matching or inferred matching. Explicit matching can be performed when an object in the business data has the same namespace as the data set 120 or an object in the data set 120. In this case, the object in the business data can be explicitly matched with the object in the data set 120 by name. Inferred matching can be performed when some metadata describing the business data and some metadata describing the data set 120 indicate that the business data and the data set 120 may be identical or nearly identical to each other.

[0096] In step 410, the selected business-related metadata 123 associated with the matching business data stored in the business environment 103 is mapped to the operational metadata 133 of the one or more matching data sets 120 stored in the data management system 106 determined in step 409. In some cases, the business-related metadata 123 is received from the business environment 103 without context about how the business-related metadata 123 is related to a specific data set 120. In one embodiment, the data set signature mapper 136 is used to determine a mapping between the business-related metadata 123 and one or more data sets 120 based on a relationship or correlation between the business-related metadata 123 and the one or more data sets 120. For example, the business-related metadata 123 may include the business monetary impact 359 of the data set 120 and identification information of the data set 120. The data set signature mapper 136 is used to map the identification information of the data set 120 to the identification information of the locally stored data set 120 to map the business monetary impact 359 to the data set 120. In some embodiments, the business-related metadata 123 is mapped to the data set 120 based on a relationship between explicit identification metadata, intrinsic identification metadata, or inferred metadata.

[0097] In one embodiment, after receiving the business-related metadata 123, the data set signature mapping program 136 further filters the received business-related metadata 123 to determine the business-related metadata 123 related to the data set 120 based on the parameters set for the data set 120. In one embodiment, the business environment 103 sets parameters that define the type of metadata that should be stored in the data set signature 140 of the data set 120. The data set signature mapping program 136 filters the received business-related metadata 123 based on the parameters set by the business environment 103. In another embodiment, the data set signature mapping program 136 can set the parameters, or can be guided to set the parameters through an interface, to define the type of metadata that should be stored in the data set signature 140 of the data set 120.

[0098] For example, the business environment 103 sets a parameter indicating that the business monetary impact 359 of the data set 120 should be received from the business data catalog 115 and stored in the data set signature 140 of the data set 120. In this example, the data set signature mapper 136 filters the business-related metadata 123 received from the business environment 103 to retrieve the business monetary impact 359 of the data set 120. The storage manager 126 creates the data set signature 140 including the business monetary impact 359 of the data set 120. In another embodiment, filtering the business-related metadata 123 is performed by the business environment before the data set signature mapper 136 receives the business-related metadata 123.

[0099] In step 412, the selected business-related metadata 123 is received from the business environment 103 determined in step 406. For example, the Tx / Rx 210 of the data set signature mapping program 136 receives the business-related metadata 123 from the business environment 103.

[0100] In step 415, one or more data set signatures 140 including operational metadata 133 describing one or more data sets 120 and selected business-related metadata 123 are stored locally in the storage directory 240. In one embodiment, the data set signature 140 includes the business-related metadata 123 after mapping, matching, and filtering are performed on the business-related metadata 123.

[0101] In step 418, the dataset signature 140 is periodically updated and maintained in the storage catalog 240 by retrieving additional or updated business-related metadata 123 from the business environment 103. In one embodiment, the dataset signature mapping program 136 periodically requests and retrieves any updates or changes to the locally stored business-related metadata 123 from the business data catalog 115 of the business environment 103. In another embodiment, the business environment 103 automatically and periodically sends updates or changes to the business-related metadata 123 to the dataset signature mapping program 136 of the data management system 106. The dataset signature mapping program 136 updates the dataset signature 140 of the dataset 120 based on the updates received from the business environment 103. In another embodiment, business-level or storage-level events can trigger the update and maintenance of the dataset signature 140. For example, such an event can be a refresh of the business-related metadata 123 initiated by the business application 117, or the completion of a security operation based on the classification of the dataset content to mark the operational metadata 133.

[0102] Figure 4B Flowchart of a method 450 for generating a data set signature 140 provided by various embodiments of the present invention. The method 450 including an explicit mapping range may be implemented by the information processing unit 200, the storage manager 126, or the data set signature mapping program 136. The method 450 may be implemented after the data set 120 is stored in the data storage area 129 of the data management system 106.

[0103] In step 453, a filter of the storage data catalog 240 (referred to as a storage-side mapping scope) is determined to identify and select the storage data sets 120 of interest. The storage-side mapping scope refers to one or more conditions and combinations of conditions of the operational metadata 133. The storage-side mapping scope is applied to the operational metadata 133 of the storage data sets 120 and can be used to select one or more data sets 120 (also referred to as "targets"). For example, the data set signature mapper 136 determines the filter or storage-side mapping scope of the storage data catalog 240.

[0104] In step 456, the business data directory 115 is queried through an API or other mechanism to determine the business data sets in the business side mapping range that correspond or may correspond to the storage side mapping range. In one embodiment, given the storage side mapping range determined in step 453, the corresponding business side mapping is derived to identify the business data sets in the business data directory 115 that may correspond to the storage data sets 120 in the storage side mapping range. In one embodiment, there may be business data sets in the business data directory 115 of the business side mapping range that do not correspond to the storage data sets 120 in the storage side mapping range.

[0105] The business-side mapping scope represents a source for mapping, matching, and combining business-related metadata 123 to data set signatures 140. In one embodiment, requesting metadata about a data set in a business-side mapping scope is not necessarily the same as requesting metadata associated with or describing one or more specific data sets 120. The names and granularity of the data sets may be different in the business-related metadata 123 and the operational metadata 133. In a sense, data sets may be different at the business level and the operational level.

[0106] The business data set is virtual in a sense because the business data set is represented by the business-related metadata 123 in the business data directory 115, business application 117, or business environment 103. In some cases, the business data set is actually stored in an operational data set managed by one or more data managers. For example, actual access to a portion of a business data set may involve accessing one or more underlying stored data sets directly or through a copy (e.g., an intermediate cache). In this embodiment, the storage manager 126 or the data set signature mapping program 136 queries the business data directory 115 through an API or other interface to determine the business data sets within the business side mapping range.

[0107] In step 459, the business data set determined in step 456 is mapped to the corresponding storage data set 120 by a combination of the data set name and other metadata, optionally using previously discovered correspondences from the storage data directory 240. In one embodiment, step 459 is combined with the above Figure 4A Some steps described in step 410 are similar, wherein the data set signature mapping program 136 is used to map the business data set to the corresponding storage data set 120.

[0108] In step 462, the selected metadata about the business data set within the business side mapping range is matched and propagated from the business data directory 115 to the storage data directory 240. The selected metadata is associated with the corresponding storage data set 120 to create or modify the data set signature 140 in the storage data directory 240. In one embodiment, step 462 is combined with the above Figure 4A Some steps are similar to those described in step 410, wherein the data set signature mapping program 136 is used to match the business-related metadata 123 about the business data set within the business side mapping scope from the business data directory 115 to the storage data directory 240, and then associate the business-related metadata 123 with the corresponding storage data set 120 to create or modify the data set signature 140.

[0109] In one embodiment, the selected metadata may be transformed or modified before being propagated to the storage data directory. When selecting business-related metadata 123 from the business data directory 115 to be propagated to the storage data directory 240, the data set signature mapper 136 may encode the business-related metadata 123 in a particular manner to convey more information to the storage manager 126. For example, when selecting a business name from the business data directory 115, the data set signature mapper 136 may encode the business name to additionally indicate whether the corresponding business is a domestic entity or an international entity. Transforming or modifying the metadata prior to propagation to the storage data directory 240 may standardize the metadata, or may facilitate and / or increase the speed of policy management.

[0110] In step 465, the data set signature 140 of the corresponding stored data set 120 is inserted, updated, modified, deleted, or reconciled in the stored data catalog 240 to reflect the business-level metadata changes since the last update of the data set signature 140. In one embodiment, step 465 is similar to Figure 4A Similar to step 418 , the data set signature mapping program 136 is used to periodically update the data set signature 140 in the storage data directory 240 for the corresponding data set 120 .

[0111] Figure 5 A diagram of another information processing system 500 for implementing data set signatures 140 for data impact driven storage management provided for various embodiments of the present invention. Information processing system 500 is similar to information processing system 150, except that information processing system 500 includes more than one business data directory, a primary business data directory 515 and an auxiliary business data directory 516.

[0112] In one embodiment, the operation of information handling system 500 is similar to information handling system 150, except that storage manager 126 receives business-related metadata 123 from primary business data directory 515, which includes metadata received from secondary business data directory 516. Primary business data directory 515 is associated with or belongs to business environment 103 and includes metadata describing data objects, business applications 117, processes, and other data stored in business environment 103 and used by business entities corresponding to business environment 103. In one embodiment, secondary business data directory 516 is associated with business environment 103, but includes metadata describing data objects, business applications 117, processes, and other data not described in primary business data directory 515. In another embodiment, secondary business data directory 516 is associated with another business environment 103 and includes metadata describing data objects, business applications, processes, and other data stored in another business environment 103 and used by another business entity corresponding to another business environment 103.

[0113] In one embodiment, the primary business data directory 515 receives the business-related metadata 123 from the secondary business data directory 516 and orchestrates the integration of the business-related metadata 123 with the business-related metadata within the primary business data directory 515. In one embodiment, the storage manager 126 requests the business-related metadata 123 from the primary business data directory 515 and receives the integrated business-related metadata 123 from the business environment 103. In this manner, the storage manager 126 does not have to request, retrieve, and coordinate business-related metadata 123 from two different business data directories 515 and 516, respectively. Instead, the storage manager 126 receives business-related metadata 123 from both the primary business data directory 515 and the secondary business data directory 516 in one interaction with the primary business data directory 515. Figure 5 An advantage of the illustrated embodiment is that the storage manager 126 does not have to perform additional metadata integration, since the integration of business-related metadata 123 is done within the business context.

[0114] Figure 6 A diagram of another information handling system 600 for implementing data set signatures 140 for data impact driven storage management provided for various embodiments of the present invention. Information handling system 600 is similar to information handling system 150, except that information handling system 600 includes more than one storage manager 126, primary storage manager 626 and secondary storage manager 627.

[0115] In one embodiment, the operation of the information handling system 600 is similar to the information handling system 150, except that the primary storage manager 626 can transfer the data set signature 140 to another secondary storage manager 627. The primary storage manager 626 is associated with or belongs to the data management system 106 and is used to manage the data set 120 stored locally in the data storage area 129 of the data management system 106. In one embodiment, the secondary storage manager 627 is associated with the data management system 106, but is used to manage another subset of the data set 120 that is not managed by the primary storage manager 626. For example, when the data set 120 is migrated between storage tiers, if the data set 120 is accompanied by its data set signature 140, the data set 120 can be managed more efficiently and effectively. In another embodiment, the secondary storage manager 627 is associated with another data management system 106 and is used to manage other data sets 120 that are not stored in the data management system 106. In one embodiment, the secondary storage manager 627 is associated with a cloud manager environment or a data virtualization environment. In one embodiment, the cloud manager environment can handle different views or levels of the same underlying data set 120. In one embodiment, the data virtualization environment may manage virtual data sets that are actually mapped to some storage data sets 120 managed by the primary storage manager 626 .

[0116] In one embodiment, primary storage manager 626 transmits one or more data set signatures 140 to secondary storage manager 627. Secondary storage manager 627 may store data set signatures 140 and use data set signatures 140 to perform operations on behalf of one or more business environments 103.

[0117] Figure 7 A diagram of another information processing system 700 for implementing a data set signature 140 for data impact driven storage management provided for various embodiments of the present invention. Information processing system 700 is similar to information processing system 150, except that storage manager 126 in information processing system 700 transmits data set signature 140 back to business environment 103 via link 109.

[0118] In one embodiment, the business data catalog 115 in the business environment 103 can integrate the received data set signatures 140 into its business-related metadata 123, thereby providing operational insights about the data sets 120 to users of the business environment 103. In another embodiment, the business environment 103 can send a request for one or more data set signatures 140 to the storage manager 126. The storage manager 126 transmits the requested data set signatures 140 back to the business environment 103.

[0119] Figure 8Figure 800 is provided to illustrate the use of historical or hypothetical data set signatures 140 when designing and / or adjusting a system for a workload provided by various embodiments of the present invention. The system is designed and / or adjusted using a system profile or a profile of system resources, configurations, and topologies to provide sufficient capacity, response time, throughput, and management services to meet the needs of the expected workload. In some cases, the system may be one or more of a data management system, a storage system, a cloud system, a hyper-converged infrastructure, a composite infrastructure with independently scalable blocks, and the like. A workload refers to a series of processes that use the memory and / or processing power of the system within a given time period or at a specific time instance. A workload profile 803 describes a time-varying workload that the business environment 103 hopes or expects to be executed by a new or modified system. In one embodiment, the workload profile 803 includes the specification of expected throughput, response time, and other key performance indicators related to workload execution. The system profile 806 describes the hardware (e.g., memory, processor, etc.), software, services, configurations, topologies, or resource spaces, etc. that can potentially support and execute the workload described by the workload profile 803. For example, the processes to be executed for the workload are described by the workload profile 803 , and the processes require system capabilities, which are described by the system profile 806 .

[0120] While executing the process of the workload, one or more data sets 120 may be created, accessed, deleted, modified, etc. A data set profile 840 refers to a data set signature 140 that describes one or more data sets 120 created and / or accessed during a hypothetical, simulated, or historical execution of the workload.

[0121] The workload profile 803 includes references to the data sets 120 used by the processes of the workload. In one embodiment, the system profile 806 is created based on the workload profile 803 and the data set profile 840, or based on a set of data set signatures 140 describing the data sets 120 hypothetically used by the workload corresponding to the workload profile 803, rather than creating the system profile 806 based solely on the workload profile 803. Using the data set signatures 140 to determine the system profile 806 enables the designer of the system or the tuner of the system to more accurately and specifically tune and configure the system for the workload corresponding to the workload profile 803. This is because the data set signatures 140 provide a more detailed and refined description of the data sets 120 used by the processes of the workload, which can be used to more accurately configure the necessary system hardware, software, and other resources to meet the needs of the workload. In this way, when creating the system profile 806 for the workload, using the data set signatures 140 within the data set profile 840 as an intermediate representation in addition to the workload profile 803 can produce a more accurately designed and / or tuned system. In one embodiment, the designed and tuned system may be optimized to minimize acquisition and operating costs while still providing desired workload throughput and response times. In one embodiment, multiple workload profiles 803 and / or data set profiles 840 and / or system profiles 806 may be considered for tuning and optimization purposes.

[0122] Fig. 9 Flowchart of method 900 for determining system profile 806 for a workload based on one or more data set signatures 140 provided for various embodiments of the invention. Method 900 may be implemented using historically captured, hypothetical, or simulated data set signatures 140 of one or more data sets 120.

[0123] In step 903, a workload profile 803 of the system to be adjusted, specified, configured, or provisioned is determined. As described above, a workload profile 803 describes a series of processes executed for a corresponding workload and one or more data sets 120 used by the workload during execution. In another embodiment, as described above, the workload profile 803 includes a specification of expected throughput, response time, and other key performance indicators related to future workload execution.

[0124] In step 906, one or more system profiles 806 are determined to support the workload profile 803. As described above, the system profile 806 describes the hardware, software, services, configurations, topologies, resource spaces, and other capabilities within the system that can be used to support and execute the workload processes described in the workload profile 803.

[0125] In step 909, a data set profile 840 including one or more data set signatures 140 is determined for the workload profile 803 based on the data sets 120 identified in the workload profile 803 as being accessed or used during a hypothetical, simulated, or historical execution of the workload profile 803. In one embodiment, the data set signatures 140 included in the data set profile 840 are determined based on previous experience with previous workloads or workload profiles 803 having similar processes.

[0126] In step 912, one or more system profiles 806 are evaluated against the data set profiles 840 determined for the workload profile 803 to eliminate, modify, or improve one or more system profiles 806 based on business attributes. The business attributes refer to performance, cost, flexibility, or any other objectives defined by the business environment 103 according to which the workload should be optimized. Detailed information contained in the data set signature 140 of the data set profile 840 may indicate that certain aspects of the system profile 806 may not be required to perform a particular process using the data set 120. In this case, the system profile 806 may be modified or reduced to consume fewer resources than previously indicated.

[0127] In one embodiment, one of the one or more system profiles 806 is selected based on business requirements, business attributes, and trade-off analysis. In the future, the data management system 106 may be tuned, configured, and provisioned to execute the workload corresponding to the workload profile 803. In another embodiment, step 906 may be omitted, and one or more system profiles may be determined based on the data set profile 840 and the workload profile 803 at the beginning of step 912.

[0128] Fig.10A diagram showing various examples of system management policies 243 that are called, executed, or orchestrated based on data set signatures 140 provided by various embodiments of the present invention. In one embodiment, system management policies 243 may be called, executed, or orchestrated by a policy manager 137. While some system management policies 243 call and execute operations on data sets 120, other policies are applicable to storage management, cloud management, and system management. As described above, data set signatures 140 include detailed metadata, operational metadata 133, and business-related metadata 123 about data sets 120 stored in a data management system 106. Due to the additional detailed information stored in the data set signatures 140, the policy manager 137 may utilize system management policies 243 to call fine-grained operations on data sets 120 more efficiently, more accurately, with lower latency, more refined, more comprehensive, and / or more timely. In other embodiments, system management policies 243 may improve the matching and / or mapping steps in creating data set signatures 140 for data sets 120 stored in a data management system 106, and / or the identification of relationships between data sets 120. In other embodiments, system management policies 243 enable management software to improve the efficiency, performance, automation, correctness, and / or results of system management, workload management, and / or event management processes.

[0129] In various embodiments, system management policy 243 is implemented based on certain parameters associated with data set 120. In one embodiment, system management policy 243 is applied to data set 120 based on metadata contained in data set signature 140 describing data set 120. System management policy 243 may be applied to data set 120 based on metadata contained in data set signature 140 describing data set 120 and a current location of data set 120. System management policy 243 may be applied to data set 120 based on metadata contained in data set signature 140 describing data set 120 and metadata contained in a second data set signature 140 describing a second data set 120, where second data set 120 is related to data set 120. System management policy 243 may be applied to data set 120 based on metadata contained in data set signature 140 describing data set 120 and operations performed on data set 120. System management policy 243 may be applied to data set 120 based on metadata contained in data set signature 140 describing data set 120 and a user requesting access to data set 120. System management policies 243 may be applied to data sets 120 based on metadata contained within data set signatures 140 describing data sets 120 and workload profiles indicating workloads to be performed by the data management system accessing data sets 120. In other embodiments, system management policies 243 may be scheduled for application or presented to policy manager 137 for application, rather than being immediately applied as described above. As described above, in other embodiments, system management policies 243 may be applied to data management, storage management, system management, cloud management, workload management, and / or event management.

[0130] Fig.10Listed are examples of system management policies 243 that indicate operations that can be implemented, executed, or orchestrated by the policy manager 137 using data set signatures 140 in a more efficient, accurate, low latency, comprehensive, and / or timely manner. The various operations indicated by the system management policies 243 described herein may be collectively referred to herein as data manipulation operations. In one embodiment, the preferred location policy 1001 indicates that a data set 120 described by a particular type of metadata (as indicated by the data set signature 140) should be stored in a particular type of location or a particular location inside the data management system 106 or outside the data management system 106. The preferred location policy 1001 may indicate a data manipulation operation, or include a set of instructions stored in the data management system 106 that instruct the policy manager 137 to store the data set 120 in a certain type of location or a particular location inside the data management system 106 or outside the data management system 106 based on the data set signature 140 of the data set 120. The location specified for the data set 120 may be determined to improve the performance of accessing important data sets 120 and the security of important data sets 120, and to balance the load of the data sets 120 stored in the data management system 106.

[0131] In one embodiment, the data mobility policy 1002 indicates that a data set 120 described by a specific type of metadata (as identified in the data set signature 140) and stored in a specific location should be migrated to another type of location or a specific location inside the data management system 106 or outside the data management system 106. The data mobility policy 1002 may indicate a data manipulation operation, or include a set of instructions stored in the data management system 106 that instruct the policy manager 137 to change the location of the data set 120 to another location based on the data set signature 140 of the data set 120 and the current location of the data set 120. The instructions may instruct the policy manager 137 to change the location of the data set 120 periodically or based on a triggering event in response to the data set 120 being associated with the data set signature 140. The data mobility policy 1002 may be used to move the data set 120 or copy the data set 120 to another location periodically or based on a triggering event.

[0132] In one embodiment, deduplication policy 1003 indicates that a data set 120 described by a particular type of metadata (as identified in data set signature 140) should be deduplicated such that any duplicates of data set 120 are deleted from storage to save space. Deduplication policy 1003 may indicate a data manipulation operation, or include a set of instructions stored in data management system 106 that instruct policy manager 137 to dedupe data set 120 in response to data set 120 being associated with data set signature 140. In other embodiments, deduplication policy 1003 indicates when deduplication should be performed, and / or which of the duplicates found should be retained (e.g., retaining the latest copy).

[0133] In one embodiment, data compression policy 1004 indicates that a data set 120 described by a particular type of metadata (as identified in data set signature 140) should be compressed to reduce the size of data set 120. Data compression policy 1004 may indicate a data manipulation operation, or include a set of instructions stored in data management system 106 that instruct policy manager 137 to compress data set 120 in response to data set 120 being associated with data set signature 140. In one embodiment, data compression policy 1004 is triggered when data set 120 is first stored in data management system 106.

[0134] In one embodiment, the namespace mapping policy 1005 indicates that there is a namespace mapping between a data set 120 described by the business-related metadata 123 and one or more other data sets 120 described by the operational metadata 133. Alternatively, one data set 120 described by the operational metadata 133 may have a namespace mapping to one or more data sets 120 described by the business-related metadata. Such mappings, which may be 1:1, 1:N, M:1, or rarely M:N, precede the combination of metadata to create or update the data set signature 140, such as Figure 4A The namespace mapping policy 1005 may indicate a data manipulation operation, or include a set of instructions stored in the data management system 106 that instructs the data set signature mapping program 136 to assert a namespace mapping between two or more data sets 120. Such a policy connects the storage view of the data set 120 to the business view of the data set 120 to identify and / or monetize high business impact data sets 120.

[0135] In one embodiment, the tagging policy 1006 indicates that a data set 120 described by a particular type of metadata (as identified in the data set signature 140) should be tagged or identified as a particular type of data set 120. The tagging policy 1006 may indicate a data manipulation operation, or include a set of instructions stored in the data management system 106 that instruct the policy manager 137 to tag a data set 120 in response to the data set signature 140 being associated with the data set 120. Once a data set 120 is tagged or identified as a particular type of data set 120, the data management system 106 and the business environment 103 may treat the data set 120 differently from other data sets 120, for example, in terms of its access, location, copy, and / or update. The tagging policy 1006 that tags data sets 120 based on business importance identified or reflected in the data set signature allows for optimization of high-impact data for the business.

[0136] In one embodiment, the relationship identification policy 1007 indicates that a relationship exists between a data set 120 described by a particular type of metadata (as identified in the data set signature 140) and one or more other data sets also described by the particular type of metadata identified in the data set signature 140. The relationship identification policy 1007 may indicate a data manipulation operation, or include a set of instructions stored in the data management system 106, the instructions instructing the policy manager 137 to assert a relationship between one or more data sets 120 based on the data set signature 140 associated with each data set 120. The instructions may instruct the policy manager 137 to first determine whether a relationship exists between the first data set 120 and the second data set 120 based on the data set signature 140 of the first data set 120 and the data set signature 140 of the second data set 120. Then, the instructions may instruct the policy manager 137 to update the operational metadata 133 and / or the business metadata 123 within the data set signature 140 of the first data set 120 and / or the data set signature 140 of the second data set 120 in response to the relationship between the first data set 120 and the second data set 120. For example, the relationship between the data sets 120 may be that the two data sets 120 are copies of each other, one data set 120 is a different version of another data set 120, one data set 120 is a subset of another data set 120, one data set 120 is a transformation of another data set 120, etc. Then, detecting and maintaining records of the relationship between the data sets 120 in the metadata can advantageously trigger additional policy calls or operations on one or more data sets 120.

[0137] In one embodiment, the trigger operation policy 1008 indicates that when an operation is performed on a data set 120 described by a particular type of metadata (as identified in the data set signature 140), other operations may also need to be performed on the data set 120. For example, when a data set 120 having a data set signature 140 is accessed (a first operation), the data set 120 may also be copied to a cache (a second operation). In this manner, the trigger operation policy 1008 may indicate a data manipulation operation, or include a set of instructions stored in the data management system 106 that instruct the policy manager 137 to perform one or more additional operations on the data set 120 in response to the data set 120 being associated with the data set signature 140 and the first operation being performed on the data set 120.

[0138] In one embodiment, the propagation operation policy 1009 indicates that when an operation is performed on a first data set 120 described by a particular type of metadata (as identified in the data set signature 140), other operations may also need to be performed on other data sets 120 related to the first data set 120. For example, when a first data set 120 is deleted, other related data sets 120 (such as copies, subsets, transformations, etc.) of the first data set 120 may also need to be deleted. The propagation operation policy 1009 may indicate a data manipulation operation, or include a set of instructions stored in the data management system 106, which instruct the policy manager 137 to perform one or more additional operations on other related data sets 120 in response to the first data set 120 being associated with the data set signature 140 and the first operation performed on the first data set 120.

[0139] In one embodiment, the access management policy 1010 indicates whether to allow or deny access to a data set 120 stored in a specific location with specific metadata (as identified by the data set signature 140) based on the requested operation and the user making the request (whether human or not). The access management policy 1010 may indicate a data manipulation operation, or include a set of instructions stored in the data management system 106 that instructs the policy manager 137 to determine whether to allow or deny access to the data set 120 based on the data set signature 140, the location of the data set 120, the operation requested to be performed on the data set 120, and / or the user requesting to perform the operation on the data set 120. The access management policy 1010 can help to more selectively implement access policies based on the metadata in the data set signature 140, and can also help to consistently implement access policies across heterogeneous data stores.

[0140] In one embodiment, the access enablement policy 1011 indicates how to allow access to a data set 120 stored in a specific location with specific metadata (as identified by the data set signature 140) based on the requested operation and the user making the request. The access enablement policy 1011 may indicate a data manipulation operation or include a set of instructions stored in the data management system 106 that instructs the policy manager 137 to allow limited access to the data set 120 or determine whether to transform, mask, or allow access to the data set 120 based on the data set signature 140, the location of the data set 120, the requested operation on the data set 120, and / or the user requesting the access operation to be performed on the data set 120. The access enablement policy 1011 can help cleanse or anonymize the requested data set 120, especially when the requested data set 120 contains sensitive personally identifiable information, medical details, financial details, historical details, or the like.

[0141] In one embodiment, the configuration policy 1012 indicates the configuration of an existing or future system or its workload parameters based in part on one or more data sets 120 stored in a particular location with specific metadata (as identified by a data set signature 140) and optionally associated with a particular workload profile 803. For an existing system, the configuration policy 1012 may indicate a data manipulation operation, or include a set of instructions stored in or external to the database management system 106 that instruct a management tool to modify the configuration or workload parameters of the existing system. In one embodiment, the configuration policy 1012 optimizes storage and cloud configuration for expected future workloads. The configuration of the future system or its workload parameters may describe the expected placement or processing of the data set 120, or modify the configuration of the system profile 806, and / or describe where and how the workload should be placed.

[0142] In one embodiment, the provisioning policy 1013 indicates how to provision storage tiers of various types and capacities for workloads within the data management system 106 to store one or more data sets 120 with specific metadata (as identified by the data set signature 140). The provisioning policy 1013 may indicate data manipulation operations, or include a set of instructions stored in the data management system 106 that instruct the policy manager 137 to provision storage tiers as described in the provisioning policy 1013. The provisioning policy 1013 provisions storage based on user profiles, expected data volumes and types, and predicted workloads to store various types of data sets 120 in potentially more than one location. In one embodiment, the provisioning policy 1013 orchestrates the provisioning of system resources such as compute and / or network based on specific metadata (as identified by the data set signature 140).

[0143] In one embodiment, a workload migration policy 1014 indicates that, based on one or more data sets 120 described by a particular type of metadata (as identified in a data set signature 140) and associated with a particular workload, all or part of the workload should be migrated to another location, such as a cloud server. This is different from migrating any or all data sets 120 accessed by the particular workload. The workload migration policy 1014 may indicate data manipulation operations, or include a set of instructions stored in or outside the data management system 106 that instruct system management and / or workload management tools to migrate the workload. In one embodiment, a workload migration policy may orchestrate workload migration once, multiple times, periodically, or in specific circumstances. The workload migration policy 1014 provides decisions and instructions to management tools to migrate workloads when deemed appropriate to better access data, improve performance, and balance the load of the data management system 106 or other systems.

[0144] In one embodiment, the event response policy 1015 indicates that when an event occurs at a data set 120 described by a particular type of metadata (as identified in the data set signature 140), applicable policies may be executed or orchestrated for the data set 120 and / or other systems or objects. The event response policy 1015 may indicate a data manipulation operation, or include a set of instructions stored in the data management system 106 that instruct the policy manager 137 to execute policies for the data set 120 when an event occurs at the data set 120 based on the data set signature 140, the workload accessing the data set 120, the location of the data set 120, and / or the user accessing the data set 120. The event response policy 1015 provides instructions to automatically respond to events at the data set 120 to which the policy applies. In one embodiment, when an event occurs at the data set 120 or the data set 120 is affected by an event, applicable policies may be executed or orchestrated for the data set 120 and / or other systems or objects. For example, during recovery of a previous state of the data management system (an event), access to a data set 120 associated with a particular business organization may not be allowed.

[0145] In one embodiment, the event identification policy 1016 indicates that when one or more events occur at a data set 120 described by a particular type of metadata (as identified in a data set signature 140), another event can be detected and identified as the root cause of the one or more events. The event identification policy 1016 can indicate a data manipulation operation, or include a set of instructions stored in the data management system 106, which instructs the policy manager 137 to identify an event occurring at a data set 120 as a root cause or failure of other events or failures occurring at the data management system 106. The event can be identified based on the data set signature 140 and / or the workload accessing the data set 120. The event identification policy 1016 is used to debug or determine the fingerprint of a system bottleneck that may occur at the data management system 106. Complex events can be difficult to identify and track the root cause, and considering one or more data set signatures 140 can provide valuable insights into such identification and tracking. For example, by considering the data set signatures 140 of one or more data sets 120 owned by a particular organization within the business environment 103, unauthorized attempts to access these data sets 120 can be more quickly identified.

[0146] In one embodiment, the contextual policies 1017 indicate that specific policies or operations may be executed or orchestrated for a data set 120 described by a specific type of metadata (as identified in the data set signature 140) to control the performance and operation of the data management system 106 relative to optimal performance and operation based on machine learning or training data. The contextual policies 1017 may indicate data manipulation operations, or include a set of instructions stored in the data management system 106 that instruct the policy manager 137 to orchestrate or execute one or more policies based on the data set signature 140, the workload to be performed using the data set 120, the operations to be performed or performed on the data set 120, the location of the data set 120, and / or the user requesting access to the data set 120. In one embodiment, the system management policies 243 may be organized in a multi-dimensional table that covers all or most situations. In one embodiment, the contextual policies 1017 applicable to the current situation may be executed or orchestrated for the data set 120 and / or other systems or objects.

[0147] In one embodiment, the trade-off strategy 1018 indicates preferences and / or weights for certain business attributes based at least in part on a data set 120 described by a particular type of metadata (as identified in the data set signature 140), stored in a particular location, and accessed by a workload. The trade-off strategy 1018 allows the data management system 106 and / or other management systems to consider the data set signature 140 when setting preference levels for optimizing impact, performance, cost, or other business attributes. Once the preference levels are set, management policies that access and consider these preference levels can be activated so that operations performed on the data management system or other systems can be orchestrated. In short, the improved knowledge about the data set 120 captured in the data set signature 140 can be used to determine the trade-offs in optimizing preferences.

[0148] In one embodiment, the policy creation policy 1019 enables the creation of additional system management policies 243 for a data set 120 described by a particular type of metadata (as identified in the data set signature 140). The policy creation policy 1019 may indicate a data manipulation operation, or include a set of instructions stored in the data management system 106 that instruct the policy manager 137 to assert and activate a new system management policy 243 for the data set 120 based on the data set signature 140, the operations that have been or are to be performed on the data set 120, the location of the data set 120, and / or the workload accessing the data set 120. In one embodiment, the policy creation policy 1019 may identify through machine learning common actions or orchestrations that are required of system management tools that are performed within the data management system 106 and are directly or indirectly related to data set 120 management, and then an automated system management policy may be created to initiate these actions or orchestrations in similar situations. In one embodiment, these actions or orchestrations may be taken against other systems and / or may involve cloud management, workload management, and / or system configuration.

[0149] Fig.11 Flowchart of a method 1100 for implementing a data set signature 140 for data impact driven storage management provided by various embodiments of the present invention. The method 1100 may be implemented by the information processing unit 200 or the storage manager 126 in the data management system 106. The method 1100 may be implemented after the data set 120 is stored in the data storage area 129 of the data management system 106.

[0150] In step 1103, the operational metadata 133 describing the data set 120 is stored in the memory 233 of the data management system 106, for example, in the data directory 240. The data directory 240 may be stored in the memory 233 of the information processing unit 200, which may be implemented as the storage manager 126. For example, the data set signature mapping program 136 stores the operational metadata 133 describing the data set 120 in the memory 233 of the information processing unit 200 or the data management system 106.

[0151] In step 1104, business-related metadata 123 describing business-impacting characteristics of the data set 120 is received from one or more business environments 103. The business environments 103 are external to the data management system 106. For example, the Tx / Rx 210 of the information processing unit 200 implemented as a storage manager 126 receives the business-related metadata 123 from one or more business data directories 115 associated with the one or more business environments 103. In one embodiment, the data set signature mapping program 136 receives the business-related metadata 123 from the business environment 103.

[0152] In step 1106, a data set signature 140 including a combination of the operational metadata 133 and the business-related metadata 123 describing the data set 120 is generated and stored in the data management system 106. For example, the data set signature mapping program 136, when executed by the processor 230 of the information processing unit 200 implemented as the storage manager 126, generates the data set signature 140 including the operational metadata 133 and the business-related metadata 123. The data set signature 140 can also be stored in the memory 233 of the information processing unit 200 implemented as the storage manager 126.

[0153] In step 1108, a data manipulation operation is performed on the data set 120 using a combination of the operational metadata 133 and the business-related metadata 123 in the data set signature 140. Fig.10 Examples of various types of operations that are enhanced to be performed more efficiently, more accurately, with lower latency, more comprehensively, and / or more timely are further described. For example, the policy manager 137, when executed by the processor 230 of the information processing unit 200, performs or requests an operation on the data set based on the data set signature 140. The operation performed on the data set 120 can be a read, write, copy, move, transform, or any other type of operation performed by the data management system 106 on the data set 120.

[0154] The embodiments described herein describe a technical solution for connecting a semantically meaningful information context (i.e., layer) to an operational context below it by extending a profile of a managed object in a subordinate context with information related to a superior context (e.g., creating a data signature profile 140 including a combination of operational metadata 133 and business-related metadata 123 describing a data set 120). This can be applied to other object and system management situations where such contexts exist. A subordinate context can refer to a storage data set 120. A second context above the storage data set 120 can include on-demand data virtualization. A third context above the storage data set 120, transformation capabilities and data copy or move operations, can include layers of DataOps processes and process flows. A superior context above the storage data set 120, stream processing capabilities and instant analytical model evaluation, can include advanced Internet of Things (IoT) state characterization and event recognition.

[0155] The embodiments described herein also combine capabilities in processes. However, the individual steps in the process can be split up to apply to different environments where not all steps are required. For example, using the embodiments disclosed herein, metadata extensions for objects can be split up based on the value representations of the objects observed, inferred, or captured in the various contexts described above. The embodiments disclosed herein can also be applied to split up data directory mapping and integration capabilities across the above contexts and systems. In addition, the embodiments disclosed herein can also be used to split up the services of enhanced data profiles into a broader set of management systems.

[0156] Fig.12 A diagram of an apparatus 1200 for implementing a data set signature 140 for data impact driven storage management provided for various embodiments of the present invention. The apparatus 1200 includes a means for storing 1203, a means for receiving 1206, and a means for executing 1209. The means for storing 1203 includes means for storing operational metadata 133 describing a data set 120 in a memory 233 of the apparatus 1200. The means for receiving 1206 includes means for receiving business-related metadata 123 describing business impact characteristics of the data set 120 from one or more business environments 103, the business environment 103 being external to the apparatus 1200. The means for storing 1203 also includes means for storing a data set signature 140 including a combination of operational metadata 133 describing the data set 120 and business-related metadata 123. The means for executing 1209 includes means for executing a data manipulation operation on the data set 120 using the combination of operational metadata 133 and business-related metadata 123 in the data set signature 140.

[0157] Although several embodiments have been provided in the present invention, it should be understood that the systems and methods disclosed in the present invention may be embodied in many other specific forms without departing from the spirit or scope of the present invention. The examples of the present invention should be considered illustrative rather than restrictive, and the present invention is not limited to the details given herein. For example, various elements or components may be combined or merged in another system, or certain features may be omitted or not implemented.

[0158] In addition, without departing from the scope of the present invention, the techniques, systems, subsystems and methods described and illustrated as discrete or separate in various embodiments may be combined or merged with other systems, modules, techniques or methods. Other items shown or discussed as coupled may be directly coupled or may be indirectly coupled or communicated electrically, mechanically or otherwise via an interface, device or intermediate component. Other example variations, substitutions and modifications may be determined by those skilled in the art without departing from the spirit and disclosed scope of the present invention.

Claims

1. A method, characterized in that include: A data set signature mapping program of a data management system stores operational metadata describing a data set, the operational metadata and the data set being stored in a memory of the data management system; The data set signature mapping program of the data management system receives business-related metadata describing the data set from a business environment, the business environment being external to the data management system; The data set signature mapping program of the data management system stores a data set signature, the data set signature comprising a combination of the operational metadata and the business-related metadata; A policy manager of the data management system performs a data manipulation operation on the data set using the combination of the operational metadata and the business-related metadata in the data set signature.

2. The method according to claim 1, characterized in that The operation metadata includes at least one of an operation name, an operation type, an operation structure, an operation size, an operation security class or tag, an operation timestamp, an operation content type, an operation content class or tag, an operation index existence, an operation source, an operation version status, an operation encryption class, an operation encoding class, an operation movement status, an operation access class, an operation access heat class, an operation access skew, an operation access blockiness, an operation local entropy, an operation global entropy, or an operation relationship.

3. The method according to any one of claims 1 to 2, characterized in that The business-related metadata includes at least one of business name, business type, natural language description, business owner, business manager, business content class / tag, business permission workflow, business organization level, business data geography, business structure type, business structure, business language, business size, business security class, business timestamp, business source or origin, business cleansing or maintenance, business dataset profile, business virtualization status, business sorting order, business retention date, business compliance obligations, business relationships, business usage counts, business utility ratings, business user authenticity ratings, business data scientist utility ratings, business data scientist comments, business decision impact, business monetary impact, business aggregate dataset impact, or average impact of business dataset friends.

4. The method according to any one of claims 1 to 2, characterized in that Also includes: The data set signature mapping program of the data management system determines the data set from multiple data sets stored locally in the data management system based on interest attributes associated with the data set, wherein the interest attributes associated with the data set include at least one of an operation name, a time when the data set was last accessed, an event associated with the data set, a previous operation performed on the data set, a current operation performed on the data set, or a future operation performed on the data set.

5. The method according to any one of claims 1 to 2, characterized in that Also includes: The data set signature mapping program of the data management system determines a business environment to receive the business-related metadata from the business environment based on a business data catalog of the business environment, wherein the business data catalog stores information describing data belonging to the business environment.

6. The method according to any one of claims 1 to 2, characterized in that Also includes: The data set signature mapping program of the data management system sends a query to the business environment requesting the business-related metadata describing one or more data sets stored by the business environment; The data set signature mapping program of the data management system matches the data set stored locally in the data management system with the business data stored in the business environment; The data set signature mapping program of the data management system maps the business-related metadata describing the business data stored in the business environment to the operational metadata stored locally in the data management system; The data set signature mapping program of the data management system generates and maintains the data set signature that describes the data set.

7. The method according to any one of claims 1 to 2, characterized in that The business-related metadata is received from a single business data catalog or a plurality of different business data catalogs.

8. The method according to any one of claims 1 to 2, characterized in that The business-related metadata from two or more external business data directories are combined by one of the two or more external business data directories and then combined with the operational metadata to generate the data set signature.

9. The method according to any one of claims 1 to 2, characterized in that The data set signature is generated by at least one of the data management system, cloud management software, management software or user in the following manner: receiving the business-related metadata or storage-oriented operational metadata from the storage manager of the data management system, and combining the operational metadata with the business-related metadata or storage-oriented operational metadata.

10. The method according to any one of claims 1 to 2, characterized in that Also includes: The storage manager of the data management system transmits the data set signature to another storage manager.

11. The method according to any one of claims 1 to 2, characterized in that Also includes: The storage manager of the data management system transmits the data set signature to the business environment to store the data set signature in a business data directory associated with the business environment.

12. The method according to any one of claims 1 to 2, characterized in that The business-related metadata is received from a business data catalog associated with the business environment, wherein the business data catalog describes data stored in a cloud computing environment or a data virtualization environment.

13. The method according to any one of claims 1 to 2, characterized in that Also includes: A storage manager of the data management system transmits the data set signature to a cloud-based environment or a data virtualization environment.

14. The method according to any one of claims 1 to 2, characterized in that Also includes: The data set signature mapping program of the data management system uses machine learning and training data to determine other metadata describing the data set based on the business-related metadata and the operational metadata.

15. The method according to any one of claims 1 to 2, characterized in that Also includes: The dataset signature mapping program of the data management system determines a system profile of a workload profile based on a dataset profile; wherein the workload profile describes a time-varying workload requested by a business environment to be executed by the system; wherein the system profile describes at least one of hardware, software, services, configurations, topologies, or resource spaces to support and execute the workload described by the workload profile; wherein the dataset profile includes one or more dataset signatures, and the one or more dataset signatures are based on at least one of an assumed dataset signature or a historical dataset signature of the dataset that the workload described in the workload profile is expected to access.

16. The method according to any one of claims 1 to 2, characterized in that The policy manager of the data management system uses the combination of the operational metadata and the business-related metadata in the data set signature to perform data manipulation operations on the data set, which also includes: the policy manager of the data management system determines a system management policy, and the system management policy indicates the data manipulation operations to be performed on one or more systems or data sets based on the metadata contained in the data set signature.

17. The method according to any one of claims 1 to 2, characterized in that The policy manager of the data management system uses the combination of the operational metadata and the business-related metadata in the data set signature to perform data manipulation operations on the data set, which also includes: the policy manager of the data management system determines a system management policy, and the system management policy indicates the data manipulation operation to be performed on the data set based on the metadata contained in the data set signature and the current location of the data set.

18. The method according to any one of claims 1 to 2, characterized in that The policy manager of the data management system uses the combination of the operational metadata and the business-related metadata in the data set signature to perform data manipulation operations on the data set, which also includes: the policy manager of the data management system determines a system management policy, and the system management policy indicates the data manipulation operations to be performed on one or more data sets based on the metadata contained in the data set signature and the metadata contained in a second data set signature, wherein the second data set signature describes a second data set related to the first data set.

19. The method according to any one of claims 1 to 2, characterized in that The policy manager of the data management system uses the combination of the operational metadata and the business-related metadata in the data set signature to perform data manipulation operations on the data set, which also includes: the policy manager of the data management system determines a system management policy, and the system management policy indicates the data manipulation operation to be performed on the data set based on the metadata contained in the data set signature and the operation performed on the data set.

20. The method according to any one of claims 1 to 2, characterized in that The policy manager of the data management system uses the combination of the operational metadata and the business-related metadata in the data set signature to perform data manipulation operations on the data set, which also includes: the policy manager of the data management system determines a system management policy, and the system management policy indicates the data manipulation operations to be performed on one or more systems or logs based on the metadata contained in the data set signature and the user requesting access to the data set.

21. The method according to any one of claims 1 to 2, characterized in that The policy manager of the data management system uses the combination of the operational metadata and the business-related metadata in the data set signature to perform data manipulation operations on the data set, which also includes: the policy manager of the data management system determines a system management policy and a workload profile, the system management policy indicating the data manipulation operations to be performed on one or more systems and the data set based on the metadata contained in the data set signature, and the workload profile indicating the workload to be performed by the data management system to access the data set.

22. The method according to any one of claims 1 to 2, characterized in that The policy manager of the data management system uses the combination of the operational metadata and the business-related metadata in the data set signature to perform data manipulation operations on the data set, which also includes: the policy manager of the data management system determines a system management policy, and the system management policy indicates the data manipulation operation to be performed on the data set based on the metadata contained in the data set signature, wherein the data manipulation operation includes storing the data set in a specific location or a specific type of location based on the metadata contained in the data set signature.

23. The method according to any one of claims 1 to 2, characterized in that The policy manager of the data management system uses the combination of the operational metadata and the business-related metadata in the data set signature to perform a data manipulation operation on the data set, which also includes: the policy manager of the data management system determines a system management policy, and the system management policy indicates the data manipulation operation to be performed on the data set based on the metadata contained in the data set signature and one or more related data sets, wherein the data manipulation operation includes marking the data set based on the metadata contained in the data set signature and the one or more related data sets.

24. The method according to any one of claims 1 to 2, characterized in that The policy manager of the data management system uses the combination of the operational metadata and the business-related metadata in the data set signature to perform a data manipulation operation on the data set, which also includes: the policy manager of the data management system determines a system management policy, and the system management policy indicates the data manipulation operation to be performed on the data set based on the metadata contained in the data set signature and the metadata contained in a second data set signature, wherein the second data set signature describes a second data set related to the first data set, and the data manipulation operation includes asserting a relationship between the data set and the second data set based on the data set signature of the data set and the second data set signature of the second data set.

25. The method according to any one of claims 1 to 2, characterized in that The data management system is included in an information processing system including multiple contexts, wherein a first subordinate context includes a storage data set, a second context includes on-demand data virtualization, a third context includes a data operations (DataOps) process, and a fourth upper-level context includes Internet of Things (IoT) state representation and event identification; wherein the data set signature includes metadata from at least one of the first subordinate context, the second context, the third context or the fourth context.

26. The method according to any one of claims 1 to 2, characterized in that Also includes: Metadata augmentation of the object is unpacked based on at least one of a value representation of the object observed, inferred, or captured in another context.

27. The method according to any one of claims 1 to 2, characterized in that Also includes: Unbundle data catalog mapping and integration capabilities across tiers and systems.

28. The method according to any one of claims 1 to 2, characterized in that Also includes: Unbundle enhanced data profile services into a broader management system.

29. A data management system, characterized in that: include: A memory for storing instructions; A processor, coupled to the memory and configured to execute the instructions, such that the processor is configured to: storing operational metadata describing a data set, the operational metadata and the data set being stored in a memory of the data management system; receiving business-related metadata describing the data set from a business environment, the business environment being external to the data management system; storing a data set signature including a combination of the operational metadata and the business-related metadata; An operation is performed on the data set using the combination of the operational metadata and the business-related metadata in the data set signature.

30. A device, characterized in that: The device comprises: means for storing operational metadata describing a data set, said operational metadata and said data set being stored in said device; means for receiving business-related metadata describing said data set from a business environment, said business environment being external to said apparatus; means for storing a signature of a data set comprising a combination of said operational metadata and said business-related metadata; A tool for performing an operation on the data set using the combination of the operational metadata and the business-related metadata in the data set signature.

Images