Data management method, system and computer device for applications

Abstract

The present application provides a data management method, system, and computer device for an application, to solve the problem in the prior art that targeted data management cannot be achieved for a specific application. The method provided by the present application formulates and executes a specific storage policy for the specific information of the application by obtaining information of the application, where the information of the application includes, but is not limited to, one or more of the I / O operation information of the application, the running information of the application, and the information of the process of the application. Since the formulated storage policy is based on the information of the application, executing the storage policy can optimize the storage of the data read and written by the application in the storage system and / or improve the efficiency of the data read and written by the application. Compared with the prior art that configures a policy applicable to all applications based on a preset method, different policies can be formulated for different applications respectively, effectively improving the compatibility of data storage and data reading and writing with different applications.

Description

Technical Field

[0001] This application relates to the field of information technology, and particularly to a data management method, system, and computer device for an application. Background Art

[0002] In current distributed storage systems, the data storage method is basically pre-configured or specified by the user when creating a directory. For example, in a file system, the default number of stripes for a file is 1, and the default stripe size is 1 megabyte (Mbyte, MB). After the user creates a directory, a special command can be used to specify the number of stripes and / or stripe size of the files created in that directory.

[0003] This data storage method may have good performance for specific input / output (I / O) patterns or specific applications. However, when the I / O pattern of the application itself is complex, or multiple different applications use this distributed file system simultaneously, the fixed data layout method is difficult to meet the requirements of different applications. Summary of the Invention

[0004] Embodiments of this application provide a data management method, system, and computer device for an application to solve the problem in the prior art that targeted data management cannot be achieved for specific applications.

[0005] In a first aspect, embodiments of this application provide a data management method for an application. The data of the application is stored in a storage system, and the storage system includes at least one storage node. The method includes:

[0006] Obtain information about the application, where the information about the application includes at least one of the following: the I / O operation information of the application, the running information of the application, and the information of the process of the application;

[0007] Formulate a storage strategy for the application according to the information about the application;

[0008] Execute the storage strategy.

[0009] The above method can formulate and execute a storage strategy for the application according to the information about the application, and can optimize the storage of the data read and written by the application in the storage system and / or improve the access efficiency. Compared with the prior art of configuring a strategy applicable to all applications based on a preset method, the computer device can formulate different strategies for different applications respectively, effectively improving the compatibility of data storage and data reading and writing with different applications.

[0010] Optionally, the storage node may be a server or computer device including storage resources. The storage resources may include storage resources such as memory, solid-state drive (SSD), hard disk drive (HDD), or magnetic tape.

[0011] Optionally, the I / O operation information of the application includes at least one of the following information: the data size of the I / O operation performed by the application, the location of the data of the I / O operation performed by the application in the storage system, the manner of the I / O operation performed by the application, the identifier of the I / O operation performed by the application, the type of the I / O operation performed by the application, the I / O mode (sequential, strided, random, read-only, write-only, read-write mixed, etc.) of each process in the application accessing the data, the type of the data of the I / O operation performed by the application, the duration of the I / O operation performed by the application, the life cycle of the data of the I / O operation performed by the application, the redundancy policy of the data of the I / O operation performed by the application, or the sharing information of the data of the I / O operation performed by the application, etc.

[0012] Optionally, the running information of the application includes at least one of the following information: the node identifier running the application, the name of the node running the application, the IP address of the node running the application, the name of the node where the data of the I / O operation performed by the application is located, or the node identifier where the data of the I / O operation performed by the application is located, etc.

[0013] Optionally, the information of the process executing the application includes at least one of the following information: the identifier of the job executing the application, the characteristics of the job executing the application, the process identifier executing the application, or the information about which processes on which nodes of the application process which data, etc. Among them, the job executing the application may include one or more processes.

[0014] In some possible implementation manners, the method further includes:

[0015] Before the application runs or when receiving a data processing request of the application, formulating the storage policy for the application.

[0016] In some possible implementation manners, the method further includes: obtaining characteristic information of the storage system;

[0017] According to the information of the application and the characteristic information of the storage system, formulating the storage policy for the application.

[0018] In some possible implementation manners, the characteristic information of the storage system includes at least one of the following information: the hardware structure of the storage system, the hardware components of the storage system, the storage capacity of the storage system, or the data layout manner of the storage system.

[0019] Among them, the hardware structure of the storage system is the positional relationship or connection relationship between different components in the storage system, and the hardware components of the storage system are the physical forms of different components in the storage system. For example, the hardware components of the storage system may include SSDs and HDDs; the hardware structure of the storage system indicates that the SSD is a secondary storage resource, the HDD is a tertiary storage resource, and there is a connection relationship between the SSD and the HDD.

[0020] Formulating a storage policy for the application by combining the information of the application and the characteristic information of the storage system can better fit the characteristics of the storage system, can further optimize the storage of the data of the application I / O operation in the storage system, and improve the efficiency of the application executing the I / O operation.

[0021] In some possible implementation manners, the storage policy of the application includes a data layout policy, and the data layout policy is used to determine the storage manner and / or storage location of the data written by the application to the storage system.

[0022] In some possible implementation manners, the storage manner of the application writing to the storage system includes at least one of the following information:

[0023] The structure type of the data that the application needs to store, the number of copies of the data that the application needs to store, or the verification manner of the data that the application needs to store.

[0024] For example, the data layout policy may include allocating continuous storage space for the data to be written by a certain application to the storage device to improve the data writing efficiency; it may also be to determine whether the data to be written is stored as a single copy or multiple copies, and it may also be to determine the erasure code (EC) or redundant array of independent disks (RAID) of the data to be written, etc.

[0025] In some possible implementation manners, the storage policy of the application includes a caching policy, which is used to determine the data read by the application from the storage system and the caching manner of the read data. The caching manner of the data includes at least one of the following: full caching, partial caching, prefetching, write-through caching when writing data, or a cache cleaning mechanism. For example, the caching policy may include pre-writing specific data to be read by a certain application into the fastest read-write level-1 cache to improve the data reading efficiency of the application; or for a request initiated by a certain application that only writes data to the storage device, the data is directly written into the storage space of the storage device in a bypass or pass-through manner without passing through the cache, reducing the cache space occupancy and thus improving the cache utilization rate.

[0026] In some possible implementation manners, the storage policy of the application further includes a hierarchical storage policy, which is used to store data with different access frequencies accessed by the application in storage media with different read-write speeds. For example, the hierarchical storage policy may include storing frequently accessed data, that is, data with a high access frequency, in storage media with a high read-write speed, such as storage class memory (SCM) or SSD, which can improve the efficiency of accessing this data and avoid the resource occupancy and waste caused by copying data between different storage media.

[0027] In some possible implementation manners, the storage policy of the application further includes a redundancy policy, which is used to determine the backup manner and / or the number of backups of the data to be written by the application to the storage system. For example, the redundancy policy may include: for one-time data, no backup is required; for data that needs to be read subsequently, a backup is required.

[0028] In some possible implementation manners, a reference policy is saved in advance; the method further includes:

[0029] Formulating the storage policy for the application with reference to the reference policy.

[0030] Optionally, the reference policy is a policy stored in a policy information library. The policy information library may be stored in the storage system or may be an independent database, such as an independent database device.

[0031] Optionally, the method may further include: first obtaining the policy of the application already stored in the policy information library, and combining the information of the application obtained this time to determine the storage policy for the application.

[0032] If the storage policy of the application is not stored in the policy information repository, the storage policy of an application similar to the application in the policy information repository may be referred to. For example, according to the type of the application or job parameters, the storage policy of an application with the same type or job parameters as the application recorded in the policy information repository may be referred to, or the storage policy of an application similar to the type or job parameters of the application may be referred to, to determine the storage policy of the application.

[0033] Under normal circumstances, the read-write mode of an application is relatively fixed. Therefore, referring to the storage policy in the policy information repository to determine the storage policy of the application does not require determining duplicate policies for the same content each time, which can improve the efficiency of determining policies and reduce the occupation of resources such as the central processing unit (CPU) caused by determining the corresponding policies.

[0034] In some possible implementation manners, the method further includes: storing the storage policy of the application.

[0035] Optionally, the storage policy of the application may be stored in the above-mentioned policy information repository.

[0036] Optionally, after determining the policy of the application, the storage policy may be stored in the policy information repository immediately. When storing the storage policy in the policy information repository, the storage policy may be stored in an updated manner, that is, the information of the storage policy with changes is stored in the policy information repository.

[0037] In some possible implementation manners, the information of the application is information after being normalized according to a preset format and / or content. Optionally, normalizing the information of the application according to a preset format includes, but is not limited to, processing the information of the application according to a unified expression manner or arrangement order. Normalizing the information of the application according to a preset content includes, but is not limited to, adding necessary information. For example, adding context information, etc. The context information includes, but is not limited to: the identifier of the node where the application is located, the name of the node where the application is located, the Internet Protocol (IP) address of the node where the application is located, the name of the application, application parameters, or job descriptions, etc.

[0038] By normalizing the information of the application, the unified storage of information of different applications can be simplified, the manner of storing the information of the application can be simplified, and the efficiency of storing and transmitting the information of the application can be improved; at the same time, when determining a policy according to the information of the application, the information of the application can also be quickly identified and relevant policy decisions can be made, improving the efficiency of determining specific policies.

[0039] In some possible implementations, the execution of the storage strategy includes: executing part of the storage strategy before the application initiates a read or write request. For example, when a hierarchical storage strategy is determined for the application, the strategy can be executed upon receipt, so that when the application initiates a related read or write request, the related data can be quickly obtained, and the utilization rate of the hierarchical storage medium can be improved. After the hierarchical storage strategy is determined for the application, when a read data request from the application is received, the data to be read is cached in the corresponding medium based on the specific data corresponding to the read request, so that the hit rate of the cached data can be improved and the read and write efficiency of the data can be improved.

[0040] In some possible implementations, the storage strategy executed before the application initiates a read / write request includes a hierarchical storage strategy, and the hierarchical storage strategy is used to store the data to be read by the application in a storage medium with a high read / write speed in advance.

[0041] In some possible implementations, the method further includes: obtaining information of the application from the application through an extended interface; or,

[0042] The formatted information of the application is obtained from a computing device on which the application is running.

[0043] In some possible implementations, obtaining the information of the application from the application through the extended interface includes one of the following methods:

[0044] Obtain application information through the extended interface library;

[0045] The information of the application is obtained through a dedicated file, and the application or a scheduler corresponding to the application writes the information of the application into the dedicated file;

[0046] Acquire the application information through a remote procedure call; or,

[0047] The information of the application is obtained through RESTful representation state transfer.

[0048] In a second aspect, an embodiment of the present application provides a computer device, including a processor and a memory, wherein the memory stores a computer-readable executable program, and the processor is configured to read the computer-readable executable program to perform the following steps:

[0049] Acquire application information, the application information including at least one of the following information: I / O operation information of the application, running information of the application, and process information of the application; wherein data of the application is stored in a storage system, and the storage system includes at least one storage node;

[0050] Formulate a storage policy for the application according to the information of the application;

[0051] Execute the storage policy.

[0052] The above computer device can formulate a storage policy for the application according to the information of the application and execute it, which can optimize the storage of the data read and written by the application in the storage system and / or improve the efficiency of the access. Compared with the prior art that configures a policy applicable to all applications based on a preset method, the computer device can formulate different policies for different applications respectively, effectively improving the compatibility of data storage and data reading and writing with different applications.

[0053] Optionally, the storage node may be a server or a computer device including storage resources. The storage resources may include storage resources such as memory, SSD, HDD, or tape.

[0054] Optionally, the I / O operation information of the application includes at least one of the following information: the data size of the I / O operation executed by the application, the location of the data of the I / O operation executed by the application in the storage system, the manner of the I / O operation executed by the application, the identifier of the I / O operation executed by the application, the type of the I / O operation executed by the application, the I / O mode (sequential, strided, random, read-only, write-only, read-write mixed, etc.) of each process accessing data in the application, the type of the data of the I / O operation executed by the application, the duration of the I / O operation executed by the application, the life cycle of the data of the I / O operation executed by the application, the redundancy policy of the data of the I / O operation executed by the application, or the sharing information of the data of the I / O operation executed by the application, etc.

[0055] Optionally, the running information of the application includes at least one of the following information: the node identifier running the application, the name of the node running the application, the IP address of the node running the application, the name of the node where the data of the I / O operation executed by the application is located, or the node identifier where the data of the I / O operation executed by the application is located, etc.

[0056] Optionally, the information of the process executing the application includes at least one of the following information: the identifier of the job executing the application, the characteristics of the job executing the application, the process identifier executing the application, or the information about which processes on which nodes of the application process which data, etc. Among them, the job executing the application may include one or more processes.

[0057] In some possible implementation manners, the processor is further configured to read the computer-readable executable program to execute the following steps:

[0058] Before the application runs or when a data processing request of the application is received, formulate the storage policy for the application.

[0059] In some possible implementation manners, the processor is further configured to read the computer-readable executable program to perform the following steps:

[0060] Obtain the feature information of the storage system;

[0061] According to the information of the application and the feature information of the storage system, formulate the storage policy for the application.

[0062] In some possible implementation manners, the feature information of the storage system includes at least one of the following information: the hardware structure of the storage system, the hardware composition of the storage system, the storage capacity of the storage system, or the data layout manner of the storage system.

[0063] Among them, the hardware structure of the storage system is the positional relationship or connection relationship between different components in the storage system, and the hardware composition of the storage system is the physical form of different components in the storage system. For example, the hardware composition of the storage system may include SSD and HDD; the hardware structure of the storage system indicates that the SSD is a secondary resource, the HDD is a tertiary resource, and there is a connection relationship between the SSD and the HDD.

[0064] In some possible implementation manners, the storage policy of the application includes a data layout policy, and the data layout policy is used to determine the storage manner and / or storage location of the data written by the application to the storage system.

[0065] In some possible implementation manners, the storage manner of the application written to the storage system includes at least one of the following information:

[0066] The structure type of the data that the application needs to store, the number of copies of the data that the application needs to store, or the verification method of the data that the application needs to store.

[0067] For example, the data layout policy may include allocating continuous storage space for the data to be written to the storage device by a certain application to improve the data writing efficiency; it may also be to determine whether the data to be written is stored in a single copy or multiple copies, or it may also be to determine the erasure code (EC) or redundant array of independent disks (RAID) of the data to be written, etc.

[0068] In some possible implementation manners, the storage policy of the application includes a caching policy, which is used to determine the data read by the application from the storage system and the caching manner of the read data. The caching manner of the data includes at least one of the following: full caching, partial caching, prefetching, write-through caching when writing data, or a cache cleaning mechanism. For example, the caching policy may include pre-writing specific data to be read by a certain application into the first-level cache with the fastest read and write speed to improve the data reading efficiency of the application; or for a request from a certain application that only writes data to the storage device, the data is directly written into the storage space of the storage device in a bypass or pass-through manner without passing through the cache, reducing the space occupied by the cache to improve the utilization rate of the cache.

[0069] In some possible implementation manners, the storage policy of the application further includes a hierarchical storage policy, which is used to store data with different access frequencies accessed by the application in storage media with different read and write speeds. For example, the hierarchical storage policy may include storing frequently accessed data, that is, data with a high access frequency, in storage media with a high read and write speed, such as SCM or SSD, which can improve the efficiency of accessing this data and avoid the resource occupation and waste caused by copying data between different storage media.

[0070] In some possible implementation manners, the storage policy of the application further includes a redundancy policy, which is used to determine the backup manner and / or the number of backups of the data to be written by the application to the storage system. For example, the redundancy policy may include: for one-time data, no backup is required; for data that needs to be read subsequently, a backup is required.

[0071] In some possible implementation manners, a reference policy is pre-stored; then the processor is further configured to read the computer-readable executable program to execute the following steps:

[0072] Formulate the storage policy for the application with reference to the reference policy.

[0073] Optionally, the processor is further configured to read the computer-readable executable program to execute the following steps: first obtain the policy of the application already stored in the policy information library, and combine the information of the application obtained this time to determine the storage policy for the application.

[0074] If the storage policy of the application is not stored in the policy information repository, the storage policy of an application similar to the application in the policy information repository may be referred to. For example, according to the type or job parameters of the application, the storage policy of an application with the same type or job parameters as recorded in the policy information repository may be referred to, or the storage policy of an application similar to the type or job parameters of the application may be referred to, to determine the storage policy of the application.

[0075] Under normal circumstances, the read-write mode of an application is relatively fixed. Therefore, referring to the storage policy in the policy information repository to determine the storage policy of the application does not require determining duplicate policies for the same content each time, which can improve the efficiency of policy determination and reduce the occupation of resources such as the CPU caused by determining the corresponding policy.

[0076] In some possible implementation manners, the processor is further configured to read the computer-readable executable program to perform the following steps: storing the storage policy of the application.

[0077] Optionally, the processor may store the storage policy of the application in the above-mentioned policy information repository.

[0078] Optionally, after determining the policy of the application, the processor may immediately store the storage policy in the policy information repository. When storing the storage policy in the policy information repository, the storage policy may be stored in an updated manner, that is, the information of the changed storage policy is stored in the policy information repository.

[0079] In some possible implementation manners, the information of the application is the information after being normalized according to a preset format and / or content. Optionally, normalizing the information of the application according to a preset format includes, but is not limited to, processing the information of the application according to a unified expression manner or arrangement order. Normalizing the information of the application according to a preset content includes, but is not limited to, adding necessary information. For example, adding context information, etc. The context information includes, but is not limited to: the identifier of the node where the application is located, the name of the node where the application is located, the IP address of the node where the application is located, the name of the application, application parameters, or job descriptions, and other context information.

[0080] By normalizing the information of the application, the unified storage of the information of different applications can be simplified, the manner of storing the information of the application can be simplified, and the efficiency of storing and transmitting the information of the application can be improved; at the same time, when determining the policy according to the information of the application, the information of the application can also be quickly identified and relevant policy decisions can be made, improving the efficiency of determining the specific policy.

[0081] In some possible implementation manners, the processor reads the computer-readable executable program to execute the storage policy, including: before the application initiates a read / write request, the processor reads the computer-readable executable program to execute some policies in the storage policy. For example, when a hierarchical storage policy is determined for the application, it can be executed after being received. In this way, when the relevant read / write requests initiated by the application occur, relevant data can be quickly obtained, and the utilization rate of the hierarchical storage medium can be improved. After a hierarchical storage policy is determined for the application, when a read data request of the application is received, based on the specific data corresponding to the read request, the data to be read is cached in the corresponding medium, which can improve the hit rate of the cached data and the read / write efficiency of the data.

[0082] In some possible implementation manners, the some policies include a hierarchical storage policy, and the hierarchical storage policy is used to store the data to be read by the application in advance in a storage medium with high read / write speed.

[0083] In some possible implementation manners, the storage system further includes at least one computing node, the application runs on the at least one computing node, the data read and written by the application is stored on the at least one storage node, and the at least one storage node includes the computer device.

[0084] In some possible implementation manners, the processor obtains the information of the application through an extended interface from the application; or,

[0085] The processor obtains the information of the application that has been formatted from the computing node where the application runs.

[0086] In some possible implementation manners, the at least one storage node includes the computer device, and each storage node includes storage resources and computing resources;

[0087] The processor obtaining the information of the application includes:

[0088] The processor obtains the information of the application through an extended interface.

[0089] In some possible implementation manners, the processor obtaining the information of the application through an extended interface from the application includes one of the following manners:

[0090] The processor obtains the information of the application through an extended interface library;

[0091] The processor obtains the information of the application through a dedicated file, and the application or the scheduler corresponding to the application writes the information of the application into the dedicated file;

[0092] The processor obtains information about the application through a remote procedure call; or,

[0093] The processor obtains information about the application through Representational State Transfer RESTful.

[0094] In a third aspect, an embodiment of the present application provides a data management system, which includes an application awareness module, a policy determination module, and a policy execution module, where:

[0095] The application awareness module is used to obtain information about the application, and the information about the application includes at least one of the following information: the I / O operation information of the application, the running information of the application, and the information of the process of the application; wherein, the data of the application is stored in a storage system, and the storage system includes at least one storage node;

[0096] The policy determination module is used to formulate a storage policy for the application according to the information;

[0097] The policy execution module is used to execute the policy.

[0098] The above data management system can formulate and execute a storage policy for the application according to the information of the application, and can optimize the storage of the data read and written by the application in the storage system and / or improve the efficiency of the access. Compared with the prior art that configures a policy applicable to all applications based on a preset method, the computer device can formulate different policies for different applications respectively, effectively improving the compatibility of data storage and data reading and writing with different applications.

[0099] In some possible implementation manners, before the application runs or when a data processing request of the application is received, the policy determination module formulates the storage policy for the application.

[0100] In some possible implementation manners, the policy determination module is further used to obtain feature information of the storage system, and formulate the storage policy for the application according to the information of the application and the feature information of the storage system.

[0101] In some possible implementation manners, the feature information of the storage system includes at least one of the following information: the hardware structure of the storage system, the hardware composition of the storage system, the storage capacity of the storage system, or the data layout mode of the storage system.

[0102] In some possible implementation manners, the storage policy of the application includes a data layout policy, and the data layout policy is used to determine the storage mode and / or storage location of the data written by the application into the storage system.

[0103] In some possible implementations, the storage method in which the application is written into the storage system includes at least one of the following information:

[0104] The structure type of the data that the application needs to store, the number of copies of the data that the application needs to store, or the verification method of the data that the application needs to store.

[0105] In some possible implementations, the storage strategy of the application includes a cache strategy, which is used to determine the data read by the application from the storage system and the cache method of the read data, and the cache method of the data includes at least one of the following: full cache, partial cache, pre-reading method, direct cache when writing data, or a cache cleanup mechanism.

[0106] In some possible implementations, the storage strategy of the application further includes a hierarchical storage strategy, and the hierarchical storage strategy is used to store data of different frequencies accessed by the application in storage media with different read and write speeds.

[0107] In some possible implementations, the storage policy of the application further includes a redundancy policy, and the redundancy policy is used to determine a backup method and / or a backup quantity of data to be written by the application into the storage system.

[0108] In some possible implementations, the policy determination module is further configured to formulate the storage policy for the application with reference to a pre-saved reference policy.

[0109] In some possible implementations, the application information is information that has been normalized according to a preset format and / or content.

[0110] In some possible implementations, the executing the storage policy includes: executing a portion of the storage policy before the application initiates a read or write request.

[0111] In some possible implementations, the policy determination module is further configured to:

[0112] Obtaining information of the application from the application through an extended interface; or,

[0113] The formatted information of the application is obtained from a computing device on which the application is running.

[0114] In some possible implementations, the policy determination module obtains the information of the application from the application through the extended interface in one of the following ways:

[0115] Obtain application information through the extended interface library;

[0116] Obtain the information of the application through a dedicated file, and the application or the scheduler corresponding to the application writes the information of the application into the dedicated file;

[0117] Obtain the information of the application through a remote procedure call; or,

[0118] Obtain the information of the application through Representational State Transfer (RESTful).

[0119] In a fourth aspect, the present application provides a chip, which is used to execute the steps of the method in the first aspect above. Alternatively, the chip is used to implement the content that the processor needs to execute in the second aspect above, or one or more of the functions of the application awareness module, the policy determination module, and the policy implementation module in the third aspect above. Optionally, the chip includes a processor and a chip interface. The processor may be a CPU, a microprocessor, an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), etc.

[0120] In a fifth aspect, the present application provides a computer-readable storage medium, in which instructions are stored, and the instructions direct a computer device to execute the method described in the first aspect above.

[0121] In a sixth aspect, the present application provides a computer program product containing instructions, which, when running on a computer device, causes the computer device to execute the method described in the first aspect above.

[0122] Based on the implementation manners provided in the above aspects of the present application, further combinations can be made to provide more implementation manners. Description of the Drawings

[0123] The following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0124] Figure 1 It is a schematic structural diagram of a storage system provided by an embodiment of the present application;

[0125] Figure 2 It is a schematic structural diagram of another storage system provided by an embodiment of the present application;

[0126] Figure 3ASchematic diagram of the logical architecture of a software module for implementing an application data management method provided by an embodiment of this application;

[0127] Figure 3B Provided by an embodiment of this application Figure 3A Schematic diagram of the implementation manner of the application awareness module 301 in

[0128] Figure 3C Provided by an embodiment of this application Figure 3A Schematic diagram of the implementation manner of the policy determination module 302 in

[0129] Figure 3D Provided by an embodiment of this application Figure 3A Schematic diagram of the implementation manner of the policy execution module 303 in

[0130] Figure 3E Provided by an embodiment of this application Figure 3A Schematic diagram of the implementation manner of the policy determination module 302 and the policy execution module 303 in

[0131] Figure 3F Provided by an embodiment of this application Figure 3A Schematic diagram of the implementation manner of the application awareness module 301, the policy determination module 302, and the policy execution module 303 in

[0132] Figure 4 Flowchart of an application data management method provided by an embodiment of this application;

[0133] Figure 5 Flowchart of a method for optimizing the data management method of a storage system according to application information provided by an embodiment of this application;

[0134] Figure 6 Schematic diagram of a specific application scenario for implementing an application data management method in a storage system provided by an embodiment of this application;

[0135] Figure 7A Flowchart of the specific implementation manner for Application A provided by an embodiment of this application to achieve application-aware data reading and writing;

[0136] Figure 7B Flowchart of the specific implementation manner for Application B provided by an embodiment of this application to achieve application-aware data reading and writing;

[0137] Figure 7C Flowchart of the specific implementation manner for Application C provided by an embodiment of this application to achieve application-aware data reading and writing;

[0138] Figure 8Schematic diagram of a logical implementation of the data layout decision engine provided by an embodiment of the present application;

[0139] Figure 9 Schematic flow chart of a method for data management of an application provided by an embodiment of the present application;

[0140] Figure 10 Schematic diagram of the structure of a data management system 1000 provided by an embodiment of the present application;

[0141] Figure 11 Schematic diagram of the structure of a computer device 2000 provided by an embodiment of the present application; Detailed implementation manners

[0142] The embodiments of the present invention will be described below with reference to the accompanying drawings.

[0143] Terms such as "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and do not necessarily have to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments described herein can be implemented in an order different from that shown or described herein. In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of such features.

[0144] In the description and claims of this application, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or modules need not be limited to those steps or modules clearly listed, but may include other steps or modules not clearly listed or inherent to these processes, methods, products or devices. The naming or numbering of steps that appear in this application does not mean that the steps in the method flow must be executed in the chronological / logical order indicated by the naming or numbering. The named or numbered process steps can be changed in the order of execution according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved. The division of units that appears in this application is a logical division. In actual implementation, there may be other division methods. For example, multiple units can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed coupling, direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection between units can be electrical or other similar forms, which are not limited in this application. Moreover, the units or subunits described as separate components may or may not be physically separated, may or may not be physical units, or may be distributed into multiple circuit units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this application.

[0145] It should be understood that the terms used in the description of various examples in this application and claims are only for describing specific examples and are not intended to be limiting. As used in the description of various examples and the appended claims, the singular forms "a", "an" and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0146] It should also be understood that the term "and / or" used in the description and claims of this application refers to and encompasses any and all possible combinations of one or more of the associated listed items. The term "and / or" is a description of the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this application generally represents an "or" relationship between the front and back associated objects.

[0147] It should be understood that determining B based on A does not mean determining B only based on A. B can also be determined based on A and / or other information.

[0148] It should also be understood that the term "comprises" (also referred to as "includes", "including", "comprises" and / or "comprising") when used in this specification specifies the presence of the stated features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0149] It should also be understood that the term "if" can be interpreted to mean "when" ("when" or "upon") or "in response to determining" or "in response to detecting". Similarly, depending on the context, the phrase "if determined..." or "if detected [the stated condition or event]" can be interpreted to mean "when determining..." or "in response to determining..." or "when detecting [the stated condition or event]" or "in response to detecting [the stated condition or event]".

[0150] It should be understood that the "one embodiment", "an embodiment", "a possible implementation" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment or implementation are included in at least one embodiment of the present application. Therefore, the "in one embodiment" or "in an embodiment", "a possible implementation" that appear throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner.

[0151] In the embodiments of the present application, an application refers to a program or service that realizes certain functions through electronic devices or electronic means such as a computer, a computer, a mobile phone, a network or a cloud service. For example, the application can be an Office application (such as the Word tool, etc.), a video application (such as Youku, iQIYI, Facebook, etc.) or a social application (such as WeChat, Weibo, etc.). The embodiments of the present application do not limit the specific form and presentation form of the application. As long as it is a program or tool that can realize data reading and writing, or software that realizes the same or similar functions, it is within the scope disclosed in the embodiments of the present application.

[0152] A storage system is a system for storing programs and / or data. The system can include at least one storage node and a control component. The program of the application can be stored in the storage system, and the computer device runs the above application by reading the program in the storage system. It should be noted that the storage system can be integrated into the computer device or independent of the computer device. When the storage system is independent of the computer device, each computer device in the cluster can share the storage system to form a cluster storage system.

[0153] Considering various requirements such as read-write performance, capacity, and cost, multiple levels of storage devices are generally configured in a storage system. Specifically, a cache (short for Cache), a main memory (also known as internal memory, abbreviated as memory), and a secondary memory (also known as external memory, abbreviated as external storage) can be configured in the storage system.

[0154] The storage system provides different storage resources through the above-mentioned memories. Specifically, the memory can be directly accessed by the processor of the computer device, directly providing data and instructions for the processor, and storing the data processed by the processor. Its read-write speed is fast, but the capacity is relatively small. The external storage cannot be directly accessed by the processor, so it is mainly used to store data and programs that the processor does not need temporarily. Its read-write speed is lower than that of the memory, but the capacity is relatively large. The cache is located between the processor and the memory and is used to solve the problem of the mismatch between the operation speed of the processor and the read-write speed of the memory. Specifically, the data that the processor will access is stored in the cache, and this data is part of the data in the memory.

[0155] In existing distributed storage systems, many have "hierarchical storage" or "Cache". Its main function is to store the data that is being used or will be used in the system (also known as hot data) in a medium with a faster read-write speed, and store the data that is rarely used (also known as cold data) in a medium with a lower read-write speed.

[0156] The hierarchical storage technology or Cache technology, in essence, blindly perceives the application's access to data and roughly predicts the hot and cold changes of the data. It has a high degree of "speculativeness" or "randomness" and is difficult to accurately predict all the hot and cold changes of the data. If the algorithm is not good, it is easy to cause hot data to be stored in a medium with a low read-write speed, while cold data is instead written into a medium with a high read-write speed. For the hot data stored in a medium with a low read-write speed, its read-write efficiency is low. Moreover, the medium with a high read-write speed often has a higher cost, and the cold data occupying the space of the medium with a high read-write speed will increase the cost.

[0157] In addition, in a distributed storage system, to improve the system throughput, the data of a file is split into blocks of the same size and stored on different storage nodes in units of blocks. This approach can, on the one hand, make full use of the bandwidth of multiple nodes and multiple storage devices. On the other hand, various redundancy strategies, such as EC and multiple replicas, can be constructed on this basis to ensure data reliability. This default data sharding method is not suitable for the way applications use data in some scenarios. For example, when the default sharding strategy is one shard per file, if the application accessing (reading / writing) this file runs on multiple nodes, especially when applications on multiple nodes need to write to this file, on the one hand, it cannot guarantee the file I / O bandwidth, and on the other hand, it will generate many I / O conflicts, seriously slowing down the read / write progress of each application and affecting the efficiency of application read / write.

[0158] In the above implementation, since the storage system is unaware of the I / O mode of the application, it is unable to select an appropriate data layout or formulate an accurate cache strategy for different application I / O modes, which affects the effective management of stored data and the efficiency of application access.

[0159] The embodiment of this application provides a method for data management in a storage system, which can enable the storage system to perceive information such as the I / O mode of the application, select reasonable strategies such as data layout according to the I / O mode of the application, and use a differentiated cache strategy and / or a hierarchical storage scheme to improve the performance of the storage system and the efficiency of application access such as reading and writing data.

[0160] Figure 1 FIG. is a schematic structural diagram of a storage system provided by an embodiment of this application. As Figure 1 shown, the storage system consists of two parts: storage nodes and a storage client. Figure 1 Taking the example that it includes storage node 111, storage node 112, and storage node 113, and the storage client runs on computing node 101 and computing node 102 respectively for illustration.

[0161] Multiple storage nodes form a storage backend, which is composed of media with different bandwidths or characteristics inside, such as it can be composed of memory, SCM, SSD, HDD, or tape, etc. Figure 1 Taking the example that the storage node includes memory, SSD, and HDD for illustration.

[0162] Storage media with different characteristics in storage nodes form storage pools at multiple tiers, also known as hierarchical storage management (HSM). Usually, frequently accessed data and recently used data are placed in faster tiers (such as media at the memory or SSD level), while data with fewer accesses is stored in slower tiers (such as media at the HDD and tape levels). For example Figure 1 in, the memory in at least two storage nodes can form a memory pool of the storage system, the SSDs in at least two storage nodes can form a flash pool of the storage system, and the HDDs in at least two storage nodes can form a hard disk pool of the storage system, so as to constitute storage media at different levels for storing data with different access frequencies.

[0163] The storage client is an interface between the compute node and the storage node, used to receive I / O requests initiated by the application and implement the interaction between the compute node and the storage node. Figure 1 Taking the storage client running on the compute node as an example for illustration. Many different applications can run on each compute node, and different applications can connect to the storage system through the same storage client. Figure 1 Taking two applications running separately on each compute node as an example for illustration.

[0164] It should be noted that Figure 1 only exemplarily shows the composition and connection relationship of the storage node and the compute node. In a specific implementation, the storage node (storage node 111, storage node 112, and storage node 113) may further include hardware resources such as a processor and a bus, as well as software or software modules related to data read / write control implemented through the processor, etc. The compute node (compute node 101 and compute node 102) may further include hardware components such as a processor, a memory, and a bus. The applications and storage clients in each compute node can implement corresponding functions through the processor, etc. This application embodiment will not be elaborated further.

[0165] Figure 2 This is a schematic structural diagram of another storage system provided by the embodiments of this application. Figure 2 The shown scenario is an implementation scenario of a hyper-converged infrastructure (HCI). In the HCI scenario, computing and storage coexist on each HCI node. For example, node 201 includes both computing and storage parts. The computing part includes applications, and the storage part includes storage media such as memory, SSD, and HDD. The applications in the computing part are connected to the storage part in this node and implement data read / write. Applications in different nodes can also access data stored in other nodes.

[0166] It should be noted that, for the convenience of description, Figure 2 only part of the composition of each node in [] is shown. In the specific implementation, the computing part of each node may further include hardware resources such as a processor, and the storage part may further include hardware resources such as a bus and software resources for implementing data read / write control, etc., which will not be elaborated here.

[0167] Figure 1 and Figure 2 The scenarios shown are only examples and should not be regarded as a limitation to the embodiments of the present application. In the specific implementation, it can also be implemented in other similar scenarios. For example, the embodiments of the present application can also be applied in a distributed storage system, including but not limited to a distributed file storage system, a distributed block storage system, or a distributed object storage system; it can also be applied in a system such as cloud storage or the platform as a service (PaaS) layer of a public cloud. As an optional implementation manner, the solution provided by the embodiments of the present application is more suitable for being applied in a storage system with more applications, or a storage system applicable to applications with variable I / O modes.

[0168] Figure 3A is a logical schematic diagram of a software module for implementing the data management method of an application in a storage system provided by the embodiments of the present application. As Figure 3A shown, the software module for implementing the embodiments of the present application includes an application awareness module 301, a policy determination module 302, and a policy implementation module 303. Among them:

[0169] The application awareness module 301 is used to obtain information of the application, including static attribute information and / or dynamic variable information of the application, and provide the obtained information to the policy determination module 302. The information of the application includes, but is not limited to, one or more of the following information: the application I / O operation information, the running information of the application, the process information for executing the application, etc. Among them:

[0170] The application I / O operation information includes, but is not limited to, at least one of the following information: the data size of the application executing the I / O operation, the location of the data of the application executing the I / O operation in the storage system, the way of the application executing the I / O operation, the identifier of the application executing the I / O operation, the type of the application executing the I / O operation, the I / O mode (sequential, strided, random, read-only, write-only, read-write mixed, etc.) of each process in the application accessing the data, the type of the data of the application executing the I / O operation, the duration of the application executing the I / O operation, the life cycle of the data of the application executing the I / O operation, the redundancy policy of the data of the application executing the I / O operation, or the sharing information of the data of the application executing the I / O operation, etc.;

[0171] The running information of the application includes, but is not limited to, at least one of the following information: the node identifier of the node running the application, the name of the node running the application, the IP address of the node running the application, the node name where the data for the I / O operation performed by the application is located, or the node identifier where the data for the I / O operation performed by the application is located, etc.

[0172] The information of the process executing the application includes, but is not limited to, at least one of the following information: the identifier of the job executing the application, the characteristics of the job executing the application, the process identifier executing the application, or information on which processes on which nodes of the application process which data. Among them, the job executing the application may include one or more processes.

[0173] The policy determination module 302 is configured to determine a storage policy for the application according to the information provided by the application perception module 301.

[0174] The storage policy includes, but is not limited to, a data layout policy, a Cache policy, a hierarchical storage policy, or a redundancy policy, etc.

[0175] In one implementation, the policy determination module 302 is further configured to store the storage policy for the application in a policy information library. The policy information library is used to store the storage policies determined by the policy determination module 302 for each application. Correspondingly, when the policy determination module 302 determines a specific policy for the application according to the information provided by the application perception module 301, it can refer to the policies already determined in the policy information library and combine the information of the application received this time to determine the data management policy this time. Usually, the read-write mode of an application is relatively fixed. Therefore, referring to the policies in the policy information library to determine the storage policy of the application does not require determining duplicate policies for the same content each time, which can improve the efficiency of the policy determination module 302 in determining the storage policy and reduce the occupancy of resources such as the CPU caused by determining the corresponding policy.

[0176] Exemplarily, the policy determination module 302 may also store the updated or changed storage policy in the policy information library after determining the storage policy of the application. That is, once the policy determination module 302 has determined the policy, it will send the determined storage policy to the policy information library.

[0177] The policy implementation module 303 is configured to execute the storage policy determined by the policy determination module 302.

[0178] Figure 3A The software modules shown can be deployed independently or in combination in different computer devices to implement the corresponding functions. The following takes Figures 3B to 3F as an example to illustrateFigure 3A The different implementations of the software modules shown in a computer device. It can be understood that Figures 3B to 3F This is only an exemplary illustration. In specific implementations, other methods can also be used.

[0179] Such as Figure 3B As shown, the first computer device 310 includes a bus 311, a processor 312, a communication interface 313, and a memory 314. The processor 312, the memory 314, and the communication interface 313 communicate with each other through the bus 311. The bus 311 can be a peripheral component interconnect (PCI) bus, a peripheral component interconnect express (PCIe) bus, or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, Figure 3B only a thick line is used to represent it in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface 313 is used for external communication, such as receiving operation attribute information and object attribute information of I / O operations, etc.

[0180] Among them, the processor 312 can be a CPU, a Graphics Processing Unit (GPU), a general-purpose GPU (GPGPU), a Tensor Processing Unit (TPU), a Data Processing Unit (GPU), or other chips with computing capabilities. The memory 314 can include volatile memory, such as random access memory (RAM). The memory 314 can also include non-volatile memory, such as read-only memory (ROM), flash memory, HDD, or SSD.

[0181] The memory 314 stores programs or instructions, such as the programs or instructions required to implement the functions of the application awareness module 301. The processor 312 executes the programs or instructions to obtain information about the application. Of course, the memory 314 can also store data, including but not limited to the data that the first computer device 310 needs to store; it can also store the information about the application obtained by the application awareness module 301.

[0182] In one implementation, the information of the application stored in the memory 314 can be periodically summarized into a dedicated database. The dedicated database can be an independent database device dedicated to storing databases related to applications.

[0183] As Figure 3C shown, the second computer device 320 includes a bus 321, a processor 322, a communication interface 323, and a memory 324. The processor 322, the memory 324, and the communication interface 323 communicate with each other via the bus 321. The memory 324 stores programs or instructions, such as the programs or instructions required to implement the functions of the policy determination module 302. Of course, the memory 324 can also store data, such as data to be read and written, etc.

[0184] Figure 3C The processor, memory, bus, etc. in Figure 3B are similar to those in

[0185] Optionally, Figure 3C the memory 324 of Figure 1 can also store the above-mentioned policy information library. The policy determination module 302 can obtain the policies related to the application locally from the second computer device 320 and formulate a storage policy for the application by combining the information of the application obtained this time. The policy determination module 302 can also store the formulated storage policy of the application or the updated storage policy of the application in the policy information library. Of course, the policy information library in the embodiments of the present application can also be stored in an independent database, such as an independent computer device for storing data; it can also be distributedly stored on different storage nodes, such as

[0186] As Figure 3D shown, the third computer device 330 includes a bus 331, a processor 332, a communication interface 333, and a memory 334. The processor 332, the memory 334, and the communication interface 333 communicate with each other via the bus 331. The memory 334 stores programs or instructions, such as the programs or instructions required to implement the functions of the policy execution module 303. Of course, the memory 334 can also store data, such as data to be read and written, etc.

[0187] Figure 3D The processor, memory, bus, etc. in Figure 3B are similar to those in

[0188] As Figure 3EAs shown, the fourth computer device 340 includes a bus 341, a processor 342, a communication interface 343, and a memory 344. The processor 342, the memory 344, and the communication interface 343 communicate with each other via the bus 341. Programs or instructions are stored in the memory 344, such as the programs or instructions required to implement the functions of the policy determination module 302 and the policy execution module 303. Of course, data such as data to be read and written can also be stored in the memory 344.

[0189] Figure 3E The processor, memory, bus, etc. in Figure 3B are similar to those in the above, and will not be elaborated here.

[0190] As Figure 3F shown, the fifth computer device 350 includes a bus 351, a processor 352, a communication interface 353, and a memory 354. The processor 352, the memory 354, and the communication interface 353 communicate with each other via the bus 351. Programs or instructions are stored in the memory 354, such as the programs or instructions required to implement the functions of the application awareness module 301, the policy determination module 302, and the policy execution module 303. Of course, data such as data to be read and written can also be stored in the memory 354.

[0191] Figure 3F The processor, memory, bus, etc. in Figure 3B are similar to those in the above, and will not be elaborated here.

[0192] The above Figures 3B to 3F computer devices can be selectively applicable to the above Figure 1 or Figure 2 shown scenarios. Exemplarily, Figure 3B the shown computer device can be applicable to Figure 1 the computing nodes in (such as computing nodes 101 or 102), Figure 3C , Figure 3D or Figure 3E the shown computer device can be applicable to Figure 1 the storage nodes shown (such as storage nodes 111, 112, or 113). Figure 3F The shown computer device can be applicable to Figure 2 the nodes described in (such as nodes 201, 202, or 203).

[0193] In some possible implementations, the functions of the foregoing application awareness module 301, policy determination module 302, and policy implementation module 303 may also be implemented by one or more chips. Each chip includes a processor and a chip interface. Optionally, the processor may be a CPU, a microprocessor, an ASIC, an FPGA, etc. Among them, the chip interface is used to receive instructions and transmit them to the processor. The processor executes the foregoing instructions to perform the steps of the foregoing method for implementing data management. Of course, when the functions of the foregoing application awareness module 301, policy determination module 302, and policy implementation module 303 are implemented by multiple chips, the chip interface of each chip receives partial instructions, and the processor of each chip implements corresponding functions according to partial steps of the foregoing partial method for implementing data management. In this way, multiple processors may implement the steps of the method for managing data in the storage system provided in the embodiments of the present application.

[0194] Next, taking Figure 1 the scenario shown in Figure 3B where the first computer device shown is applicable to Figure 1 the computing node 101 in Figure 3E and the fourth computer device shown is applicable to Figure 1 the storage node 111 in

[0195] as an example, the method for implementing data management of an application in a storage system provided in the embodiments of the present application will be described.

[0195] As Figure 4 shown in

[0196] Step 400: The application awareness module 301 receives information of an application;

[0197] Exemplarily, the application awareness module 301 may receive information from an application through an extended interface.

[0198] In one implementation, the application and the application awareness module 301 may implement information transfer through more than one extended interface. The more than one extended interface is an interface added by an extended method and serves as an interface for transferring information between the application and the application awareness module 301. These extended interfaces may be more than one different interface and are different from the existing interfaces of the application.

[0199] In specific implementation, the extension of the interface may be implemented in the following manner, but the embodiments of the present application are not limited to the following implementation manner:

[0200] 1) The application awareness module 301 provides a dedicated interface library to implement interface expansion; among them, the interface library can be an application programming interface (API) library. The interface library can be a dynamic link library, and when the application is compiled, a link to the dynamic link library is added, and the dynamic link library is linked to the executable file of the application. In this way, when the application runs, it can pass the information of the application to the application awareness module 301 by calling the dynamic link library.

[0201] This method can be applied to scenarios where the code of the application can be modified. That is, by modifying the code of the application, the application and the application awareness module 301 can transfer information through the interface library.

[0202] 2) The application awareness module 301 provides a command-line tool, and the application can directly call the command-line tool to call the interface library provided by the application awareness module 301 through the command-line tool;

[0203] This method is applicable to scenarios where the code of the application cannot be modified. Exemplarily, the application or the relevant scheduler calls the command-line tool to implement the call of the interface library, and further realizes the transfer of information between the application and the application awareness module 301.

[0204] 3) The storage system provides information reception and display files (such as the file systems procfs and sysfs mechanisms), and the application or the relevant scheduler can transfer information to the application awareness module 301 by writing data to such files.

[0205] In this case, the application awareness module 301 can access the above file system and obtain the information of the application, that is, the application awareness module 301 needs to have the ability to access the file system.

[0206] Similarly, if the code of the application can be modified, the application can directly write information to the above file system; if the code of the application cannot be modified, the information can be written to the above file system by calling the command-line tool.

[0207] 4) The application awareness module 301 provides remote procedure call (RPC) or Representational State Transfer (RESTful).

[0208] That is, the application awareness module 301 requests the information of the application from the application through RPC or RESTful.

[0209] As an alternative implementation, the above application information may be transmitted by the application, or may be transmitted by other entities independent of the application (such as a scheduler). Specifically, when the application transmits the above application information, it may transmit the information of the application to the application awareness module 301 in the above manner before the application initiates an I / O request. When the above application information is transmitted by the scheduler, it may transmit the information of the application in the above manner before or during the operation of the application.

[0210] It should be noted that the information of the application transmitted between the application and the application awareness module 301 includes, but is not limited to, at least one of the following information: the application I / O operation information, the running information of the application, and the information of the application process. For example, the information of the application may be information such as the file type to be accessed by the application, the node to be accessed by the application, or the read / write mode of the application; these information are static attribute information determined by the modeling module of the application, and the application awareness module 301 may obtain the information of the application in the above manner before or just when the application runs. The application awareness module 301 may also obtain the dynamic variable information of the application when the application just runs. For example, based on the requirements of a special scenario, the application needs to access specific data, and the information of these specific data is not included in the static attribute information.

[0211] Optionally, when the application awareness module 301 is implemented on a computer device serving as a storage node, as Figure 3F shown in the implementation manner, the application awareness module 301 may obtain the information of the application when the application initiates a read / write request.

[0212] Step 402: The application awareness module 301 sends the obtained information of the application to the policy determination module 302;

[0213] The application awareness module 301 sends the obtained information of the application to the policy determination module 302, so that the policy determination module can formulate a policy for the application according to the information of the application.

[0214] In one implementation, the application awareness module 301 can normalize the obtained information of the application and send the normalized information to the policy determination module 302. The normalization process is to convert the information of the application into a unified format and store it. The way of converting to a unified format includes, but is not limited to, processing the information of the application according to a unified expression or arrangement order, or adding necessary information, such as adding context information, etc. The context information includes, but is not limited to: the identifier of the node where the application is located, the name of the node where the application is located, the IP address of the node where the application is located, the name of the application, application parameters, or job descriptions, etc. For example, information such as the identifier of the node where the application is located can be obtained through existing functional modules in the prior art.

[0215] Normalizing the information of the application can simplify the unified storage of information of different applications, simplify the way of storing application information, and improve the efficiency of storing and transmitting application information; at the same time, it can also facilitate the policy determination module 302 to quickly identify the information of the application and make relevant policy decisions, and improve the efficiency of the policy determination module 302 to determine specific policies.

[0216] For example, the application awareness module 301 can transmit different information through multiple interfaces with the policy determination module 302; that is, the application awareness module 301 can pass different messages through different interfaces to the policy determination module 302. Of course, the application awareness module 301 can also pass the information of the application to the policy determination module 302 through a unified interface.

[0217] The format of the message transmitted by the application awareness module 301 to the policy determination module 302 can adopt the Extensible Markup Language (XML) or JavaScript Object Notation (JSON) format, or can also be a custom data structure, as long as it can be recognized by the policy determination module 302. Optionally, when the format of the message is an extensible message format, it can enable the policy determination module 302 to be changed or added according to the change or addition of requirements. In this way, it can improve the applicability to new business forms. When adding information of new services in the future, it is not necessary to modify the format, but only need to modify the content of the message format, or directly add new information to achieve. In specific implementation, the embodiments of the present application do not limit the specific format of the message.

[0218] The ways for the application awareness module 301 and the policy determination module 302 to transmit information include, but are not limited to: 1) direct function call; 2) shared memory; 3) inter-process communication; 4) dynamic library call; 5) network communication, such as remote procedure call (RPC), etc.; 6) Hypertext Transfer Protocol (HTTP) or RESTFUL, etc. Among them, the ways 1)-4) are applicable to the scenario where the application awareness module 301 and the policy determination module 302 are implemented on one node, and the ways 5)-6) are applicable to the scenario where the application awareness module 301 and the policy determination module 302 are implemented on different nodes.

[0219] Step 404: The policy determination module 302 determines the storage policy of the application according to the received information;

[0220] Based on the information of the application received, the policy determination module 302 determines a specific storage policy to implement data layout, cache, hierarchical storage, redundancy backup, etc. of the application.

[0221] Exemplarily, the policy determination module 302 may obtain the policies in the policy information repository, draw on the previously constructed policies and combine them with the information of the application to determine the storage policy of the application. The policy determination module 302 may obtain the storage policy of the application recorded in the policy information repository. If the storage policy of the application is not recorded in the policy information repository, it may refer to the storage policies of similar applications. When the policy determination module 302 refers to the storage policies of similar applications, the policy determination module 302 refers to the storage policies of applications with the same type or job parameters as the application recorded in the policy information repository according to the type or job parameters of the application, or refers to the storage policies of applications similar to the type or job parameters of the application. In one implementation, the policy determination module 302 formulates a storage policy for the application according to the obtained information of the application and combines it with the characteristic information of the storage system. Since the hardware structure, hardware composition, storage capacity, and data layout of each storage system are not exactly the same. For example, some storage systems include a memory pool, an SSD pool, and an HDD pool, while some storage systems only include an SSD pool and an HDD pool, that is, their hardware compositions are different; in some storage systems, the SSD pool is a secondary storage resource and is directly connected to the HDD pool; in some storage systems, the SSD pool is a primary storage resource and is directly connected to the HDD pool, that is, the hardware structures of the storage systems are different; some storage systems have a storage capacity of 100 GB, while some storage systems have a storage capacity of 100 TB, that is, the storage capacities of the storage systems are different; in some storage systems, the block size at the bottom layer of the storage medium is different and the adopted redundancy backup mechanism is different, that is, the data layouts of the storage systems are different. Therefore, when determining the storage policy of the application, the policy determination module 302 can better fit the characteristics of the storage system by combining the information of the application and the characteristic information of the storage system, and can further optimize the storage of the data of the I / O operation of the application in the storage system and improve the efficiency of the application performing the I / O operation.

[0222] After the policy determination module 302 determines the storage policy of the application, it may also add or update the determined storage policy of the application to the policy information repository.

[0223] The storage policy of the application determined by the policy determination module 302 includes, but is not limited to, data layout policy, Cache policy, hierarchical storage policy, redundancy policy, etc. The policy determination module 302 may send them to the corresponding policy engine or policy determination sub-module according to the category of the received information of the application for corresponding decision-making. For example:

[0224] A data layout policy engine is used to determine the layout of data to be written (data to be written to a storage device) in a storage system. Specifically, the policy layout policy engine can combine application information to determine the storage method and / or storage location of the data written by the application to the storage system. The storage method of the data written by the application to the storage system includes at least one of the following information: the structural type of the data that the application needs to store, the number of copies of the data that the application needs to store, or the verification method of the data that the application needs to store. Exemplarily, the policy layout policy engine can allocate continuous storage space for the data to be written by a certain application to a storage device to improve the efficiency of data writing; it can also be to determine whether the data to be written is stored as a single copy or multiple copies, or it can also be to determine the EC or RAID method of the data to be written, etc.

[0225] A Cache policy engine is used to determine specific cached data and / or caching methods, that is, to pre-cache the cached data (data read from a storage device) in a specific caching medium so that the application can obtain the data to be read in a timely manner. Specifically, the Cache policy engine can combine application information to determine the data read by the application from the storage system and the caching method of the read data. The caching method of the data includes but is not limited to: full caching, partial caching, cache cleaning mechanism, prefetching method, or bypassing the cache when writing data, etc. Exemplarily, the specific data to be read by a certain application is pre-written into the fastest read-write level 1 cache to improve the efficiency of the application reading data; or for a request initiated by a certain application that only writes data to a storage device, the data is directly written into the storage space of the storage device through bypass or pass-through without passing through the cache, reducing the space occupancy of the cache to improve the utilization rate of the cache.

[0226] A hierarchical storage policy engine is used to store data with different access frequencies in media of different tiers. Specifically, the hierarchical storage policy engine can combine application information to determine that the data with different access frequencies accessed by the application is stored in storage media with different read-write speeds. The hierarchical storage of the data accessed by the application determined by the hierarchical storage policy engine includes but is not limited to: storing the data with high access frequency of the application in a storage medium with high read-write speed or storing the data with low access frequency in a storage medium with low read-write speed. Exemplarily, data with high access frequency is stored in an SSD with high read-write speed, and data with low access frequency is stored in an HDD with low read-write speed.

[0227] A redundancy policy engine is used to determine the backup method and / or the number of backups for the data to be written by the application to the storage system. Specifically, the redundancy policy engine can combine the information of the application to determine the redundancy policy for the data accessed by the application, including but not limited to one or more of whether to back up, the backup method, or the number of backups, etc. Exemplarily, for one-time data, there is no need to back up; for data that needs to be read subsequently, backup is required. The redundancy policy mainly optimizes the storage method of data from the perspective of the data life cycle.

[0228] It should be noted that the above is only an example of some of the policies determined by the policy determination module 302. In specific implementation, the policy determination module 302 can also be used to determine other storage policies, that is, other policies for optimizing the layout of data in the storage system or improving the data access efficiency of the storage system. Moreover, the policies formulated by the above hierarchical storage policy engine or the policies formulated by the above redundancy policy engine are part of the policies formulated by the above data layout policy engine. In the embodiments of the present application, the hierarchical storage policy engine and the redundancy policy engine are described separately because the proportions of hierarchical storage and redundancy backup in optimizing the storage of data to be accessed by the application and improving the data access efficiency of the application are relatively large. Using independent policy engines to formulate relevant policies can balance and optimize the policy formulation process and improve the efficiency of policy formulation. In specific implementation, the functions completed by the hierarchical storage policy engine and the redundancy policy engine can be implemented through the data layout policy engine, that is, the data layout policy engine, the hierarchical storage policy engine, and the redundancy policy engine can be combined into one; or the corresponding policies can be independently formulated by the hierarchical storage policy engine and the redundancy policy engine respectively, that is, the data layout policy engine, the hierarchical storage policy engine, and the redundancy policy engine can be independently implemented respectively. The embodiments of the present application do not limit the specific implementation method.

[0229] Step 406: The policy execution module 303 obtains the storage policy for executing the application from the policy determination module 302;

[0230] After the policy determination module 302 determines the storage policy of the application, it can send the determined storage policy to the policy execution module 303, or when the policy execution module 303 sends a request to obtain the policy to the policy determination module 302, send the storage policy of the application to the policy execution module 303. The policy execution module 303 can also obtain the storage policy of the application from the policy library according to the information of the application, such as the job identifier, etc., and execute it.

[0231] After the policy execution module 303 obtains the storage policy for the application, in combination with the specific policy of the specific application, it can first execute part of the policy and then execute part of the storage policy after receiving the read / write request of the application. That is, part of the storage policy can be executed first, and step 410 is executed after obtaining the storage policy of the application; the other part of the storage policy can be executed when receiving the access request initiated by the application, that is, step 410 is executed after receiving the access request initiated by the application. For example, for the hierarchical storage policy of the application determined by the policy determination module 302, it can be executed after receiving this policy. In this way, when the relevant read / write requests initiated by the application occur, relevant data can be quickly obtained, and the utilization rate of the hierarchical storage medium can be improved. For the Cache policy determined by the policy determination module 302, when receiving the read data request of the application, based on the specific data corresponding to the read request, the data to be read is cached in the corresponding medium.

[0232] Exemplarily, after the policy execution module 303 receives the hierarchical storage policy of the application determined by the policy determination module 302, before the access of the read / write request initiated by the application, it executes this hierarchical storage policy, which can be implemented in the following two scenarios:

[0233] One scenario is: after the policy determination module 302 formulates the hierarchical storage policy for the application, the policy execution module 303 determines whether the trigger condition of the hierarchical storage policy is triggered. One implementation of the trigger condition is that the scheduler of the application pre-fetches the data to be accessed by the application. When the policy execution module 303 determines that the scheduler of the application is going to fetch the data of the application, it executes this hierarchical storage policy and pre-stores the data that the application needs to read in a preset storage medium. For example, the data that the application needs to read is pre-read from the HDD and stored in the SSD. In this way, when the application initiates a read request, the corresponding data can be quickly obtained.

[0234] Another scenario is that before the application initiates an access request for reading data, it first notifies the storage system through the interface with the storage system that it is about to initiate a read operation and the data to be read. The policy execution module 303 executes the hierarchical storage policy that has been formulated for this application according to the request of the data to be accessed initiated by the application, for example, pre-reads the data from the HDD and stores it in the SSD, so as to quickly obtain relevant data when the application initiates a read request.

[0235] Step 408: The policy execution module 303 receives the access request of the application;

[0236] The policy execution module 303 can call the access request initiated by the application to the storage system through the relevant interface.

[0237] Exemplarily, the policy execution module 303 determines whether the execution of the access request is related to the relevant storage policy. If so, it executes the relevant storage policy; for example, it executes the relevant Cache policy.

[0238] Step 410: The policy execution module 303 executes the obtained storage policy of the application in the storage system.

[0239] The policy execution module 303 executes the relevant policy in the storage system according to the obtained access request of the application and in combination with the obtained storage policy of the application.

[0240] For example, during the I / O execution of the application, the policy execution module 303 combines specific policies to execute specific optimization measures according to the identifier (ID) of the node where the application is located, the job ID or process information of the application, and information such as the specific data object and range to be read and written.

[0241] Figure 4 The method shown is used for the processing of data and read / write requests of the application by obtaining application information and determining specific policies for the application based on the application information. For example, the layout optimization of data in the storage system and the pre-caching of data can effectively optimize the data layout of the storage system and improve the efficiency of data access. Compared with the prior art that configures policies applicable to all applications in a preset manner, the computer device can formulate different policies for different applications respectively, effectively improving the fit between data storage and data read / write for different applications. As described above Figure 2 Taking the HCI scenario shown as an example, if the storage policy of the storage system leads to unreasonable data allocation, it is possible that the application often needs to obtain data from other nodes, increasing the network latency. By using the method provided in the embodiment of the present application, the storage system can sense the application information, make full use of the locality of data access, place the data closer to the calculation, and optimize the data layout of the storage system. At the same time, for each specific application on each HCI node, the data range it accesses is the data in the address space stored on this node, which can reduce the network latency and improve the efficiency of data access.

[0242] Figure 4 The method shown can be implemented independently of the existing storage policy implementation method or in combination with the existing storage policy scheme. When combined with the existing storage policy scheme, it can further optimize the policy determination and execution when the application accesses data on the basis of the existing storage policy scheme; at the same time, for those that have been Figure 4 shown, the method further optimizes the policy determination and execution when the application accesses data; at the same time, for those that have been in accordance with Figure 4When determining the storage policy for related applications using the method shown, it is no longer necessary to follow the existing implementation method of the storage policy. Instead, based on the storage policy determined according to the application information and combined with Figure 4 the method shown, determine the policy for the application when specifically performing I / O operations and execute it. When implemented independently of the existing storage policy, it can be achieved only through Figure 4 the method shown to determine and execute the storage policy when the application accesses data.

[0243] Figure 5 This is a schematic diagram of a specific implementation for optimizing the data management method of the storage system according to the application information provided by the embodiments of this application. As Figure 5 shown, Application 1, Application 2, and Application 3 are different applications, which communicate with the policy determination module and the policy execution module in the storage system through the application awareness module respectively. The application awareness module obtains the information of each application (for example, it can be I / O hints). The policy determination module realizes the data layout and caching of different applications (for example: Application 1, Application 2, and Application 3) in the storage system through data layout policies (such as optimizing storage through Smart Placement), Cache policies (such as pre-Caching), or hierarchical storage policies (such as different levels of storage methods). Figure 5 For the storage medium of the primary storage in, for example, the memory resource pool, it can be a memory Pool, and its data reading and writing speed is faster than the other two levels. For the storage medium of the secondary storage, for example, the Flash resource pool (which can be a Flash Pool), its data reading and writing speed is lower than that of the primary storage but higher than that of the tertiary storage. For the storage medium of the tertiary storage, for example, a hard disk or a Disk, it can be a Disk Pool, and its data reading and writing speed is lower than that of the primary storage and the secondary storage.

[0244] Exemplarily, if the information passed by Application 1 to the application awareness layer is: "IOPattern:SeqRead; IOSize:8KB; FileSet: / my / file / dir / file1, / my / file / dir2 / fileX". Among them, IOPattern:SeqRead shows the way Application 1 reads and writes data, IOSize:8KB shows the size of the data for which Application 1 performs I / O operations, and FileSet: / my / file / dir / file1, / my / file / dir2 / fileX shows the location of the data that Application 1 wants to read and write in the storage system.

[0245] After the policy determination module receives the above information of Application 1 passed by the application awareness module, it calls the engines of each policy (such as the processing functions of each policy) to formulate policies for this application: After receiving the information, the data layout policy engine identifies the IOPattern as sequential read, considers it unnecessary to process, and then ignores the information; after the Cache policy engine receives the information, it identifies the IOPattern as sequential read, the IOSize as 8KB, and the FileSet as / my / file / dir / file1, / my / file / dir2 / fileX. On the one hand, it adds an entry AppA:Read:Prefetch; Seq; 128KB (indicating that in the case of future read operations of the application, sequential prefetch actions should be performed with a granularity of 128KB) to the policy information library. On the other hand, it passes this policy to the policy execution module (ExecutionEngiene), and reads the first 128KB of data of file1 and fileX into the cache (primary storage) in advance, that is, the policy execution module calls the prefetch instruction of the Cache module to load the file data into the Cache in advance. It should be noted that 128KB in the aforementioned formulated policy is determined according to the block size of the underlying storage system. Since the underlying block sizes of different storage systems are different, 128KB is only an example. In specific implementation, it can be flexibly adjusted according to the specific situation of the storage system (different underlying block sizes).

[0246] Figure 5 The application awareness module, policy determination module, and policy implementation module shown in can be implemented by referring to the above implementation methods of the application awareness module 301, policy determination module 302, and policy execution module 303 respectively, and will not be elaborated here.

[0247] Figure 6 It is a structural schematic diagram of a specific application scenario for implementing the data management method of the storage system provided by an embodiment of the present application. As Figure 6 shown, different applications are running on three nodes (such as computing devices), and these applications include Application A, Application B, and Application C. Among them, Application A is a read-write hybrid application (such as a database application) and runs on Node 1, Node 2, and Node 3. Application B is a read-only application (such as a video-on-demand system) and runs on Node 1 and Node 3. Application C is a write-only application (such as a logging system) and runs on Node 2.

[0248] Distributed private clients (DPC) are deployed on all three nodes. The DPC can sense information such as the type of application, I / O mode, node number, or thread identifier, and pass this information to the storage server along with the I / O request. Figure 6The DPC in the example is a specific implementation of the application perception module in the embodiment of the present application.

[0249] Figure 7A , Figure 7B and Figure 7C By way of example, the process of determining and executing policies for application A, application B, and application C is shown respectively. It can be understood that Figure 7A , Figure 7B and Figure 7C This is just an example of different applications. In actual implementation, the implementation of the application-aware interface, the determination of relevant strategies and the implementation of the strategies will have different implementations based on specific scenarios. Figure 7A , Figure 7B and Figure 7C The application awareness interface may be a specific implementation of the application awareness module, the data layout engine, the cache policy engine and the hierarchical storage engine may be a specific implementation of the policy determination module, and the policy execution engine may be a specific implementation of the policy execution module.

[0250] like Figure 7A As shown, the information of application A obtained by the application-aware interface is as follows:

[0251] Job ID: 12345;

[0252] Job characteristics: Frequent operation, frequent data updates, and frequent use of old data;

[0253] Running nodes: N1, N2, N3;

[0254] Process ID: 24,1137,6578;

[0255] Job I / O type: random read / write, IOPS type, frequent modification;

[0256] I / O block size: 512B-4KB;

[0257] Access datasets: / path / to / appA / file1, / path / to / appA / file2,…,

[0258] / path / to / appA / file10000;

[0259] I / O file size: single file 64K-2M, total data set size 2G, total I / O volume 30G.

[0260] The characteristics of the read-write hybrid application A are as follows: it runs frequently, the data is updated frequently, the old data is also frequently used, and the read-write operations are for small files (e.g., 512B - 4KB); the job type is random read-write, IOPS type, and the modification is frequent. After the application awareness module obtains this information through the application awareness interface, it passes this information to the policy determination module. The policy determination module formulates policies for application A through the data layout policy engine, the Cache policy engine, and the hierarchical storage policy engine respectively. For example, the data layout policy engine determines that the data structure type to be stored by application A is a log data structure to reduce possible read and write amplifications; determines that the storage method of the data to be stored by application A is to store the data in a packed manner to improve the utilization rate of the storage space; determines that the verification method of the data to be stored by application A is to increase the proportion of verification data to enhance the reliability of the data. Exemplarily, based on the characteristics that application A has multi-node I / O and all files are small files, the data layout policy engine adopts a data layout policy of a log data structure to reduce the read amplification and write amplification caused by random writes; and, since the information of application A belongs to small I / O (block size 4KB) and the data is frequently modified, the Container mechanism can be adopted for the data of this application, packed into an object with 6 + 2EC and a block size of 4KB. Then, during the I / O process, the Cache system will be prompted not to flush the disk frequently. In this way, the data accessed by the same application can be placed together as much as possible according to the information of application A, making full use of spatial and temporal locality. Among them, 6 in 6 + 2EC represents the number of data blocks in EC, and 2 represents the number of parity blocks in EC; in this way, the verification data is relatively large, which can enhance the data reliability.

[0261] For application A, which belongs to frequent random small I / O, frequent modification, small dataset, and relatively large I / O volume compared to the dataset, the Cache policy engine formulates a Cache policy to improve the hit rate of reading data by application A in a full-cache manner, batch processes write operations to increase the merging opportunity, and reduce the number of I / Os sent to the backend. Optionally, when the read operation of application A is a random read, the Cache policy engine can determine to turn off the prefetch policy for application A and clean the Cache related to application A to avoid Cache pollution and space waste.

[0262] For application A, whose dataset is frequently used, does not age, and has a small volume, the hierarchical storage policy engine formulates a policy of adopting Cache and SSD to improve the read and write efficiency of the data.

[0263] The policy execution engine, that is, the policy execution module, can query the policy of application A from the application policy library according to at least one of the node ID (NodeID), job ID (JobID), or process ID (Pid), and execute the obtained policy.

[0264] As shown Figure 7B below, the information of Application B obtained through the application awareness interface is as follows:

[0265] Job ID: 65432;

[0266] Job characteristics: No data update, running continuously;

[0267] Running nodes: N1, N2;

[0268] Process IDs: 654, 3297;

[0269] Job I / O types: Mostly sequential continuous read, sequential skip read, and read-only after writing;

[0270] Rarely write, and will not read immediately after writing;

[0271] I / O block size: 1MB - 4MB;

[0272] Accessed data sets: / path / to / appB / vid100000.avi, … / path / to / appB / vid99999.avi;

[0273] I / O file size: 500MB - 5 gigabytes (GB) per single file, and 4 terabytes (TB) for the total data set.

[0274] The characteristics of the read-only type Application B are: Most are sequential continuous read and sequential skip read, and Application B rarely modifies after writing data, will not read data immediately, and the data to be read and written is large. After obtaining this information through the application awareness interface, the application awareness module passes this information to the policy determination module. The policy determination module formulates policies for Application B through the data layout policy engine, Cache policy engine, and hierarchical storage policy engine.

[0275] Among them, since Application B involves multi-node I / O and only reads data after writing data, the policy formulated by the data layout policy engine is: store it in the files in the access directory of Application B and adopt a multi-copy mechanism, which is conducive to improving the data reading efficiency of Application B and ensuring data reliability. Since Application B mostly performs sequential reads and the written data is not read immediately, the policy determined by the Cache policy engine is: trigger sequential progressive prefetch during read operations, with the prefetch increment granularity of 5MB, to improve the data reading efficiency of Application B; adopt the least recently used (LRU) policy for Cache eviction to reduce the occupancy of the cache, and bypass the cache for write operation data and directly write it to the HDD resource pool to avoid occupying other storage resources and improve the write data efficiency. Because Application B has a large amount of data and performs sequential large-block reads, the policy determined by the hierarchical storage policy engine is: aggressive Cache prefetch and eviction, and persist the data to the HDD storage pool.

[0276] The policy execution engine, that is, the policy execution module, can query the storage policy of Application B from the application policy library according to at least one of the Node ID (NodeID), Job ID (JobID), or Process ID (Pid), and execute the obtained storage policy.

[0277] It should be noted that Figure 7B The dotted line in shows the way of directly writing the data that Application B needs to write to the storage pool to the HDD, which is described for the case where Application B only writes once and then reads data subsequently. For the case where Application B only reads data without writing data, it is also possible not to write data to the storage pool, that is, there is no write operation for Application B to write data to the HDD storage pool.

[0278] As Figure 7C shown, the information of Application C obtained by the application awareness interface is as follows:

[0279] Job ID: 2467;

[0280] Job characteristics: Runs for a long time on only one node;

[0281] Running node: N1;

[0282] Process ID: 3257;

[0283] Job I / O type: Write-only type, and the written data is compressed text;

[0284] I / O block size: 64KB - 1MB;

[0285] Access to data sets: / path / to / appC / log123, …, / path / to / appC / longXXX;

[0286] I / O file size: 64KB - 1GB per file, a large number of files, the written data is rarely read but stored long-term.

[0287] The characteristics of the write-only application C are: only write data into the storage pool, and the written data is compressed text. After the application awareness module obtains this information through the application awareness interface, it passes this information to the policy determination module. The policy determination module formulates policies for application C through the data layout policy engine, Cache policy engine, and hierarchical storage policy engine. For example, because the amount of data that application C needs to write into the storage pool is large, it is rarely read after writing, and it is required to be stored long-term, so the data layout policy engine can determine to store the data that application C needs to store in the 16 + 2 EC method, where 16 represents the number of data blocks in EC, and 2 represents the number of parity blocks in EC; in this way, the space occupied by the parity data is small, and storage space is saved as much as possible while ensuring reliability. The Cache policy engine determines the policy that there is no need to cache the data written by application C according to the characteristics of the write-only type of application C, so as to avoid possible occupation of the cache space. The hierarchical storage policy engine determines to directly write the data that application C needs to write into the HDD storage pool according to the characteristics of the write-only application C and the fact that the data is rarely read after being written into the storage pool. In this way, it will not occupy the cache and can improve the efficiency of writing data into the storage pool.

[0288] The policy execution engine, that is, the policy execution module, can query the storage policy of application C from the application policy library according to at least one of the node ID (NodeID), job ID (JobID), or process ID (Pid), and execute the obtained storage policy.

[0289] The above is an explanation of the implementation method of the application data management method provided by the embodiments of the present application by taking the specific situations of three different applications as examples. Next, taking an application running on multiple nodes as an example, a further explanation of an implementation method of the application data management method provided by the embodiments of the present application is made.

[0290] Figure 8 It is a logical schematic diagram of a data layout decision-making process provided by the embodiments of the present application. As Figure 8 shown, after the application awareness module obtains the information of the application, when the policy determination module determines the policy of the application, it first judges whether the application is running on one node or multiple nodes.

[0291] When the application runs on a single node, determine whether the application has reliability requirements for data access. If there are no reliability requirements, it can be determined that the data layout strategy of the application is a single-copy strategy to reduce storage space occupancy. If the application has reliability requirements for data, then determine the size of the data to be stored by the application. When the data is less than 1 GB, determine a two-copy strategy to improve the degree of data reliability without occupying too much storage space. If the data is greater than 1 GB, then determine an EC layout strategy to avoid excessive occupation of storage space. It can be understood that the aforementioned 1 GB and two-copy are merely examples and do not constitute a limitation on the embodiments of the present application.

[0292] When the application runs on multiple nodes, the policy determination module first determines whether the application is an application in a read-write mixed mode. If the application is a write-only application, then determine that the number of copies of the data to be written is the number of nodes on which the application runs to improve data reliability. If the application is an application in a read-write mixed mode, then determine whether there are conflicts between its reads and writes. When there are conflicts, determine an EC data layout strategy. When there are no conflicts, then adopt a copy data layout strategy.

[0293] In a possible implementation manner, Figure 8 The scenario shown may further include a scheduler for scheduling jobs of the application. The scheduler can, according to the I / O situations of each job in the cluster, schedule applications with more I / O to more nodes for execution and avoid I / O conflicts between different jobs.

[0294] It can be understood that Figure 8 This is only an example of the data layout decision engine. In specific implementation, specific policies can also be determined according to the characteristics of the storage system.

[0295] In the solution provided by the embodiments of the present application, the application awareness module receives information from the application through a series of interfaces. The policy determination module formulates appropriate storage policies according to the application information, and takes targeted measures for application requests through the policy execution module, optimizing data storage, improving the efficiency of data access, and enhancing the data access performance of the storage system.

[0296] In a general storage system, some storage policies are usually set by default. For example, the default caching policy of the Cache system involves water levels, disk flushing times, cache amounts, prefetching, etc. The storage backend will usually default to setting some strip sizes, EC ratios, etc. Generally speaking, these default settings are only suitable for individual scenarios. In a production environment, it is necessary to conduct manual tests, continuously try various parameters, and configure the system according to empirical values to achieve sub-optimal performance. Moreover, once set, the parameters are difficult to change. When the business model changes, the performance will "degrade".

[0297] In the above implementation provided by the embodiments of the present application, an application-aware module is added to the storage system, which can receive information from the application in real time, detect changes in the application's requirements and access patterns, and adjust the behavior of the storage system according to this information, optimize the data access path and storage method, so that the storage system can flexibly adapt to various scenarios. For example, for scenarios where it is not suitable to use Cache, such as large-block (1MB+) write-only applications, the cache layer is directly bypassed, and the DirectIO technology is used to write data directly to the storage backend. This not only saves a large amount of Cache space, but also shortens the I / O path, improves Cache utilization, and reduces the overall latency. For data access of the type of input / output operations per second (IOPS), appropriate caching policies, such as full caching + LRU eviction, and appropriate data layouts (such as small file packaging) can be adopted to improve the performance of data access. For read-only data, an effective prefetching strategy can be adopted to preload data into the cache in advance and shorten the response time to improve IOPS.

[0298] Figure 9 It is a schematic flow chart of a data management method for an application in an embodiment of the present application. The data of the application is stored in a storage system, and the storage system includes at least one storage node. As Figure 9 shown, the method includes:

[0299] Step 900: Obtain information of the application, where the information of the application includes at least one of the following information: the I / O operation information of the application, the running information of the application, and the information of the process of the application;

[0300] Step 902: Develop a storage policy for the application according to the information of the application,;

[0301] Step 904: Execute the storage policy.

[0302] The above method obtains information about an application and formulates a storage policy for the application based on the information about the application, for the storage and read / write processing of data read and written by the application, to optimize the storage of data read and written by the application in a storage system and / or improve the efficiency of reading and writing data by the application. Compared with the prior art that configures a policy applicable to all applications based on a preset method, the method can formulate different storage policies for different applications respectively, can improve the efficiency of application access, optimize the data storage method, and improve the resource utilization rate of storage devices, effectively improving the compatibility of data storage and data read / write with different applications. For example, for scenarios where it is not suitable to use Cache, such as large data block write-only applications, directly bypass the cache layer and use the DirectIO technology to write data directly to the storage backend, etc. This can not only save a large amount of Cache space, but also shorten the I / O path, improve the Cache utilization rate, and reduce the overall latency.

[0303] Figure 9 The specific implementation manner of the method shown can be implemented by referring to the above implementation manner of the embodiments of the present application. For example, it can be implemented by referring to the Figure 4 implementation method shown above, or it can be implemented by referring to the Figure 5 、 Figure 6 、 Figures 7A - 7B and Figure 8 specific implementation manner shown above, which will not be elaborated here.

[0304] Figure 10 FIG. is a schematic structural diagram of a data management system 1000 provided by an embodiment of the present application. As Figure 10 shown, the data management system 1000 includes an application awareness module 1001, a policy determination module 1002, and a policy execution module 1003, where:

[0305] The application awareness module 1001 is configured to obtain information about an application, and the information about the application includes at least one of the following information: the I / O operation information of the application, the running information of the application, and the information of the process of the application; wherein, the data of the application is stored in a storage system, and the storage system includes at least one storage node;

[0306] The policy determination module 1002 is configured to formulate a storage policy for the application according to the information about the application;

[0307] The policy execution module 1003 is configured to execute the storage policy.

[0308] The above data management system 1000 can optimize the storage of data read and written by the application and / or improve the access efficiency of the application in the storage system by obtaining information about the application and formulating a storage policy for the application based on the information about the application. Compared with the prior art that configures a policy applicable to all applications in a preset manner, the data management system 1000 can formulate different storage policies for different applications respectively, which can improve the efficiency of reading and writing application data, optimize the data storage method, and improve the resource utilization rate of the storage device, effectively improving the compatibility of data storage and data reading and writing with different applications.

[0309] Figure 10 For the specific implementation manner of the data management system 1000 shown, reference can be made to the implementation manner described above in the embodiments of the present application. For example, reference can be made to the above Figures 3A - 3F or the implementation method shown above Figure 4 to implement, or reference can also be made to the above Figure 5 、 Figure 6 、 Figures 7A - 7B and Figure 8 to implement, which will not be elaborated here.

[0310] Figure 11 FIG. shows a schematic structural diagram of a computer device 2000 provided in an embodiment of the present application. As Figure 11 shown, it includes a processor 2002 and a memory 2001, and the processor 2002 and the memory 2001 are connected through a bus 2003. The memory 2001 is used to store a computer-readable executable program, and the processor 2002 is used to execute the computer-readable executable program in the memory 2001 to implement the method described above Figure 4 or Figure 9 . For example, the processor 2002 is used to read the computer-readable executable program in the memory 2001 to execute the following steps:

[0311] Obtain information about the application, where the information about the application includes at least one of the following information: the input / output I / O operation information of the application, the running information of the application, and the information of the process of the application; wherein, the data of the application is stored in a storage system, and the storage system includes at least one storage node;

[0312] Formulate a storage policy for the application according to the information about the application;

[0313] Execute the storage policy.

[0314] It can be understood that Figure 11The illustrated processor 2002 may be a processor located in different devices. For example, the processor 2002 may be a processor located in a client of a distributed storage system or a processor in a storage node of a distributed storage system. Similarly, Figure 11 The illustrated memory 2001 may be a memory located in different devices. For example, the memory 2001 may be a memory located in a client of a distributed storage system or a memory in a storage node of a distributed storage system.

[0315] The above computer device 2000 can formulate a storage policy for the application according to the information of the application, optimize the storage of the data read and written by the application in the storage system and / or improve the efficiency of the data read and written by the application. Compared with the prior art that configures a policy applicable to all applications based on a preset method, the computer device can formulate different storage policies for different applications respectively, effectively improving the compatibility of data storage and data reading and writing with different applications.

[0316] Figure 11 For the specific implementation manner of the illustrated data management system 2000, reference may be made to the above implementation manners in the embodiments of the present application. For example, reference may be made to the above Figures 3A - 3F , or the above Figure 4 illustrated implementation method to implement, or reference may also be made to the above Figure 5 , Figure 6 , Figures 7A - 7B and Figure 8 illustrated specific implementation manners to implement, which will not be elaborated herein.

[0317] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described according to their functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0318] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments, which will not be elaborated herein.

[0319] In several embodiments provided by this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed couplings or direct couplings or communication connections to each other can be indirect couplings or communication connections through some interfaces, devices, or units, and can also be electrical, mechanical, or other forms of connection.

[0320] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiments of the present invention.

[0321] In addition, each functional unit in various embodiments of the present invention can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0322] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0323] The above is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of various equivalent modifications or substitutions, and these modifications or substitutions should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A data management method for an application, characterized in that, the data of the application is stored in a storage system, the storage system includes at least one storage node, and the method includes: obtaining information of the application, the information of the application including input / output (I / O) operation information of the application, node information associated with the application, and information of the processes of the application, the information of the processes of the application including information of jobs executing the application or information of processes related to jobs executing the application; formulating a storage policy for the application according to the information of the application, the storage policy being used to indicate that if the application runs on multiple nodes, the application is a read-write hybrid application and there are conflicts in the read and write of the application, then store the data to be stored by the application in the form of an erasure code (EC) data layout policy; executing the storage policy.

2. The method according to claim 1, characterized in that, the method further includes: formulating the storage policy for the application before the application runs or when a data processing request of the application is received.

3. The method according to claim 1, characterized in that, the method further includes: obtaining characteristic information of the storage system; formulating the storage policy for the application according to the information of the application and the characteristic information of the storage system.

4. The method according to claim 3, characterized in that, the characteristic information of the storage system includes at least one of the following information: the hardware structure of the storage system, the hardware composition of the storage system, the storage capacity of the storage system, or the way of data layout of the storage system.

5. The method according to claim 1, characterized in that, the data layout policy is used to determine the storage method and / or storage location of the data written by the application to the storage system.

6. The method according to claim 5, characterized in that, the storage method of the application writing to the storage system includes at least one of the following information: the structure type of the data that the application needs to store, the number of copies of the data that the application needs to store, or the verification method of the data that the application needs to store.

7. The method according to any one of claims 1-6, characterized in that, the storage policy of the application includes a caching policy, the caching policy being used to determine the data read by the application from the storage system and the caching method of the read data, the caching method of the data including at least one of the following: full caching, partial caching, prefetching, write-through caching when writing data, or cache cleaning mechanism.

8. The method according to any one of claims 1-6, characterized in that, the storage policy of the application further includes a hierarchical storage policy, the hierarchical storage policy being used to store data accessed by the application at different frequencies in storage media with different read and write speeds.

9. The method according to any one of claims 1-6, characterized in that, the storage policy of the application further includes a redundancy policy, the redundancy policy being used to determine the backup method and / or backup quantity of the data to be written by the application to the storage system.

10. The method according to any one of claims 1 to 6, It is characterized in that Pre-save reference strategies; The method further comprises: The storage policy is formulated for the application with reference to the reference policy.

11. The method according to any one of claims 1 to 6, It is characterized in that The method further includes storing a storage policy of the application.

12. The method according to any one of claims 1 to 6, It is characterized in that The application information is information that has been normalized according to a preset format and / or content.

13. The method according to any one of claims 1 to 6, It is characterized in that The executing the storage policy includes: executing part of the storage policy before the application initiates a read or write request.

14. The method according to claim 13, It is characterized in that The storage strategy executed before the application initiates a read / write request includes a hierarchical storage strategy, and the hierarchical storage strategy is used to store the data to be read by the application in a storage medium with a high read / write speed in advance.

15. The method according to any one of claims 1 to 6, It is characterized in that The method further includes: obtaining information of the application from the application through an extended interface; or, The formatted information of the application is obtained from a computing device on which the application is running.

16. The method according to claim 15, It is characterized in that The obtaining the information of the application from the application through the extended interface comprises one of the following methods: obtaining the information of the application through an extended interface library; The information of the application is obtained through a dedicated file, and the application or a scheduler corresponding to the application writes the information of the application into the dedicated file; Acquiring information of the application through a remote procedure call; or, The information of the application is obtained through RESTful representation state transfer.

17. The method according to any one of claims 1 to 6, It is characterized in that The node information associated with the application includes at least one of the following information: the node identifier of the node running the application, the name of the node running the application, the IP address of the node running the application, the node name where the data on which the application performs I / O operations is located, or the node identifier where the data on which the application performs I / O operations is located.

18. The method according to any one of claims 1 to 6, It is characterized in that The information of executing the job of the application includes at least one of the following information: the identification of executing the job of the application, the characteristics of executing the job of the application; the information of the process related to executing the job of the application includes the process identification related to executing the job of the application.

19. A data management system, It is characterized in that The data management system includes an application perception module, a policy determination module and a policy execution module, wherein: The application awareness module is used to obtain information of an application, where the information of the application includes input / output (I / O) operation information of the application, node information associated with the application, and information of the process of the application; wherein, data of the application is stored in a storage system, and the storage system includes at least one storage node; the information of the process of the application includes information of a job for executing the application or information of a process related to the job for executing the application; The policy determination module is used to formulate a storage policy for the application according to the information, and the storage policy is used to indicate that if the application runs on multiple nodes, the application is a read-write hybrid application and there is a conflict between reading and writing of the application, then store the data to be stored by the application in the form of an erasure code (EC) data layout policy; The policy execution module is used to execute the policy.

20. The data management system according to claim 19, wherein, Before the application runs or when a data processing request of the application is received, the policy determination module formulates the storage policy for the application.

21. The data management system according to claim 19, wherein, The policy determination module is further used to obtain characteristic information of the storage system, and formulate the storage policy for the application according to the information of the application and the characteristic information of the storage system.

22. The data management system according to claim 21, wherein, The characteristic information of the storage system includes at least one of the following information: the hardware structure of the storage system, the hardware composition of the storage system, the storage capacity of the storage system, or the way of data layout of the storage system.

23. The data management system according to claim 19, wherein, The data layout policy is used to determine the storage method and / or storage location of the data written by the application to the storage system.

24. The data management system according to claim 23, wherein, The storage method of the application writing to the storage system includes at least one of the following information: the structure type of the data that the application needs to store, the number of copies of the data that the application needs to store, or the verification method of the data that the application needs to store.

25. The data management system according to any one of claims 19-24, wherein, The storage policy of the application includes a caching policy, and the caching policy is used to determine the data read by the application from the storage system and the caching method of the read data, and the caching method of the data includes at least one of the following: full caching, partial caching, prefetching method, pass-through caching when writing data, or cache cleaning mechanism.

26. The data management system according to any one of claims 19-24, wherein, The storage policy of the application further includes a hierarchical storage policy, and the hierarchical storage policy is used to store data accessed by the application at different frequencies in storage media with different read-write speeds.

27. The data management system according to any one of claims 19-24, wherein, The storage strategy of the application also includes a redundancy strategy, and the redundancy strategy is used to determine the backup method and / or backup quantity of the data to be written into the storage system by the application.

28. The data management system according to any one of claims 19 to 24, It is characterized in that The policy determination module is also used to refer to a pre-saved reference policy to formulate the storage policy for the application.

29. The data management system according to any one of claims 19 to 24, It is characterized in that The application information is information that has been normalized according to a preset format and / or content.

30. The data management system according to any one of claims 19 to 24, It is characterized in that The executing the storage policy includes: executing part of the storage policy before the application initiates a read or write request.

31. A data management system according to any one of claims 19 to 24, It is characterized in that The strategy determination module is also used for: Obtaining information of the application from the application through an extended interface; or, The formatted information of the application is obtained from a computing device on which the application is running.

32. The data management system according to claim 31, It is characterized in that The policy determination module obtains the information of the application from the application through the extended interface in one of the following ways: Obtain application information through the extended interface library; The information of the application is obtained through a dedicated file, and the application or a scheduler corresponding to the application writes the information of the application into the dedicated file; Acquiring information of the application through a remote procedure call; or, The information of the application is obtained through RESTful representation state transfer.

33. A data management system according to any one of claims 19 to 24, It is characterized in that The node information associated with the application includes at least one of the following information: the node identifier of the node running the application, the name of the node running the application, the IP address of the node running the application, the node name where the data on which the application performs I / O operations is located, or the node identifier where the data on which the application performs I / O operations is located.

34. A data management system according to any one of claims 19 to 24, It is characterized in that The information of executing the job of the application includes at least one of the following information: the identification of executing the job of the application, the characteristics of executing the job of the application; the information of the process related to executing the job of the application includes the process identification related to executing the job of the application.

35. A computer device, It is characterized in that The method comprises a processor and a memory, wherein the memory stores a computer-readable executable program, and the processor is used to read the computer-readable executable program to execute the steps of the method according to any one of claims 1 to 18.

36. A computer readable storage medium, It is characterized in that The computer-readable storage medium stores instructions, and when the instructions are executed on a computer device, the computer device executes the method according to any one of claims 1 to 18.

37. A computer program product comprising instructions, wherein, when the instructions are run on a computer device, the computer device is caused to execute the method according to any one of claims 1-18.

Images