Data management method and system and computer equipment

Abstract

The invention provides a data management method and system and computer equipment, and aims to solve the problem that targeted data management cannot be realized for specific applications in the prior art. According to the method provided by the invention, the information of the application is obtained, the information comprises but is not limited to one or more of I / O operation information of the application, running information of the application and process information of the application, and a specific storage strategy is formulated and executed according to the specific information of the application. As the formulated storage strategy is formulated based on the information of the application, the execution of the storage strategy can optimize the storage of the data read and written by the application in the storage system and / or improve the efficiency of the data read and written by the application. Compared with the prior art that strategies suitable for all applications are configured based on a preset mode, different strategies can be formulated for different applications, and the integrating degree of data storage and data reading and writing on different applications is effectively improved.

Description

[0001] This application is a divisional application. The application number of the original application is 202011052507.3, and the original application date is September 29, 2020. The entire contents of the original application are incorporated into this application by reference. Technical Field

[0002] The present application relates to the field of information technology, and in particular to a data management method, system and computer equipment. Background Art

[0003] In current distributed storage systems, the data storage method is basically pre-configured, or the user specifies the data storage strategy when creating a directory. For example, in a file system, the default number of file stripes is 1, and the default stripe size is 1 megabyte (MB). After creating a directory, the user can use special commands to specify the number and / or stripe size of the files created in the directory.

[0004] This data storage method may have better performance for specific input / output (I / O) modes or specific applications. However, when the I / O mode of the application itself is complex, or when multiple different applications use the same distributed file system at the same time, the fixed data layout method is difficult to meet the needs of different applications. Summary of the invention

[0005] The embodiments of the present application provide a data management method, system and computer device to solve the problem in the prior art that targeted data management cannot be implemented for specific applications.

[0006] In a first aspect, an embodiment of the present application provides a data management method, wherein the data of the application is stored in a storage system, the storage system includes at least one storage node, and the method includes:

[0007] Acquire application information, the application information including at least one of the following information: I / O operation information of the application, running information of the application, and process information of the application;

[0008] formulating a storage strategy for the application according to the information of the application;

[0009] The storage policy is executed.

[0010] The above method can formulate and execute a storage strategy for the application based on the information of the application, and can optimize the storage of the data read and written by the application in the storage system and / or improve the efficiency of the access. Compared with the prior art that configures strategies applicable to all applications based on a preset manner, the computer device can formulate different strategies for different applications, effectively improving the compatibility of data storage and data reading and writing for different applications.

[0011] Optionally, the storage node may be a server or computer device including storage resources, such as memory, solid-state drive (SSD), hard disk drive (HDD), or magnetic tape.

[0012] Optionally, the I / O operation information of the application includes, but is not limited to, at least one of the following information: the size of data on which the application performs the I / O operation, the location of the data on which the application performs the I / O operation in the storage system, the manner in which the application performs the I / O operation, an identifier of the I / O operation performed by the application, the type of I / O operation performed by the application, the I / O mode in which each process in the application accesses data (sequential, stride, random, read-only, write-only, mixed read and write, etc.), the type of data on which the application performs the I / O operation, the duration of the application performing the I / O operation, the life cycle of the data on which the application performs the I / O operation, the redundancy strategy of the data on which the application performs the I / O operation, or sharing information of the data on which the application performs the I / O operation.

[0013] Optionally, the running information of the application includes, but is not limited to, at least one of the following information: the node identifier of the node running the application, the name of the node running the application, the IP address of the node running the application, the name of the node where the data on which the application performs I / O operations is located, or the node identifier where the data on which the application performs I / O operations is located.

[0014] Optionally, the information of the process executing the application includes, but is not limited to, at least one of the following information: an identifier of the job executing the application, characteristics of the job executing the application, an identifier of the process executing the application, or information on which processes of the application on which nodes process which data, etc. The job executing the application may include one or more processes.

[0015] In some possible implementations, the method further includes:

[0016] The storage policy is formulated for the application before the application is run or when a data processing request of the application is received.

[0017] In some possible implementations, the method further includes: acquiring characteristic information of the storage system;

[0018] The storage policy is formulated for the application according to the information of the application and the characteristic information of the storage system.

[0019] In some possible implementations, the characteristic information of the storage system includes at least one of the following information: the hardware structure of the storage system, the hardware composition of the storage system, the storage capacity of the storage system, or the data layout method of the storage system.

[0020] The hardware structure of the storage system refers to the positional relationship or connection relationship between different components in the storage system, and the hardware composition of the storage system refers to the physical form of different components in the storage system. For example, the hardware composition of the storage system may include an SSD and an HDD; the hardware structure of the storage system indicates that the SSD is a secondary storage resource, the HDD is a tertiary storage resource, and there is a connection relationship between the SSD and the HDD.

[0021] Combining the information of the application and the characteristic information of the storage system to formulate a storage strategy for the application can better match the characteristics of the storage system, further optimize the storage of data of the application I / O operation in the storage system and improve the efficiency of the application in performing I / O operations.

[0022] In some possible implementations, the storage policy of the application includes a data layout policy, and the data layout policy is used to determine the storage method and / or storage location of the data written by the application to the storage system.

[0023] In some possible implementations, the storage method in which the application is written into the storage system includes at least one of the following information:

[0024] The structure type of the data that the application needs to store, the number of copies of the data that the application needs to store, or the verification method of the data that the application needs to store.

[0025] For example, the data layout strategy may include allocating continuous storage space for data to be written to a storage device by a certain application to improve the efficiency of data writing; it may also be determining whether the data to be written is stored in a single copy or multiple copies, or it may be determining the error correction code (EC) or redundant array of independent disks (RAID) of the data to be written.

[0026] In some possible implementations, the storage strategy of the application includes a cache strategy, which is used to determine the data read by the application from the storage system and the cache method of the read data, and the cache method of the data includes at least one of the following: full cache, partial cache, pre-reading method, direct cache when writing data, or cache cleaning mechanism. For example, the cache strategy may include pre-writing specific data to be read by an application into a first-level cache with the fastest read and write speed to improve the efficiency of the application reading data; or for a request initiated by an application to only write data to a storage device, the data is directly written to the storage space of the storage device by direct (bypass) or transparent transmission without passing through the cache, thereby reducing the space occupied by the cache and improving the utilization of the cache.

[0027] In some possible implementations, the storage strategy of the application further includes a hierarchical storage strategy, and the hierarchical storage strategy is used to store data of different frequencies accessed by the application in storage media with different read and write speeds. For example, the hierarchical storage strategy may include storing frequently accessed data, i.e., data with high access frequency, in a storage medium with high read and write speeds, such as storage class memory (SCM) or SSD, so as to improve the efficiency of accessing the data and avoid the occupation and waste of resources caused by copying the data in different storage media.

[0028] In some possible implementations, the storage strategy of the application also includes a redundancy strategy, which is used to determine the backup method and / or backup quantity of the data to be written by the application to the storage system. For example, the redundancy strategy may include: for one-time data, no backup is required; for data that needs to be read later, backup is required.

[0029] In some possible implementations, the reference strategy is saved in advance; the method further includes:

[0030] The storage policy is formulated for the application with reference to the reference policy.

[0031] Optionally, the reference policy is a policy stored in a policy information base. The policy information base may be stored in the storage system or may be an independent database, such as an independent database device.

[0032] Optionally, the method may further include: first obtaining a policy of the application that has been stored in a policy information library, and determining a storage policy for the application in combination with the information of the application obtained this time.

[0033] If the storage policy of the application is not stored in the policy information base, the storage policy of an application similar to the application in the policy information base may be referenced. For example, according to the type or operation parameters of the application, the storage policy of the application recorded in the policy information base with the same type or operation parameters as the application may be referenced, or the storage policy of an application similar to the type or operation parameters of the application may be referenced to determine the storage policy of the application.

[0034] Usually, the read and write mode of an application is relatively fixed. Therefore, referring to the storage policy in the policy information library to determine the storage policy of the application does not require determining repeated policies for the same content each time. This can improve the efficiency of determining policies and reduce the occupancy of resources such as the central processing unit (CPU) caused by determining the corresponding policies.

[0035] In some possible implementations, the method further includes: storing a storage policy of the application.

[0036] Optionally, the storage policy of the application may be stored in the policy information library.

[0037] Optionally, after determining the application policy, the storage policy may be stored in the policy information base. When storing the storage policy in the policy information base, the storage policy may be stored in an updated manner, that is, information of the storage policy that has been changed is stored in the policy information base.

[0038] In some possible implementations, the information of the application is normalized according to a preset format and / or content. Optionally, the information of the application is normalized according to a preset format, including but not limited to processing the information of the application according to a unified expression method or arrangement order. The information of the application is normalized according to preset content, including but not limited to adding necessary information. For example, adding context information, etc. The context information includes but is not limited to: the identifier of the node where the application is located, the name of the node where the application is located, the Internet Protocol (IP) address of the node where the application is located, the name of the application, application parameters or job description and other context information.

[0039] By normalizing the information of the application, the unified storage of information of different applications can be simplified, the method of storing application information can be simplified, and the efficiency of information storage and transmission of the application can be improved; at the same time, when determining the strategy based on the information of the application, the information of the application can also be quickly identified and relevant strategy decisions can be made, thereby improving the efficiency of determining specific strategies.

[0040] In some possible implementations, the execution of the storage strategy includes: executing part of the storage strategy before the application initiates a read or write request. For example, when a hierarchical storage strategy is determined for the application, the strategy can be executed upon receipt, so that when the application initiates a related read or write request, the related data can be quickly obtained, and the utilization rate of the hierarchical storage medium can be improved. After the hierarchical storage strategy is determined for the application, when a read data request from the application is received, the data to be read is cached in the corresponding medium based on the specific data corresponding to the read request, so that the hit rate of the cached data can be improved and the read and write efficiency of the data can be improved.

[0041] In some possible implementations, the storage strategy executed before the application initiates a read / write request includes a hierarchical storage strategy, and the hierarchical storage strategy is used to store the data to be read by the application in a storage medium with a high read / write speed in advance.

[0042] In some possible implementations, the method further includes: obtaining information of the application from the application through an extended interface; or,

[0043] The formatted information of the application is obtained from a computing device on which the application is running.

[0044] In some possible implementations, obtaining the information of the application from the application through the extended interface includes one of the following methods:

[0045] Obtain application information through the extended interface library;

[0046] The information of the application is obtained through a dedicated file, and the application or a scheduler corresponding to the application writes the information of the application into the dedicated file;

[0047] Acquire the application information through a remote procedure call; or,

[0048] The information of the application is obtained through RESTful representation state transfer.

[0049] In a second aspect, an embodiment of the present application provides a computer device, including a processor and a memory, wherein the memory stores a computer-readable executable program, and the processor is configured to read the computer-readable executable program to perform the following steps:

[0050] Acquire application information, the application information including at least one of the following information: I / O operation information of the application, running information of the application, and process information of the application; wherein data of the application is stored in a storage system, and the storage system includes at least one storage node;

[0051] formulating a storage strategy for the application according to the information of the application;

[0052] The storage policy is executed.

[0053] The computer device can formulate and execute storage strategies for the application based on the application information, and can optimize the storage of the data read and written by the application in the storage system and / or improve the efficiency of the access. Compared with the prior art that configures strategies applicable to all applications based on a preset manner, the computer device can formulate different strategies for different applications, effectively improving the compatibility of data storage and data reading and writing for different applications.

[0054] Optionally, the storage node may be a server or computer device including storage resources. The storage resources may include storage resources such as memory, SSD, HDD or magnetic tape.

[0055] Optionally, the I / O operation information of the application includes, but is not limited to, at least one of the following information: the size of data on which the application performs the I / O operation, the location of the data on which the application performs the I / O operation in the storage system, the manner in which the application performs the I / O operation, an identifier of the I / O operation performed by the application, the type of I / O operation performed by the application, the I / O mode in which each process in the application accesses data (sequential, stride, random, read-only, write-only, mixed read and write, etc.), the type of data on which the application performs the I / O operation, the duration of the application performing the I / O operation, the life cycle of the data on which the application performs the I / O operation, the redundancy strategy of the data on which the application performs the I / O operation, or sharing information of the data on which the application performs the I / O operation.

[0056] Optionally, the running information of the application includes, but is not limited to, at least one of the following information: the node identifier of the node running the application, the name of the node running the application, the IP address of the node running the application, the name of the node where the data on which the application performs I / O operations is located, or the node identifier where the data on which the application performs I / O operations is located.

[0057] Optionally, the information of the process executing the application includes, but is not limited to, at least one of the following information: an identifier of the job executing the application, characteristics of the job executing the application, an identifier of the process executing the application, or information on which processes of the application on which nodes process which data, etc. The job executing the application may include one or more processes.

[0058] In some possible implementations, the processor is further configured to read the computer-readable executable program to perform the following steps:

[0059] The storage policy is formulated for the application before the application is run or when a data processing request of the application is received.

[0060] In some possible implementations, the processor is further configured to read the computer-readable executable program to perform the following steps:

[0061] Acquiring characteristic information of the storage system;

[0062] The storage policy is formulated for the application according to the information of the application and the characteristic information of the storage system.

[0063] In some possible implementations, the characteristic information of the storage system includes at least one of the following information: the hardware structure of the storage system, the hardware composition of the storage system, the storage capacity of the storage system, or the data layout method of the storage system.

[0064] The hardware structure of the storage system is the positional relationship or connection relationship between different components in the storage system, and the hardware composition of the storage system is the physical form of different components in the storage system. For example, the hardware composition of the storage system may include SSD and HDD; the hardware structure of the storage system indicates that the SSD is a secondary resource, the HDD is a tertiary resource, and there is a connection relationship between the SSD and the HDD.

[0065] In some possible implementations, the storage policy of the application includes a data layout policy, and the data layout policy is used to determine the storage method and / or storage location of the data written by the application to the storage system.

[0066] In some possible implementations, the storage method in which the application is written into the storage system includes at least one of the following information:

[0067] The structure type of the data that the application needs to store, the number of copies of the data that the application needs to store, or the verification method of the data that the application needs to store.

[0068] For example, the data layout strategy may include allocating continuous storage space for data to be written to a storage device by a certain application to improve the efficiency of data writing; it may also be determining whether the data to be written is stored in a single copy or multiple copies, or it may be determining the error correction code (EC) or redundant array of independent disks (RAID) of the data to be written.

[0069] In some possible implementations, the storage strategy of the application includes a cache strategy, which is used to determine the data read by the application from the storage system and the cache method of the read data, and the cache method of the data includes at least one of the following: full cache, partial cache, pre-reading method, direct cache when writing data, or cache cleaning mechanism. For example, the cache strategy may include pre-writing specific data to be read by an application into a first-level cache with the fastest read and write speed to improve the efficiency of the application reading data; or for a request initiated by an application to only write data to a storage device, the data is directly written to the storage space of the storage device by direct (bypass) or transparent transmission without passing through the cache, thereby reducing the space occupied by the cache and improving the utilization of the cache.

[0070] In some possible implementations, the storage strategy of the application also includes a hierarchical storage strategy, which is used to store data of different frequencies accessed by the application in storage media with different read and write speeds. For example, the hierarchical storage strategy may include storing frequently accessed data, that is, data with high access frequency, in a storage medium with high read and write speeds, such as SCM or SSD, so as to improve the efficiency of accessing such data and avoid the occupation and waste of resources caused by copying data in different storage media.

[0071] In some possible implementations, the storage strategy of the application also includes a redundancy strategy, which is used to determine the backup method and / or backup quantity of the data to be written by the application to the storage system. For example, the redundancy strategy may include: for one-time data, no backup is required; for data that needs to be read later, backup is required.

[0072] In some possible implementations, the reference strategy is saved in advance; the processor is further configured to read the computer-readable executable program to perform the following steps:

[0073] The storage policy is formulated for the application with reference to the reference policy.

[0074] Optionally, the processor is further used to read the computer-readable executable program to execute the following steps: first obtain the policy of the application already stored in the policy information library, and determine the storage policy for the application in combination with the information of the application obtained this time.

[0075] If the storage policy of the application is not stored in the policy information base, the storage policy of an application similar to the application in the policy information base may be referenced. For example, according to the type or operation parameters of the application, the storage policy of the application recorded in the policy information base with the same type or operation parameters as the application may be referenced, or the storage policy of an application similar to the type or operation parameters of the application may be referenced to determine the storage policy of the application.

[0076] Usually, the read and write mode of an application is relatively fixed. Therefore, referring to the storage policy in the policy information library to determine the storage policy of the application does not require determining repeated policies for the same content each time. This can improve the efficiency of determining policies and reduce the occupancy of resources such as CPU caused by determining the corresponding policies.

[0077] In some possible implementations, the processor is further configured to read the computer-readable executable program to execute the following step: storing a storage policy of the application.

[0078] Optionally, the processor may store the storage policy of the application in the policy information library.

[0079] Optionally, the processor may store the storage policy in the policy information base after determining the policy of the application. When storing the storage policy in the policy information base, the storage policy may be stored in an updated manner, that is, information of the storage policy that has changed is stored in the policy information base.

[0080] In some possible implementations, the information of the application is normalized according to a preset format and / or content. Optionally, the information of the application is normalized according to a preset format, including but not limited to processing the information of the application according to a unified expression method or arrangement order. The information of the application is normalized according to preset content, including but not limited to adding necessary information. For example, adding context information, etc. The context information includes but is not limited to: the identifier of the node where the application is located, the name of the node where the application is located, the IP address of the node where the application is located, the name of the application, application parameters or job description and other context information.

[0081] By normalizing the information of the application, the unified storage of information of different applications can be simplified, the method of storing application information can be simplified, and the efficiency of information storage and transmission of the application can be improved; at the same time, when determining the strategy based on the information of the application, the information of the application can also be quickly identified and relevant strategy decisions can be made, thereby improving the efficiency of determining specific strategies.

[0082] In some possible implementations, the processor reading the computer-readable executable program to execute the storage policy includes: before the application initiates a read or write request, the processor reading the computer-readable executable program to execute part of the storage policy. For example, when a hierarchical storage policy is determined for the application, the policy can be executed upon receipt, so that when the application initiates a related read or write request, the related data can be quickly acquired, and the utilization rate of the hierarchical storage medium can be improved. After the hierarchical storage policy is determined for the application, when a read data request from the application is received, the data to be read is cached in the corresponding medium based on the specific data corresponding to the read request, so that the hit rate of the cached data can be improved and the efficiency of reading and writing data can be improved.

[0083] In some possible implementations, the partial strategy includes a hierarchical storage strategy, and the hierarchical storage strategy is used to store the data to be read by the application in advance in a storage medium with a high reading and writing speed.

[0084] In some possible implementations, the storage system further includes at least one computing node, the application runs on the at least one computing node, the data read and written by the application is stored on the at least one storage node, and the at least one storage node includes the computer device.

[0085] In some possible implementations, the processor obtains the information of the application from the application through an extended interface; or,

[0086] The processor obtains formatted information of the application from a computing node where the application is running.

[0087] In some possible implementations, the at least one storage node includes the computer device, and each of the storage nodes includes storage resources and computing resources;

[0088] The processor obtaining the information of the application includes:

[0089] The processor obtains the information of the application through the extended interface.

[0090] In some possible implementations, the processor obtains the information of the application from the application through the extended interface in one of the following ways:

[0091] The processor obtains application information through an extended interface library;

[0092] The processor obtains the information of the application through a dedicated file, and the application or a scheduler corresponding to the application writes the information of the application into the dedicated file;

[0093] The processor obtains the application information through a remote procedure call; or,

[0094] The processor obtains the information of the application through RESTful representation state transfer.

[0095] In a third aspect, an embodiment of the present application provides a data management system, the data management system comprising an application perception module, a policy determination module and a policy execution module, wherein:

[0096] The application perception module is used to obtain application information, wherein the application information includes at least one of the following information: I / O operation information of the application, running information of the application, and process information of the application; wherein the data of the application is stored in a storage system, and the storage system includes at least one storage node;

[0097] The policy determination module is used to formulate a storage policy for the application according to the information;

[0098] The policy execution module is used to execute the policy.

[0099] The above data management system can formulate and execute storage strategies for the application according to the information of the application, and can optimize the storage of the data read and written by the application in the storage system and / or improve the efficiency of the access. Compared with the prior art that configures strategies applicable to all applications based on a preset manner, the computer device can formulate different strategies for different applications, effectively improving the compatibility of data storage and data reading and writing for different applications.

[0100] In some possible implementations, the policy determination module formulates the storage policy for the application before the application is run or when a data processing request from the application is received.

[0101] In some possible implementations, the policy determination module is further configured to obtain characteristic information of the storage system, and formulate the storage policy for the application according to the information of the application and the characteristic information of the storage system.

[0102] In some possible implementations, the characteristic information of the storage system includes at least one of the following information: the hardware structure of the storage system, the hardware composition of the storage system, the storage capacity of the storage system, or the data layout method of the storage system.

[0103] In some possible implementations, the storage policy of the application includes a data layout policy, and the data layout policy is used to determine the storage method and / or storage location of the data written by the application to the storage system.

[0104] In some possible implementations, the storage method in which the application is written into the storage system includes at least one of the following information:

[0105] The structure type of the data that the application needs to store, the number of copies of the data that the application needs to store, or the verification method of the data that the application needs to store.

[0106] In some possible implementations, the storage strategy of the application includes a cache strategy, which is used to determine the data read by the application from the storage system and the cache method of the read data, and the cache method of the data includes at least one of the following: full cache, partial cache, pre-reading method, direct cache when writing data, or a cache cleanup mechanism.

[0107] In some possible implementations, the storage strategy of the application further includes a hierarchical storage strategy, and the hierarchical storage strategy is used to store data of different frequencies accessed by the application in storage media with different read and write speeds.

[0108] In some possible implementations, the storage policy of the application further includes a redundancy policy, and the redundancy policy is used to determine a backup method and / or a backup quantity of data to be written by the application into the storage system.

[0109] In some possible implementations, the policy determination module is further configured to formulate the storage policy for the application with reference to a pre-saved reference policy.

[0110] In some possible implementations, the application information is information that has been normalized according to a preset format and / or content.

[0111] In some possible implementations, the executing the storage policy includes: executing a portion of the storage policy before the application initiates a read or write request.

[0112] In some possible implementations, the policy determination module is further configured to:

[0113] Obtaining information of the application from the application through an extended interface; or,

[0114] The formatted information of the application is obtained from a computing device on which the application is running.

[0115] In some possible implementations, the policy determination module obtains the information of the application from the application through the extended interface in one of the following ways:

[0116] Obtain application information through the extended interface library;

[0117] The information of the application is obtained through a dedicated file, and the application or a scheduler corresponding to the application writes the information of the application into the dedicated file;

[0118] Acquire the application information through a remote procedure call; or,

[0119] The information of the application is obtained through RESTful representation state transfer.

[0120] In a fourth aspect, the present application provides a chip, which is used to execute the steps of the method in the first aspect. Alternatively, the chip is used to implement the content that the processor in the second aspect needs to provide to you, or one or more of the functions of the application perception module, the policy determination module and the policy implementation module in the third aspect. Optionally, the chip includes a processor and a chip interface. The processor can be a CPU, a microprocessor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc.

[0121] In a fifth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, wherein the instructions instruct a computer device to execute the method described in the first aspect above.

[0122] In a sixth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computer device, enables the computer device to execute the method described in the first aspect above.

[0123] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0124] The following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0125] Figure 1 A schematic diagram of the structure of a storage system provided in an embodiment of the present application;

[0126] Figure 2 A schematic diagram of the structure of another storage system provided in an embodiment of the present application;

[0127] Figure 3AA schematic diagram of the logical architecture of a software module for implementing a data management method provided in an embodiment of the present application;

[0128] Figure 3B Provided in the embodiments of this application Figure 3A A structural diagram of the implementation method of the application perception module 301;

[0129] Figure 3C Provided in the embodiments of this application Figure 3A A structural diagram of the implementation of the strategy determination module 302;

[0130] Figure 3D Provided in the embodiments of this application Figure 3A A structural diagram of the implementation method of the strategy execution module 303;

[0131] Figure 3E Provided in the embodiments of this application Figure 3A A schematic diagram of the structure of the implementation of the policy determination module 302 and the policy execution module 303;

[0132] Figure 3F Provided in the embodiments of this application Figure 3A A schematic diagram of the structure of the implementation of the application perception module 301, the policy determination module 302 and the policy execution module 303;

[0133] Figure 4 A flowchart of a data management method provided in an embodiment of the present application;

[0134] Figure 5 A flowchart of a method for optimizing storage system data management according to application information provided in an embodiment of the present application;

[0135] Figure 6 A structural diagram of a specific application scenario of a method for data management in a storage system provided in an embodiment of the present application;

[0136] Fig. 7A A flowchart of a specific implementation method for application A to implement application-aware data reading and writing provided in an embodiment of the present application;

[0137] Figure 7B A flowchart of a specific implementation method for application B to implement application-aware data reading and writing provided in an embodiment of the present application;

[0138] Figure 7C A flowchart of a specific implementation method for application C to implement application-aware data reading and writing provided in an embodiment of the present application;

[0139] Figure 8 A schematic diagram of a logical implementation of a data layout decision engine provided in an embodiment of the present application;

[0140] Fig. 9 A flowchart of a method for data management of an application according to an embodiment of the present application;

[0141] Fig.10 A schematic diagram of the structure of a data management system 1000 provided in an embodiment of the present application;

[0142] Fig.11 A schematic diagram of the structure of a computer device 2000 provided in an embodiment of the present application; DETAILED DESCRIPTION

[0143] The embodiments of the present invention are described below in conjunction with the accompanying drawings.

[0144] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "first" and "second" are used only for descriptive purposes and are not to be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features.

[0145] In the specification and claims of this application, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules that are clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices. The naming or numbering of steps in this application does not mean that the steps in the method flow must be executed in the time / logical sequence indicated by the naming or numbering. The named or numbered process steps can change the execution order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved. The division of units in this application is a logical division. There may be other division methods when it is implemented in actual applications. For example, multiple units can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, and the indirect coupling or communication connection between units can be electrical or other similar forms, which are not limited in this application. Furthermore, the units or sub-units described as separate components may or may not be physically separated, may or may not be physical units, or may be distributed in multiple circuit units, and some or all of the units may be selected according to actual needs to achieve the purpose of the present application.

[0146] It should be understood that the terms used in the description of various examples in the specification and claims of this application are only for describing specific examples and are not intended to be limiting. As used in the description of various examples and the appended claims, the singular forms "a", "an" and "the" are intended to include plural forms as well, unless the context clearly indicates otherwise.

[0147] It should also be understood that the term "and / or" used in the specification and claims of this application refers to and covers any and all possible combinations of one or more of the associated listed items. The term "and / or" is a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this application generally indicates that the associated objects before and after are in an "or" relationship.

[0148] It should be understood that determining B based on A does not mean determining B only based on A. B can also be determined based on A and / or other information.

[0149] It should also be understood that the term “comprise” (also known as “includes,” “including,” “comprises” and / or “compris ing”) when used in this specification specifies the presence of stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0150] It should also be understood that the term "if" may be interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting." Similarly, the phrase "if it is determined that ..." or "if [a stated condition or event] is detected" may be interpreted to mean "upon determining that ..." or "in response to determining that ..." or "upon detecting [a stated condition or event]" or "in response to detecting [a stated condition or event]," depending on the context.

[0151] It should be understood that the references to "one embodiment", "an embodiment", or "a possible implementation" throughout the specification mean that specific features, structures, or characteristics related to the embodiment or implementation are included in at least one embodiment of the present application. Therefore, the references to "in one embodiment" or "in an embodiment", or "a possible implementation" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0152] In the embodiments of the present application, an application refers to a program or service that implements a certain function through an electronic device or electronic method such as a computer, a computer, a mobile phone, a network or a cloud service. For example, the application can be an Office application (such as a Word tool, etc.), a video application (such as Youku, iQiyi, Facebook, etc.) or a social application (such as WeChat weichat, Weibo, etc.). The embodiments of the present application do not limit the specific form and presentation form of the application. As long as it is a program or tool that can realize data reading and writing, or software that realizes the same or similar functions, it is within the scope disclosed in the embodiments of the present application.

[0153] A storage system is a system for storing programs and / or data, and the system may include at least one storage node and a control component. Application programs may be stored in the storage system, and the computer device runs the application by reading the program in the storage system. It should be noted that the storage system may be integrated into the computer device or may be independent of the computer device. When the storage system is independent of the computer device, each computer device in the cluster may share the storage system to form a cluster storage system.

[0154] Considering the requirements of read / write performance, capacity, cost, etc., a storage system can generally be configured with multi-level storage devices. Specifically, a storage system can be configured with a cache memory (Cache, referred to as cache), a main memory (Main Memory, also known as internal memory, referred to as memory), and a secondary memory (Secondary Memory, also known as external memory, referred to as external memory).

[0155] The storage system provides different storage resources through the above-mentioned memories. Specifically, the internal memory can be directly accessed by the processor of the computer device, directly providing data and instructions to the processor, and storing the data processed by the processor. It has a fast read and write speed, but a relatively small capacity. The external memory cannot be directly accessed by the processor, so it is mainly used to store data and programs that the processor does not use temporarily. Its read and write speed is lower than that of the internal memory, but its capacity is relatively large. The cache is located between the processor and the memory, and is used to solve the problem of mismatch between the processor's operating speed and the memory's read and write speed. Specifically, the data that the processor will access is stored in the cache, and this data is part of the data in the memory.

[0156] Many existing distributed storage systems have "tiered storage" or "Cache". Its main function is to store the data that is being used or will be used in the system (or hot data) in a medium with a faster read / write speed, and to store the rarely used data (or cold data) in a medium with a lower read / write speed.

[0157] In essence, hierarchical storage technology or cache technology roughly predicts the hot and cold changes of data by "blindly" sensing the access of applications to data. It is "speculative" or "random" to a large extent, and it is difficult to accurately predict the hot and cold changes of all data. If the algorithm is not good, it is easy to cause hot data to be stored in a medium with low read and write speeds, while cold data is written to a medium with high read and write speeds. For hot data stored in a medium with low read and write speeds, its read and write efficiency is low. In addition, media with high read and write speeds are often more expensive, and cold data occupying the space of media with high read and write speeds will increase costs.

[0158] In addition, in order to improve the system throughput, the distributed storage system will divide the file data into blocks of the same size and store them in different storage nodes in blocks. This approach can fully utilize the bandwidth of multiple nodes and multiple storage devices. On the other hand, it can also be used to construct a variety of redundancy strategies, such as EC and multiple copies, to ensure data reliability. This default data sharding method is not suitable for the way applications use data in some scenarios. For example, when the default sharding strategy is one shard for each file, if the application that accesses (reads / writes) this file is running on multiple nodes, especially when there are applications on multiple nodes that want to write to this file, on the one hand, the file I / O bandwidth cannot be guaranteed, and on the other hand, many I / O conflicts will occur, which will seriously slow down the reading and writing progress of each application and affect the efficiency of application reading and writing.

[0159] In the above implementation, since the storage system is unaware of the I / O mode of the application, it is impossible to select the appropriate data layout or formulate an accurate cache strategy for the I / O mode of different applications, which affects the effective management of storage data and the efficiency of application access.

[0160] An embodiment of the present application provides a method for data management in a storage system, which enables the storage system to perceive information such as the I / O mode of the application, and select reasonable data layout and other strategies according to the I / O mode of the application, and use differentiated caching strategies and / or hierarchical storage solutions to improve the performance of the storage system and improve the efficiency of application access such as reading and writing data.

[0161] Figure 1 A schematic diagram of the structure of a storage system provided in an embodiment of the present application. Figure 1 As shown, the storage system consists of two parts: storage nodes and storage clients. Figure 1 In the description, an example is taken in which storage nodes 111, 112 and 113 are included, and storage clients are respectively running on computing nodes 101 and 102.

[0162] Multiple storage nodes make up the storage backend, which is internally composed of media with different bandwidths or characteristics, such as memory, SCM, SSD, HDD or tape. Figure 1 The storage nodes include memory, SSD and HDD as an example.

[0163] Storage media with different characteristics in storage nodes form storage pools of multiple tiers, also known as hierarchical storage management (HSM). Usually, frequently accessed data and recently used data are placed in faster tiers (such as memory or SSD-level media), while data with fewer accesses is stored in slower tiers (such as HDD and tape-level media). For example Figure 1 In the storage system, the memories in at least two storage nodes can constitute a memory pool of the storage system, the SSDs in at least two storage nodes can constitute a flash memory pool of the storage system, and the HDDs in at least two storage nodes can constitute a hard disk pool of the storage system, so as to form storage media of different levels for storing data with different access frequencies.

[0164] The storage client is the interface between the computing node and the storage node. It is used to receive I / O requests initiated by applications and implement interaction between the computing node and the storage node. Figure 1 The following example uses the storage client running on the computing node as an example. Many different applications can run on each computing node, and different applications can connect to the storage system through the same storage client. Figure 1 In the example, two applications are run on each computing node.

[0165] It should be noted that Figure 1 It is only exemplary to show the composition of storage nodes and computing nodes and their connection relationship. In the specific implementation, the storage nodes (storage nodes 111, storage nodes 112 and storage nodes 113) may also include hardware resources such as processors and buses, and software or software modules related to data read and write control implemented by the processors. The computing nodes (computing nodes 101 and computing nodes 102) may also include hardware components such as processors, memories and buses, and the applications and storage clients in each computing node may implement corresponding functions through processors, etc. The embodiments of the present application will not be repeated.

[0166] Figure 2 A schematic diagram of the structure of another storage system provided in an embodiment of the present application. Figure 2 The scenario shown is an implementation scenario of a hyper-converged infrastructure (HCI). In the HCI scenario, computing and storage coexist on each HCI node. For example, node 201 includes computing and storage. The computing part includes applications, and the storage part includes storage media such as memory, SSD and HDD. The application in the computing part is connected to the storage part in this node to read and write data. Applications in different nodes can also access data stored in other nodes.

[0167] It should be noted that, just for the convenience of description, Figure 2 Only part of the components of each node are shown. In the specific implementation, the computing part of each node may also include hardware resources such as a processor, and the storage part may also include hardware resources such as a bus and software resources for realizing data reading and writing control, etc., which will not be repeated here.

[0168] Figure 1 and Figure 2 The scenarios shown are only examples and should not be used as a limitation to the embodiments of the present application. In specific implementations, it can also be implemented in other similar scenarios. For example, the embodiments of the present application can also be applied to distributed storage systems, including but not limited to distributed file storage systems, distributed block storage systems, or distributed object storage systems; they can also be applied to cloud storage systems or the platform as a service (PaaS) layer of public clouds. As an optional implementation method, the solution provided in the embodiments of the present application is more suitable for use in storage systems with more applications, or storage systems applicable to applications with variable I / O modes.

[0169] Figure 3A The following is a logical diagram of a software module for implementing a data management method in a storage system provided in an embodiment of the present application. Figure 3A As shown, the software modules provided for implementing the embodiment of the present application include an application perception module 301, a policy determination module 302 and a policy implementation module 303.

[0170] in:

[0171] The application perception module 301 is used to obtain application information, including static attribute information and / or dynamic variable information of the application, and provide the obtained information to the policy determination module 302. The application information includes, but is not limited to: one or more of the application I / O operation information, the application operation information, and the process information of executing the application. Among them:

[0172] The application I / O operation information includes, but is not limited to, at least one of the following information: the size of the data on which the application performs the I / O operation, the location of the data on which the application performs the I / O operation in the storage system, the manner in which the application performs the I / O operation, the identifier of the application performing the I / O operation, the type of the I / O operation performed by the application, the I / O mode in which each process in the application accesses data (sequential, stride, random, read-only, write-only, mixed read-write, etc.), the type of the data on which the application performs the I / O operation, the duration of the application performing the I / O operation, the life cycle of the data on which the application performs the I / O operation, the redundancy strategy of the data on which the application performs the I / O operation, or the sharing information of the data on which the application performs the I / O operation;

[0173] The running information of the application includes but is not limited to at least one of the following information: the node identifier of the node running the application, the name of the node running the application, the IP address of the node running the application, the name of the node where the data of the I / O operation performed by the application is located, or the node identifier where the data of the I / O operation performed by the application is located;

[0174] The information of the process executing the application includes, but is not limited to, at least one of the following information: the identification of the job executing the application, the characteristics of the job executing the application, the identification of the process executing the application, or which processes of the application on which nodes process which data, etc. The job executing the application may include one or more processes.

[0175] The policy determination module 302 is used to determine a storage policy for the application according to the information provided by the application awareness module 301;

[0176] The storage strategy includes but is not limited to a data layout strategy, a cache strategy, a hierarchical storage strategy or a redundancy strategy.

[0177] In one implementation, the policy determination module 302 is also used to store the storage policy for the application in a policy information library. The policy information library is used to store the storage policy determined by the policy determination module 302 for each application. Accordingly, when the policy determination module 302 determines a specific policy for the application based on the information provided by the application perception module 301, it can refer to the policy that has been determined in the policy information library, and determine the data management policy for this time in combination with the information of the application received this time. Normally, the read and write mode of an application is relatively fixed. Therefore, referring to the policy in the policy information library to determine the storage policy of the application does not require determining repeated policies for the same content each time, which can improve the efficiency of the policy determination module 302 in determining the storage policy and reduce the occupation of resources such as CPU caused by determining the corresponding policy.

[0178] Exemplarily, after determining the storage policy of the application, the policy determination module 302 may also store the updated or changed storage policy in the policy information repository. That is, once the policy determination module 302 completes the policy determination, the determined storage policy is sent to the policy information repository.

[0179] The policy implementation module 303 is used to execute the storage policy determined by the policy determination module 302 .

[0180] Figure 3A The software modules shown can be deployed independently or in combination in different computer devices to achieve corresponding functions. FIG. 3B to FIG. 3F Take this as an example to explain Figure 3A The software modules shown are implemented differently in a computer device. It is understood that FIG. 3B to FIG. 3F This is just an exemplary description, and in actual implementation, it can also be achieved through other methods.

[0181] like Figure 3B As shown, the first computer device 310 includes a bus 311, a processor 312, a communication interface 313 and a memory 314. The processor 312, the memory 314 and the communication interface 313 communicate with each other through the bus 311. The bus 311 can be a peripheral component interconnect (PCI) bus, a peripheral component interconnect express (PCIe) bus or an extended industry standard architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3B The communication interface 313 is used to communicate with the outside, for example, to receive operation attribute information and object attribute information of I / O operation, etc.

[0182] The processor 312 may be a CPU, a graphics processor (GPU), a general-purpose GPU (GPGPU), a tensor processor (TPU), a data processor (GPU), or other chips with computing capabilities. The memory 314 may include a volatile memory, such as a random access memory (RAM). The memory 314 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a HDD, or a SSD.

[0183] The memory 314 stores programs or instructions, such as programs or instructions required to implement the functions of the application awareness module 301, and the processor 312 executes the programs or instructions to obtain application information. Of course, the memory 314 can also store data, including but not limited to data that the first computer device 310 needs to store; it can also store application information obtained by the application awareness module 301.

[0184] In one implementation, the application information stored in the memory 314 may be periodically aggregated into a dedicated database, which may be an independent database device specifically used to store application-related databases.

[0185] like Figure 3C As shown, the second computer device 320 includes a bus 321, a processor 322, a communication interface 323, and a memory 324. The processor 322, the memory 324, and the communication interface 323 communicate with each other via the bus 321. The memory 324 stores programs or instructions, such as programs or instructions required to implement the functions of the policy determination module 302. Of course, the memory 324 can also store data, such as data to be read and written.

[0186] Figure 3C Processor, memory, bus, etc. Figure 3B The implementation in is similar and will not be repeated here.

[0187] Optional, Figure 3CThe above-mentioned policy information library can also be stored in the memory 324. The policy determination module 302 can obtain the policy related to the application locally from the second computer device 320 and formulate a storage policy for the application in combination with the information of the application obtained this time. The policy determination module 302 can also store the formulated storage policy of the application or the updated storage policy of the application in the policy information library. Of course, the policy information library in the embodiment of the present application can also be stored in an independent database, such as an independent computer device for storing data; it can also be stored in a distributed manner on different storage nodes, such as Figure 1 In the scenario shown, the data may be stored on any two or three of the storage nodes 111 , 112 , and 113 .

[0188] like Figure 3D As shown, the third computer device 330 includes a bus 331, a processor 332, a communication interface 333, and a memory 334. The processor 332, the memory 334, and the communication interface 333 communicate with each other through the bus 331. The memory 334 stores programs or instructions, such as programs or instructions required to implement the functions of the policy execution module 303. Of course, the memory 334 can also store data, such as data to be read and written.

[0189] Figure 3D Processor, memory, bus, etc. Figure 3B The implementation in is similar and will not be repeated here.

[0190] like Figure 3E As shown, the fourth computer device 340 includes a bus 341, a processor 342, a communication interface 343, and a memory 344. The processor 342, the memory 344, and the communication interface 343 communicate with each other through the bus 341. The memory 344 stores programs or instructions, such as programs or instructions required to implement the functions of the policy determination module 302 and the functions of the policy execution module 303. Of course, the memory 344 can also store data, such as data to be read and written.

[0191] Figure 3E Processor, memory, bus, etc. Figure 3B The implementation in is similar and will not be repeated here.

[0192] like Figure 3FAs shown, the fifth computer device 350 includes a bus 351, a processor 352, a communication interface 353, and a memory 354. The processor 352, the memory 354, and the communication interface 353 communicate with each other through the bus 351. The memory 354 stores programs or instructions, such as programs or instructions required to implement the functions of the application perception module 301, the policy determination module 302, and the policy execution module 303. Of course, the memory 354 can also store data, such as data to be read and written.

[0193] Figure 3F Processor, memory, bus, etc. Figure 3B The implementation in is similar and will not be repeated here.

[0194] Above FIG. 3B to FIG. 3F The computer equipment in the above Figure 1 or Figure 2 In the scenario shown. For example, Figure 3B The computer equipment shown can be applied to Figure 1 A computing node in (such as computing node 101 or 102), Figure 3C , Figure 3D Or the computer equipment shown in Figure E can be applied to Figure 1 The storage nodes shown (such as storage nodes 111, 112 or 113). Figure 3F The computer equipment shown can be applied to Figure 2 The node (such as node 201, 202 or 203).

[0195] In some possible implementations, the functions of the above-mentioned application perception module 301, policy determination module 302 and policy implementation module 303 can also be implemented by one or more chips. Each chip includes a processor and a chip interface. Optionally, the processor can be a CPU, a microprocessor, an ASIC, an FPGA, etc. Among them, the chip interface is used to receive instructions and transmit them to the processor. The processor executes the above-mentioned instructions to execute the steps of the aforementioned method for implementing data management. Of course, when the functions of the above-mentioned application perception module 301, the policy determination module 302 and the policy implementation module 303 are implemented by multiple chips, the chip interface of each chip receives part of the instructions, and the processor of each chip implements the corresponding functions according to the partial steps of the above-mentioned method for implementing data management. In this way, multiple processors can implement the steps of the storage system data management method provided in the embodiment of the present application.

[0196] Below Figure 1 The scene shown, Figure 3B The first computer device is suitable for Figure 1 Compute node 101 in Figure 3E The fourth computer device shown is suitable for Figure 1 Taking the storage node 111 in the example, the data management method for implementing the storage system provided by the embodiment of the present application is described.

[0197] like Figure 4 As shown, the data management method provided in the embodiment of the present application includes:

[0198] Step 400: the application sensing module 301 receives application information;

[0199] Exemplarily, the application awareness module 301 may receive information from an application through an extended interface.

[0200] In one implementation, the application and the application awareness module 301 may implement information transmission through one or more extended interfaces. The one or more extended interfaces are interfaces added by extension, serving as interfaces for transmitting information between the application and the application awareness module 301. These extended interfaces may be one or more different interfaces, and may be different from the existing interfaces of the application.

[0201] In specific implementation, the interface expansion can be implemented in the following manner, but the embodiments of the present application are not limited to the following implementation manner:

[0202] 1) The application awareness module 301 provides a dedicated interface library to implement interface expansion; wherein the interface library may be an application programming interface (API) library. The interface library may be a dynamic link library, and a link to the dynamic link library is added when the application is compiled, and the dynamic link library is linked to the executable file of the application. In this way, the application can pass the application information to the application awareness module 301 by calling the dynamic link library when the application is running.

[0203] This method can be applied to the scenario where the code of the application can be modified, that is, by modifying the code of the application, the application and the application perception module 301 can transmit information through the interface library.

[0204] 2) The application-aware module 301 provides a command line tool, and the application can directly call the command line tool to call the interface library provided by the application-aware module 301 through the command line tool;

[0205] This method is suitable for scenarios where the application code cannot be modified. Exemplarily, the application or the related scheduler calls the command line tool to implement the call of the interface library, and then implements the transmission of information between the application and the application perception module 301.

[0206] 3) The storage system provides information reception and display files (such as file system procfs, sysfs mechanism), and the application or related scheduler can pass information to the application awareness module 301 by writing data to such files.

[0207] In this case, the application awareness module 301 can access the above file system and obtain the information of the application, that is, the application awareness module 301 needs to have the ability to access the file system.

[0208] Likewise, if the code of the application can be modified, the application can directly write the information into the file system; if the code of the application cannot be modified, the information can be written into the file system by calling a command line tool.

[0209] 4) The application-aware module 301 provides remote procedure call (RPC) or representational state transfer (RESTful).

[0210] That is, the application awareness module 301 requests the application information from the application through RPC or RESTful.

[0211] As an optional implementation, the application information may be transmitted by the application, or may be transmitted by other entities independent of the application (e.g., a scheduler). Specifically, when the application transmits the application information, the application information may be transmitted to the application perception module 301 in the above manner before the application initiates an I / O request. When the application information is transmitted through the scheduler, the application information may be transmitted in the above manner before the application runs or during the application runs.

[0212] It should be noted that the information of the application transmitted between the application and the application perception module 301 includes but is not limited to at least one of the following information: the application I / O operation information, the application running information and the application process information. For example, the application information may be information such as the file type to be accessed by the application, the node to be accessed by the application or the read-write mode of the application; this information is static attribute information that has been determined by the modeling module of the application, and the application perception module 301 can obtain the application information through the above-mentioned extended interface before the application runs or when the application just runs. The application perception module 301 can also obtain dynamic variable information of the application when the application just runs. For example, the application needs to access specific data based on the needs of special scenarios, and the information of these specific data is not included in the static attribute information.

[0213] Optionally, when the application awareness module 301 is implemented on a computer device serving as a storage node, Figure 3F In the implementation shown, the application awareness module 301 can obtain information about the application when the application initiates a read or write request.

[0214] Step 402: the application sensing module 301 sends the acquired information of the application to the policy determination module 302;

[0215] The application perception module 301 sends the acquired information of the application to the policy determination module 302, so that the policy determination module can formulate a policy for the application according to the information of the application.

[0216] In one implementation, the application perception module 301 may normalize the acquired information of the application and send the normalized information to the policy determination module 302. The normalization process is to convert the application information into a unified format and store it. The method of converting to a unified format includes but is not limited to processing the application information according to a unified expression method or arrangement order, or adding necessary information, such as adding context information. The context information includes but is not limited to: the identifier of the node where the application is located, the name of the node where the application is located, the IP address of the node where the application is located, the name of the application, application parameters or job description and other context information. Exemplarily, obtaining information such as the identifier of the node where the application is located can be obtained through existing functional modules in the prior art.

[0217] Normalizing the information of the application can simplify the unified storage of information of different applications, simplify the method of storing application information, and improve the efficiency of application information storage and transmission; at the same time, it can also facilitate the policy determination module 302 to quickly identify application information and make relevant policy decisions, thereby improving the efficiency of the policy determination module 302 in determining specific policies.

[0218] Exemplarily, the application awareness module 301 can transmit different information respectively through multiple interfaces with the policy determination module 302; that is, the application awareness module 301 can pass different messages to the policy determination module 302 through different interfaces. Of course, the application awareness module 301 can also pass the application information to the policy determination module 302 through a unified interface.

[0219] The format of the message transmitted by the application perception module 301 to the policy determination module 302 can be in the format of Extensible Markup Language (XML) or JavaScript Object Notation (JSON), or it can be a custom data structure, as long as it can be recognized by the policy determination module 302. Optionally, when the format of the message is an extensible message format, the policy determination module 302 can be changed or added according to the change or increase of demand, which can improve the applicability to new business forms and facilitate the addition of new business information in the future. There is no need to modify the format, only the content of the message format needs to be modified, or the new information can be directly added. In the specific implementation, the embodiment of the present application does not limit the specific format of the message.

[0220] The ways in which the application awareness module 301 and the policy determination module 302 transmit information include but are not limited to: 1) direct function call; 2) shared memory; 3) inter-process communication; 4) dynamic library call; 5) network communication, such as remote procedure call (RPC); 6) Hypertext Transfer Protocol (HTTP) or RESTFUL. Among them, ways 1)-4) are applicable to the scenario where the application awareness module 301 and the policy determination module 302 are implemented on one node, and ways 5)-6) are applicable to the scenario where the application awareness module 301 and the policy determination module 302 are implemented on different nodes.

[0221] Step 404: the policy determination module 302 determines the storage policy of the application according to the received information;

[0222] The policy determination module 302 determines a specific storage policy for implementing data layout, cache, hierarchical storage or redundant backup of the application based on the received information of the application.

[0223] Exemplarily, the policy determination module 302 may obtain policies in the policy information library, refer to the policies constructed in the early stage and combine the information of the application to determine the storage policy of the application. The policy determination module 302 may obtain the storage policy of the application recorded in the policy information library. If the storage policy of the application is not recorded in the policy information library, the storage policy of similar applications may be referred to. When the policy determination module 302 refers to the storage policy of similar applications, the policy determination module 302 refers to the storage policy of the application with the same type or operation parameters as the application recorded in the policy information library, or refers to the storage policy of the application with similar type or operation parameters as the application, according to the type or operation parameters of the application. In one implementation, the policy determination module 302 formulates a storage policy for the application based on the acquired information of the application and the characteristic information of the storage system. Since the hardware structure, hardware composition, storage capacity and storage system data layout of each storage system are not exactly the same, for example, some storage systems include memory pool, SSD pool and HDD pool, and some storage systems only include SSD pool and HDD pool, that is, their hardware composition is different; in some storage systems, SSD pool is a secondary storage resource and is directly connected to HDD pool; in some storage systems, SSD pool is a primary storage resource and is directly connected to HDD pool, that is, the hardware structure of the storage system is different; some storage systems have a storage capacity of 100GB, and some storage systems have a storage capacity of 100TB, that is, the storage capacity of the storage system is different; in some storage systems, the underlying block size of the storage medium is different, and the redundant backup mechanism used is different, that is, the data layout of the storage system is different. Therefore, when determining the storage strategy of the application, the policy determination module 302 combines the information of the application and the characteristic information of the storage system to better fit the characteristics of the storage system, and can further optimize the storage of the data of the application I / O operation in the storage system and improve the efficiency of the application in performing I / O operations.

[0224] After the policy determination module 302 determines the storage policy of the application, it may also add or update the determined storage policy of the application to the policy information library.

[0225] The storage strategy of the application determined by the strategy determination module 302 includes, but is not limited to, data layout strategy, cache strategy, hierarchical storage strategy or redundancy strategy. The strategy determination module 302 can send the received information of the application to the corresponding strategy engine or strategy determination submodule according to the category of the received information to make corresponding decisions. For example:

[0226] A data layout policy engine is used to determine the layout of the data to be written (data to be written to the storage device) in the storage system. Specifically, the policy layout policy engine can determine the storage method and / or storage location of the data written by the application to the storage system in combination with the application information. The storage method of the application writing to the storage system includes at least one of the following information: the structure type of the data that the application needs to store, the number of copies of the data that the application needs to store, or the verification method of the data that the application needs to store. Exemplarily, the policy layout policy engine can allocate continuous storage space for data to be written to the storage device by a certain application to improve the efficiency of data writing; it can also determine whether the data to be written is stored in a single copy or multiple copies, and it can also determine the EC or RAID method of the data to be written.

[0227] The cache policy engine is used to determine specific cache data and / or cache methods, that is, to pre-cache cache data (data read from the storage device) in a specific cache medium so that the application can obtain the data to be read in time. Specifically, the cache policy engine can determine the data read by the application from the storage system and the cache method of the read data in combination with the application information. The cache method of the data includes but is not limited to: full cache, partial cache, cache cleaning mechanism, pre-reading method or direct (bypass) cache when writing data. One or more. Exemplarily, the specific data to be read by an application is pre-written into the first-level cache with the fastest read and write speed to improve the efficiency of the application reading data; or for a request initiated by an application to write data only to the storage device, the data is directly written into the storage space of the storage device by direct (bypass) or transparent transmission without passing through the cache, thereby reducing the space occupied by the cache and improving the utilization of the cache.

[0228] A hierarchical storage strategy engine is used to store data with different access frequencies in media with different tiers. Specifically, the hierarchical storage strategy engine can combine application information to determine that data with different frequencies accessed by the application is stored in storage media with different read and write speeds. The hierarchical storage of data accessed by the application determined by the hierarchical storage strategy engine includes but is not limited to: storing data with high access frequency of the application in a storage medium with high read and write speed or storing data with low access frequency in a storage medium with low read and write speed. Exemplarily, data with high access frequency is stored in an SSD with high read and write speed, and data with low access frequency is stored in an HDD with low read and write speed.

[0229] A redundancy policy engine is used to determine the backup method and / or the number of backups of the data to be written by the application to the storage system. Specifically, the redundancy policy engine can combine the application information to determine the redundancy policy of the data accessed by the application, including but not limited to: whether to back up, the backup method or the number of backups, etc. For example, for one-time data, no backup is required; for data that needs to be read subsequently, a backup is required. The redundancy policy mainly optimizes the storage method of data from the perspective of the data life cycle.

[0230] It should be noted that the above is only an example of a part of the strategies determined by the strategy determination module 302. In specific implementation, the strategy determination module 302 can also be used to determine other storage strategies, that is, other strategies for optimizing the layout of data in the storage system or improving the data access efficiency of the storage system. In addition, the strategy formulated by the above-mentioned hierarchical storage strategy engine or the strategy formulated by the above-mentioned redundant strategy engine is a part of the strategy formulated by the above-mentioned data layout strategy engine. The embodiment of the present application separately describes the hierarchical storage strategy engine and the redundant strategy engine, because hierarchical storage and redundant backup have a relatively large proportion in optimizing the storage of data to be accessed by applications and improving the efficiency of application access to data. Using an independent strategy engine to formulate related strategies can balance and optimize the process of strategy formulation and improve the efficiency of strategy formulation. In specific implementation, the functions completed by the hierarchical storage strategy engine and the redundant strategy engine can be realized by the data layout strategy engine, that is, the data layout strategy engine, the hierarchical storage strategy engine and the redundant strategy engine can be integrated; corresponding strategies can also be independently formulated by the hierarchical storage strategy engine and the redundant strategy engine, that is, the data layout strategy engine, the hierarchical storage strategy engine and the redundant strategy engine can be independently implemented, and the specific implementation method is not limited by the embodiment of the application.

[0231] Step 406: the policy execution module 303 obtains the storage policy for executing the application from the policy determination module 302;

[0232] After the policy determination module 302 determines the storage policy of the application, it can send the determined storage policy to the policy execution module 303, or when the policy execution module 303 sends a request to obtain the policy to the policy determination module 302, the storage policy of the application is sent to the policy execution module 303. The policy execution module 303 can also obtain the storage policy of the application from the policy library and execute it according to the information of the application, such as the job identifier.

[0233] After the policy execution module 303 obtains the storage policy for the application, it can execute part of the policy first in combination with the specific policy of the specific application, and then execute part of the storage policy after receiving the read and write request of the application. That is, part of the storage policy can be executed first, and step 410 is executed after the storage policy of the application is obtained; the other part of the storage policy can be executed after receiving the access request initiated by the application, that is, step 410 is executed after receiving the access request initiated by the application. For example, the hierarchical storage policy of the application determined by the policy determination module 302 can be executed after receiving the policy, so that when the relevant read and write request is initiated by the application, the relevant data can be quickly obtained, and the utilization rate of the hierarchical storage medium can be improved. For the cache policy determined by the policy determination module 302, when receiving the read data request of the application, the data to be read is cached in the corresponding medium based on the specific data corresponding to the read request.

[0234] Exemplarily, after receiving the hierarchical storage policy of the application determined by the policy determination module 302, the policy execution module 303 executes the hierarchical storage policy before the application initiates the access of the read / write request, which can be implemented in the following two scenarios:

[0235] One scenario is: after the policy determination module 302 formulates a hierarchical storage strategy for the application, the policy execution module 303 determines whether the triggering condition of the hierarchical storage strategy is triggered. One implementation of the triggering condition is that the scheduler of the application pre-retrieves the data that the application will access. When the policy execution module 303 determines that the scheduler of the application is to retrieve the data of the application, the hierarchical storage strategy is executed, and the data that the application needs to read is pre-stored in a preset storage medium. For example, the data that the application needs to read is pre-read from the HDD and stored in the SSD. In this way, when the application initiates a read request, the corresponding data can be quickly obtained.

[0236] Another scenario is that before the application initiates a request to read data, it first notifies the storage system through the interface with the storage system that a read operation is about to be initiated and the data to be read. The policy execution module 303 executes the hierarchical storage policy that has been formulated for the application according to the request for the data to be accessed initiated by the application, such as pre-reading data from the HDD and storing it in the SSD, so that the relevant data can be quickly obtained when the application initiates a read request.

[0237] Step 408: The policy execution module 303 receives the access request of the application;

[0238] The policy execution module 303 may call the access request initiated by the application to the storage system through a related interface.

[0239] Exemplarily, the policy execution module 303 determines whether the execution of the access request is related to a relevant storage policy, and if so, executes the relevant storage policy; for example, executes the relevant Cache policy.

[0240] Step 410: The policy execution module 303 executes the acquired storage policy of the application in the storage system.

[0241] The policy execution module 303 executes relevant policies in the storage system according to the obtained access request of the application and the obtained storage policy of the application.

[0242] For example, during the I / O execution process of the application, the policy execution module 303 executes specific optimization measures in combination with specific policies based on the identifier (ID) of the node where the application is located, the job ID or process information of the application, and the specific data objects and ranges that need to be read and written.

[0243] Figure 4 The method shown, by obtaining application information and determining a specific strategy for the application based on the application information, is used for processing the application data and processing read and write requests, such as optimizing the layout of data in the storage system and pre-caching data, which can effectively optimize the data layout of the storage system and improve the efficiency of data access. Compared with the prior art that configures strategies applicable to all applications based on a preset manner, the computer device can formulate different strategies for different applications, effectively improving the compatibility of data storage and data reading and writing for different applications. Figure 2 Taking the HCI scenario shown as an example, if the storage policy of the storage system leads to unreasonable data allocation, it is possible that the application will often need to obtain data from other nodes, which will increase network latency. By adopting the above method provided by the embodiment of the present application, the storage system can perceive the information of the application, and can make full use of the locality of data access, bring data closer to calculation, and optimize the data layout of the storage system. At the same time, the data range accessed by each specific application on each HCI node is the data in the address space stored on this node, which can reduce network latency and improve the efficiency of data access.

[0244] Figure 4 The method shown can be implemented independently of the existing storage strategy or in combination with the existing storage strategy. When combined with the existing storage strategy, the method can be implemented based on the existing storage strategy. Figure 4 The method shown further optimizes the policy determination and execution when the application accesses data; at the same time, for Figure 4When the storage policy of the related application is determined by the method shown in the figure, the existing storage policy implementation method may no longer be followed. The storage policy determined based on the application information and combined with the storage policy Figure 4 The method shown in the figure determines the strategy of the application when performing specific I / O operations and executes it. When it is implemented independently of the existing storage strategy, it can be realized only by Figure 4 The method shown implements storage policy determination and execution when an application accesses data.

[0245] Figure 5 The present invention provides a schematic diagram of a specific implementation of a method for optimizing storage system data management based on application information. Figure 5 As shown, application 1, application 2 and application 3 are different applications, which communicate with the policy determination module and the policy execution module in the storage system through the application perception module. The application perception module obtains information of each application (for example, it can be I / O hints), and the policy determination module implements data layout and caching of different applications (for example, application 1, application 2 and application 3) in the storage system through data layout strategy (for example, optimizing storage through Smart Placement), cache strategy (for example, through advance cache) or hierarchical storage strategy (for example, different levels of storage methods). Figure 5 The storage medium of the first-level storage, such as the memory resource pool, can be the memory Pool, and its data read and write speed is faster than the other two levels. The storage medium of the second-level storage, such as the Flash resource pool (which can be the Flash Pool), has a data read and write speed lower than the first-level storage but higher than the third-level storage. The storage medium of the third-level storage, such as the hard disk or Disk, can be the Disk Pool, and its data read and write speed is lower than the first-level storage and the second-level storage.

[0246] For example, if the information that application 1 passes to the application perception layer is: "IOPattern: SeqRead; IOSize: 8KB; FileSet: / my / file / dir / file1, / my / file / dir2 / fileX", IOPattern: SeqRead shows the way application 1 reads and writes data, IOSize: 8KB shows the size of the data that application 1 performs the I / O operation on, and FileSet: / my / file / dir / file1, / my / file / dir2 / fileX shows the location of the data that application 1 wants to read and write in the storage system.

[0247] After the policy determination module receives the above information of application 1 transmitted by the application perception module, it calls the engine of each policy (for example, the processing function of each policy) to formulate a policy for the application: after receiving the information, the data layout policy engine identifies the IOPattern as sequential read and considers that no processing is required, so it ignores the information; after receiving the information, the cache policy engine identifies the IOPattern as sequential read, the IOSize is 8KB, and the FileSet is / my / file / dir / file1, / my / file / dir2 / fileX. On the one hand, it adds an AppA:Read:Prefetch; Seq; 128KB to the policy information library (indicating that when the application has a read operation in the future, a sequential pre-read action with a granularity of 128KB must be performed); on the other hand, it passes the policy to the policy execution module (Execution Engine) to read the first 128KB of file1 and fileX into the cache (primary storage) in advance, that is, the policy execution module calls the pre-read instruction of the cache module to load the file data into the cache in advance. It should be noted that the 128KB in the aforementioned strategy is determined based on the underlying block size of the storage system. Since different storage systems have different underlying block sizes, 128KB is just an example. In specific implementation, it can be flexibly adjusted according to the specific situation of the storage system (different underlying block sizes).

[0248] Figure 5 The application perception module, policy determination module and policy implementation module shown in the figure can be implemented by referring to the above-mentioned implementation methods of the application perception module 301, the policy determination module 302 and the policy execution module 303, and will not be repeated here.

[0249] Figure 6 This is a structural diagram of another specific application scenario of a storage system data management method provided by an embodiment of the present application. Figure 6 As shown, different applications are respectively run on three nodes (such as computing devices), including application A, application B, and application C. Application A is a read-write hybrid application (such as a database application) and runs on nodes 1, 2, and 3. Application B is a read-only application (such as a video on demand system) and runs on nodes 1 and 3. Application C is a write-only application (such as a log system) and runs on node 2.

[0250] Distributed private clients (DPCs) are deployed on all three nodes. DPCs can sense application type, I / O mode, node number or thread ID, and pass this information along with the I / O request to the storage server. Figure 6The DPC in the example is a specific implementation of the application perception module in the embodiment of the present application.

[0251] Fig. 7A , Figure 7B and Figure 7C By way of example, the process of determining and executing policies for application A, application B, and application C is shown respectively. It can be understood that Fig. 7A , Figure 7B and Figure 7C It is just an example of different applications. In actual implementation, the implementation of application-aware interfaces, the determination of relevant strategies and the implementation of strategies will have different implementations based on specific scenarios. Fig. 7A , Figure 7B and Figure 7C The application awareness interface may be a specific implementation of the application awareness module, the data layout engine, the cache policy engine and the hierarchical storage engine may be a specific implementation of the policy determination module, and the policy execution engine may be a specific implementation of the policy execution module.

[0252] like Fig. 7A As shown, the information of application A obtained by the application-aware interface is as follows:

[0253] Job ID: 12345;

[0254] Job characteristics: Frequent operation, frequent data updates, and frequent use of old data;

[0255] Running nodes: N1, N2, N3;

[0256] Process ID: 24,1137,6578;

[0257] Job I / O type: random read / write, IOPS type, frequent modification;

[0258] I / O block size: 512B-4KB;

[0259] Access datasets: / path / to / appA / file1, / path / to / appA / file2,…,

[0260] / path / to / appA / file10000;

[0261] I / O file size: single file 64K-2M, total data set size 2G, total I / O volume 30G.

[0262] The characteristics of read-write hybrid application A are: frequent operation, frequent data updates, old data is also frequently used, read and write operations belong to small files (for example, 512B-4KB); the job type is random read and write, IOPS type, and frequent modification. After the application perception module obtains this information through the application perception interface, it passes this information to the policy determination module. The policy determination module formulates policies for application A through the data layout policy engine, cache policy engine, and hierarchical storage policy engine. For example, the data layout policy engine determines that the data structure type to be stored by application A is a log-type data structure to reduce possible read and write amplification; determines that the storage method of the data to be stored by application A is to package and store the data to improve the utilization of storage space; determines that the verification method of the data to be stored by application A is to increase the proportion of verification data to improve the reliability of the data. Exemplarily, the data layout strategy engine adopts a data layout strategy of log-type data structure based on the characteristics of application A multi-node I / O and small files, to reduce read amplification and write amplification caused by random writes; and, since the information of application A belongs to small I / O (block size 4KB) and the data is frequently modified, the data of the application can be packaged into a 6+2EC object with a block size of 4KB using the Container mechanism, and then during the I / O process, the Cache system will be prompted not to flush the disk frequently. In this way, data accessed by the same application can be put together as much as possible according to the information of application A, making full use of spatial and temporal locality. Among them, 6 in 6+2EC represents the number of data blocks in EC, and 2 represents the number of check blocks in EC; in this way, there are relatively more check data, which can improve data reliability.

[0263] The cache policy engine formulates a cache policy based on the characteristics of application A, which is frequent random small I / O, frequent modification, small data set, and relatively large I / O volume. The cache policy is to use full cache to improve the hit rate of application A's read data, batch write operations to increase the chance of merging, and reduce the number of I / Os sent to the backend. Optionally, when the read operation of application A is random read, the cache policy engine can determine to close the pre-reading policy for application A and clean up the cache related to application A to avoid cache pollution and space waste.

[0264] The hierarchical storage strategy engine formulates a strategy of using cache and SSD to improve data reading and writing efficiency, considering that application A data sets are frequently used, do not age, and are small in size.

[0265] The policy execution engine, ie, the policy execution module, can query the policy of application A from the application policy library according to at least one of a node ID (NodeID), a job ID (JobID), or a process ID (Pid), and execute the acquired policy.

[0266] like Figure 7B As shown, the information of application B obtained by the application-aware interface is as follows:

[0267] Job ID: 65432;

[0268] Operation characteristics: No data update, always running;

[0269] Running nodes: N1, N2;

[0270] Process ID: 654,3297;

[0271] Job I / O type: Mostly sequential reads, sequential skip reads, read-only after writes; rarely writes.

[0272] It will not be read immediately after writing;

[0273] I / O block size: 1MB-4MB;

[0274] Access dataset: / path / to / appB / vid100000.avi,…

[0275] / path / to / appB / vid99999.avi;

[0276] I / O file size: 500MB-5 gigabytes (GB) per file, 4 total datasets

[0277] Terabyte (TB).

[0278] The characteristics of read-only application B are: most of them are sequential continuous reading and sequential skip reading, and application B rarely modifies the data after writing it, and will not read the data immediately, and the data to be read and written is large. After the application perception module obtains this information through the application perception interface, it passes this information to the policy determination module. The policy determination module formulates a policy for application B through the data layout policy engine, cache policy engine, and hierarchical storage policy engine.

[0279] Among them, because application B involves multi-node I / O and only reads data after writing it, the data layout strategy engine formulates the strategy: store it in the file of the directory accessed by application B and adopt a multi-copy mechanism, which is conducive to improving the efficiency of application B in reading data and ensuring data reliability. Since application B mostly reads sequentially and the written data will not be read immediately, the strategy determined by the cache strategy engine is: when the read operation is triggered, the sequential progressive retrieval is adopted, and the retrieval increment granularity is 5MB to improve the efficiency of application B in reading data; cache elimination adopts the least recently used (LRU) strategy to reduce the occupancy of the cache, and the write operation data bypasses the cache and is directly written to the HDD resource pool to avoid occupying other storage resources and improve the efficiency of writing data. Because the data volume of application B is large and is read in large blocks sequentially, the strategy determined by the hierarchical storage strategy engine is: aggressive cache retrieval and elimination, and data is persisted in the HDD storage pool.

[0280] The policy execution engine, ie, the policy execution module, can query the storage policy of application B from the application policy library according to at least one of the node ID (NodeID), the job ID (JobID), or the process ID (Pid), and execute the acquired storage policy.

[0281] It should be noted that Figure 7B The middle dotted line shows the method of writing the data that application B needs to write to the storage pool directly to the HDD, which is for the case where application B only writes once and reads data afterwards. If application B does not need to write data but only reads data, it is also not necessary to write data to the storage pool, that is, there is no write operation of application B writing data to the HDD storage pool.

[0282] like Figure 7C As shown, the information of application C obtained by the application-aware interface is as follows:

[0283] Job ID: 2467;

[0284] Job characteristics: Running for a long time on only one node;

[0285] Running node: N1;

[0286] Process ID: 3257

[0287] Job I / O type: write-only, written data is compressed text;

[0288] I / O block size: 64KB-1MB;

[0289] Access datasets: / path / to / appC / log123,…, / path / to / appC / longXXX;

[0290] I / O file size: 64KB-1GB for a single file, many files, and data rarely read after being written

[0291] Take, but store for a long time.

[0292] The characteristics of write-only application C are: it only writes data to the storage pool, and the written data is compressed text. After the application perception module obtains this information through the application perception interface, it passes this information to the policy determination module. The policy determination module formulates policies for application C through the data layout policy engine, the cache policy engine and the hierarchical storage policy engine. For example, because the amount of data that application C needs to write to the storage pool is large, it is rarely read after writing, and long-term storage is required. Therefore, the data layout policy engine can determine that the data to be stored by application C is stored in a 16+2 EC mode, where 16 represents the number of data blocks in EC and 2 represents the number of check blocks in EC; in this way, the check data occupies less space, and storage space is saved as much as possible on the basis of ensuring reliability. The cache policy engine determines that the data written by application C does not need to be cached in view of the write-only characteristics of application C, so as to avoid possible occupation of cache space. The hierarchical storage policy engine determines that the data to be written by application C is directly written to the HDD storage pool in view of the fact that application C is a write-only application and the data is rarely read after being written to the storage pool, so that the cache will not be occupied and the efficiency of data writing to the storage pool can be improved.

[0293] The policy execution engine, ie, the policy execution module, can query the storage policy of the application C from the application policy library according to at least one of the node ID (NodeID), the job ID (JobID), or the process ID (Pid), and execute the acquired storage policy.

[0294] The above is an explanation of the implementation of the data management method provided in the embodiment of the present application, taking the specific situations of three different applications as examples. The following is an example of an application running on multiple nodes to further explain an implementation of the data management method provided in the embodiment of the present application.

[0295] Figure 8 A logical diagram of a data layout decision process provided by an embodiment of the present application. Figure 8 As shown, after the application perception module obtains the information of the application, the policy determination module first determines whether the application is running on one node or on multiple nodes when determining the policy of the application.

[0296] When the application runs on a node, determine whether the application has reliability requirements for accessing the data. If there is no reliability requirement, the data layout strategy of the application can be determined as a single copy strategy to reduce the storage space occupied. If the application has requirements for data reliability, determine the size of the data to be stored by the application. When the data is less than 1GB, determine a 2-copy strategy, so that the degree of data reliability is improved without occupying too much storage space. If the data is larger than 1GB, determine the EC layout strategy, so as to avoid excessive occupation of storage space. It can be understood that the aforementioned 1GB and 2 copies are merely examples, and do not constitute a limitation on the embodiments of the present application.

[0297] When the application runs on multiple nodes, the policy determination module first determines whether the application is a read-write hybrid application. If the application is a write-only application, the number of replicas of the data to be written is determined to be the number of nodes on which the application runs to improve data reliability. If the application is a read-write hybrid application, it is determined whether there is a conflict between its read and write operations. When there is a conflict, the EC data layout strategy is determined. When there is no conflict, the replica data layout strategy is adopted.

[0298] In one possible implementation, Figure 8 The scenario shown may also include a scheduler for scheduling the jobs of the application. The scheduler may schedule applications with more I / O to execute on more nodes according to the I / O conditions of each job in the cluster, and avoid I / O conflicts between different jobs.

[0299] Understandably, Figure 8 This is just an example of a data layout decision engine. In actual implementation, a specific strategy may also be determined based on the characteristics of the storage system.

[0300] The solution provided in the embodiment of the present application is that the application perception module receives information from the application through a series of interfaces, the policy determination module formulates an appropriate storage strategy based on the application information, and the policy execution module takes targeted measures for the application request, thereby optimizing data storage, improving data access efficiency, and improving the data access performance of the storage system.

[0301] Some storage policies are set by default in common storage systems. For example, the default cache policy of the cache system involves water level, disk flushing timing, cache amount, pre-reading, etc. The storage backend will set some stripe size, EC ratio, etc. by default. Generally speaking, these default settings are only suitable for individual scenarios. In a production environment, manual testing, continuous attempts at various parameters, and configuration of the system based on empirical values ​​are required to achieve suboptimal performance. Moreover, once set, the parameters are difficult to change. When the business model changes, performance will be "degraded".

[0302] In the above implementation provided by the embodiment of the present application, an application perception module is added to the storage system, which can receive information from the application in real time, detect changes in the application's needs and access patterns, and adjust the behavior of the storage system based on this information, optimize the access path and storage method of the data, so that the storage system can flexibly adapt to various scenarios. For example, for scenarios that are not suitable for using Cache, such as large block (1MB+) write-only applications, the cache layer is directly bypassed, and the data is written to the storage backend using DirectIO technology. This not only saves a lot of Cache space, but also shortens the I / O path, improves Cache utilization, and reduces overall latency. For data access with the number of read / write operations per second (input / output operations per second, IOPS), a suitable cache strategy can be adopted, such as full cache + LRU elimination, and a suitable data layout (such as small file packaging) to improve data access performance. For read-only data, an effective pre-reading strategy can be adopted to load data into the cache in advance, shorten the response time, and improve IOPS.

[0303] Fig. 9 1 is a flow chart of a data management method according to an embodiment of the present application. The data of the application is stored in a storage system, and the storage system includes at least one storage node. Fig. 9 As shown, the method includes:

[0304] Step 900: Acquire application information, where the application information includes at least one of the following information: I / O operation information of the application, running information of the application, and process information of the application;

[0305] Step 902: Formulate a storage strategy for the application according to the information of the application;

[0306] Step 904: Execute the storage policy.

[0307] The above method obtains application information and formulates a storage strategy for the application based on the application information, which is used for the storage and read-write processing of the data read and written by the application, optimizes the storage of the data read and written by the application in the storage system and / or improves the efficiency of the application reading and writing data. Compared with the prior art that configures policies applicable to all applications based on a preset method, the method can formulate different storage strategies for different applications, improve the efficiency of application access, optimize the data storage method and improve the resource utilization of storage devices, and effectively improve the compatibility of data storage and data reading and writing for different applications. For example, for scenarios that are not suitable for using Cache, such as large data block write-only applications, directly bypass the cache layer and use DirectIO technology to write data to the storage backend. This not only saves a lot of Cache space, but also shortens the I / O path, improves Cache utilization, and reduces overall latency.

[0308] Fig. 9 The specific implementation of the method shown can be implemented by referring to the implementation of the embodiment of the present application, for example, Figure 4 The implementation method shown in the figure can also be used to refer to the above Figure 5 , Figure 6 , Figure 7A-7B as well as Figure 8 The specific implementation method shown is used for implementation and will not be described in detail.

[0309] Fig.10 Schematic diagram of a data management system 1000 provided in an embodiment of the present application. Fig.10 As shown, the data management system 1000 includes an application perception module 1001, a policy determination module 1002 and a policy execution module 1003, wherein:

[0310] The application sensing module 1001 is used to obtain application information, wherein the application information includes at least one of the following information: I / O operation information of the application, running information of the application, and process information of the application; wherein the data of the application is stored in a storage system, and the storage system includes at least one storage node;

[0311] The policy determination module 1002 is used to formulate a storage policy for the application according to the information of the application;

[0312] The policy execution module 1003 is used to execute the storage policy.

[0313] The above-mentioned data management system 1000 can optimize the storage of the data read and written by the application in the storage system and / or improve the efficiency of the application access by obtaining the information of the application and formulating a storage strategy for the application according to the information of the application. Compared with the prior art that configures a strategy applicable to all applications based on a preset method, the data management system 1000 can formulate different storage strategies for different applications, which can improve the efficiency of reading and writing application data, optimize the data storage method and improve the resource utilization of the storage device, effectively improving the compatibility of data storage and data reading and writing for different applications.

[0314] Fig.10 The specific implementation of the data management system 1000 shown in the figure can be implemented by referring to the implementation of the embodiment of the present application, for example, Figure 3A-Figure 3F , or the above Figure 4 The implementation method shown in the figure can also be used to refer to the above Figure 5 , Figure 6 , Figure 7A-7B as well as Figure 8 The specific implementation method shown is used for implementation and will not be described in detail.

[0315] Fig.11 A schematic diagram of the structure of a computer device 2000 provided in an embodiment of the present application is shown in FIG. Fig.11 As shown, the system includes a processor 2002 and a memory 2001, and the processor 2002 and the memory 2001 are connected via a bus 2003. The memory 2001 is used to store a computer-readable executable program, and the processor 2002 is used to execute the computer-readable executable program in the memory 2001 to implement the above Figure 4 or Fig. 9 For example, the processor 2002 is used to read the computer-readable executable program in the memory 2001 to perform the following steps:

[0316] Acquire application information, the application information including at least one of the following information: input / output (I / O) operation information of the application, running information of the application, and process information of the application; wherein data of the application is stored in a storage system, and the storage system includes at least one storage node;

[0317] formulating a storage strategy for the application according to the information of the application;

[0318] The storage policy is executed.

[0319] Understandably, Fig.11The processor 2002 shown may be a processor located in a different device, for example, the processor 2002 may be a processor located in a distributed storage system client, or a processor in a distributed storage system storage node. Fig.11 The memory 2001 shown may be a memory located in a different device. For example, the memory 2001 may be a memory located in a distributed storage system client or a memory in a storage node of the distributed storage system.

[0320] The computer device 2000 can formulate a storage strategy for the application based on the information of the application, optimize the storage of the data read and written by the application in the storage system and / or improve the efficiency of the application in reading and writing data. Compared with the prior art that configures a strategy applicable to all applications based on a preset method, the computer device can formulate different storage strategies for different applications, effectively improving the compatibility of data storage and data reading and writing for different applications.

[0321] Fig.11 The specific implementation of the data management system 2000 shown in the figure can be implemented by referring to the implementation of the embodiment of the present application, for example, Figure 3A-Figure 3F , or the above Figure 4 The implementation method shown in the figure can also be used to refer to the above Figure 5 , Figure 6 , Figure 7A-7B as well as Figure 8 The specific implementation method shown is used for implementation and will not be described in detail.

[0322] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0323] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0324] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.

[0325] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.

[0326] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0327] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0328] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A data management method, characterized in that: The method comprises: Acquire application information, the application information including at least one of the following information of the application: input / output (I / O) operation information, operation information, and process information; wherein the application data is stored in a storage system, the storage system including one or more storage nodes; formulating a storage strategy for the application according to the information of the application; The storage policy is executed.

2. The method according to claim 1, characterized in that The method further comprises: The storage policy is formulated for the application before the application is run or when a data processing request of the application is received.

3. The method according to claim 1 or 2, characterized in that: The method further includes: acquiring characteristic information of the storage system; The storage policy is formulated for the application according to the information of the application and the characteristic information of the storage system.

4. The method according to claim 3, characterized in that The characteristic information of the storage system includes at least one of the following information: the hardware structure of the storage system, the hardware composition of the storage system, the storage capacity of the storage system, or the data layout method of the storage system.

5. The method according to any one of claims 1 to 4, characterized in that: The storage policy of the application includes a data layout policy, and the data layout policy is used to determine the storage method and / or storage location of the data written by the application to the storage system.

6. The method according to claim 5, characterized in that The storage method in which the application is written into the storage system includes at least one of the following information: The structure type of the data that the application needs to store, the number of copies of the data that the application needs to store, or the verification method of the data that the application needs to store.

7. The method according to any one of claims 1 to 6, characterized in that: The storage strategy of the application includes a cache strategy, which is used to determine the data read by the application from the storage system and the cache method of the read data. The cache method of the data includes at least one of the following: full cache, partial cache, pre-read mode, direct cache when writing data, or a cache cleanup mechanism.

8. The method according to any one of claims 1 to 7, characterized in that: The storage strategy of the application further includes a hierarchical storage strategy, and the hierarchical storage strategy is used to store data of different frequencies accessed by the application in storage media with different read and write speeds.

9. The method according to any one of claims 1 to 8, characterized in that: The storage strategy of the application also includes a redundancy strategy, and the redundancy strategy is used to determine the backup method and / or backup quantity of the data to be written into the storage system by the application.

10. The method according to any one of claims 1 to 9, characterized in that: Pre-save reference strategies; The method further comprises: The storage policy is formulated for the application with reference to the reference policy.

11. The method according to any one of claims 1 to 10, characterized in that: The method further includes storing a storage policy of the application.

12. The method according to any one of claims 1 to 11, characterized in that: The application information is information that has been normalized according to a preset format and / or content.

13. The method according to any one of claims 1 to 12, characterized in that: The executing the storage policy includes: executing part of the storage policy before the application initiates a read or write request.

14. The method according to claim 13, characterized in that The storage strategy executed before the application initiates a read / write request includes a hierarchical storage strategy, and the hierarchical storage strategy is used to store the data to be read by the application in a storage medium with a high read / write speed in advance.

15. The method according to any one of claims 1 to 14, characterized in that: The method further includes: obtaining information of the application from the application through an extended interface; or, The formatted information of the application is obtained from a computing device on which the application is running.

16. The method according to claim 15, characterized in that The obtaining the information of the application from the application through the extended interface includes one of the following methods: Obtain application information through the extended interface library; The information of the application is obtained through a dedicated file, and the application or a scheduler corresponding to the application writes the information of the application into the dedicated file; Acquiring information of the application through a remote procedure call; or, The information of the application is obtained through RESTful representation state transfer.

17. A data management system, characterized in that: The system comprises: An application sensing module, used to obtain application information, wherein the application information includes at least one of the following information of the application: input / output (I / O) operation information, operation information, and process information; wherein the application data is stored in a storage system, and the storage system includes one or more storage nodes; A policy determination module, used to formulate a storage policy for the application according to the information; A policy execution module is used to execute the policy.

18. The data management system according to claim 17, characterized in that: Before the application is run or when a data processing request from the application is received, the policy determination module formulates the storage policy for the application.

19. The data management system according to claim 17 or 18, characterized in that: The policy determination module is further used to obtain characteristic information of the storage system, and formulate the storage policy for the application according to the information of the application and the characteristic information of the storage system.

20. The data management system according to claim 19, characterized in that: The characteristic information of the storage system includes at least one of the following information: the hardware structure of the storage system, the hardware composition of the storage system, the storage capacity of the storage system, or the data layout method of the storage system.

21. The data management system according to any one of claims 17 to 20, characterized in that: The storage policy of the application includes a data layout policy, and the data layout policy is used to determine the storage method and / or storage location of the data written by the application to the storage system.

22. The data management system according to claim 21, characterized in that: The storage method in which the application is written into the storage system includes at least one of the following information: The structure type of the data that the application needs to store, the number of copies of the data that the application needs to store, or the verification method of the data that the application needs to store.

23. The data management system according to any one of claims 17 to 22, characterized in that: The storage strategy of the application includes a cache strategy, which is used to determine the data read by the application from the storage system and the cache method of the read data. The cache method of the data includes at least one of the following: full cache, partial cache, pre-read mode, direct cache when writing data, or a cache cleanup mechanism.

24. The data management system according to any one of claims 17 to 23, characterized in that: The storage strategy of the application further includes a hierarchical storage strategy, and the hierarchical storage strategy is used to store data of different frequencies accessed by the application in storage media with different read and write speeds.

25. The data management system according to any one of claims 17 to 24, characterized in that: The storage strategy of the application also includes a redundancy strategy, and the redundancy strategy is used to determine the backup method and / or backup quantity of the data to be written into the storage system by the application.

26. The data management system according to any one of claims 17 to 25, characterized in that: The policy determination module is also used to refer to a pre-saved reference policy to formulate the storage policy for the application.

27. The data management system according to any one of claims 17 to 26, characterized in that: The application information is information that has been normalized according to a preset format and / or content.

28. The data management system according to any one of claims 17 to 27, characterized in that: The executing the storage policy includes: executing part of the storage policy before the application initiates a read or write request.

29. The data management system according to any one of claims 17 to 28, characterized in that: The strategy determination module is also used for: Obtaining information of the application from the application through an extended interface; or, The formatted information of the application is obtained from a computing device on which the application is running.

30. The data management system according to claim 29, characterized in that: The policy determination module obtains the information of the application from the application through the extended interface in one of the following ways: Obtain application information through the extended interface library; The information of the application is obtained through a dedicated file, and the application or a scheduler corresponding to the application writes the information of the application into the dedicated file; Acquiring information of the application through a remote procedure call; or, The information of the application is obtained through RESTful representation state transfer.

31. A computer device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer-readable executable program, and the processor is used to read the computer-readable executable program to execute the steps of any one of the methods of claims 1 to 16.

Images