Data management method and apparatus, device, and readable storage medium

Abstract

Disclosed are a data management method and apparatus, a device, and a readable storage medium, relating to the technical field of storage. The method is used for a computing device, and the computing device reads and writes target data when running an application, the application comprising attribute information of the target data. On this basis, before the computing device reads and writes the target data, the attribute information of the target data is acquired, a management policy for the target data is determined, and the determined management policy is executed for the target data. Because the management policy is determined on the basis of the attribute information of the target data, different management policies can be formulated for different applications, and the degree of match between the management policies and the applications can be improved, so that the management policies can meet different requirements of different applications for I / O performance. Thus, the I / O performance of the computing device is effectively improved, the time for I / O operation is shortened, and the I / O performance is prevented from becoming a performance bottleneck for the computing device, thereby effectively improving the running efficiency of applications, and achieving the effect of accelerating applications.

Description

Data management method, device, equipment and readable storage medium

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on June 30, 2023, with application number 202310801356.4 and application name “Data management method, device, equipment and readable storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of storage technology, and in particular to a data management method, apparatus, device, and readable storage medium. Background Art

[0003] Currently, to improve the input / output (I / O) performance of computing devices, a data optimization policy is typically set for the device's I / O data (i.e., the data read and written by the device). This data optimization policy is then used to optimize the device's I / O data when running different applications.

[0004] However, this method is difficult to meet the different I / O performance requirements of computing devices when running different applications, resulting in an inability to effectively improve the operating efficiency of applications.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a data management method, apparatus, device, and readable storage medium, which can effectively improve the operating efficiency of applications.

[0007] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0008] In a first aspect, a data processing method is provided, which is applied to a computing device, and reads and writes target data while the computing device runs an application program, and the application program includes attribute information of the target data; the method comprises: obtaining the attribute information of the target data; determining a management policy for the target data based on the attribute information; and executing the management policy on the target data.

[0009] In this solution, the target data is read while the computing device is running the application, and the target data's attribute information is configured in the application. Based on this, the computing device can obtain the application's attribute information while running the application, use it to determine the target data's management policy, and then execute the determined management policy on the target data.

[0010] On the one hand, since the management strategy of application data is determined based on the attribute information of the data, different management strategies can be determined for different applications. This helps to improve the matching degree between the management strategy and the application, so that the management strategy can meet the different requirements of different applications for I / O performance, thereby effectively improving the I / O performance of the computing device when running the application, reducing the time of I / O operations, and avoiding I / O performance becoming a performance bottleneck of the computing device, thereby effectively improving the operating efficiency of the application and achieving the effect of application acceleration.

[0011] On the other hand, writing data attribute information into the application allows it to be automatically retrieved when the application is running, making it easier and less difficult to obtain attribute information. Writing target data attribute information into the application allows for the definition of management policies for specific application data (i.e., target data). For example, for data that significantly impacts I / O performance when running an application, this not only provides a more targeted management policy (i.e., a management policy for the target data) but also reduces the difficulty of data management for the application, compared to defining a management policy for all application data.

[0012] In addition, after the computing device obtains the attribute information of the target data, it can automatically determine the management policy of the target data. Compared with the user specifying the management policy for the target data, it helps to improve the efficiency of determining the management policy and reduce the requirements for the user when determining the management policy.

[0013] In a possible implementation, the application further includes a data description interface; and obtaining the attribute information of the target data includes: obtaining the attribute information of the target data based on the data description interface.

[0014] In this implementation, by configuring the data description interface in the application, the attribute information in the application is obtained by calling the data description interface in the application during the running of the application. On the one hand, it helps to reduce the difficulty of obtaining attribute information. On the other hand, compared with modifying the native interface in the application to obtain data description information, it can not only avoid interfering with the native function of the native interface, but also distinguish the interface used to obtain attribute information from other interfaces in the application (such as the I / O interface for reading and writing data) to avoid related interference.

[0015] In another possible implementation, the application also includes running stage information, which is used to indicate the running stage of the application; based on the attribute information of the target data, the management strategy of the target data is determined, including: based on the attribute information and running stage information of the target data, the management strategy of the target data is determined.

[0016] In this implementation, the management strategy of the target data is determined based on the attribute information of the target data in combination with the operation stage information, which enriches the parameters used to determine the management strategy and helps to improve the accuracy of the determined management strategy.

[0017] In another possible implementation, the running stage information includes first stage information; based on the attribute information of the target data, the management strategy of the target data is determined, including: obtaining the first stage information; the first stage information is used to indicate that the application is in the first running stage; if the first running stage is earlier than the target running stage, the management strategy of the target data is determined based on the attribute information of the target data; the target running stage is the running stage of the application when the target data is read and written.

[0018] This implementation method provides an implementation method for determining the management policy of target data before reading and writing the target data. Specifically, the running stage information configured in the application includes first stage information, indicating that the application is in the first running stage. Based on this, when the first stage information is obtained, it means that the application has run to the first running stage. When the first running stage is earlier than the target running stage, the management policy of the target data is determined. This can achieve the determination of the management policy before the target data is read and written, thereby helping to ensure that the management policy is executed on the target data when reading and writing the target data or before reading and writing the target data, thereby helping to avoid the inability to read and write based on the management policy when reading and writing the target data.

[0019] In another possible implementation, the computing device also reads and writes non-target data while running the application program, and the impact of the non-target data on the I / O performance is smaller than that of the target data.

[0020] In this implementation, writing the attribute information of target data into the application, compared to writing the attribute information of non-target data into the application, allows for the development and management of management policies for data that has a greater impact on I / O performance during application execution, thereby more effectively improving the I / O performance of the computing device. Furthermore, compared to writing the attribute information of both target and non-target data into the application, this helps reduce the amount of data that requires management.

[0021] In another possible implementation, the computing device also reads and writes non-target data while running the application program, and the storage space occupied by the non-target data is smaller than the storage space occupied by the target data.

[0022] In this implementation, writing the attribute information of target data into the application, compared to writing the attribute information of non-target data into the application, allows for the development and management of management policies for large amounts of data (i.e., data that occupies a large amount of storage space) read and written during application execution, thereby more effectively improving the I / O performance of the computing device. Furthermore, compared to writing the attribute information of both target and non-target data into the application, this helps reduce the amount of data that requires management.

[0023] In another possible implementation, the computing device is configured with a first storage area, and the read and write speed of the first storage area is greater than the read and write speed of the non-first storage area configured with the computing device; the attribute information indicates that the operation type of the target data includes a write operation; and the management policy includes writing the target data to the first storage area when performing a write operation on the target data.

[0024] In this implementation, by setting a management policy for the target data, including writing the target data to the first storage area when performing a write operation on the target data, compared to writing the target data to a non-first storage area, it helps to increase the write speed of the target data, reduce the I / O time when running the application, and thus help improve the I / O performance of the computing device when running the application.

[0025] In another possible implementation, the attribute information indicates that the target data is a permanent file, and the management strategy includes flushing the target data written in the first storage area to a non-first storage area.

[0026] In this implementation, when the target data is a permanent file, setting a management policy includes flushing the target data written in the first storage area to a non-first storage area and deleting the target data in the first storage area, thereby preventing the target data from occupying the storage space of the first storage area for a long time, thereby helping to improve the utilization rate of the fast storage area.

[0027] In another possible implementation, the attribute information indicates that the operation type of the target data includes a read operation and the stage in which the target data is read is a target operation stage; the management strategy includes pre-reading the target data to a first storage area of ​​the computing device in an operation stage before the target operation stage.

[0028] In this implementation, the running stage when the target data is read (i.e., the target data stage) is indicated by setting attribute information. In this way, when formulating a management strategy for pre-reading target data, it can be clearly stated in which running stages of the application the pre-reading is performed (i.e., the running stage before the target running stage), thereby helping to ensure the timely execution of the pre-reading management strategy, that is, pre-reading the target data into the first storage area before the target data is read.

[0029] In another possible implementation, the management policy further includes deleting the target data in the first storage area after the life cycle of the target data ends.

[0030] In this implementation, useless target data can be prevented from occupying the storage space of the first storage area without affecting the normal operation of the application, thereby helping to improve the utilization rate of the first storage area.

[0031] In another possible implementation, the management strategy further includes deleting the target data pre-read into the first storage area in an operation phase after the target operation phase.

[0032] In this implementation, since the target data reading operation phase is the target operation phase, this means that the target data will no longer be read after the target operation phase ends. Based on this, by setting the management policy to include deleting the target data in the first storage area during the operation phase after the target operation phase, it is possible to prevent useless target data from occupying the storage space of the first storage area without affecting the normal operation of the application, thereby helping to improve the utilization rate of the first storage area.

[0033] In another possible implementation, executing the management policy on the target data includes executing the management policy on the target data based on the operation stage information.

[0034] In this implementation, when executing a management policy on target data, combining the running stage information in the application program helps to ensure the accuracy and timeliness of the execution of the management policy.

[0035] In another possible implementation, the running stage information also includes second-stage information; executing a management policy on the target data includes: obtaining the second-stage information; the second-stage information is used to indicate that the application is in the second running stage; if the second running stage is earlier than the target running stage, pre-reading the target data into the first storage area; if the second running stage is later than the target running stage, deleting the target data pre-fetched into the first storage area.

[0036] In this implementation, when the second-stage information is obtained, it indicates that the application has entered the second operating stage. If the second operating stage is earlier than the target operating stage, the target data is pre-read into the first storage area, thereby achieving the goal of pre-reading the target data into the first storage area before reading the target data in the operating stage, thereby facilitating the target data to be read from the first storage area at any time during the target operating stage. If the second operating stage is later than the target operating stage, the target data pre-fetched into the first storage area is deleted, thereby achieving the goal of timely deleting the target data in the first storage area after the target data's usage cycle ends, thereby helping to improve the utilization rate of the first storage area.

[0037] In another possible implementation, determining the management policy of the target data based on the attribute information of the target data includes: determining the management policy of the target data based on the attribute information of the target data and the storage capacity of each tier storage device in the first storage area.

[0038] In this implementation, when determining the management strategy corresponding to a read operation or a write operation, the storage capacity of each hierarchical storage device in the first storage area can be comprehensively considered to select a suitable placement location for the target data, thereby avoiding the situation where some hierarchical storage devices are overused while other hierarchical storage devices are not fully utilized.

[0039] In another possible implementation, a management policy for the target data is determined based on the attribute information of the target data and the storage capacity of each hierarchical storage device in the first storage, including: determining the management policy for the target data based on the attribute information and the first storage capacity of each hierarchical storage device in the first storage area; the first storage capacity is the storage capacity of each hierarchical storage device at the first moment; updating the management policy for the target data based on the second storage capacity of each hierarchical storage device; the second storage capacity is the storage capacity of each hierarchical storage device at the second moment, the second moment being later than the first moment.

[0040] In this implementation, before reading or writing target data, a target data management policy is first determined based on the first storage capacity of each tiered storage device. Subsequently, the target data management policy is updated based on the second storage capacity of each tiered storage device. Because the second storage capacity represents the most recent storage capacity of each tiered storage device, updating the target data management policy based on the second storage capacity helps ensure that the target data management policy is determined based on the most recent storage capacity of each tiered storage device, thereby helping to ensure the accuracy and rationality of the target data management policy.

[0041] In another possible implementation, the attribute information indicates the storage space occupied by the target data.

[0042] In this implementation, the storage space occupied by the target data is indicated by setting attribute information. In this way, when determining the management strategy based on the storage space occupied by the target data and the storage capacity of each level of storage devices, a suitable level can be determined for the target data in the first storage area, and it can be ensured that the determined level matches the size of the target data.

[0043] In another possible implementation, the management strategy includes writing or pre-reading the target data to a target storage device in the first storage area; and the available storage space of the target storage device is larger than the available storage space of the non-target storage devices.

[0044] In this implementation, a management policy is set to include writing or pre-reading target data to a target storage device in a first storage area, and the available storage space of the target storage device is larger than the available storage space of non-target storage devices. This helps to balance the available storage space of storage devices at each level, avoiding excessive use of some level storage devices and underuse of other level storage devices.

[0045] In another possible implementation, the attribute information indicates a target number, which is used to indicate the number of times the target data is read during the running of the application; the management policy also includes deleting the target data in the first storage area when the number of times the target data is read is equal to the target number.

[0046] In this implementation, the target number of times is indicated by setting attribute information, and the management policy includes deleting the target data in the first storage area when the number of times the target data is read is equal to the target number of times. In this way, during the process of the computing device running the application, when the actual number of times the computing device reads the target data is equal to the target number of times, that is, when the computing device no longer reads the target data, the target data in the first storage area can be deleted, thereby avoiding useless target data occupying the storage space of the first storage area without affecting the normal operation of the application, thereby helping to improve the utilization rate of the first storage area.

[0047] In another possible implementation, executing a management policy on the target data includes: executing a management policy on the target data based on I / O operation information of the target data; the I / O operation information is used to indicate one or more of the open operation, close operation and unlink operation performed on the target data.

[0048] In this implementation, since the I / O operation information indicates the I / O operations of the computing device on the target data, such as open operations, close operations, and unlink operations, executing management policies on the target data based on the I / O operation information helps to improve the timeliness and effectiveness of the execution of management policies.

[0049] In another possible implementation, a management strategy is executed on the target data based on the I / O operation information of the target data, including: obtaining the I / O operation information of the target data; determining the number of times the target is read based on the I / O operation information; if the number of times the target data is read is equal to the target number, deleting the target data pre-read into the first storage area.

[0050] In this implementation, since the I / O operation information indicates the I / O operations performed by the computing device on the target data, determining the current read count of the target data based on the I / O operation information helps improve the accuracy of the current read count. Furthermore, when the current read count equals the target read count, deleting the target data from the first storage area can determine that the computing device is no longer reading the target data, thereby improving the accuracy of management policy execution.

[0051] In another possible implementation, the application is run by multiple computing devices; the attribute information indicates that the target data is a shared file of the multiple computing devices, and the management strategy includes writing or pre-reading the target data to a shared storage device of the multiple computing devices.

[0052] In this implementation, when the target data is a shared file of multiple computing devices, the management strategy includes writing or pre-reading the target data to a shared storage device of multiple computing devices. In this way, multiple computing devices can all read the target data from the shared storage device, avoiding the need for multiple computing devices to store the target data separately, thereby avoiding duplicate storage.

[0053] In another possible implementation, the target data includes multiple data items, and the attribute information of the target data includes attribute information of the data items and / or attribute information of a group, wherein the group includes multiple data items with the same attributes.

[0054] In this implementation, since the smallest unit of reading and writing data during the running of the application is the data item, setting the attribute information of the target data to include the attribute information of the data item helps to improve the richness and completeness of the attribute information of the target data. A group includes multiple data items with the same attributes, and setting the attribute information of the target data to include the attribute information of the group can simultaneously describe the attribute information of multiple data items through the attribute information of the group. In this way, not only can the repeated writing of the same attribute information in the application be avoided, thereby avoiding application redundancy, but also the storage space occupied by the application can be reduced. In addition, the workload when setting the attribute information of multiple data items can be reduced. In addition, when multiple data items need to be managed, the attribute information of multiple data items can be obtained at one time, thereby helping to improve the efficiency of obtaining attribute information.

[0055] In another possible implementation, determining a management strategy for target data based on attribute information includes: creating a namespace for an application; the namespace includes the target data; and determining a management strategy for the target data in the namespace of the application based on the attribute information.

[0056] In this implementation, since different namespaces can set different management permissions, creating an application namespace and determining the management policy of the target data in the application namespace can make the management policy of the target data only applicable to the application. In this way, the management policies of different applications can be isolated from each other, thereby preventing the management policies of different applications from interfering with each other, and further preventing the I / O operations of different applications from interfering with each other when the computing device runs multiple applications at the same time.

[0057] In another possible implementation, the method further includes: performing a read operation or a write operation on the target data based on a native I / O interface in the application.

[0058] In this implementation, since read or write operations can be performed on target data based on the native I / O interface in the application, that is, the data process of the application can be managed without modifying the native I / O interface. Therefore, it is possible to avoid the need to perform a large number of operations to modify the native I / O interface when there are many I / O operations, thereby improving the speed of I / O operations. In addition, since there are many types of I / O interfaces in computing devices, and the data management method of the embodiment of the application can manage the data of the application without modifying the native I / O interface, it helps to ensure the scalability of the I / O interface of the computing device.

[0059] In a second aspect, a data management device is provided, comprising: functional units for executing any one of the methods provided in the first aspect, wherein the actions performed by each functional unit are implemented via hardware or via hardware executing corresponding software implementations. For example, the data management device may include: an acquisition unit, a determination unit, and an execution unit; the acquisition unit is configured to acquire attribute information of target data; the determination unit is configured to determine a management policy for the target data based on the attribute information; and the execution unit is configured to execute the management policy on the target data.

[0060] In a third aspect, a data management device is provided, comprising: a processor and a memory, wherein the processor is connected to the memory, the memory is used to store computer-executable instructions, and the processor executes the computer-executable instructions stored in the memory, thereby implementing any one of the methods provided in the first aspect.

[0061] In a fourth aspect, a computing device is provided, comprising: the data management apparatus provided in the second aspect, or the data management apparatus provided in the third aspect.

[0062] In a fifth aspect, a computer system is provided, which includes a computing device and a storage system; the computing device is used to obtain attribute information of target data; the computing device is also used to determine a management policy for the target data based on the attribute information; the computing device is also used to send the management policy to the storage system to instruct the storage system to apply the management policy to the target data.

[0063] In the fifth aspect, the computing device may also be used to execute any possible implementation of the first aspect.

[0064] In a sixth aspect, a chip is provided, comprising: a processor and an interface circuit; the interface circuit is used to receive code instructions and transmit them to the processor; and the processor is used to run the code instructions to execute any one of the methods provided in the first aspect above.

[0065] In a seventh aspect, a computer-readable storage medium is provided, which stores computer execution instructions. When the computer execution instructions are run on a computer, the computer executes any one of the methods provided in the first aspect above.

[0066] In an eighth aspect, a computer program product is provided, comprising: computer execution instructions, which, when executed on a computer, cause the computer to execute any one of the methods provided in the first aspect.

[0067] Among them, the technical effects brought about by any implementation method in the second to eighth aspects can refer to the technical effects brought about by different implementation methods in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] FIG1A is a schematic diagram of a computing device provided in an embodiment of the present application;

[0069] FIG1B is a schematic diagram of a computer system provided in an embodiment of the present application;

[0070] FIG2 is a schematic diagram of a storage system provided in an embodiment of the present application;

[0071] FIG3 is a schematic diagram of another computer system provided in an embodiment of the present application;

[0072] FIG4 is a schematic diagram of another computer system provided in an embodiment of the present application;

[0073] FIG5 is a schematic diagram of a data orchestration engine provided in an embodiment of the present application;

[0074] FIG6 is a flow chart of a data management method provided in an embodiment of the present application;

[0075] FIG7 is a flow chart of another data management method provided in an embodiment of the present application;

[0076] FIG8 is a schematic diagram of a data management principle provided by an embodiment of the present application;

[0077] FIG9 is a schematic diagram of attribute information of a group provided in an embodiment of the present application;

[0078] FIG10 is a schematic diagram of another data management method according to an embodiment of the present application;

[0079] FIG11 is a schematic diagram of a mobile data item provided in an embodiment of the present application;

[0080] FIG12 is a flowchart of an HPC application operation provided by an embodiment of the present application;

[0081] FIG13 is a flow chart of data management for an HPC application provided in an embodiment of the present application;

[0082] FIG14 is a schematic diagram of a data management device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0083] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0084] In the description of this application, unless otherwise specified, " / " indicates that the objects associated before and after are in an "or" relationship, for example, A / B can represent A or B; "and / or" in this application is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural.

[0085] Furthermore, in the description of this application, unless otherwise specified, "plurality" means two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0086] In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. Those skilled in the art will understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit differences. At the same time, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or design schemes. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way for easy understanding.

[0087] The following is a brief introduction to the relevant terms involved in the embodiments of this application.

[0088] High performance computing (HPC) refers to the computing power provided by supercomputers or computer clusters, mainly including extremely fast data processing capabilities and huge data storage capacity.

[0089] An application program is a computer program that is designed to complete one or more specific tasks.

[0090] Remote procedure call (RPC) is a method of inter-process communication that allows a program to remotely call a procedure or function in another address space without requiring the programmer to explicitly code the details of the remote call. This allows applications to call remote procedures and services as if they were local methods. RPC is applicable to many scenarios, including distributed services, distributed computing, and remote service calls.

[0091] The following is an illustrative introduction to the application scenarios of the embodiments of the present application.

[0092] With the large-scale emergence of high-performance computing (HPC) and the integration of artificial intelligence (AI) and HPC, the computational capacity of computing devices (such as those in supercomputing centers or computer clusters) is increasing, gradually transitioning to the exascale level, that is, capable of performing hundreds of billions of calculations per second. The computational scale of HPC is strongly correlated with the amount of I / O data (or I / O volume). In other words, the process of providing high-performance computing services by computing devices generates a large amount of I / O data. For example, the I / O data volume of exabytes (TB) to petabytes (PB) of I / O data for exascale computing devices can typically reach terabytes (TB) to petabytes (PB). Since the larger the amount of I / O data, the more time it takes to read and write I / O data, and the correspondingly lower computing performance, higher I / O performance can shorten this time. Therefore, the I / O performance of computing devices has an increasingly greater impact on computing performance.

[0093] In related technologies, to improve the I / O performance of computing devices, data optimization strategies are typically implemented for the device's I / O data. For example, I / O data is classified as hot data or cold data, and different optimization strategies are set for hot and cold data. Subsequently, when the computing device runs different applications, the device's I / O data is optimized using these data optimization strategies. However, this approach struggles to meet the varying I / O performance requirements of different applications, resulting in an inability to effectively improve application efficiency.

[0094] In view of this, an embodiment of the present application provides a data management method, which reads the target data in the process of the computing device running the application and configures the attribute information of the target data in the application. On this basis, the attribute information in the application can be obtained in the process of the computing device running the application, which is used to determine the management policy of the target data and execute the determined management policy on the target data. On the one hand, since the management policy of the application data is determined based on the attribute information of the data, different management policies can be determined for different applications. This helps to improve the matching degree between the management policy and the application, so that the management policy can meet the different requirements of different applications for I / O performance, thereby effectively improving the I / O performance of the computing device when running the application, reducing the time of I / O operations, and avoiding the I / O performance from becoming the performance bottleneck of the computing device, thereby effectively improving the operating efficiency of the application and achieving the effect of application acceleration.

[0095] On the other hand, writing data attribute information into the application allows it to be automatically retrieved when the application is running, making it easier and less difficult to obtain attribute information. Writing target data attribute information into the application allows for the definition of management policies for specific application data (i.e., target data). For example, data that has a greater impact on I / O performance when running an application than other data can provide a more targeted management policy (i.e., a management policy for the target data) and reduce the difficulty of data management for the application, compared to defining a management policy for all application data.

[0096] In addition, after the computing device obtains the attribute information of the target data, it can automatically determine the management policy of the target data. Compared with the user specifying the management policy for the target data, it helps to improve the efficiency of determining the management policy and reduce the requirements for the user when determining the management policy.

[0097] In addition, determining the management strategy based on the attribute information of the target data improves the accuracy of data management and achieves more fine-grained management of data compared to managing data based on cold data and hot data.

[0098] The following is an exemplary introduction to the system architecture of the embodiment of the present application.

[0099] As shown in FIG1A , it is a schematic diagram of a computing device provided in an embodiment of the present application.

[0100] 1A , a computing device 110 includes a processor 111. The computing device 110 is configured to run an application program.

[0101] The processor is the control center of the computing device 110 and is used to run applications. The processor can be a computing unit with computing capabilities, such as a central processing unit (CPU), a graphics processing unit (GPU), a data processing unit (DPU), a neural processing unit (NPU), and an embedded neural-network processing unit (NPU).

[0102] The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), system on chip (SOC) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or any conventional processor, etc.

[0103] It should be noted that the above is only an exemplary description, and the embodiments of the present application do not limit the specific form of the processor. Below, the embodiments of the present application are described by taking the processor as a GPU as an example.

[0104] In the embodiment of the present application, the computing device 110 can be a mobile phone, a tablet computer, a handheld computer, a PC, a cellular phone, a personal digital assistant (PDA), a wearable device (such as a smart watch, a smart bracelet, etc.), a smart home device (such as a television, etc.), a car computer (such as a car computer, etc.), a smart screen, a game console, headphones, AI speakers, augmented reality (AR) / virtual reality (VR) devices, an ultra-mobile personal computer (UMPC), a laptop computer, a netbook, a desktop computer or an all-in-one computer, etc.

[0105] It should be noted that the above is merely an exemplary description, and the embodiments of the present application do not limit the specific device form of the computing device 110.

[0106] In the embodiment of the present application, an input device 130 is further provided, and the input device 130 is connected to the computing device 110. The input device 130 is used to input attribute information of data, running stage information of application programs, etc. to the computing device 110.

[0107] For example, the attribute information of the data and the running stage information of the application can be stored in the application, so that the computing device 110 can obtain the attribute information of the data, the running stage information of the application, etc. from the application during the running of the application.

[0108] In the embodiment of the present application, a display device 140 is further provided, and the display device 140 is connected to the computing device 110. The display device 140 is used to display attribute information of data, running stage information of the application, and calculation results obtained by running the application.

[0109] In the embodiment of the present application, the computing device 110 may further include an access layer, which may include multiple levels of cache, such as a level 1 cache (L1 cache), a level 2 cache (L2 cache), and a level 3 cache (L2 cache). The access layer is located between the processor and the memory of the computing device and is connected to the processor and the memory respectively.

[0110] For example, when running an application, the processor can write data to the memory by first writing the data to the cache layer and then writing the data in the cache to the memory. When the processor reads data from the memory, it can first read the data in the memory to the cache layer and then read the data from the cache layer.

[0111] Based on this, the data management method provided in the embodiment of the present application can be used to manage data in the access layer.

[0112] FIG1B is a schematic diagram of a computer system according to an embodiment of the present application.

[0113] 1B , a computer system 100 includes a computing device 110 and a storage system 120. A processor 111 of the computing device 110 is connected to the storage system 120. The storage system is used to store application data.

[0114] The data management method provided in the embodiment of the present application can be used to manage data of applications in a storage system.

[0115] Thus, the data management method provided in the embodiments of the present application can be used to manage application data in a storage system, or it can also be used to manage data in an access layer. Below, the data management method provided in the embodiments of the present application is described using the data management method used to manage application data in a storage system as an example.

[0116] The application data includes data written to the storage system 120 and data read from the storage system 120 during the process of the computing device 110 running the application. The data written to the storage system 120 includes temporary data or permanent data generated during the process of running the application.

[0117] It can be understood that the data of the application can also be referred to as I / O data during the process of the computing device 110 running the application.

[0118] It is understood that writing data to the storage system 120 can be considered as performing a write operation on the data. Reading data from the storage system 120 can be considered as performing a read operation on the data. Data written to the storage system 120 can be referred to as an output file of the application, and data read from the storage system 120 can be referred to as an input file of the application.

[0119] In the embodiments of the present application, the application may be an application requiring high data processing capabilities, such as HPC applications for weather forecasting, aircraft manufacturing, energy exploration, collision simulation, equation solving, etc. (hereinafter referred to as HPC applications), or may be other types of applications, such as applications in the field of artificial intelligence. It will be understood that the above description is merely illustrative, and the embodiments of the present application do not limit the specific types of applications.

[0120] In the embodiments of the present application, the application can be stored in the storage system 120, or can also be stored in the computing device 110. In other words, the embodiments of the present application do not limit the storage location of the application. For ease of description, the following examples illustrate the embodiments of the present application using the example of an application stored in the storage system 120.

[0121] FIG2 is a schematic diagram of a storage system according to an embodiment of the present application.

[0122] 2 , storage system 120 includes multiple layers of storage devices. The multiple layers of storage devices may include a first-tier storage device and a second-tier storage device. The first-tier storage device may be a memory (MEM), and the second-tier storage device may be a solid-state drive (SSD).

[0123] In the embodiment of the present application, the storage system 120 may further include a controller. The controller in the storage system 120 is connected to the processor of the computing device 110 and is configured to receive read and write instructions and data management policies sent by the processor and execute the read and write instructions and data management policies.

[0124] In the embodiment of the present application, the multi-tier storage device may further include a third-tier storage device and a fourth-tier storage device. The third-tier storage device may be a hard disk drive (HDD) or a burst buffer (BB), and the fourth-tier storage device may be a parallel file system (PFS).

[0125] As you can understand, first-tier storage devices are the highest, second-tier storage devices are the next, third-tier storage devices are the next, and fourth-tier storage devices are the lowest. Higher-tier storage devices tend to have higher I / O speeds (also known as read and write speeds), lower latency, and smaller storage capacities than lower-tier storage devices.

[0126] It should be noted that the embodiments of the present application do not limit the number of tiers of storage system 120, and the above description is merely an example. In the following embodiments, the embodiments of the present application are described using an example in which multiple tiers of storage devices include a first tier storage device, a second tier storage device, a third tier storage device, and a fourth tier device.

[0127] The multi-tiered storage device may include a first storage area and a second storage area. The first storage area may be referred to as a fast storage area relative to the second storage area. The second storage area is typically a distributed storage structure and serves as the initial storage location for application data. The second storage area may also be considered a non-first storage area.

[0128] Illustratively, the first storage area may include a first-tier storage device, a second-tier storage device, and a third-tier storage device, and the second storage area may include a fourth-tier storage device.

[0129] In an embodiment of the present application, the storage system 120 can be integrated into a computing device, or it can be independent of the computing device. Alternatively, a portion of the hierarchical storage devices of the storage system 120 can be integrated into the computing device, while another portion of the hierarchical storage devices can be independent of the computing device.

[0130] FIG3 is a schematic diagram of another computer system provided in an embodiment of the present application.

[0131] 3 , a computer system 300 includes a computing device 310 and a storage system 320 , wherein a portion of the hierarchical storage devices of the storage system 320 are integrated with the computing device 310 , while another portion of the hierarchical storage devices are independent of the computing device 310 .

[0132] Computing device 310 includes a processor 3101, and a storage system includes a first-tier storage device 3201, a second-tier storage device 3202, a third-tier storage device 3203, and a fourth-tier storage device 3204. Processor 3101 is connected to first-tier storage device 3201, second-tier storage device 3202, third-tier storage device 3203, and fourth-tier storage device 3204, respectively, to facilitate writing data to each tier of storage device or reading data from each tier of storage device.

[0133] Among them, the first-tier storage device 3201 and the second-tier storage device 3202 are integrated into the computing device, and the third-tier storage device 3103 and the fourth-tier storage device 3104 are independent of the computing device.

[0134] It should be noted that the embodiment of the present application does not limit the number of levels of storage devices 310 integrated into the computing device and the number of levels of storage devices 310 independent of the computing device. The above is only an exemplary description.

[0135] In an embodiment of the present application, storage devices independent of the computing device 310, such as the third-tier storage device 3203, the fourth-tier storage device 3204, etc., can be mounted on the computing device 310. In this way, the storage device independent of the computing device 310 can be equivalent to the local storage device of the computing device 310.

[0136] In an embodiment of the present application, storage devices independent of the computing device 310, such as the third-tier storage device 3203, the fourth-tier storage device 3204, etc., can be used by a computing device alone, or can be shared by multiple computing devices in the computer system.

[0137] FIG4 is a schematic diagram of another computer system provided for the implementation of this application.

[0138] In the computer system, some hierarchical storage devices are shared by multiple computing devices in a computer cluster.

[0139] 4 , a computer system 400 includes multiple computing devices 410 and a storage system 420 . The multiple computing devices 400 include a first computing device 411 and a second computing device 412 . The first computing device 403 includes a first processor 413 , and the second computing device 412 includes a second processor 414 .

[0140] The storage system 420 includes a first-tier storage device 421 and a second-tier storage device 422 of the first computing device 411, a first-tier storage device 423 and a second-tier storage device 424 of the second computing device 412, and a shared storage device of the first computing device 411 and the second computing device 412, which includes a third-tier storage device 425 and a fourth-tier storage device 426.

[0141] The first-tier storage device 421 and the second-tier storage device 422 of the first computing device 411 are integrated into the first computing device 411. The first processor 413 is connected to the first-tier storage device 421 and the second-tier storage device 422, respectively.

[0142] The first-tier storage device 423 and the second-tier storage device 424 of the second computing device 412 are integrated into the second computing device 412. The second processor 414 is connected to the first-tier storage device 423 and the second-tier storage device 424, respectively.

[0143] The third-tier storage device 425 and the fourth-tier storage device 426 are shared by the first computing device 411 and the second computing device 412. The first processor 413 is connected to the third-tier storage device 425 and the fourth-tier storage device 426, respectively. The second processor 414 is connected to the third-tier storage device 425 and the fourth-tier storage device 426, respectively.

[0144] It should be noted that the embodiment of the present application does not limit the number of computing devices in the computer system, and the above computer system including two computing devices is only an example.

[0145] FIG5 is a schematic diagram of a data orchestration engine provided in an embodiment of the present application.

[0146] The data orchestration engine 500 is a software module for implementing the data management method provided in the embodiments of the present application. The data orchestration engine 500 can be executed by a computing device and / or a storage system to implement the data management method provided in the embodiments of the present application. For example, the data orchestration engine 500 can be executed by a processor of a computing device and / or a controller of a storage system.

[0147] 5 , the data orchestration engine 500 includes a data orchestration policy library (DOPL) module 510 , an application session manager (ASM) module 520 , a data placement engine 530 , a data movement engine 540 , and a metadata management module 550 .

[0148] Data Orchestration Policy Library Module 510: This module is used to obtain data attribute information, determine data management policies based on the data attribute information, periodically determine the data storage location, and uniformly manage the data lifecycle. This uniform management of the data lifecycle includes deleting data from the first storage area after the data lifecycle has expired.

[0149] The data orchestration strategy library module 510 includes a handler pool and a strategy pool.

[0150] The processor pool includes multiple processor modules, and different processor modules are used to process different information.

[0151] Exemplarily, the information processed by the processor module may include dynamic information and static information, wherein the static information includes attribute information and the like, and the dynamic information includes operation stage information and the like.

[0152] For example, processing of static information may include integrating different attribute information of one data, allocating attribute information of a group to each data in the group, etc.

[0153] The policy pool includes multiple management policies, such as pre-fetching to the first storage area, flushing to the non-first storage area, etc.

[0154] Application session management module 520: used to create a namespace for an application program, thereby preventing I / O operations and data management policies of different applications from interfering with each other.

[0155] In addition, the application session management module can provide an interface for the data orchestration policy module to access the database. That is, the data orchestration policy library module can read metadata in the database through the application session management module, or write metadata to the database.

[0156] Data placement engine 530: Determines the placement location of data in the initial data management policy determined by the data orchestration policy library module 510 and the capacity of each tier of storage devices in the storage system. For example, if the initial data management policy is to write data to the first storage area, the data placement engine 530 determines to place the data on the first tier of storage device in the first storage area and returns the identifier of the first tier of storage device to the data orchestration policy library module 510, so that the data orchestration policy library module 510 can determine the data management policy.

[0157] Data move engine 540: used to delete data items in the storage system, move data in the storage system, etc., for example, move data in the fourth-tier storage device to the second-tier storage device.

[0158] Metadata management module 550: used to record data information of the current session and data information of historical sessions, such as data written and read in the current session, and data written and read in historical sessions.

[0159] The current session refers to the entire lifecycle of the currently running application, and the historical session refers to the entire lifecycle of the previously running application.

[0160] It should be noted that the above division of the data orchestration engine and the naming of each module are only exemplary. The embodiments of the present application do not limit how to divide the data orchestration engine and how to name each module.

[0161] Exemplarily, the data orchestration policy library module 510 , application session management module 520 , data placement engine 530 , and metadata management module 550 may be executed by a computing device, and the data movement engine 540 may be executed by a storage system.

[0162] On this basis, embodiments of the present application also provide a data definition interface (DDI). The DDI can be stored in the application. The DDI is a software module that can be executed by the processor of the computing device and is used to receive data attribute information and / or group attribute information, and application runtime information; write the data attribute information and / or group attribute information and application runtime information into the application; and send the data attribute information and application runtime information to the data orchestration engine 500.

[0163] In an embodiment of the present application, by providing a data description interface, the attribute information of the specified data (i.e., the target data in the following embodiment) and the running stage information of the application are stored in a computing device or a storage system before running the application. In this way, the computing device can obtain the attribute information of the specified data and the running stage information of the application for indicating the stage of the application during the running of the application, thereby specifying a management policy for the specified data based on the data information of the specified data, or based on the attribute information and the running stage information of the specified data, and by executing the management policy on the specified data, the management policy is applied to the specified data, thereby managing the specified data.

[0164] On this basis, embodiments of the present application also provide a hierarchical storage management (HSM) module. The HSM module, executed by a processor of a computing device, is used to uniformly manage multiple layers of storage devices in a storage system. For example, it monitors the remaining storage space of each layer of storage devices based on the storage system's configuration information.

[0165] The configuration information of the storage system may be written to the computing device via a configuration file, for example, written to a daemon process and an application process of the computing device, so as to be used when the processor runs the HSM module.

[0166] Taking the above storage system as an example, HSM can be used to manage the MEM, SSD, HDD / BB and PFS of the storage system.

[0167] Exemplarily, the configuration information of the storage system includes information such as storage devices at each level of the storage system, the mount points of storage devices at each level, capacity, bandwidth, latency, and whether they are shared storage devices.

[0168] For example, the configuration information of the storage system included in the configuration file may be as follows:

[0169] The storage system described in the above configuration information includes four layers: memory, local disk, burst buffer, and pfs (parallel file system).

[0170] The mount point of memory is " / tmpfs", CAP (capacity) is 10240 bytes, BANDWIDTH (bandwidth) is 5000 mbps (megabits per second), LATENCY (latency) is 15 ms (milliseconds), and IS_SHARED_STORAGE (shared storage) is false.

[0171] The local disk's mount point is / local / tmp, its CAP (capacity) is 20480 bytes, its BANDWIDTH (bandwidth) is 300 Mbps, its LATENCY (latency) is 250000 ms, and its IS_SHARED_STORAGE (shared storage) is incorrect.

[0172] The burst buffer's mount point is / mnt / bb, its CAP is 40960 bytes, its BANDWIDTH is 150 Mbps, its LATENCY is 500000 ms, and its IS_SHARED_STORAGE is correct.

[0173] The mount point of pfs is " / pfs", CAP (capacity) is 102400 bytes, BANDWIDTH (bandwidth) is 70 Mbps, LATENCY (latency) is 1000000 ms, and IS_SHARED_STORAGE (shared storage) is correct.

[0174] It should be noted that the system architecture and application scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field can know that with the evolution of the system architecture and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0175] For ease of understanding, the data management method provided in the embodiment of the present application is introduced below in combination with the above system architecture.

[0176] The following embodiments of this application will be divided into three parts to exemplarily introduce the scheme of the data management method.

[0177] The first part, in conjunction with FIG6 , introduces the process of writing the attribute information of the target data and the running stage information of the application into the application.

[0178] The second part, in conjunction with FIG. 7 to FIG. 11 , introduces the process of managing application data based on the attribute information of the data in the first part and the running stage information of the application.

[0179] The third part, in conjunction with FIG. 12 to FIG. 13 , introduces the process of managing the data of the target application based on the first and second parts.

[0180] The following introduces the first part of the embodiment of the present application in conjunction with FIG6 .

[0181] Fig. 6 is a flow chart showing a data management method according to an exemplary embodiment. Exemplarily, the data management method includes S601-S602.

[0182] S601: Receive attribute information of target data and running stage information of the application.

[0183] The attribute information of the target data is determined by the user based on prior knowledge of the application program and is used to indicate the I / O attributes of the target data.

[0184] It is understandable that the prior knowledge of the application may include the data read and written at different running stages of the application, the number of times different data is read and written during the running of the application, the life cycle of different data of the application, and whether the data of the application is a shared file, etc.

[0185] Exemplarily, the attribute information of the target data may include the operation type, I / O times, life cycle, the running stage of the application when performing I / O operations on the target data, shared attributes, etc.

[0186] The operation type may include write operations and / or read operations. The I / O count refers to the number of times data is read and written during the running of the application. The life cycle is used to indicate whether the data is a permanent file or a temporary file. The running stage of the application when the I / O operation is performed on the data is used to indicate the running stage of the application when the read operation or write operation is performed on the data. The sharing attribute is used to indicate whether the data is a shared file or a non-shared file of multiple computing devices.

[0187] The running stage information of the application is information determined by the user based on prior knowledge of the application, and is used to indicate the current running stage of the application.

[0188] It can be understood that the prior knowledge of the application may also include the locations of different running stages in the application.

[0189] Exemplarily, the running stages of the application may include an initial running stage, an intermediate running stage, and a final running stage.

[0190] Since the attribute information of the target data is determined by the user based on prior knowledge of the application, the user can determine the data that needs to be managed (i.e., target data) based on specific I / O problems that arise during the running of the application, such as the impact of target data on I / O performance is greater than the impact of non-target data on I / O performance, such as data that has a greater impact on I / O performance, data that causes I / O performance to become a performance bottleneck of the computing device, etc., and write the attribute information of this part of data (i.e., target data) into the application, so that this part of data can be managed while the computing device is running the application. This not only helps to manage the application data more intelligently, more specifically and more accurately, thereby more effectively improving the I / O performance of the computing device, but also helps to reduce the amount of data that needs to be managed, thereby reducing the amount of data management tasks.

[0191] In the embodiments of this application, the application data can be divided into three levels from top to bottom: DataSet, Group, and Data Entity. A DataSet contains multiple Groups, and a Group contains multiple Data Entities. A DataSet has DataSet attributes, a Group has Group attributes, and a Data Entity has Data Entity attributes.

[0192] A data item is an independent data entity accessed during the execution of an application. For example, a file, an object, or a remote call result. A data item can also be called a data entity.

[0193] A group is a collection of multiple data items with the same attribute information. In other words, a group includes multiple data items with the same attribute information. The attribute information of a group is the same attribute information across the multiple data items in the group. A group can also be referred to as a data group.

[0194] A dataset is a collection of all data items read and written during the execution of an application. A dataset can also be referred to as the application's data. In other words, an application's data consists of multiple groups, and each group contains multiple data items.

[0195] When describing the attributes of a data item, users can describe the attributes of a single data item, or group multiple data items with the same attributes and then describe the attributes of the group. This helps reduce the user's workload when managing a large number of data items in an application.

[0196] Based on this, the attribute information of the data in S601 may include attribute information of the data item and / or attribute information of the group. In other words, when multiple data items need to be managed, some of the data items can be put into a group and the attribute information of the group can be used to describe the multiple data items at the same time, while the attribute information of other data items can be described separately.

[0197] It is understood that the target data is the data in the application that needs to be managed. The target data may include one data item, or may include multiple data items. The embodiment of the present application does not limit the number of data items included in the target data. In addition, the target data may also include at least one group. The embodiment of the present application also does not limit the number of groups included in the target data.

[0198] In an embodiment of the present application, a user can input attribute information of target data and running stage information of an application to a computing device through an input device, and the computing device receives the attribute information of the target data and running stage information of the application in response to the user's input operation.

[0199] Exemplarily, the attribute information and the operating stage information may be input to a processor of the computing device, and the processor receives the attribute information and the operating stage information in response to the input operation.

[0200] In addition, the computing device can also output the attribute information of the target data and the running stage information of the application to the display device, so that the display device can display the attribute information of the target data and the running stage information of the application input by the user. The user can view the input attribute information and running stage information through the display device, which helps to ensure the accuracy of the attribute information and running stage information.

[0201] S602: Based on the data description interface, write the attribute information and running stage information of the target data into the application.

[0202] In an embodiment of the present application, after the computing device receives the attribute information of the target data and the running stage information of the application, it writes the attribute information of the target data and the running stage information of the application into the application based on the data description interface.

[0203] Here, writing the attribute information of the target data and the running stage information of the application into the application can also be considered as writing into the source code of the application.

[0204] In this embodiment, the attribute information and runtime information of the target data are written into the application. In this way, when the computing device runs the application, the attribute information and runtime information of the target data can be directly obtained from the application. Compared with obtaining the attribute information and runtime information of the target data from its file, this helps to improve the convenience and efficiency of obtaining the attribute information and runtime information of the target attributes.

[0205] It can be understood that the attribute information and operation stage information of the target data in the embodiments of the present application can also be stored in the computing device or storage system.

[0206] In addition, the embodiments of the present application can also write the attribute information and runtime information of the target data into the application through other methods besides the data description interface. That is to say, the embodiments of the present application do not limit the method of writing the attribute information and runtime information of the target data into the application. The above is only an exemplary description.

[0207] Exemplarily, the processor of the computing device may write the attribute information and the runtime information into the application based on the data description interface.

[0208] In an embodiment of the present application, the running stage information can be written into different locations of the application, wherein the running stage information in different locations of the application is used to indicate different running stages of the application.

[0209] In an embodiment of the present application, data sets, groups, and data items can all have their attribute information described through a data description interface. That is, the computing device can write the attribute information of the data set, the attribute information of the group, and the attribute information of the data item into the application through the data description interface.

[0210] In the embodiment of the present application, the data description interface includes at least one of the following interfaces: DDI_Init (DDI initialization) interface, DDI_Create_Droup (DDI create group) interface, DDI_Add_Data_Entity (DDI add data attribute information) interface, DDI_Set_Stage (DDI set operation stage) interface, and DDI_Finalize (DDI end). Based on this, the data description interface can be considered as a collection of interfaces.

[0211] DDI_Init is used to initialize the DDI, or to initialize resources for a session / DataSet. DDI_Create_Droup is used to create a group, or to initialize resources for a drop. DDI_Add_Data_Entity is used to add data attributes. DDI_Set_Stage is used to set the application's running stage. DDI_Finalize is used by the DDI to indicate the end of a task.

[0212] In this embodiment, the data description interface is configured to include a DDI end interface, and the DDI end interface is added after " / / Application program execution ends". In this way, when the processor executes the DDI end interface, the DDI end interface can send task completion information to the data orchestration engine to notify the application program execution ends. Based on this, the data orchestration engine can delete the target data in the first storage area, thereby preventing useless data from occupying the first storage area, which helps to improve the utilization rate of the fast storage area.

[0213] For example, the data description interface in the application, the attribute information of the target data, and the running stage information of the application may be as follows:

[0214] / / 1.session(DataSet) initialization

[0215] DDI Init Para para;

[0216] para.proc_init = false;

[0217] std::unique_ptr <ddi>test_ddi=std::make_unioque <ddi>();

[0218] test_ddi->DDI_Init(para);

[0219] / / 2.Group creation

[0220] std::string groupName_1="group1";

[0221] GroupAttr test_groupAttr_1;

[0222] test_groupAttr_1.ioPattern=IOPattern::RANDOM;

[0223] ...(additional Group attributes)

[0224] test_ddi->DDI_Creat_Group(groupName_1,test_groupAttr_1);

[0225] / / 3. Add Data Entity1 information

[0226] EntityAttr entity1;

[0227] entity1.entityName=" / home / entity1.txt";

[0228] entity1.operation=Operation::READ;

[0229] entity1.scope=ScopeType::PROCESS;

[0230] entity1.ioStage=Stage::MIDDLE;

[0231] ...(other properties added)

[0232] test_ddi->DDI_Add_Data_Entity(entity1);

[0233] / / Add other Data Entity information

[0234] / / 4. Task running stage settings

[0235] test_ddi->DDI_Set_Stage(xxx)

[0236] / / Application calculation process

[0237] ......

[0238] Open / read / write / close...

[0239] / / Application ends

[0240] / / 5.DDI task ends

[0241] test_ddi->DDI_Finalize();

[0242] In the above code, / / 1. Session (DataSet) initialization is performed using the DDI_Init interface, / / 2. Group creation is performed using the DDI_Create_Droup interface, / / 3. Data Entity 1 information is added using the DDI_Add_Data_Entity interface, / / 4. The task execution stage is set using the DDI_Set_Stage interface, and / / 5. The DDI task ends using the DDI_Finalize interface. / / The application's calculation process and / / application execution end are represented by the application code.

[0243] It should be noted that the application may include both / / 2.Group creation and / / 3.Data Entity1 information addition, or may only include / / 2.Group creation or / / 3.Data Entity1.

[0244] During the application's execution, when the " / / 1.session(DataSet) initialization" portion of code is executed, the DDI_Init interface initializes the DDI session (dataset). When the " / / 2.Group creation" portion of code is executed, the DDI_Create_Droup interface is used to create a group, and the group's attribute information is sent to the data orchestration engine based on the DDI_Create_Droup interface. When the " / / 3.Data Entity1 information addition" portion of code is executed, the attribute information of the data item is sent to the data orchestration engine based on the DDI_Add_Data_Entity interface. When the " / / 4.Task running stage setting" portion of code is executed, the running stage information is sent to the data orchestration engine based on the DDI_Set_Stage interface. When the " / / 5.DDI task end" portion of code is executed, the DDI_Finalize interface is used to send task end information to the data orchestration engine to indicate the task end.

[0245] For example, when the DDI_Create_Droup interface creates a group, it includes: creating an identifier for the group, wherein the identifier of the group corresponds to the group one by one, that is, different groups have different identifiers.

[0246] In an embodiment of the present application, during the execution of an application, the computing device reads and writes target data and non-target data of the application, wherein the storage space occupied by the target data is greater than the storage space occupied by the non-target data.

[0247] In this embodiment, by receiving the attribute information of the target data and writing the attribute information of the target data into the application, a management policy can be determined for the target data and the target data can be managed, thereby achieving management of the large data (i.e., data that occupies a large storage space) that is read and written during the operation of the application. Compared with writing the attribute information of non-target data into the application, a management policy can be formulated and managed for the large data (i.e., data that occupies a large storage space) that is read and written during the operation of the application, which helps to more effectively improve the I / O performance of the computing device. Compared with writing the attribute information of both target data and non-target data into the application, this helps to reduce the amount of data that needs to be managed.

[0248] The second part of the embodiment of the present application is introduced below with reference to FIG. 7 to FIG. 11 .

[0249] Fig. 7 is a flow chart showing a data management method according to an exemplary embodiment. Exemplarily, the data management method includes S701-S703.

[0250] S701: Acquire attribute information of target data.

[0251] The data type of the target data may be file, audio, video, image or binary data, etc. It should be noted that the above description is only exemplary, and the embodiments of the present application do not limit the specific form of the target data.

[0252] In an embodiment of the present application, during the execution of an application program, before performing read and write operations on target data, the computing device obtains attribute information of the target data based on a data description interface.

[0253] Illustratively, while the processor of the computing device is running an application, it sends the attribute information of the target data to the data orchestration engine via the data description interface in the application. For example, the attribute information of a group may be sent to the data orchestration engine via the DDI_Create_Droup interface in the data description interface, or the attribute information of a data item may be sent to the data orchestration engine via the DDI_Add_Data_Entity interface in the data description interface.

[0254] Among them, sending the attribute information of the target data to the data orchestration engine through the data description interface can be considered as the processor calling the data description interface to send the attribute information of the target data to the data orchestration engine, or it can also be considered as the processor sending the attribute information of the target data to the data orchestration engine when running the data description interface.

[0255] In this embodiment, a user's prior knowledge of an application (i.e., target data attribute information and runtime information) is written into the application via a data description interface, thereby providing the target data attribute information and runtime information to the computing device. Furthermore, during the application's execution, the target data attribute information and application runtime information are sent to the data orchestration engine via the data description interface, thereby establishing a bridge between the user's prior knowledge and the data orchestration engine. This allows the data orchestration engine to formulate a management policy for the target data based on the target data attribute information and the application's runtime information, and manage the application's target data in accordance with the formulated management policy.

[0256] As shown in FIG8 , it is a schematic diagram of a data management method provided in an embodiment of the present application.

[0257] Referring to FIG8 , during the application program execution, the attribute information of the target data is sent to the data orchestration engine through the data description interface and the RPC service, for example, to the data orchestration policy library module of the data orchestration engine, so that the data orchestration engine can manage the target data through the file system.

[0258] In the embodiment of the present application, the attribute information includes multiple situations, which are explained below through situations 1 to 6.

[0259] Case 1: The attribute information includes an operation type of the target data, and the operation type includes a write operation and / or a read operation.

[0260] Exemplarily, the operation type of the target data is a write operation, which means that the target data is generated during the running of the application program. After the target data is generated, the processor writes the target data into the storage system.

[0261] It is understood that the target data generated during the running of an application can be temporary files, non-temporary files, and permanent files. Temporary files refer to files used once during the running of an application, while non-temporary files refer to files used throughout the life cycle of the application (i.e., used multiple times), usually configuration files. Permanent files refer to files output during the running of an application, usually used to record the running results of the application.

[0262] Exemplarily, the operation type of the target data is a read operation, which means that the target data is data stored in a storage system, and the target data will be read from the storage system during the running of the application program.

[0263] Exemplarily, the target data operation types are write and read operations, also known as a write-before-read type or a write-and-read type. This refers to the process of generating target data during application execution. After the target data is generated, the processor writes the target data to the storage system. Subsequently, during the application execution, the processor also reads the previously written target data from the storage system.

[0264] Case 2: The attribute information includes a target count, which indicates the number of times the target data is read during the running of the application.

[0265] Exemplarily, the target number of times may be 1, or may be N, where N is a positive integer greater than 1.

[0266] Case 3: The attribute information includes a target running phase, which is used to indicate a phase of performing an I / O operation on target data during the running of the application.

[0267] It can be understood that performing an I / O operation on target data can also be considered as performing a write operation / read operation on the target data.

[0268] When the target data operation type is a write operation, the first operation phase is an operation phase for executing the write operation on the target data. When the operation type is a read operation, the first operation phase can be considered an operation phase for executing the read operation on the target data.

[0269] Exemplarily, the running phases of the application program may include an initial phase, an intermediate phase, and an end phase. The target running phase may be any one of the initial phase, the intermediate phase, and the end phase.

[0270] It should be noted that the division of the above-mentioned application's running stages is only for illustrative purposes and is not limited in this embodiment of the present application.

[0271] By setting the attribute information to include the first operating stage, the application's usage requirements for the target data can be sent to the data orchestration engine, so that the data orchestration engine can determine the application's usage requirements for the target data based on the attribute information of the target data, that is, at which specific stage of the application the target data is used. Then, based on the application's usage requirements for the target data, the data orchestration engine can determine the target data management strategy and the timing for executing the management strategy on the target data, which helps to improve the accuracy and diversity of the target data management strategy.

[0272] Case 4: The attribute information includes a life cycle, which is used to indicate whether the data item is a permanent file or a temporary file.

[0273] Permanent files refer to files that require persistent storage, while temporary files refer to files that do not require persistent storage.

[0274] Case 5: The attribute information further includes a sharing attribute, which is used to indicate whether the target data is a shared file or a non-shared file.

[0275] In an embodiment of the present application, when an application is run by multiple computing devices, the sharing attribute of the target data is used to indicate whether the target data is a shared file or a non-shared file of the multiple computing devices.

[0276] Exemplarily, the multiple computing devices include a first computing device and a second computing device, and the target data generated by the first computing device during the execution of an application is data required to be used by the second computing device during the execution of the application, or the first computing device and the second computing device in the multiple computing devices both use the target data during the execution of the application, then the sharing attribute of the target data item indicates that the target data is a shared file, that is, the target data is a shared file of the first computing device and the second computing device.

[0277] Case 6: The attribute information includes the storage space occupied by the target data.

[0278] The storage space occupied by the target data may also be referred to as the size of the target data.

[0279] It should be noted that the attribute information of the target data may include one or more of the above-mentioned situations 1 to 6, and the embodiment of the present application does not limit this.

[0280] In an embodiment of the present application, the attribute information of the target data may further include the storage path of the target data. Thus, when a pre-read management policy is executed on the target data, the initial storage location of the target data can be determined based on the storage path of the target data, thereby pre-reading the target data from the initial storage location to the storage location specified in the management policy.

[0281] In the embodiment of the present application, the attribute information of the target data may include attribute information of the data item and / or attribute information of the group.

[0282] Herein, a group includes multiple data items with corresponding attribute information.

[0283] When the target data's attribute information includes the data item's attribute information, the target data includes the data item. Since the smallest unit of data read and written during application execution is a data item, setting the target data's attribute information to include the data item's attribute information helps improve the richness and completeness of the target data's attribute information.

[0284] When the target data's attribute information includes group attribute information, the target data includes multiple data items with corresponding attribute information. Since a group includes multiple data items with the same attribute information, setting the target data's attribute information to include the group's attribute information allows the group's attribute information to simultaneously describe the attribute information of multiple data items. This not only avoids duplicate storage of the same attribute information, thereby reducing the storage space occupied by the attribute information, but also reduces the workload associated with setting the attribute information for multiple data items. Furthermore, when multiple data items need to be managed, the attribute information for multiple data items can be obtained simultaneously, thereby improving the efficiency of obtaining attribute information.

[0285] FIG9 is a schematic diagram of the attribute information of a group provided in an embodiment of the present application.

[0286] 9 , the attributes of a group may include path (PAth), operation (Operation), open count (Opencount), life cycle (Lifecycle), stage (Stage), and the like.

[0287] Among them, the path is used to represent the native path of the target data (that is, the storage path), the business is used to represent the operation type of the target data, the number of opens is used to represent the number of times the target data is read and written during the running of the application, and the stage is used to represent the running stage of the application when reading and writing the target data.

[0288] It should be noted that the attribute information of the group shown in FIG9 is only an example. The attribute information of the group can refer to the relevant description of the attribute information of the target data mentioned above, which will not be repeated here.

[0289] For example, referring to FIG9 , the path is " / input / input.txt ( / input / input file)", that is, the target data (i.e., the input.txt file) is stored in the input directory. The operation is "read operation", that is, the operation type of the target data is a read operation. The number of times it is opened is "read only once", that is, the target data is read once during the running of the application. The life cycle is "a temporary file generated during the running of the application", that is, the target data is a temporary file, not a permanent file. The stage is "an I / O request is made during the first running stage of the application", that is, the target data is read during the first running stage of the application.

[0290] It can be understood that the target data in the group shown in Figure 9 has a native path of / input / input.txt. A read operation is performed on it during the running of the application, the number of reads is 1, the life cycle is short, it is a temporary file, and it is read in a specific running stage (i.e., the first running stage) of the running application.

[0291] In the embodiment of the present application, the attribute information and the value of the attribute information can be represented by a key-value pair. In particular, the attribute information and the value of the attribute information of a group, or the attribute information and the value of the attribute information of a data item can be represented by a key-value pair.

[0292] Exemplarily, the attribute information is the key in a key-value pair, and the value of the attribute information is the value in the key-value pair. Taking the attribute information shown in FIG9 as an example, the attribute information includes path, business, number of opens, life cycle, stage, etc., and the value of the attribute information includes / input / input.txt, read operation, read only once, temporary files generated during the application execution, and I / O request made during the first execution phase of the application execution.

[0293] S702: Determine a management strategy for the target data based on the attribute information of the target data.

[0294] In an embodiment of the present application, after the computing device obtains the attribute information of the target data, it determines a management strategy for the target data based on the attribute information of the target data.

[0295] Exemplarily, after receiving the target data attribute information sent via the data description interface, the data orchestration engine determines a management policy for the target data based on the target data attribute information. For example, a data orchestration policy library module of the data orchestration engine may receive the target data attribute information and determine the target data management policy. Alternatively, the data orchestration policy library module and the data placement engine may determine the target data management policy.

[0296] In the embodiment of the present application, a management strategy is formulated for the target data based on the attribute information of the target data, which not only helps to improve the matching degree between the management strategy and the target data, but also helps to improve the richness of the management strategy. In addition, the data management method provided in the embodiment of the present application can manage the target data, that is, all data corresponding to the attribute information in the application can be managed. Compared with managing only a small amount of data that is accessed frequently, by writing the attribute information of the data that has a greater impact on I / O performance into the application, more targeted management of improving I / O performance can be achieved, thereby further helping to improve I / O performance, and further helping to improve the operating efficiency of the application.

[0297] There are multiple implementations for determining the management strategy for target data. Two implementations are exemplified below.

[0298] In a first implementation manner, when a read or write instruction of target data is received, a management strategy for the target data is determined based on attribute information of the target data.

[0299] In the second implementation, the management strategy of the target data is determined based on the operation stage information and the attribute information of the target data. This implementation is exemplified below through the following steps 1 and 2.

[0300] The running stage information includes first stage information, which is used to indicate that the application is in the first running stage.

[0301] Step 1: Get the first stage information of the application.

[0302] In an embodiment of the present application, during the process of running the application, the computing device obtains the first-stage information of the application based on the data description interface.

[0303] Exemplarily, during the execution of an application by a processor of a computing device, the first stage information of the application is sent to the data orchestration engine via a data description interface. For example, the first stage information of the application may be sent to the data orchestration engine via a DDI_Set_Stage interface in the data description interface.

[0304] Step 2: If the first operation stage is earlier than the target operation stage, a management strategy for the target data is determined based on the attribute information.

[0305] The target running phase is the running phase of the application when the target data is read and written.

[0306] In an embodiment of the present application, if the first operating stage is earlier than the target operating stage, the computing device determines a management strategy for the target data based on attribute information of the target data.

[0307] Exemplarily, the data orchestration engine may determine the management policy of the target data based on the attribute information of the target data.

[0308] In an embodiment of the present application, it is possible to determine whether the current running stage of the application is earlier than the running stage of the application when the target data is read and written based on the target running stage indicated by the attribute information.

[0309] In this embodiment, the operating stage information configured in the application includes first stage information, indicating that the application is in the first operating stage. Based on this, when the first stage information is obtained, it can be determined that the application has reached the first operating stage. When the first operating stage is earlier than the target operating stage, the management policy is determined. This enables the management policy of the target data to be determined before the target data is read or written, thereby helping to ensure that the management policy is executed on the target data when or before the computing device reads or writes the target data, thereby helping to avoid failures to read or write the target data based on the management policy when reading or writing the target data.

[0310] In an embodiment of the present application, different attribute information of the target data can be used to determine different management strategies for the target data. The following, in combination with situations 1 to 6 corresponding to the attribute information of the target data in S701 above, various situations of the management strategy of the target data are exemplified through the following situations a to f.

[0311] In case a, the management strategy includes: when performing a write operation on target data, writing the target data into the first storage area.

[0312] Based on the fact that the operation type of the target data in the above situation 1 includes a write operation, the target data management strategy may include: when a write operation is performed on the target data, writing the target data into the first storage area.

[0313] In this way, when a write operation is performed on the target data, the target data can be written to the first storage area. Compared with writing the target data to a non-first storage area, this helps to increase the writing speed of the target data, thereby helping to reduce the time of I / O operations when running applications and improving the I / O performance of the computing device when running applications.

[0314] In case b, the management strategy includes: flushing the target data written in the first storage area to a non-first storage area.

[0315] Based on the fact that the operation type of the target data in the above-mentioned situation 1 includes a write operation and the life cycle in situation 4 indicates that the target data is a permanent file, the management strategy may include: flushing the target data written in the first storage area to a non-first storage area, and deleting the target data in the first storage area. This helps to avoid the permanent files written in the first storage area occupying the storage space of the first storage area, thereby helping to improve the utilization rate of the first storage area.

[0316] In case c, the management strategy includes: pre-reading the target data into the first storage area before performing a read operation on the target data.

[0317] Based on the fact that the operation type of the target data in the above situation 1 includes a read operation, the management strategy may include: pre-reading the target data into the first storage area before performing the read operation on the target data.

[0318] In this way, the target data can be pre-read into the first storage area before performing a read operation on the target data, and then the target data can be read from the first storage area when performing a read operation on the target data. Compared with reading the target data from a non-first storage area, this helps to increase the reading speed of the target data, and thus helps to reduce the time of I / O operations when running applications, thereby improving the I / O performance of the computing device when running applications.

[0319] In addition, based on the fact that the operation type of the target data in the above situation 1 includes a read operation and the stage in which the read operation is performed on the target data during the running application in the above situation 3 is the target running stage, the management strategy may include: pre-reading the target data to the first storage area in the running stage before the target running stage.

[0320] For example, when running to the second running stage of the application, if the second running stage is earlier than the target running stage, the target data that needs to be read in the running stage after the second running stage (that is, the target running stage) is pre-read into the first storage area, so that when running to the target running stage of the application, the target data can be read directly from the first storage area.

[0321] The operation phase before the target operation phase may be any operation phase before the target operation phase.

[0322] Exemplarily, the application program's running phase may include an initial phase, an intermediate phase, and an end phase. If the target running phase is the end phase, the target data may be pre-read into the first storage area during the initial phase or the intermediate phase. If the target running phase is the intermediate phase, the target data may be pre-read into the first storage area during the initial phase.

[0323] In this embodiment, based on the attribute information indicating the running stage when the target data is read (i.e., the target running stage), when formulating the management strategy for pre-reading the target data, it is clear in which running stages of the application the pre-reading is performed (i.e., the running stage before the target running stage). This helps to ensure the timely execution of the pre-reading management strategy, that is, the target data is pre-read into the first storage area before the target data is read.

[0324] Since the target data is pre-read into the first storage area in the operation stage before the target operation stage, compared with pre-reading the target data into the first access area in the operation stage in which the read operation is performed on the target data (i.e., the first operation stage), it helps to ensure that the target data can be read from the first storage area when the read operation is performed on the target data at any time point in the first operation stage.

[0325] In case d, the management policy includes: deleting target data in the first storage area during a running phase subsequent to a target running phase of the application program.

[0326] Based on the fact that the operation type of the target data in the above situation 1 is a read operation and the stage of performing the read operation on the target data during the running of the application in situation 3 is the target running stage, the management strategy of the target data may include: after the target running stage of the application ends, deleting the target data in the first storage area. This helps to avoid useless target data occupying the storage space of the first storage area, thereby helping to improve the utilization rate of the first storage area.

[0327] In case e, the management policy includes: when the number of times the target data is read is equal to the target number, deleting the target data in the first storage area.

[0328] Based on the fact that the operation type of the target data in the above situation 1 is a read operation and that the target data is read a target number of times during the running of the application in situation 2, the management strategy of the target data may include: when the target data is read a target number of times, deleting the data item in the first storage area. This helps to avoid useless target data occupying the storage space of the first storage area, thereby helping to improve the utilization rate of the first storage area.

[0329] Case f: The management policy includes: writing or pre-reading the target data into the shared storage device in the first storage area.

[0330] Based on the sharing attribute in the above case 1 and case 5 indicating that the target file is a shared file, the target data management strategy includes: writing or pre-reading the target data to a shared storage device.

[0331] Since the shared storage device is a storage layer shared by multiple computing devices, that is, any computing device can access the shared storage device, therefore, only one target file needs to be stored, and multiple computing devices can read it, avoiding different computing devices from storing shared files separately, thereby avoiding shared files taking up too much storage space.

[0332] It should be noted that the management strategy of the target data may include one or more of the above-mentioned situations a to f. The embodiment of the present application does not limit the number of management strategies for the target data.

[0333] In an embodiment of the present application, the policy pool of the data orchestration policy library module includes multiple management policies. Therefore, the processor can select a management policy for the target data from the policy pool of the data orchestration policy library module based on the attribute information of the target data.

[0334] In the embodiment of the present application, for the above-mentioned situation a and situation c, S702 may include: determining a management policy for the target data based on the attribute information of the target data and the storage capacity of each tier storage device in the first storage area.

[0335] It should be noted that this solution can be used in combination with step 1 and step 2 included in the above S702, or can be used alone, and the embodiment of the present application does not limit this.

[0336] Illustratively, when a management policy is determined based on the attribute information and the storage capacity of each tier of storage devices, the determined management policy may specifically include writing or pre-reading the target data into the target tier storage device in the first storage area.

[0337] Among them, the target-level storage device can be the storage device with the largest available storage space in the first storage area, or it can be the storage device with the second largest available storage space in the first storage area, or it can be any storage device with non-smallest available storage space in the first storage area.

[0338] In this embodiment, when determining the management strategy corresponding to a read operation or a write operation, the storage capacity of each hierarchical storage device in the first storage area can be comprehensively considered in combination with the storage capacity of each hierarchical storage device, so as to select a suitable placement location for the target data, thereby avoiding the situation where some hierarchical storage devices are overused while other hierarchical storage devices are not fully utilized.

[0339] Optionally, the target data management policy includes a first management policy and a second management policy, and determining the target data management policy based on attribute information of the target data and storage capacity of each tier storage device in the first storage area includes:

[0340] Step 3: Determine a management policy for the target data based on the attribute information and the first storage capacity of each tier storage device in the first storage area, wherein the first storage capacity is the storage capacity of each tier storage device at a first moment.

[0341] In the embodiment of the present application, the time before reading or writing the target data includes a first time and a second time, and the second time is later than the first time. Based on this, the computing device first determines the management policy of the target data based on the attribute information and the first storage capacity of each tier storage device in the first storage area at the first time.

[0342] Step 4: Update the target data management policy based on the second storage capacity of each level of storage device, wherein the second storage capacity is the storage capacity of each level of storage device at the second moment.

[0343] Based on the above, at the second moment, the computing device updates the management policy of the target data based on the second storage capacity of each tier storage device to obtain an updated management policy.

[0344] It should be noted that step four can be performed after the management policy determined in step three is executed, or it can be performed before the management policy determined in step three is executed. The embodiment of the present application does not limit this.

[0345] It should be noted that the above steps three and four can be used in combination with the above steps one and two, or can be used alone, and the embodiments of the present application do not limit this.

[0346] In this embodiment, the target data management policy is first determined based on the first storage capacity of each tiered storage device, and then the target data management policy is updated based on the second storage capacity of each tiered storage device. Since the second storage capacity is the most recent storage capacity of each tiered storage device, updating the target data management policy based on the second storage capacity helps ensure that the target data management policy is determined based on the most recent storage capacity of each tiered storage device, thereby helping to ensure the accuracy and rationality of the target data management policy.

[0347] Optionally, the above step 4 may include: determining a management policy for the target data based on the storage space occupied by the target data and the available storage space of each tier storage device in the first storage area.

[0348] The management strategy includes writing or pre-reading the target data to a target storage device in the first storage area; and the available storage space of the target storage device is larger than the available storage space of the non-target storage devices.

[0349] It can be understood that the available storage space of the storage device can also be considered as the remaining storage space of the storage device.

[0350] That is to say, when the management policy of the target data is determined to be written or pre-read into the first storage area, the target storage device to which the target data is written / pre-read into the first storage area can be determined based on the size of the target data and the available storage space of each level of storage devices in the first storage area.

[0351] In an embodiment of the present application, the data orchestration policy library module and the data placement engine may determine a management policy for the target data based on the storage space occupied by the target data and the available storage space of each tier storage device in the first storage area.

[0352] Exemplarily, when the data orchestration policy library module determines that the initial management policy for the target data is to write or pre-read the target data into the first storage area, the data orchestration policy library module sends the initial management policy for the target data to the data placement engine. The data placement engine determines the target storage device for writing or pre-reading the target data into the first storage area based on the size of the target data and the available storage space of each tier storage device, that is, determines the placement location of the target data in the first storage area, and returns the identifier of the target storage device to the data orchestration policy library module, so that the data orchestration policy library module determines the management policy for the target data based on the identifier of the target storage device.

[0353] It should be noted that the implementation of the above step three can refer to the implementation of step four, which will not be repeated here.

[0354] In the embodiment of the present application, the target storage device includes multiple implementation methods, which are described below using methods a to c.

[0355] Hereinafter, methods a to c are described by taking the example that the first storage area includes a first-tier storage device, a second-tier storage device, and a third-tier storage device.

[0356] Method a: The available storage space of the target storage device is greater than the available storage space of the non-target storage device.

[0357] Exemplarily, when the first-tier storage device is the target storage device, the available storage space of the first-tier storage device is greater than the available storage space of the second-tier storage device and the third-tier storage device.

[0358] Method b: The available storage space of the target storage device is greater than or equal to a preset threshold.

[0359] Exemplarily, when the available storage space of the first-tier storage device is greater than or equal to a preset threshold, and the available storage space of the second-tier storage device and the third-tier storage device is less than the predicted threshold, the first-tier storage device is the target storage device.

[0360] For example, if the available storage space of the second-tier storage device and the third-tier storage device is greater than or equal to a preset threshold, and the available storage space of the first-tier storage device is less than the preset threshold, either the second-tier storage device or the third-tier storage device can be the target storage device. In other words, the target storage device can be any storage device in any tier of the universe whose available storage space is greater than or equal to the preset threshold.

[0361] It should be noted that in the embodiments of the present application, there is no limitation on the specific value of the preset threshold value. The preset threshold value can be determined based on parameters such as the size of the target data (i.e., the storage space occupied by the target data), the available storage space of each level of storage devices in the first storage area, and the total storage space of each level of storage devices.

[0362] Method c: The available storage space of the target storage device is larger than the storage space occupied by the target data.

[0363] For example, when the first-tier storage space, the second-tier storage space, and the third-tier storage space are all larger than the storage space occupied by the target data, the target storage device can be a storage device at any one of the first-tier storage space, the second-tier storage space, and the third-tier storage space.

[0364] It should be noted that the above-mentioned methods a to c can be used simultaneously, or can be used separately, and the embodiments of the present application do not limit this.

[0365] It should be noted that the first storage area in the embodiment of the present application may also include two levels of storage devices. In this case, the relevant description of the target storage device can refer to the above-mentioned methods a to c and will not be repeated here.

[0366] It should be noted that when the first storage area includes multiple shared storage devices, the implementation method of "writing or pre-reading the target data to the shared storage device in the first storage area" can refer to the implementation method of "writing / pre-reading the target data to the first storage area". The principles of the two are the same and will not be repeated here.

[0367] In the embodiment of the present application, after determining the management policy of the target data, the computing device can store the management policy of the target data. In this way, during the running of the application, the target data can be managed according to the management policy determined in advance for the target data.

[0368] Illustratively, the management policy of the target data may be associated with the target data through an identifier of the target data. Thus, when the management policy of the target data needs to be used, the management policy of the target data may be determined according to the identifier of the target data.

[0369] It should be noted that the embodiment of the present application does not limit the storage location of the management policy of the target data. For example, it can be stored in a storage system, or it can also be stored in a computing device.

[0370] It is understandable that the target data may have one or more management policies, and the present embodiment of the application does not limit the number of management policies for the target data. In addition, the management policies for different target data may be the same, or they may be different, and the present embodiment of the application does not limit this.

[0371] Optionally, S702 may include the following steps five to six.

[0372] Step 5: Create an application namespace; the namespace includes the target data.

[0373] In an embodiment of the present application, a namespace for an application may be created by a processor.

[0374] Exemplarily, when the data orchestration engine receives the attribute information of the target data sent through the data description interface, the data orchestration engine sends notification information to the application session management module. After receiving the notification information, the application session management module creates a namespace for the application.

[0375] The name of the namespace includes the name of the application and the creation time. For example, if the application is named A and the creation time is 15:31 on January 21, 2023, the namespace name can be A2023.1.21.15.31.

[0376] It should be noted that the embodiment of the present application does not limit the accuracy of the creation time. For example, the creation time can be accurate to the minute, or it can also be accurate to the second, etc.

[0377] By including the creation time in the namespace name, different applications running at different times have different names, helping to avoid duplication of namespace names across different applications. Furthermore, the namespace name includes the application name, allowing you to accurately identify the application to which the namespace belongs. This helps distinguish the namespaces of different applications when the processor is running multiple applications simultaneously.

[0378] Step 6: Based on the attribute information of the target data, determine the management strategy of the target data in the namespace.

[0379] In an embodiment of the present application, the processor may determine a management policy for the target data in a namespace based on the attribute information of the target data, so that the management policy is only applicable to applications in the namespace.

[0380] It should be noted that steps five to six can be used simultaneously with the above steps one to two, and steps three to six, or they can be used separately. This embodiment of the present application does not limit this.

[0381] Since different namespaces can be set with different management permissions, that is, the management policies of different namespaces are isolated from each other, therefore, by creating a namespace for the application and determining the management policy of the target data in the namespace, the management policy of the target data can be made to apply only to the application to which the namespace belongs. In this way, on the one hand, the management policies of different applications can be isolated from each other, thereby preventing the management policies of different applications from interfering with each other, and further preventing the I / O operations of different applications from interfering with each other when the computing device runs multiple applications at the same time. On the other hand, since the namespace can be automatically created and the management policy of the target data in the namespace can be determined, the user does not need to specify the application to which the target data belongs in the attribute information, which helps to reduce the requirements for users and improve the user experience.

[0382] S703: Execute the management policy on the target data.

[0383] There are multiple ways to implement management policies on target data, two of which are described below.

[0384] In method 1, the computing device applies the management policy to the target data, thereby implementing the management policy on the target data.

[0385] For example, a processor of a computing device may apply the management policy to the target data.

[0386] In mode 2, the computing device sends the management policy to the storage system to instruct the storage system to apply the management policy to the target data, thereby implementing the management policy on the target data.

[0387] For example, the computing device processor may send the management policy to the controller of the storage system, and the controller may apply the management policy to the target data in response to the received management policy.

[0388] In an embodiment of the present application, executing a management policy on the target data may be before performing a read or write operation on the target data, or during the process of performing a read or write operation on the target data, or after performing a read or write operation on the target data.

[0389] Illustratively, the management policies in cases a and f are executed during the read and write operations on the target data. The management policies in cases b, d, and e are executed after the read and write operations on the target data. The management policy in case c is executed before the read and write operations on the target data.

[0390] There are multiple implementation methods for executing management policies. The following uses three implementation methods as examples to illustrate.

[0391] The first implementation method is to execute a management policy on the target data based on the management policy of the target data, for example, the management policies in case a and case b.

[0392] The second implementation method is to execute the management policy on the target data based on the management policy of the target data and the running stage information of the application, for example, the management policy in case c, case d, and case f.

[0393] In a third implementation, a management policy is executed on the target data based on the management policy of the target data and the I / O operation information of the target data, for example, the management policy in case e.

[0394] It should be noted that the first to third implementation modes mentioned above are applicable to the modes 1 to 2 in the above S703.

[0395] The second implementation is exemplarily described below through steps one and two.

[0396] In the embodiment of the present application, the running stage information in the application further includes second stage information, indicating that the application is in the second running stage. The second running stage can be any one of the multiple running stages of the application.

[0397] Step 1: Obtain the second stage information.

[0398] In an embodiment of the present application, the processor may obtain the second-stage information. Exemplarily, during the running of the application, the second-stage information in the application is sent to the data orchestration engine via the data description interface.

[0399] It can be understood that when the second stage information is obtained, it indicates that the application has entered the second running stage.

[0400] Step 2: Based on the information from the second stage, execute the management strategy on the target data.

[0401] In this embodiment, the second-stage information of the application is obtained so that the relationship between the current running stage of the application and the target running stage can be determined. Therefore, executing the management policy on the target data based on the second-stage information of the application helps to ensure the timeliness and reliability of the execution of the management policy of the target data.

[0402] In the embodiment of the present application, step 2 may include: if the second operating stage is earlier than the target operating stage, pre-reading the target data into the first storage area.

[0403] In this embodiment, when the second operation stage is earlier than the target operation stage, the target data is pre-read into the first storage area, thereby realizing that the target data is pre-read into the first storage area in the operation stage before the target data is read, thereby facilitating the target data to be read from the first storage area at any time point in the target operation stage.

[0404] Alternatively, in an embodiment of the present application, step two may include: the second running stage is later than the target running stage, and the target data pre-fetched into the first storage area is deleted.

[0405] In this embodiment, when the second operation stage is later than the target operation stage, the target data pre-fetched into the first storage area is deleted, thereby achieving timely deletion of the target data in the first storage area after the use cycle of the target data ends, which helps to improve the utilization rate of the first storage area.

[0406] In an embodiment of the present application, the processor sends a target data management policy to a controller of the storage system based on the second phase information of the application program, instructing the controller to apply the management policy to the target data. For example, when the second operating phase is earlier than the target operating phase, the processor sends a management policy to the controller to pre-read the target data into the first storage area; when the second operating phase is later than the target operating phase, the processor sends a management policy to the controller to delete the target data in the first storage area.

[0407] For example, the data orchestration policy library module may send the management policy of the target data to the data movement engine to instruct the data movement engine to apply the management policy to the target data.

[0408] The third implementation method is exemplified below through steps three and four.

[0409] Step 3: Get the I / O operation information of the target data.

[0410] The I / O operation information indicates one or more of an open operation, a close operation, and an unlink operation performed on target data during the running of the application.

[0411] In an embodiment of the present application, during the execution of an application, the processor of the computing device obtains I / O operation information of the target data. For example, the I / O operation information of the target data can be sent to the data orchestration engine via the I / O perception module.

[0412] As shown in Figure 8, during the application execution, the I / O perception module obtains the I / O operation information of the target data based on the native I / O operation behavior of the target data perceived during the application execution, and sends the I / O operation information to the data orchestration engine through the RPC service.

[0413] Step 4: Execute a management policy on the target data based on the I / O operation information of the target data.

[0414] In an embodiment of the present application, the processor may send a management policy for target data to a controller of the storage system to instruct the controller to execute the management policy on the target data.

[0415] Exemplarily, the data orchestration policy library module sends the management policy of the target data to the data movement engine, and the data movement engine executes the predetermined management policy on the target data, for example, pre-reading the target data into the first storage area.

[0416] In this implementation, since the I / O operation information indicates the reading and writing status of the target data by the computing device, such as opening the target data, closing the target data, unlinking the target data, etc., executing the management policy on the target data based on the I / O operation information of the target data helps to improve the timeliness and effectiveness of the execution of the management policy.

[0417] Based on situation 3 in S701 , the above step 4 may include: if the I / O operation information indicates that the current number of read and write times of the target data is equal to the target number, deleting the target data in the first storage area.

[0418] Here, one opening operation and one closing operation are performed on the target data, and it is determined that the target data is read once.

[0419] In an embodiment of the present application, the processor determines the number of times the target data is read based on the I / O operation information. If the number of times the target data is read is equal to the target number, the processor sends a delete instruction to the controller of the storage system to instruct the controller to delete the target data in the first storage area.

[0420] Exemplarily, the data orchestration policy library module sends a deletion instruction to the data move engine to instruct the data move engine to delete the target data in the first storage area.

[0421] In this embodiment, since the I / O operation information indicates the I / O operations performed by the computing device on the target data, determining the current read and write counts of the target data based on the I / O operation information helps improve the accuracy of the current read and write counts. Furthermore, when the current read and write counts equal the target counts, deleting the target data from the first storage area can determine that the computing device is no longer reading the target data, helping to improve the accuracy of management policy execution.

[0422] The following is an exemplary introduction to the data management process in the embodiment of the present application in conjunction with FIG10 .

[0423] FIG10 is a schematic diagram of another data management principle provided in an embodiment of the present application.

[0424] Referring to Figure 10, on the application side, the data description interface is called to send the attribute information (attribute information of the group and / or attribute information of the data item) and the running stage information to the data orchestration strategy library module through the RPC service (server). The I / O perception (Observer) module determines the I / O operation information of the target data based on the perceived native I / O operation behavior of the target data during the running application, such as open, close, unlink, etc., and sends the I / O operation information of the target data to the data orchestration strategy library module through the RPC service.

[0425] It should be noted that attribute information can also be called data information (data msg), which mainly includes static information of data. Runtime information can also be called control information (ctl msg), which mainly includes dynamic information of data.

[0426] On the daemon side, the data orchestration policy library module determines the target data's management policy based on the target data's attribute information, or based on the target data's attribute information and runtime information. Exemplarily, the data orchestration policy library module preprocesses the attribute information and then determines the target data's management policy from a policy pool based on the preprocessed attribute information. The data orchestration policy library module then sends the target data's management policy to the data movement engine, which then moves or deletes the target data to implement the target data's management policy.

[0427] In addition, the application session management module saves metadata information during the application program execution (eg, attribute information of target data, current storage path of target data, target data read and written during each application program execution, etc.), for example, in a database (DB).

[0428] In addition, the data orchestration strategy library module can also read the metadata in the database through the application session management module. The HSM module is used to manage storage devices at all levels of the storage system.

[0429] In the embodiment of the present application, there are multiple implementation methods for how to execute the management policy on the target data. The following is an illustrative description of the execution process of the management policy in the above-mentioned cases a to f through methods 1 to 5.

[0430] Mode 1: Based on the management policy in the above situation a, S703 may include: writing the target data into the first storage area.

[0431] In an embodiment of the present application, when a processor writes target data to a storage system, it determines the target data's management policy. If the target data's management policy is the management policy described in scenario a above, the processor writes the target data to the first storage area of ​​the storage system. For example, the processor may send the target data and the target data's management policy to a controller of the storage system, instructing the controller to write the target data to the first storage area.

[0432] Exemplarily, the processor may determine the management policy of the target data from a storage device storing the management policy according to the identifier of the target data.

[0433] FIG11 is a schematic diagram of a data management method provided by this embodiment.

[0434] Referring to Figure 11 , the first-tier storage device of the storage system is MEM, the second-tier storage device is SSD, the third-tier storage device is HDD / BB, and the fourth-tier storage device is PFS. The first storage area of ​​the storage system includes the first-tier storage devices through the third-tier storage devices, and the second storage area of ​​the storage system includes the fourth-tier storage devices.

[0435] The application data includes write.txt, which is the target data and the operation type of write.txt is a write operation. The management policy of write.txt may include: writing write.txt to the first storage area. Exemplarily, writing write.txt to the first-tier storage device in the first storage area.

[0436] Based on this, the process of executing the management policy on write.txt may include: writing write.txt to the first-tier storage device.

[0437] Mode 2: Based on the management policy in the above situation b, S703 may include: flushing the target data written in the first storage area to the second storage area.

[0438] In an embodiment of the present application, after the target data is written to the first storage area, if the target data is a permanent file, the target data in the first storage area is flushed to a non-first storage area and the target data in the first storage area is deleted. For example, the processor may send a management policy for deleting the target data in the first storage area to the controller, and the received management policy may be applied to the target data by the controller.

[0439] For example, based on method 1, if write.txt is a permanent file, the management strategy of write.txt may further include: flushing write.txt in the first storage area to the fourth-tier storage device.

[0440] Based on this, the process of executing the management policy on write.txt may further include: flashing write.txt in the first-tier storage device to the fourth-tier storage device, and deleting write.txt in the first-tier storage device.

[0441] For example, the data movement engine may flush write.txt from the first-tier storage device to the fourth-tier storage device.

[0442] Mode 3, based on the management policy in the above case c, S703 includes: before performing a read operation on the target data, pre-reading the target data into the first storage area.

[0443] The application data also includes 1.txt, 2.txt, and 3.txt, where 1.txt, 2.txt, and 3.txt are target data, and the operation types of 1.txt, 2.txt, and 3.txt are all read operations, and are all stored in the fourth-tier storage device (PFS).

[0444] The management policy for 1.txt includes a first management policy and a second management policy. The first management policy for 1.txt may include pre-reading 1.txt to a second-tier storage device (i.e., SSD) in the first storage area, and the second management policy for 1.txt may include pre-reading 1.txt to a first-tier storage device (i.e., MEM). The management policy for 2.txt may include pre-reading 2.txt to a second-tier storage device in the first storage area, and the management policy for 3.txt may include pre-reading 3.txt to a third-tier storage device (HDD / BB) in the first storage area.

[0445] It can be understood that the second management policy can be considered as a management policy obtained by updating the first management policy based on the available storage space of each layer of storage devices in the first storage device.

[0446] Based on this, referring to Figure 11, the process of executing the management policy on 1.txt is: before reading 1.txt, based on the first management policy, 1.txt is pre-read to the second-tier storage device; based on the second management policy, 1.txt is pre-read to the first-tier storage device. In this way, when a read operation is performed on 1.txt, 1.txt can be read from the first-tier storage device, which can significantly improve the reading speed compared to reading 1.txt from the fourth-tier storage device or the second-tier storage device.

[0447] It should be noted that the process of executing the management policy on 2.txt and 3.txt can refer to the above process of executing the management policy on 1.txt, and will not be repeated here.

[0448] For example, the data move engine may pre-read 1.txt to the second-tier storage device and the first-tier storage device, pre-read 2.txt to the second-tier storage device, and pre-read 3.txt to the third-tier storage device.

[0449] There are multiple implementations for how to pre-read the target data into the first storage area before performing a read operation on the target data. Methods A and B are used as examples below.

[0450] Mode A: When the attribute information of the target data includes the target operation stage, the target data is pre-read into the first storage area before the read operation is performed on the target data, which may include the following steps 1 to 2.

[0451] In the embodiment of the present application, the running stage information includes third stage information, wherein the third stage information is used to indicate the running stage of the application.

[0452] Step 1: Get the third stage information in the application.

[0453] Based on this, when the processor obtains the second-stage information in the application, it can determine the current running stage of the application according to the third-stage information.

[0454] In one example, the third stage information includes a third stage start instruction for instructing the application to execute in a third execution stage, where the third execution stage is an execution stage prior to the target execution stage. When the processor executes the third stage start instruction and obtains the third execution stage information, the processor can determine, based on the third execution stage information, that the current execution stage of the application is the third execution stage.

[0455] In another example, the third execution stage information includes a target stage start instruction, which is used to instruct the application to execute the target execution stage, that is, the stage where the application executes a read operation on target data. Based on this, when the processor executes the target stage start instruction and obtains the second stage information, the processor can determine, based on the second stage information, that the current execution stage of the application is the target execution stage.

[0456] In another example, the third running stage is the running stage preceding the target running stage, and the third stage information includes a third stage end instruction, which indicates the end of the third running stage of the application program. That is, the processor is about to run the target running stage of the application program. Based on this, when the processor executes the second stage end instruction and obtains the second stage information, the processor can determine that the third running stage of the application program has ended based on the second stage information.

[0457] Step 2: Based on the second-stage information of the application program, pre-read the target data into the first storage area.

[0458] In an exemplary embodiment, if the second stage information indicates that the current running stage of the application is the third running stage, and the third running stage is a running stage before the target running stage, the target data is pre-read into the first storage area.

[0459] Since the attribute information indicates that the running stage for performing the read operation on the target data is the target running stage, when it is determined that the current running stage of the application is the third running stage, the target data is pre-read into the first storage area. In this way, it can be ensured that when the target data is read at any time in the target running stage, the target data can be read from the first storage area of ​​the storage system.

[0460] In another example, if the second stage information indicates that the application has been executed to a target execution stage, the target data is pre-read into the first storage area.

[0461] In yet another example, if the second stage information indicates that the third running stage of the application program has ended, and the third running stage is a running stage before the target running stage, the target data is pre-read into the first storage area.

[0462] Since the attribute information of the target data indicates that the running stage for performing the read operation on the target data is the target running stage, when it is determined that the operation has reached the target running stage of the application, or the third running stage of the application has ended (that is, the processor is about to run the target running stage of the application), the target data is pre-read into the first storage area, which helps to avoid occupying the storage space of the first storage area too early, thereby helping to improve the utilization rate of the first storage area.

[0463] Mode B: Before performing a read operation on the target data, the processor pre-reads the target data into the first storage area, including: the following steps 3 to 4.

[0464] Step 3: Obtain a to-be-read instruction, which is used to indicate that target data is to be read.

[0465] Exemplarily, the application may include an instruction to read the target data. When the processor executes the instruction to read the target data, the processor may determine that a read operation is to be performed on the target data.

[0466] Step 4: In response to the read instruction, the processor pre-reads the target data into the first storage area.

[0467] Exemplarily, after the processor obtains a target data read instruction, the processor pre-reads the target data from a non-first storage area of ​​the storage system to the first storage area of ​​the storage system. In this way, when the processor performs a read operation on the target data, the target data can be read from the first storage area of ​​the storage system. For example, the processor sends a target data management policy (i.e., pre-reading the target data to the first storage area) to the controller, and the controller, in response to the received management policy, applies the management policy to the target data.

[0468] Exemplarily, the instruction to be read is associated with the target data through an identifier of the target data. After the processor obtains the instruction to be read, it determines the target data to be pre-read into the first storage area based on the identifier of the target data.

[0469] It can be understood that one instruction to be read is used to indicate one target data, that is, there is a one-to-one correspondence between the instruction to be read and the target data.

[0470] Mode 4, based on the management policy in the above situation d, S703 may include: after the target running phase of the application program ends, deleting the target data in the first storage area.

[0471] After the target running phase of the application program ends, deleting the target data in the first storage area may include steps five and six.

[0472] In the embodiment of the present application, the running stage information includes fourth stage information, wherein the fourth stage information is used to indicate the running stage of the application.

[0473] It should be noted that in the embodiment of the present application, the running stage information may include multiple stage information, and the multiple stages are respectively used to indicate the stage in which the application is located.

[0474] In addition, the first stage information, second stage information, third stage information and fourth stage information mentioned in the embodiments of the present application are for illustrative purposes of different schemes of the embodiments of the present application, and are not for distinguishing different stage information. Therefore, the first stage information, second stage information, third stage information and fourth stage information can indicate the same running stage of the application, or can also indicate different running stages of the application. The embodiments of the present application do not impose any restrictions on this.

[0475] Step 5: Get the fourth stage information in the application.

[0476] Based on this, the processor obtains the fourth stage information in the application program and can determine the current running stage of the application program according to the fourth stage information.

[0477] In one example, the fourth stage information may include a fourth stage start instruction for instructing execution of the application into the fourth execution stage, where the fourth execution stage is a stage subsequent to the target execution stage. When the processor executes the fourth stage start instruction and obtains the fourth stage information, the processor may determine, based on the fourth stage information, that the current execution stage of the application is the fourth execution stage.

[0478] In another example, the fourth current execution stage information includes a target stage end instruction, which is used to indicate that the target execution stage of the application has ended. Based on this, when the processor executes the target stage end instruction and obtains the fourth stage information, the processor can determine that the target execution stage of the application has ended based on the fourth stage information.

[0479] It should be noted that for other relevant instructions about step five, please refer to the relevant instructions of step one in the above method A, which will not be repeated here.

[0480] Step 6: Based on the fourth-stage information of the application, delete the target data in the first storage area.

[0481] In one example, if the fourth stage information indicates that the running stage of the application is the fourth running stage, since the fourth running stage is a stage after the target running stage, the target data in the first storage area is deleted.

[0482] In another example, if the fourth stage information indicates that the target running stage of the application program ends, the target data in the first storage area is deleted.

[0483] Since the attribute information of the data item indicates that the stage for performing the read operation on the target data is the target operation stage, that is, the stage for using the target data is the target operation stage, therefore, when the current operation stage is the fourth operation stage or the target operation stage of the application ends, the target data in the first storage area is deleted. In this way, useless target data can be avoided from being stored in the first storage area, thereby avoiding useless target data from occupying the storage space of the first storage area, and improving the storage space utilization rate of the first storage area.

[0484] Mode 5, based on the management policy in the above situation e, S703 may include: if the number of times the data item is read is equal to the target number, deleting the target data in the first storage area.

[0485] In an embodiment of the present application, the processor obtains I / O operation information of the target data and determines the number of times the target data has been read based on the I / O operation information. If the number of times the data item has been read is equal to the target number, the target data in the first storage area is deleted.

[0486] Exemplarily, the processor may obtain the I / O operation information of the target data in real time, or the processor may obtain the I / O operation information of the target data at a preset time interval. If the target data is opened once and closed once, it is considered that the target data is read once.

[0487] It should be noted that the embodiment of the present application does not limit the specific value of the preset time interval, and can be determined based on the frequency of the processor performing I / O operations. If the frequency of the processor performing I / O operations is relatively high, the preset time interval should be set to a smaller value so that the actual number of reads of the data item can be accurately recorded. If the frequency of the processor performing I / O operations is relatively low, the preset time interval should be set to a larger value, which helps to avoid too frequent execution of operations while ensuring that the actual number of reads is accurate.

[0488] Mode 6, based on the management policy in the above situation f, S503 includes: writing or pre-reading the target data into the shared storage device in the first storage area.

[0489] In one example, when the processor writes target data to the storage system, if the management policy corresponding to the target data is the management policy in case f above, the processor writes the target data to the shared storage device. For example, the processor sends the target data and the management policy of the target data (i.e., writing the target data to the shared storage device) to the controller. In response to the received target data and management policy, the controller applies the management policy to the target data, i.e., writes the target data to the shared storage device.

[0490] In another example, when the processor pre-reads the target data, if the management policy of the pre-read target data is the management policy in case f above, the processor pre-reads the target data into the shared storage device. For example, the processor sends the management policy of the target data (i.e., pre-reading into the shared storage device) to the controller, and the controller pre-reads the target data into the shared storage device in response to the received management policy.

[0491] Exemplarily, when the third-tier storage system is a shared storage device in the first storage area, the processor writes or pre-reads the target data into the third-tier storage system in the first storage area.

[0492] Optionally, performing a read operation or a write operation on the target data includes: performing a read operation or a write operation on the target data based on a native I / O interface in the application.

[0493] Referring to the application shown in S701, the application's " / / Application's computational flow includes Open / read / write / close..." section, where "Open / read / write / close..." refers to the native I / O interface. Therefore, when running the application's computational flow, embodiments of the present application perform read and write operations on target data based on the native I / O interface (i.e., Open / read / write / close...).

[0494] Exemplarily, performing a read operation or a write operation on target data based on a native I / O interface may be that the computing device performs a read operation or a write operation on target data based on a native I / O interface of a storage system.

[0495] Based on this, the interface for managing data (i.e., the data description interface) and the interface for reading and writing data (i.e., the native I / O interface) are independent of each other, allowing embodiments of the present application to perform read or write operations on target data based on the native I / O interface. In other words, the computing device can not only manage application data based on the data description interface, but also read and write target data based on the native I / O interface. In other words, there is no need to convert the native I / O interface into an interface that matches the data description interface.

[0496] Since the data management method provided by the embodiment of the present application does not require modification of the native I / O interface, it can manage the data of the application. In this way, on the one hand, the user does not need to learn and use a new I / O interface, nor does he need to modify the native I / O interface. On the other hand, compared with the need to modify the native I / O interface to read and write the data of the application, it avoids the need to perform a large number of operations to modify the native I / O interface when there are many I / O operations, resulting in a sharp increase in the time cost of modifying the I / O interface, thereby effectively improving the speed of I / O operations. In addition, due to the wide variety of I / O interfaces, using a universal native I / O interface for reading and writing helps to improve read and write performance and ensure the scalability of the I / O interface.

[0497] The third part of the embodiment of the present application is introduced below in conjunction with Figures 12 and 13.

[0498] The target applications are HPC applications or exascale applications. HPC applications, or HPC applications, refer to applications that utilize high-performance computing. Exascale applications can achieve exaflops of computation per second.

[0499] For example, on an exascale computing device, the amount of I / O data in an HPC application can reach terabytes. Of this, I / O operations on the computing device account for approximately 86%, while I / O operations at the cache layer and reading and writing variable data account for approximately 14%. Because I / O operations on computing devices account for a large proportion, the I / O performance of the computing device has a significant impact on computing performance. Furthermore, users of HPC applications often have a certain level of prior knowledge of the HPC application. Based on this prior knowledge, users can provide attribute information and runtime information for the HPC application data.

[0500] For example, the user's prior knowledge of the HPC application includes input files, output files, and output file sizes. For example, the input file includes in_1.vtk, which is stored on the second-tier storage device (non-volatile memory express, NVME) of the storage system. The output files include out_1.dat, ..., out_n.dat, and the output file size is 10 gigabytes (GB) and is stored on the fourth-tier storage device (PFS).

[0501] As shown in FIG12 , it is a diagram of the operation process of the HPC application in the related technology provided in the embodiment of the present application.

[0502] Referring to Figure 12 , the HPC application's execution process is as follows: After starting the HPC application, processor initialization and communication initialization are performed. The processor then runs the HPC application. During this process, the processor reads the input file (i.e., in_1.vtk) from the second-tier storage device. The HPC application's computation is then performed based on in_1.vtk, generating output files (i.e., out_1.dat, ..., out_n.da). Because the total number of output files is relatively large and the storage capacity of the various tiers of storage devices in the first storage area is insufficient, the computing device writes the output files to the fourth-tier storage device in the second storage area.

[0503] Processor initialization refers to checking and starting the availability and completeness of system resources required by HPC applications, as well as checking the system operating environment. Communication initialization refers to the preparatory operations performed by HPC applications that require communication services before information transmission, including setting network parameters, checking communication addresses, and checking network status.

[0504] As shown in FIG13 , it is a flowchart of data management for HPC applications provided in an embodiment of the present application.

[0505] Based on the data management method provided in this embodiment, the user adds attribute information of the target data of the HPC application in the initial part of the HPC application, and adds operation stage information in different operation stages of the HPC application.

[0506] As shown in Figure 13, after starting the HPC application, processor initialization and communication initialization are performed. During the HPC application, the processor loads the data description interface library (libddi.so) and sends the attribute information of the input and output files in the HPC application to the RPC service on the daemon side. The data orchestration engine on the daemon side obtains the attribute information of the input and output files through the RPC service and determines the management policy of the input and output files based on the attribute information of the input and output files.

[0507] In addition, the processor loads the data description interface library (libddi.so) and can also send the running stage information of the HPC application to the RPC service on the daemon side. The data orchestration engine on the daemon side obtains the running stage information of the HPC application through the RPC service and determines the management policy of the input file and the management policy of the output file based on the running stage information of the HPC application. In this way, the management policy can be determined for the input file before performing a read operation on the input file, and the management policy can be determined for the output file before performing a write operation on the output file.

[0508] For example, the attribute information of the input file includes the operation type, which is a read operation. The attribute information of the output file includes the operation type, the size of the output file, and the lifecycle of the output file, which is a write operation, the size of the output file is 10 GB, and the lifecycle indicates that the output file is a permanent file.

[0509] Based on the attribute information of the input file, the data orchestration engine determines that the management policy for the input file includes pre-reading the input file to the first-tier storage device (MEM). Based on the attribute information of the output file, the data orchestration engine determines that the management policy for the output file includes writing the output file to the first-tier storage device and the second-tier storage device in the first storage area, and flushing the output file from the first and second storage devices to the fourth-tier storage device in the second storage area.

[0510] In addition, when running HPC applications, the processor passes the running stage information of the HPC application to the RPC service on the daemon side by loading the data description interface library (libddi.so). After the data orchestration engine on the daemon side obtains the running stage information of the HPC application through the RPC service, it can execute management policies on input files and / or output files based on the running stage information of the HPC application.

[0511] Furthermore, while running HPC applications, the processor can load the I / O awareness library (libobserver.so) and send I / O operation information determined by the I / O awareness module to the RPC service on the daemon side. The data orchestration engine on the daemon side obtains input and output file I / O operation information through the RPC service and implements management policies based on the input and output file I / O operation information.

[0512] When running an HPC application, the input file management policy is implemented by pre-reading the input file (in_1.vtk) from the second-tier storage device to the first-tier storage device before performing a read operation on the input file. This allows the input file to be read from the first-tier storage device when performing a read operation on the input file.

[0513] Since the reading speed of the first-tier storage device is higher than that of the second-tier storage device, after managing the input files of the HPC application based on the data management method provided in the embodiment of the present application, the speed of reading the input files can be increased, thereby effectively improving the I / O performance of the computing device and reducing the time of I / O operations when running the HPC application, thereby helping to improve the operating efficiency of the HPC application and avoid I / O performance becoming a performance bottleneck of the HPC application.

[0514] After obtaining the calculation results, the process of implementing the management policy on the output files includes: when performing a write operation on the output files (out_1.dat, ..., out_n.dat), the output files are first written to the first-tier storage device and the second-tier storage device. Then, the output files in the first-tier storage device and the second-tier storage device are flushed to the fourth-tier storage device, and the output files in the first-tier storage device and the second-tier storage device are deleted.

[0515] Since the write speed of the first-tier storage device and the second-tier storage device is higher than that of the fourth-tier storage device, after managing the output files of the HPC application based on the data management method provided in this embodiment, the speed of writing the output files can be increased, thereby effectively improving the I / O performance of the computing device and reducing the time of I / O operations when running HPC applications, thereby helping to improve the operating efficiency of the HPC application and prevent I / O performance from becoming a performance bottleneck of the HPC application.

[0516] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of method. In order to realize the above functions, the data management device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0517] In the embodiment of the present application, the data management device can be divided into functional modules according to the above method. For example, the data management device can include functional modules corresponding to the functional divisions, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical functional division. In actual implementation, there may be other division methods.

[0518] Exemplarily, FIG14 shows a possible structural diagram of the data management device involved in the above embodiment (referred to as data management device 1400). For example, the data management device may be the data orchestration engine 500 in the above embodiment. The data management device 1400 includes an acquisition unit 1401, a determination unit 1402, and an execution unit 1403. The acquisition unit 1401 is used to acquire attribute information of the target data. For example, S701 shown in FIG7 . The determination unit 1402 is used to determine the management policy of the target data based on the attribute information. For example, S702 shown in FIG7 . The execution unit 1403 is used to execute the management policy on the target data. For example, S703 shown in FIG7 .

[0519] It is understandable that the acquisition unit 1401 and the determination unit 1402 may be the data orchestration policy library module 510 in the data orchestration engine 500. The execution unit 1403 may be the data movement engine 540 in the data orchestration engine 500.

[0520] Optionally, the application further includes a data description interface; the acquisition unit 1401 is specifically configured to acquire attribute information of the target data based on the data description interface.

[0521] Optionally, the application further includes running stage information, and the running stage information is used to indicate the running stage of the application; the determination unit 1402 is specifically used to: determine the management strategy of the target data based on the attribute information of the target data and the running stage information.

[0522] Optionally, the running stage information includes first stage information; the determination unit 1402 is specifically used to: obtain the first stage information; the first stage information is used to indicate that the application is in the first running stage; if the first running stage is earlier than the target running stage, the management strategy of the target data is determined based on the attribute information of the target data; the target running stage is the running stage of the application when the target data is read and written.

[0523] Optionally, the computing device is configured with a first storage area, and the read and write speed of the first storage area is greater than the read and write speed of the non-first storage area configured with the computing device; the attribute information indicates that the operation type of the target data includes a write operation, and the management policy includes writing the target data to the first storage area when performing a write operation on the target data.

[0524] Optionally, the attribute information indicates that the target data is a permanent file, and the management strategy includes flushing the target data written in the first storage area to a non-first storage area.

[0525] Optionally, the attribute information indicates that the operation type of the target data includes a read operation and the stage in which the target data is read is a target operation stage, and the management strategy includes pre-reading the target data to a first storage area configured in the computing device in the operation stage before the target operation stage.

[0526] Optionally, the management policy includes deleting the target data in the first storage area after the life cycle of the target data ends.

[0527] Optionally, the management strategy further includes deleting the target data pre-read into the first storage area in an operation phase after the target operation phase.

[0528] Optionally, the execution unit 1403 is specifically configured to execute a management policy on the target data based on the operation stage information.

[0529] Optionally, the running stage information also includes second stage information; the execution unit 1403 is specifically used to: obtain the second stage information; the second stage information is used to indicate that the application is in the second running stage; if the second running stage is earlier than the target running stage, the target data is pre-read into the first storage area; if the second running stage is later than the target running stage, the target data pre-fetched into the first storage area is deleted.

[0530] Optionally, the determining unit 1402 is specifically configured to determine a management policy for the target data based on attribute information of the target data and storage capacity of each tier storage device in the first storage area.

[0531] Optionally, the determination unit 1402 is specifically used to: determine the management policy of the target data based on the attribute information and the first storage capacity of each hierarchical storage device in the first storage area; the first storage capacity is the storage capacity of each hierarchical storage device at the first moment; based on the second storage capacity of each hierarchical storage device, update the management policy of the target data; the second storage capacity is the storage capacity of each hierarchical storage device at the second moment, and the second moment is later than the first moment.

[0532] Optionally, the attribute information indicates a target number, which is used to indicate the number of times the target data is read during the running of the application; the management strategy also includes deleting the target data pre-read into the first storage area when the number of times the target data is read is equal to the target number.

[0533] Optionally, the execution unit 1403 is specifically used to: execute a management policy on the target data based on I / O operation information of the target data; the I / O operation information is used to indicate one or more of the open operation, close operation and unlink operation performed on the target data.

[0534] Optionally, the execution unit 1403 is specifically used to: obtain I / O operation information of the target data; determine the number of times the target is read based on the I / O operation information; if the number of times the target data is read is equal to the target number, delete the target data pre-read into the first storage area.

[0535] Optionally, the target data includes multiple data items, and the attribute information of the target data includes attribute information of the data items and / or attribute information of a group, wherein the group includes multiple data items with the same attribute.

[0536] Optionally, the determining unit 1402 is specifically configured to: create a namespace of the application; the namespace includes target data; and determine a management policy for the target data in the namespace of the application based on the attribute information.

[0537] Optionally, the execution unit 1403 is further configured to: perform a read operation or a write operation on the target data based on a native I / O interface of the application program.

[0538] For the detailed description of the above optional methods, please refer to the above method embodiments, which will not be repeated here. In addition, the explanation and description of the beneficial effects of any of the above data management devices 1400 can refer to the above corresponding method embodiments, which will not be repeated here.

[0539] The present application also provides a computing device, for example, a computing device 110 as shown in Figures 1A and 1B. The computing device includes a processor and a memory, the processor is connected to the memory, the memory stores computer-executable instructions, and the processor implements the data processing method in the above embodiment when executing the computer-executable instructions. The present application does not impose any restrictions on the specific form of the computing device. For example, the computing device can be a terminal device or a network device. The terminal device can be referred to as: terminal, user equipment (UE), terminal device, access terminal, user unit, user station, mobile station, remote station, remote terminal, mobile device, user terminal, wireless communication device, user agent or user device, etc. The terminal device can be a mobile phone, augmented reality (AR) device, virtual reality (VR) device, tablet computer, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), etc. The network device can be a server, etc. The server may be a physical or logical server, or may be two or more physical or logical servers that share different responsibilities and work together to implement various functions of the server.

[0540] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is run on a computer, the computer is caused to execute the method executed by any one of the computing devices provided above.

[0541] For explanations of the relevant contents and descriptions of the beneficial effects of any of the computer-readable storage media provided above, reference may be made to the corresponding embodiments described above, and no further details will be given here.

[0542] The embodiment of the present application also provides a chip. The chip integrates a control circuit and one or more ports for implementing the functions of the above-mentioned computing device. Optionally, the functions supported by the chip can be referred to above and will not be repeated here. A person of ordinary skill in the art will understand that all or part of the steps of implementing the above-mentioned embodiment can be completed by instructing the relevant hardware through a program. The program can be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a random access memory, etc. The above-mentioned processing unit or processor can be a central processing unit, a general-purpose processor, an application specific integrated circuit (ASIC), a microprocessor (digital signal processor, DSP), a field programmable gate array (FPGA) or other programmable logic device, transistor logic device, hardware component or any combination thereof.

[0543] The present application also provides a computer program product comprising instructions, which, when executed on a computer, causes the computer to perform any of the methods described in the above embodiments. The computer program product comprises one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated therein. The available media may be magnetic media (e.g., floppy disk, hard disk, tape), optical media (e.g., DVD), or semiconductor media (e.g., SSD).

[0544] It should be noted that the above-mentioned devices for storing computer instructions or computer programs provided in the embodiments of the present application, such as but not limited to the above-mentioned memories, computer-readable storage media and communication chips, etc., are all non-transitory.

[0545] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented using a software program, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more media integrated therein. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (eg, a solid state disk (SSD)).

[0546] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art may understand and implement other variations of the disclosed embodiments by reviewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple situations. A single processor or other unit may implement several functions listed in the claims. The fact that certain measures are recorded in different dependent claims does not mean that these measures cannot be combined to produce good results.

[0547] Although the present application has been described with reference to specific features and embodiments thereof, it is apparent that various modifications and combinations may be made thereto without departing from the spirit and scope of the present application. Accordingly, this specification and the drawings are merely illustrative of the present application as defined by the appended claims and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, those skilled in the art may make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, the present application is intended to include such modifications and variations as fall within the scope of the claims of the present application and their equivalents.< / ddi> < / ddi>

Claims

1. A data management method, characterized in that: Applied to a computing device, the computing device reads and writes target data during the process of running an application program, and the application program includes attribute information of the target data; the method comprises: Acquiring attribute information of the target data; Determining a management strategy for the target data based on the attribute information of the target data; The management policy is executed on the target data.

2. The method according to claim 1, characterized in that The application also includes a data description interface; the acquisition of the attribute information of the target data includes: Based on the data description interface, attribute information of the target data is acquired.

3. The method according to claim 1 or 2, characterized in that: The application further includes running stage information, and the running stage information is used to indicate the running stage of the application; the determining the management strategy of the target data based on the attribute information of the target data includes: A management strategy for the target data is determined based on the attribute information of the target data and the operation stage information.

4. The method according to claim 3, characterized in that: The operation stage information includes first stage information; the determining the management strategy of the target data based on the attribute information of the target data and the operation stage information includes: Acquire the first stage information; the first stage information is used to indicate that the application is in the first running stage; If the first running stage is earlier than the target running stage, a management strategy for the target data is determined based on the attribute information of the target data; the target running stage is the running stage of the application when the target data is read and written.

5. The method according to any one of claims 1 to 4, characterized in that The computing device is configured with a first storage area, and the read and write speed of the first storage area is greater than the read and write speed of the non-first storage area configured on the computing device; the attribute information indicates that the operation type of the target data includes a write operation, and the management policy includes writing the target data to the first storage area configured on the computing device when performing a write operation on the target data.

6. The method according to claim 5, characterized in that The attribute information indicates that the target data is a permanent file, and the management strategy includes flushing the target data written in the first storage area to the non-first storage area.

7. The method according to any one of claims 1 to 4, characterized in that The attribute information indicates that the operation type of the target data includes a read operation and the stage in which the target data is read is a target operation stage, and the management strategy includes pre-reading the target data to a first storage area of ​​the computing device in an operation stage before the target operation stage.

8. The method according to claim 7, characterized in that The management strategy further includes deleting the target data pre-read into the first storage area in an operation phase after the target operation phase.

9. The method according to claim 7 or 8, characterized in that: The executing the management policy on the target data includes: Based on the operation stage information, the management policy is executed on the target data.

10. The method according to claim 5 or 7, characterized in that: The determining the management strategy of the target data based on the attribute information of the target data includes: A management policy for the target data is determined based on the attribute information of the target data and the storage capacity of each tier storage device in the first storage area.

11. The method according to claim 10, characterized in that The determining the management strategy of the target data based on the attribute information of the target data and the storage capacity of each level storage device in the first storage includes: Determine a management policy for the target data based on the attribute information and a first storage capacity of each hierarchical storage device in the first storage area; the first storage capacity is the storage capacity of each hierarchical storage device at a first moment; Based on the second storage capacity of each of the hierarchical storage devices, the management policy of the target data is updated; the second storage capacity is the storage capacity of each of the hierarchical storage devices at a second moment, and the second moment is later than the first moment.

12. The method according to any one of claims 7 to 11, characterized in that: The attribute information indicates a target number, which is used to indicate the number of times the target data is read during the running of the application; the management strategy also includes deleting the target data pre-read into the first storage area when the number of times the target data is read is equal to the target number.

13. The method according to claim 12, characterized in that The executing the management policy on the target data includes: The management policy is executed on the target data based on I / O operation information of the target data; the I / O operation information is used to indicate one or more of an open operation, a close operation and a link cancellation operation executed on the target data.

14. The method according to any one of claims 1 to 13, characterized in that The target data includes a plurality of data items, and the attribute information of the target data includes attribute information of the data items and / or attribute information of a group, wherein the group includes a plurality of data items with the same attribute.

15. The method according to any one of claims 1 to 14, characterized in that The determining, based on the attribute information, a management strategy for the target data includes: Creating a namespace of the application; the namespace including the target data; Based on the attribute information, a management policy for the target data in the namespace of the application is determined.

16. A data management device, characterized in that: include: An acquisition unit, used for acquiring attribute information of target data; a determining unit, configured to determine a management strategy for the target data based on the attribute information of the target data; An execution unit is used to execute the management policy on the target data.

17. A data management device, characterized in that: include: Processor and memory; The processor is connected to a memory, the memory is used to store computer-executable instructions, and the processor executes the computer-executable instructions to enable the computing device to implement the method according to any one of claims 1 to 15.

18. A computing device, characterized in that Comprising the data management device as claimed in claim 16 or claim 17.

19. A computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium, characterized in that: When the instructions are executed on a computing device, the computing device is caused to execute the method according to any one of claims 1 to 15.

Images