Data management method, device and equipment and readable storage medium

Abstract

The invention discloses a data management method, device and equipment and a readable storage medium, and relates to the technical field of storage. The method is used for the computing device, the computing device reads and writes target data when running an application program, the application program comprises attribute information of the target data, and on the basis, before the computing device reads and writes the target data, the attribute information of the target data is obtained, a management strategy of the target data is determined, and the determined management strategy is executed on the target data. The management strategy is determined based on the attribute information of the target data, so that different management strategies can be formulated for different application programs, the matching degree of the management strategies and the application programs is improved, the management strategies can meet different requirements of different application programs for I / O performance, the I / O performance of the computing equipment is effectively improved, and the user experience is improved. Therefore, the I / O operation time is shortened, the I / O performance is prevented from becoming the performance bottleneck of the computing device, the running efficiency of the application program is effectively improved, and the effect of accelerating the application program is achieved.

Description

Technical Field

[0001] The present application relates to the field of storage technology, and in particular to a data management method, device, equipment and readable storage medium. Background Art

[0002] Currently, in order to improve the input / output (I / O) performance of computing devices, a data optimization strategy is usually set for the I / O data of the computing device (i.e., the data read and written by the computing device). Afterwards, when the computing device runs different applications, the I / O data of the computing device will be optimized using the data optimization strategy.

[0003] However, this method is difficult to meet the different I / O performance requirements of computing devices when running different applications, resulting in the inability to effectively improve the operating efficiency of the applications. Summary of the invention

[0004] The embodiments of the present application provide a data management method, apparatus, device and readable storage medium, which can effectively improve the operating efficiency of an application.

[0005] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, a data processing method is provided, which is applied to a computing device, and reads and writes target data during the computing device running an application program, wherein the application program includes attribute information of the target data; the method comprises: obtaining the attribute information of the target data; determining a management policy for the target data based on the attribute information; and executing the management policy on the target data.

[0007] In this solution, the target data is read when the computing device runs the application, and the attribute information of the target data is configured in the application. On this basis, when the computing device runs the application, the attribute information in the application can be obtained to determine the management policy of the target data, and the determined management policy is executed on the target data.

[0008] On the one hand, since the management strategy of the application data is determined based on the attribute information of the data, different management strategies can be determined for different applications. This helps to improve the matching degree between the management strategy and the application, so that the management strategy can meet the different requirements of different applications for I / O performance, thereby effectively improving the I / O performance of the computing device when running the application, reducing the time of I / O operations, and avoiding I / O performance from becoming a performance bottleneck of the computing device, thereby effectively improving the operating efficiency of the application and achieving the effect of application acceleration.

[0009] On the other hand, by writing the attribute information of the data into the application, the attribute information can be automatically obtained when the application is running, which helps to improve the convenience of obtaining the attribute information and reduce the difficulty of obtaining the attribute information. By writing the attribute information of the target data into the application, it is possible to determine the management strategy for the specific data of the application (i.e., the target data). For example, when running the application, the data that has a greater impact on the I / O performance can not only provide a more targeted management strategy for the application (i.e., the management strategy for the target data), but also help to reduce the difficulty of data management for the application, compared with determining the management strategy for all the data of the application.

[0010] In addition, after the computing device obtains the attribute information of the target data, it can automatically determine the management policy of the target data. Compared with the user specifying the management policy for the target data, this helps to improve the efficiency of determining the management policy and reduce the requirements on the user when determining the management policy.

[0011] In a possible implementation, the application further includes a data description interface; obtaining the attribute information of the target data includes: obtaining the attribute information of the target data based on the data description interface.

[0012] In this implementation, by configuring the data description interface in the application, the attribute information in the application is obtained by calling the data description interface in the application during the running of the application. On the one hand, it helps to reduce the difficulty of obtaining the attribute information. On the other hand, compared with modifying the native interface in the application to obtain the data description information, it can not only avoid interfering with the native functions of the native interface, but also distinguish the interface for obtaining attribute information from other interfaces in the application (such as the I / O interface for reading and writing data) to avoid related interference.

[0013] In another possible implementation, the application also includes running stage information, and the running stage information is used to indicate the running stage of the application; based on the attribute information of the target data, the management strategy of the target data is determined, including: based on the attribute information and running stage information of the target data, the management strategy of the target data is determined.

[0014] In this implementation, the management strategy of the target data is determined based on the attribute information of the target data in combination with the operation stage information, which enriches the parameters used to determine the management strategy and thus helps to improve the accuracy of the determined management strategy.

[0015] In another possible implementation, the running stage information includes first stage information; based on the attribute information of the target data, the management strategy of the target data is determined, including: obtaining the first stage information; the first stage information is used to indicate that the application is in the first running stage; if the first running stage is earlier than the target running stage, based on the attribute information of the target data, the management strategy of the target data is determined; the target running stage is the running stage of the application when the target data is read and written.

[0016] In this implementation, an implementation is provided for determining the management policy of the target data before reading and writing the target data. Specifically, the running stage information configured in the application includes the first stage information, indicating that the application is in the first running stage. Based on this, when the first stage information is obtained, it means that the application has run to the first running stage. When the first running stage is earlier than the target running stage, the management policy of the target data is determined, and the management policy can be determined before the target data is read and written, thereby helping to ensure that the management policy is executed on the target data when reading and writing the target data or before reading and writing the target data, thereby helping to avoid the inability to read and write based on the management policy when reading and writing the target data.

[0017] In another possible implementation, the computing device also reads and writes non-target data while running the application program, and the impact of the non-target data on the I / O performance is less than the impact of the target data on the I / O performance.

[0018] In this implementation, the attribute information of the target data is written into the application, which can formulate and manage the data that has a greater impact on the I / O performance during the running of the application, compared with writing the attribute information of the non-target data into the application, so as to help improve the I / O performance of the computing device more effectively. In addition, compared with writing the attribute information of both the target data and the non-target data into the application, it helps to reduce the amount of data that needs to be managed.

[0019] In another possible implementation, the computing device also reads and writes non-target data while running the application program, and the storage space occupied by the non-target data is smaller than the storage space occupied by the target data.

[0020] In this implementation, writing the attribute information of the target data into the application program can formulate and manage the large data (i.e., data occupying a large storage space) read and written during the running of the application program, which helps to more effectively improve the I / O performance of the computing device. In addition, compared with writing the attribute information of both the target data and the non-target data into the application program, it helps to reduce the amount of data that needs to be managed.

[0021] In another possible implementation, the computing device is configured with a first storage area, and the read and write speed of the first storage area is greater than the read and write speed of the non-first storage area configured with the computing device; the attribute information indicates that the operation type of the target data includes a write operation; and the management policy includes writing the target data to the first storage area when performing a write operation on the target data.

[0022] In this implementation, by setting a management strategy for the target data, including writing the target data to the first storage area when performing a write operation on the target data, compared to writing the target data to a non-first storage area, this helps to increase the write speed of the target data, reduce the I / O time when running the application, and thus help improve the I / O performance of the computing device when running the application.

[0023] In another possible implementation, the attribute information indicates that the target data is a permanent file, and the management strategy includes flushing the target data written in the first storage area to a non-first storage area.

[0024] In this implementation, when the target data is a permanent file, setting a management policy includes flushing the target data written in the first storage area to a non-first storage area and deleting the target data in the first storage area, thereby preventing the target data from occupying the storage space of the first storage area for a long time, thereby helping to improve the utilization rate of the fast storage area.

[0025] In another possible implementation, the attribute information indicates that the operation type of the target data includes a read operation and the stage in which the target data is read is a target operation stage; the management strategy includes pre-reading the target data to a first storage area of ​​the computing device in an operation stage before the target operation stage.

[0026] In this implementation, the running stage when the target data is read (i.e., the target data stage) is indicated by setting attribute information. In this way, when formulating a management strategy for pre-reading the target data, it can be clearly stated at which running stages of the application the pre-reading is performed (i.e., the running stage before the target running stage), thereby helping to ensure the timely execution of the pre-reading management strategy, that is, pre-reading the target data into the first storage area before the target data is read.

[0027] In another possible implementation, the management strategy further includes deleting the target data in the first storage area after the life cycle of the target data ends.

[0028] In this implementation, it is possible to prevent useless target data from occupying the storage space of the first storage area without affecting the normal operation of the application program, thereby helping to improve the utilization rate of the first storage area.

[0029] In another possible implementation, the management strategy further includes deleting the target data pre-read into the first storage area in an operation phase after the target operation phase.

[0030] In this implementation, since the target data reading operation phase is the target operation phase, this means that after the target operation phase ends, the target data will no longer be read. Based on this, by setting the management policy to include deleting the target data in the first storage area in the operation phase after the target operation phase, it is possible to avoid useless target data occupying the storage space of the first storage area without affecting the normal operation of the application, thereby helping to improve the utilization rate of the first storage area.

[0031] In another possible implementation, executing the management policy on the target data includes: executing the management policy on the target data based on the operation stage information.

[0032] In this implementation, when executing the management policy on the target data, combining the running stage information in the application helps to ensure the accuracy and timeliness of the execution of the management policy.

[0033] In another possible implementation, the running stage information also includes second stage information; executing a management policy on the target data includes: obtaining the second stage information; the second stage information is used to indicate that the application is in the second running stage; if the second running stage is earlier than the target running stage, pre-reading the target data into the first storage area; if the second running stage is later than the target running stage, deleting the target data pre-fetched into the first storage area.

[0034] In this implementation, when the second stage information is obtained, it indicates that the application has reached the second operation stage. If the second operation stage is earlier than the target operation stage, the target data is pre-read into the first storage area, thereby realizing the operation stage before reading the target data, pre-reading the target data into the first storage area, thereby facilitating the target data to be read from the first storage area at any time point in the target operation stage. If the second operation stage is later than the target operation stage, the target data pre-fetched into the first storage area is deleted, thereby realizing the timely deletion of the target data in the first storage area after the use cycle of the target data ends, thereby helping to improve the utilization rate of the first storage area.

[0035] In another possible implementation, the management strategy of the target data is determined based on the attribute information of the target data, including: determining the management strategy of the target data based on the attribute information of the target data and the storage capacity of each level storage device in the first storage area.

[0036] In this implementation, when determining the management strategy corresponding to a read operation or a write operation, the storage capacity of each hierarchical storage device in the first storage area can be comprehensively considered to select a suitable placement location for the target data, thereby avoiding the situation where some hierarchical storage devices are overused while other hierarchical storage devices are not fully utilized.

[0037] In another possible implementation, a management policy for the target data is determined based on the attribute information of the target data and the storage capacity of each hierarchical storage device in the first storage, including: determining the management policy for the target data based on the attribute information and the first storage capacity of each hierarchical storage device in the first storage area; the first storage capacity is the storage capacity of each hierarchical storage device at a first moment; updating the management policy for the target data based on the second storage capacity of each hierarchical storage device; the second storage capacity is the storage capacity of each hierarchical storage device at a second moment, the second moment being later than the first moment.

[0038] In this implementation, before reading and writing the target data, the management policy of the target data is first determined based on the first storage capacity of each hierarchical storage device, and then the management policy of the target data is updated based on the second storage capacity of each hierarchical storage device. Since the second storage capacity is the storage capacity of each hierarchical storage device at the latest moment, updating the management policy of the target data based on the second storage capacity helps to ensure that the management policy of the target data is determined based on the latest storage capacity of each hierarchical storage device, which helps to ensure the accuracy and rationality of the management policy of the target data.

[0039] In another possible implementation, the attribute information indicates the storage space occupied by the target data.

[0040] In this implementation, the storage space occupied by the target data is indicated by setting attribute information. In this way, when determining the management strategy based on the storage space occupied by the target data and the storage capacity of each level of storage devices, a suitable level can be determined for the target data in the first storage area to ensure that the determined level matches the size of the target data.

[0041] In another possible implementation, the management strategy includes writing or pre-reading the target data to a target storage device in the first storage area; and the available storage space of the target storage device is larger than the available storage space of the non-target storage device.

[0042] In this implementation, a management policy is set to include writing or pre-reading target data to a target storage device in a first storage area, and the available storage space of the target storage device is larger than the available storage space of non-target storage devices. This helps to balance the available storage space of storage devices at each level and avoid excessive use of some storage devices at other levels and under-use of other storage devices at other levels.

[0043] In another possible implementation, the attribute information indicates a target number, which is used to indicate the number of times the target data is read during the running of the application; the management strategy also includes deleting the target data in the first storage area when the number of times the target data is read is equal to the target number.

[0044] In this implementation, the target number of times is indicated by setting attribute information, and the management policy includes deleting the target data in the first storage area when the number of times the target data is read is equal to the target number of times. In this way, during the process of the computing device running the application, when the actual number of times the computing device reads the target data is equal to the target number of times, that is, when the computing device no longer reads the target data, the target data in the first storage area can be deleted, thereby avoiding useless target data from occupying the storage space of the first storage area without affecting the normal operation of the application, thereby helping to improve the utilization rate of the first storage area.

[0045] In another possible implementation, executing a management policy on the target data includes: executing a management policy on the target data based on I / O operation information of the target data; the I / O operation information is used to indicate one or more of an open operation, a close operation, and a cancel link operation performed on the target data.

[0046] In this implementation, since the I / O operation information indicates the I / O operation of the computing device on the target data, such as open operation, close operation and unlink operation, etc., executing the management policy on the target data based on the I / O operation information helps to improve the timeliness and effectiveness of the execution of the management policy.

[0047] In another possible implementation, a management strategy is executed on the target data based on the I / O operation information of the target data, including: obtaining the I / O operation information of the target data; determining the number of times the target is read based on the I / O operation information; if the number of times the target data is read is equal to the target number, deleting the target data pre-read into the first storage area.

[0048] In this implementation, since the I / O operation information indicates the I / O operation status of the computing device on the target data, determining the current read count of the target data based on the I / O operation information helps to improve the accuracy of the current read count. In addition, when the current read count is equal to the target count, deleting the target data in the first storage area can determine that the computing device is no longer reading the target data, which helps to improve the accuracy of the management policy execution.

[0049] In another possible implementation, the application is run by multiple computing devices; the attribute information indicates that the target data is a shared file of the multiple computing devices, and the management strategy includes writing or pre-reading the target data to a shared storage device of the multiple computing devices.

[0050] In this implementation, when the target data is a shared file of multiple computing devices, the management strategy includes writing or pre-reading the target data to a shared storage device of the multiple computing devices, so that the multiple computing devices can all read the target data from the shared storage device, avoiding the multiple computing devices from storing the target data separately, thereby avoiding duplicate storage.

[0051] In another possible implementation, the target data includes multiple data items, and the attribute information of the target data includes attribute information of the data items and / or attribute information of a group, wherein the group includes multiple data items with the same attributes.

[0052] In this implementation, since the minimum unit of reading and writing data in the process of running the application is the data item, the attribute information of the target data is set to include the attribute information of the data item, which helps to improve the richness and completeness of the attribute information of the target data. A group includes multiple data items with the same attributes, and the attribute information of the target data is set to include the attribute information of the group. The attribute information of the group can be used to describe the attribute information of multiple data items at the same time. In this way, it is not only possible to avoid repeatedly writing the same attribute information in the application, thereby avoiding application redundancy, but also to reduce the storage space occupied by the application. In addition, it is also possible to reduce the workload when setting the attribute information of multiple data items. In addition, when it is necessary to manage multiple data items, the attribute information of multiple data items can be obtained at one time, which helps to improve the efficiency of obtaining attribute information.

[0053] In another possible implementation, the management strategy of the target data is determined based on the attribute information, including: creating a namespace of the application; the namespace includes the target data; and determining the management strategy of the target data in the namespace of the application based on the attribute information.

[0054] In this implementation, since different namespaces can set different management permissions, creating an application namespace and determining the management policy for the target data in the application namespace can make the management policy for the target data only apply to the application. In this way, the management policies of different applications can be isolated from each other, thereby preventing the management policies of different applications from interfering with each other, and further preventing the I / O operations of different applications from interfering with each other when the computing device runs multiple applications at the same time.

[0055] In another possible implementation, the method further includes: performing a read operation or a write operation on the target data based on a native I / O interface in the application.

[0056] In this implementation, since the target data can be read or written based on the native I / O interface in the application, that is, the data process of the application can be managed without modifying the native I / O interface, it is possible to avoid the need to perform a large number of operations to modify the native I / O interface when there are many I / O operations, thereby increasing the speed of the I / O operation. In addition, since there are many types of I / O interfaces for computing devices, and the data management method of the embodiment of the present application can manage the data of the application without modifying the native I / O interface, it helps to ensure the extensibility of the I / O interface of the computing device.

[0057] In a second aspect, a data management device is provided, the device comprising: a functional unit for executing any one of the methods provided in the first aspect, the actions executed by each functional unit are implemented by hardware or by hardware executing corresponding software. For example, the data management device may include: an acquisition unit, a determination unit, and an execution unit; the acquisition unit is used to acquire attribute information of target data; the determination unit is used to determine a management policy for the target data based on the attribute information; and the execution unit is used to execute the management policy on the target data.

[0058] In a third aspect, a data management device is provided, comprising: a processor and a memory, wherein the processor is connected to the memory. The memory is used to store computer-executable instructions, and the processor executes the computer-executable instructions stored in the memory, thereby implementing any one of the methods provided in the first aspect.

[0059] In a fourth aspect, a computing device is provided, comprising: the data management device provided in the second aspect, or the data management device provided in the third aspect.

[0060] In a fifth aspect, a computer system is provided, which includes a computing device and a storage system; the computing device is used to obtain attribute information of target data; the computing device is also used to determine a management policy for the target data based on the attribute information; the computing device is also used to send the management policy to the storage system to instruct the storage system to apply the management policy to the target data.

[0061] In the fifth aspect, the computing device may also be used to execute any possible implementation of the first aspect.

[0062] In a sixth aspect, a chip is provided, the chip comprising: a processor and an interface circuit; the interface circuit is used to receive code instructions and transmit them to the processor; the processor is used to run the code instructions to execute any one of the methods provided in the first aspect above.

[0063] In a seventh aspect, a computer-readable storage medium is provided, storing computer execution instructions, which, when executed on a computer, enable the computer to execute any one of the methods provided in the first aspect.

[0064] In an eighth aspect, a computer program product is provided, comprising: computer execution instructions, which, when executed on a computer, cause the computer to execute any one of the methods provided in the first aspect.

[0065] Among them, the technical effects brought about by any implementation method in the second to eighth aspects can refer to the technical effects brought about by different implementation methods in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1A A schematic diagram of a computing device provided in an embodiment of the present application;

[0067] Figure 1B A schematic diagram of a computer system provided in an embodiment of the present application;

[0068] Figure 2 A schematic diagram of a storage system provided in an embodiment of the present application;

[0069] Figure 3 A schematic diagram of another computer system provided in an embodiment of the present application;

[0070] Figure 4 A schematic diagram of another computer system provided in an embodiment of the present application;

[0071] Figure 5 A schematic diagram of a data arrangement engine provided in an embodiment of the present application;

[0072] Figure 6A flowchart of a data management method provided in an embodiment of the present application;

[0073] Figure 7 A flowchart of another data management method provided in an embodiment of the present application;

[0074] Figure 8 A schematic diagram of data management provided in an embodiment of the present application;

[0075] Fig. 9 A schematic diagram of attribute information of a group provided in an embodiment of the present application;

[0076] Fig.10 Another schematic diagram of data management provided by an embodiment of the present application;

[0077] Fig.11 A schematic diagram of a mobile data item provided in an embodiment of the present application;

[0078] Fig.12 A flow chart of an HPC application operation provided in an embodiment of the present application;

[0079] Fig.13 A flow chart of data management for an HPC application provided in an embodiment of the present application;

[0080] Fig.14 A schematic diagram of a data management device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0081] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0082] In the description of this application, unless otherwise specified, " / " indicates that the objects associated with each other are in an "or" relationship, for example, A / B can represent A or B; "and / or" in this application is merely a description of the association relationship between associated objects, indicating that three relationships may exist, for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural.

[0083] Furthermore, in the description of this application, unless otherwise specified, "plurality" means two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0084] In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first", "second" and the like are used to distinguish the same items or similar items with substantially the same functions and effects. Those skilled in the art will understand that the words "first", "second" and the like do not limit the quantity and execution order, and the words "first", "second" and the like do not necessarily limit the differences. At the same time, in the embodiments of the present application, the words "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or design solutions. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete manner for ease of understanding.

[0085] The following is a brief introduction to the relevant terms involved in the embodiments of the present application.

[0086] High performance computing (HPC) refers to the computing power provided by supercomputers or computer clusters, mainly including extremely fast data processing capabilities and huge data storage capacity.

[0087] An application program is a computer program that is designed to complete one or more specific tasks.

[0088] Remote procedure call (RPC) is a communication method between processes that allows a program to remotely call a procedure or function in another address space without the programmer explicitly coding the details of the remote call, so that the application can call remote procedures and services just like calling local methods. RPC can be applied to many scenarios such as distributed services, distributed computing, and remote service calls.

[0089] The following is an exemplary introduction to the application scenarios of the embodiments of the present application.

[0090] With the large-scale emergence of high-performance computing and the integration of artificial intelligence (AI) and high-performance computing, the computing capacity of computing devices (such as computing devices in supercomputing centers or computer clusters) is increasing, gradually transitioning to the E-scale, that is, 100 trillion calculations can be performed per second. The computing scale of high-performance computing is strongly related to the amount of I / O data (or I / O volume), that is, a large amount of I / O data is generated during the operation of high-performance computing provided by computing devices. For example, for computing devices of the E-scale, the amount of I / O data can usually reach terabytes (TB)-petabytes (PB). Since the larger the amount of I / O data, the more time is required to read and write I / O data, and the lower the computing performance will be, and higher I / O performance can shorten the required time, so the I / O performance of computing devices has an increasing impact on computing performance.

[0091] In the related art, in order to improve the I / O performance of a computing device, a data optimization strategy is usually set for the I / O data of the computing device, for example, the I / O data is divided into hot data or cold data, and different optimization strategies are set for hot data and cold data respectively. Afterwards, when the computing device runs different applications, the I / O data of the computing device will be optimized using the data optimization strategy. However, this method is difficult to meet the different requirements for I / O performance when the computing device runs different applications, resulting in the inability to effectively improve the operating efficiency of the application.

[0092] In view of this, an embodiment of the present application provides a data management method, which reads the target data in the process of the computing device running the application, and configures the attribute information of the target data in the application. On this basis, in the process of the computing device running the application, the attribute information in the application can be obtained to determine the management strategy of the target data, and the determined management strategy is executed on the target data. On the one hand, since the management strategy of the application data is determined based on the attribute information of the data, different management strategies can be determined for different applications. This helps to improve the matching degree between the management strategy and the application, so that the management strategy can meet the different requirements of different applications for I / O performance, thereby effectively improving the I / O performance of the computing device when running the application, reducing the time of I / O operations, and avoiding I / O performance from becoming a performance bottleneck of the computing device, thereby effectively improving the operating efficiency of the application and achieving the effect of application acceleration.

[0093] On the other hand, by writing the attribute information of the data into the application, the attribute information can be automatically obtained when the application is running, which helps to improve the convenience of obtaining the attribute information and reduce the difficulty of obtaining the attribute information. By writing the attribute information of the target data into the application, it is possible to determine the management strategy for the specific data of the application (i.e., the target data). For example, when the application is running, the data that has a greater impact on the I / O performance than other data can not only provide a more targeted management strategy for the application (i.e., the management strategy for the target data), but also help to reduce the difficulty of data management for the application, compared with determining the management strategy for all the data of the application.

[0094] In addition, after the computing device obtains the attribute information of the target data, it can automatically determine the management policy of the target data. Compared with the user specifying the management policy for the target data, this helps to improve the efficiency of determining the management policy and reduce the requirements on the user when determining the management policy.

[0095] In addition, determining the management strategy based on the attribute information of the target data improves the accuracy of data management and enables more fine-grained management of data compared to managing data based on cold data and hot data.

[0096] The following is an exemplary introduction to the system architecture of the embodiments of the present application.

[0097] like Figure 1A Shown is a schematic diagram of a computing device provided in an embodiment of the present application.

[0098] See also Figure 1A , the computing device 110 includes a processor 111. The computing device 110 is used to run an application program.

[0099] The processor is the control center of the computing device 110 and is used to run applications. The processor can be a computing unit with computing capabilities such as a central processing unit (CPU), a graphics processing unit (GPU), a data processing unit (DPU), a neural processing unit (NPU), and an embedded neural-network processing unit (NPU).

[0100] The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), systems on chip (SOC) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or any conventional processor, etc.

[0101] It should be noted that the above is only an exemplary description, and the embodiments of the present application do not limit the specific form of the processor. In the following, the embodiment of the present application is described by taking the processor as a GPU as an example.

[0102] In the embodiment of the present application, the computing device 110 may be a mobile phone, a tablet computer, a handheld computer, a PC, a cellular phone, a personal digital assistant (PDA), a wearable device (such as a smart watch, a smart bracelet, etc.), a smart home device (such as a television, etc.), a car computer (such as a car computer, etc.), a smart screen, a game console, a headset, an AI speaker, an augmented reality (AR) / virtual reality (VR) device, an ultra-mobile personal computer (UMPC), a laptop computer, a netbook, a desktop computer or an all-in-one computer, etc.

[0103] It should be noted that the above is only an exemplary description, and the embodiment of the present application does not limit the specific device form of the computing device 110.

[0104] In the embodiment of the present application, an input device 130 is further provided, and the input device 130 is connected to the computing device 110. The input device 130 is used to input attribute information of data, running stage information of the application program, etc. to the computing device 110.

[0105] Exemplarily, the attribute information of the data and the running stage information of the application can be stored in the application, so that the computing device 110 can obtain the attribute information of the data, the running stage information of the application, etc. from the application while running the application.

[0106] In the embodiment of the present application, a display device 140 is also provided, and the display device 140 is connected to the computing device 110. The display device 140 is used to display attribute information of data, running stage information of the application, and calculation results obtained by running the application.

[0107] In the embodiment of the present application, the computing device 110 may further include an access layer, which may include multiple levels of cache, for example, including: a first level cache (L1 cache), a second level cache (L2 cache), a third level cache (L2 cache), etc. The access layer is located between the processor and the memory of the computing device, and is connected to the processor and the memory respectively.

[0108] For example, when the processor writes data to the memory during the running of the application, the data may be first written to the cache layer, and then the data in the cache may be written to the memory. When the processor reads data from the memory, the data in the memory may be first read to the cache layer, and then the data may be read from the cache layer.

[0109] Based on this, the data management method provided in the embodiment of the present application can be used to manage data in the access layer.

[0110] like Figure 1B , which is a schematic diagram of a computer system provided in an embodiment of the present application.

[0111] See also Figure 1B The computer system 100 includes a computing device 110 and a storage system 120. The processor 111 of the computing device 110 is connected to the storage system 120. The storage system is used to store application data.

[0112] The data management method provided in the embodiment of the present application can be used to manage the data of applications in a storage system.

[0113] So far, the data management method provided in the embodiment of the present application can be used to manage the data of the application in the storage system, or it can also manage the data in the access layer. Below, the data management method provided in the embodiment of the present application is described by taking the data management method used to manage the data of the application in the storage system as an example.

[0114] The data of the application includes data written to the storage system 120 and data read from the storage system 120 during the process of the computing device 110 running the application. The data written to the storage system 120 includes temporary data or permanent data generated during the process of running the application.

[0115] It can be understood that the data of the application can also be referred to as I / O data during the process of the computing device 110 running the application.

[0116] It can be understood that writing data to the storage system 120 can be considered as performing a write operation on the data. Reading data from the storage system 120 can be considered as performing a read operation on the data. The data written to the storage system 120 can be called an output file of the application, and the data read from the storage system 120 can be called an input file of the application.

[0117] In the embodiments of the present application, the application may be an application that has high requirements for data processing capabilities, such as HPC applications such as weather forecasting, aircraft manufacturing, energy exploration, collision simulation, equation solving, etc. (hereinafter referred to as HPC applications), or may be other types of applications, such as applications in the field of artificial intelligence, etc. It can be understood that the above is only an exemplary description, and the embodiments of the present application do not limit the specific types of applications.

[0118] In the embodiment of the present application, the application can be stored in the storage system 120, or can also be stored in the computing device 110. That is to say, the embodiment of the present application does not limit the storage location of the application. In the following, for the convenience of description, the embodiment of the present application is exemplarily introduced by taking the application stored in the storage system 120 as an example.

[0119] like Figure 2 , which is a schematic diagram of a storage system provided in an embodiment of the present application.

[0120] See also Figure 2 The storage system 120 includes a multi-level storage device. The multi-level storage device may include a first-level storage device and a second-level storage device. The first-level storage device may be a memory (MEM), and the second-level storage device may be a solid state drive (SSD).

[0121] In the embodiment of the present application, the storage system 120 may further include a controller. The controller in the storage system 120 is connected to the processor of the computing device 110, and is used to receive read and write instructions and data management policies sent by the processor, and execute the read and write instructions and data management policies.

[0122] In the embodiment of the present application, the multi-level storage device may further include a third-level storage device and a fourth-level storage device. The third-level storage device may be a hard disk drive (HDD) or a burst buffer (BB), and the fourth-level storage device may be a parallel file system (PFS).

[0123] It can be understood that the first-tier storage device is the highest level, the second-tier storage device is the second, the third-tier storage device is the third, and the fourth-tier storage device is the lowest. Among them, the high-tier storage device will have a higher I / O speed (also known as read and write speed), lower latency, and smaller storage capacity than the low-tier storage device.

[0124] It should be noted that the embodiment of the present application does not limit the number of levels of the storage system 120, and the above is only an exemplary description. In the following embodiments, the multi-level storage device includes a first-level storage device, a second-level storage device, a third-level storage device, and a fourth-level device as an example to illustrate the embodiment of the present application.

[0125] The multi-level storage device may include a first storage area and a second storage area. The first storage area may be referred to as a fast storage area relative to the second storage area. The second storage area is usually a distributed storage structure and serves as the starting storage location of the application data. The second storage area may also be considered as a non-first storage area.

[0126] Exemplarily, the first storage area may include a first-tier storage device, a second-tier storage device, and a third-tier storage device, and the second storage area may include a fourth-tier storage device.

[0127] In an embodiment of the present application, the storage system 120 may be integrated into a computing device, or may be independent of the computing device. Alternatively, a portion of the hierarchical storage devices of the storage system 120 may be integrated into the computing device, while another portion of the hierarchical storage devices may be independent of the computing device.

[0128] like Figure 3 , which is a schematic diagram of another computer system provided in an embodiment of the present application.

[0129] See also Figure 3 The computer system 300 includes a computing device 310 and a storage system 320 . Part of the hierarchical storage devices of the storage system 320 are integrated with the computing device 310 , and another part of the hierarchical storage devices are independent of the computing device 310 .

[0130] The computing device 310 includes a processor 3101, and the storage system includes a first-tier storage device 3201, a second-tier storage device 3202, a third-tier storage device 3203, and a fourth-tier storage device 3204. The processor 3101 is connected to the first-tier storage device 3201, the second-tier storage device 3202, the third-tier storage device 3203, and the fourth-tier storage device 3204, respectively, so as to write data to each tier storage device, or read data from each tier storage device.

[0131] Among them, the first-level storage device 3201 and the second-level storage device 3202 are integrated into the computing device, and the third-level storage device 3103 and the fourth-level storage device 3104 are independent of the computing device.

[0132] It should be noted that the embodiment of the present application does not limit the number of levels of storage devices 310 integrated into the computing device and the number of levels of storage devices 310 independent of the computing device. The above is only an exemplary description.

[0133] In an embodiment of the present application, a storage device independent of the computing device 310, such as the third-level storage device 3203, the fourth-level storage device 3204, etc., can be mounted on the computing device 310. In this way, the storage device independent of the computing device 310 can be equivalent to a local storage device of the computing device 310.

[0134] In an embodiment of the present application, a storage device independent of the computing device 310, such as the third-level storage device 3203, the fourth-level storage device 3204, etc., can be used by a computing device alone, or can be shared by multiple computing devices in a computer system.

[0135] like Figure 4 As shown, it is a schematic diagram of another computer system provided for the implementation of the present application.

[0136] In the computer system, some hierarchical storage devices are shared by multiple computing devices in a computer cluster.

[0137] See also Figure 4 The computer system 400 includes multiple computing devices 410 and a storage system 420 . The multiple computing devices 40 include a first computing device 411 and a second computing device 412 . The first computing device 403 includes a first processor 413 , and the second computing device 412 includes a second processor 414 .

[0138] The storage system 420 includes a first-tier storage device 421 and a second-tier storage device 422 of the first computing device 411, a first-tier storage device 423 and a second-tier storage device 424 of the second computing device 412, and a shared storage device of the first computing device 411 and the second computing device 412, the shared storage device including a third-tier storage device 425 and a fourth-tier storage device 426.

[0139] The first-tier storage device 421 and the second-tier storage device 422 of the first computing device 411 are integrated into the first computing device 411. The first processor 413 is connected to the first-tier storage device 421 and the second-tier storage device 422, respectively.

[0140] The first tier storage device 423 and the second tier storage device 424 of the second computing device 412 are integrated into the second computing device 412. The second processor 414 is connected to the first tier storage device 423 and the second tier storage device 424, respectively.

[0141] The third-tier storage device 425 and the fourth-tier storage device 426 are storage devices shared by the first computing device 411 and the second computing device 412. The first processor 413 is connected to the third-tier storage device 425 and the fourth-tier storage device 426. The second processor 414 is connected to the third-tier storage device 425 and the fourth-tier storage device 426.

[0142] It should be noted that the embodiment of the present application does not limit the number of computing devices in the computer system, and the above computer system includes two computing devices for exemplary purposes only.

[0143] like Figure 5 , which is a schematic diagram of a data orchestration engine provided in an embodiment of the present application.

[0144] The data orchestration engine 500 is a software module for implementing the data management method provided in the embodiment of the present application. The data orchestration engine 500 can be run by a computing device and / or a storage system to implement the data management method provided in the embodiment of the present application. For example, the data orchestration engine 500 can be run by a processor of a computing device and / or a controller of a storage system.

[0145] See also Figure 5 The data orchestration engine 500 includes: a data orchestration policy library (DOPL) module 510, an application session manager (ASM) module 520, a data placement engine 530, a data movement engine 540, and a metadata management module 550.

[0146] Data arrangement policy library module 510: used to obtain attribute information of data, determine data management policy according to the attribute information of data, periodically determine the storage location of data, and uniformly manage the life cycle of data. The uniform management of the life cycle of data includes deleting data from the first storage area after the life cycle of data ends.

[0147] The data orchestration strategy library module 510 includes a handler pool and a strategy pool.

[0148] The processor pool includes multiple processor modules, and different processor modules are used to process different information.

[0149] Exemplarily, the information processed by the processor module may include dynamic information and static information, wherein the static information includes attribute information, etc., and the dynamic information includes operation stage information, etc.

[0150] For example, the processing of static information may include integrating different attribute information of one data, assigning the attribute information of a group to each data in the group, and the like.

[0151] The policy pool includes a plurality of management policies, such as pre-fetching to the first storage area, flushing to the non-first storage area, and the like.

[0152] Application session management module 520: used to create a namespace for an application program, thereby preventing I / O operations and data management policies of different applications from interfering with each other.

[0153] In addition, the application session management module may provide an interface for the data orchestration policy module to access the database, that is, the data orchestration policy library module may read metadata in the database through the application session management module, or write metadata into the database.

[0154] Data placement engine 530: used to determine the placement position of data in the management policy according to the initial data management policy determined by the data orchestration policy library module 510 and the capacity of each level of storage devices in the storage system. For example, when the initial data management policy is to write the data into the first storage area, the data placement engine 530 determines to place the data in the first level storage device of the first storage area, and returns the identifier of the first level storage device to the data orchestration policy library module 510, so that the data orchestration policy library module 510 can determine the data management policy.

[0155] Data move engine 540: used to delete data items in the storage system, move data in the storage system, etc., for example, move data in the fourth-tier storage device to the second-tier storage device.

[0156] Metadata management module 550: used to record data information of the current session and data information of historical sessions, such as data written and read in the current session, and data written and read in historical sessions.

[0157] The current session refers to the entire life cycle of the currently running application, and the historical session refers to the entire life cycle of the historically running application.

[0158] It should be noted that the above division of the data orchestration engine and the naming of each module are only exemplary descriptions, and the embodiments of the present application do not limit how to divide the data orchestration engine and how to name each module.

[0159] Exemplarily, the data orchestration policy library module 510, the application session management module 520, the data placement engine 530, and the metadata management module 550 may be executed by a computing device, and the data movement engine 540 may be executed by a storage system.

[0160] On this basis, the embodiment of the present application also provides a data definition interface (DDI). The DDI can be stored in the application. The DDI is a software module that can be run by the processor of the computing device, and is used to receive the attribute information of the data and / or the attribute information of the group, and the running stage information of the application, write the attribute information of the data and / or the attribute information of the group, and the running stage information of the application into the application, and send the attribute information of the data, the running stage information of the application, etc. to the data orchestration engine 500.

[0161] In an embodiment of the present application, a data description interface is provided to store the attribute information of the specified data (i.e., the target data in the following embodiment) and the running stage information of the application in a computing device or a storage system before running the application. In this way, the computing device can obtain the attribute information of the specified data and the running stage information of the application for indicating the stage of the application during the running of the application, thereby specifying a management policy for the specified data based on the data information of the specified data, or based on the attribute information and the running stage information of the specified data, and by executing the management policy on the specified data, the management policy is applied to the specified data, thereby managing the specified data.

[0162] On this basis, the embodiment of the present application also provides a hierarchical storage management (HSM) module. The HSM module is run by a processor of a computing device and is used to uniformly manage the multi-level storage devices of a storage system. For example, according to the configuration information of the storage system, the remaining storage space of each level of storage device is monitored.

[0163] The configuration information of the storage system may be written into the computing device through a configuration file, for example, into a daemon process and an application process of the computing device, so as to be used when the processor runs the HSM module.

[0164] Taking the above storage system as an example, HSM can be used to manage the MEM, SSD, HDD / BB and PFS of the storage system.

[0165] Exemplarily, the configuration information of the storage system includes information such as storage devices at each level of the storage system, mount points of storage devices at each level, capacity, bandwidth, latency, and whether they are shared storage devices.

[0166] Exemplarily, the configuration information of the storage system included in the configuration file may be as follows:

[0167]

[0168]

[0169] The storage system described in the above configuration information includes four levels, namely memory, localdisk, burst buffer, and pfs (parallel file system).

[0170] The mount point of memory is " / tmpfs", CAP (capacity) is 10240 bytes, BANDWIDTH (bandwidth) is 5000 mbps (megabits per second), LATENCY (latency) is 15 ms (milliseconds), and IS_SHARED_STORAGE (shared storage) is wrong.

[0171] The mount point of the local disk is " / local / tmp", the CAP (capacity) is 20480 bytes, the BANDWIDTH (bandwidth) is 300 mbps, the LATENCY (latency) is 250000 ms, and the IS_SHARED_STORAGE (shared storage) is wrong.

[0172] The mount point of the burst buffer is " / mnt / bb", the CAP (capacity) is 40960 bytes, the BANDWIDTH (bandwidth) is 150 mbps, the LATENCY (latency) is 500000 ms, and the IS_SHARED_STORAGE (shared storage) is correct.

[0173] The mount point of pfs is " / pfs", CAP (capacity) is 102400 bytes, BANDWIDTH (bandwidth) is 70 mbps, LATENCY (latency) is 1000000ms, and IS_SHARED_STORAGE (shared storage) is correct.

[0174] It should be noted that the system architecture and application scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field can know that with the evolution of the system architecture and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0175] For ease of understanding, the data management method provided in the embodiment of the present application is introduced below in combination with the above system architecture.

[0176] The following embodiments of the present application will be divided into three parts to exemplarily introduce the scheme of the data management method.

[0177] Part I: Combination Figure 6 , introduces the process of writing the attribute information of the target data and the running stage information of the application into the application.

[0178] Part II: Combination Figures 7 to 11 , introduces the process of managing application data based on the attribute information of the data in the first part and the running stage information of the application.

[0179] Part III: Combination Figure 12 to Figure 13 ,Based on Part I and Part II, the process of managing the data of the target application is introduced.

[0180] The following, combined Figure 6 , introduces the first part of the embodiment of the present application.

[0181] Figure 6 1 is a flow chart of a data management method according to an exemplary embodiment. Exemplarily, the data management method includes S601-S602.

[0182] S601: Receive attribute information of target data and running stage information of application program.

[0183] The attribute information of the target data is determined by the user based on prior knowledge of the application program and is used to indicate the I / O attributes of the target data.

[0184] It can be understood that the prior knowledge of the application may include the data read and written at different running stages of the application, the number of times different data is read and written during the running of the application, the life cycle of different data of the application, and whether the data of the application is a shared file.

[0185] Exemplarily, the attribute information of the target data may include the operation type, I / O times, life cycle, the running stage of the application when performing I / O operations on the target data, sharing attributes, etc.

[0186] The operation type may include write operations and / or read operations. The I / O count refers to the number of times data is read and written during the running of the application. The life cycle is used to indicate whether the data is a permanent file or a temporary file. The running stage of the application when the I / O operation is performed on the data is used to indicate the running stage of the application when the read operation or write operation is performed on the data. The sharing attribute is used to indicate whether the data is a shared file or a non-shared file of multiple computing devices.

[0187] The running stage information of the application is information determined by the user based on prior knowledge of the application, and is used to indicate the current running stage of the application.

[0188] It can be understood that the prior knowledge of the application may also include the locations of different running stages in the application.

[0189] Exemplarily, the running phases of the application may include an initial running phase, an intermediate running phase, and a final running phase.

[0190] Since the attribute information of the target data is determined by the user based on the prior knowledge of the application, the user can determine the data that needs to be managed (i.e., target data) based on specific I / O problems that arise during the running of the application, such as the impact of the target data on the I / O performance is greater than the impact of non-target data on the I / O performance, such as data that has a greater impact on the I / O performance, data that causes the I / O performance to become a performance bottleneck of the computing device, etc., and write the attribute information of this part of data (i.e., target data) into the application, so that this part of data can be managed during the running of the application by the computing device. This not only helps to manage the application data more intelligently, more specifically and more accurately, thereby more effectively improving the I / O performance of the computing device, but also helps to reduce the amount of data that needs to be managed, thereby reducing the workload of data management.

[0191] In the embodiment of the present application, the data of the application can be divided into three levels from top to bottom, namely: data set (DataSet), group (Group), data item (Data Entity). Among them, the data set contains multiple groups, and a group contains multiple data items. The data set has data set attributes, the group has group attributes, and the data item has data item attributes.

[0192] A data item is an independent data entity accessed during the running of an application. For example, a file, an object, a remote call result, etc. A data item can also be called a data entity.

[0193] A group is a collection of multiple data items with the same attribute information. That is, a group includes multiple data items with the same attribute information. The attribute information of a group is the same attribute information of the multiple data items included in the group. A group can also be called a data group.

[0194] A DataSet is a collection of all data items that are read and written during the running of an application. A DataSet can also be called the data of an application. That is, the data of an application consists of multiple groups, and a group consists of multiple data items.

[0195] When describing the attribute information of a data item, the user can describe the attribute information of a single data item, or put multiple data items with the same attribute information into a group and then describe the attribute information of the group. This helps reduce the user's workload when a large number of data items in an application need to be managed.

[0196] Based on this, the attribute information of the data in S601 may include the attribute information of the data item and / or the attribute information of the group. That is to say, when multiple data items need to be managed, some of the data items may be put into the group, and the multiple data items may be described at the same time through the attribute information of the group, while the attribute information of another part of the data items may be described separately.

[0197] It is understandable that the target data is the data that needs to be managed in the data of the application. The target data may include one data item, or may include multiple data items, and the embodiment of the present application does not limit the number of data items included in the target data. In addition, the target data may also include at least one group, and the embodiment of the present application does not limit the number of groups included in the target data.

[0198] In an embodiment of the present application, a user can input attribute information of target data and running stage information of an application to a computing device through an input device, and the computing device receives the attribute information of the target data and the running stage information of the application in response to the user's input operation.

[0199] Exemplarily, the attribute information and the operating stage information may be input to a processor of the computing device, and the processor receives the attribute information and the operating stage information in response to the input operation.

[0200] In addition, the computing device can also output the attribute information of the target data and the running stage information of the application to the display device, so that the display device can display the attribute information of the target data and the running stage information of the application input by the user. The user can view the input attribute information and running stage information through the display device, which helps to ensure the accuracy of the attribute information and running stage information.

[0201] S602: Based on the data description interface, write the attribute information and running stage information of the target data into the application.

[0202] In an embodiment of the present application, after the computing device receives the attribute information of the target data and the running stage information of the application, it writes the attribute information of the target data and the running stage information of the application into the application based on the data description interface.

[0203] Here, writing the attribute information of the target data and the running stage information of the application into the application can also be considered as writing into the source code of the application.

[0204] In this embodiment, the attribute information and the running stage information of the target data are written into the application. In this way, when the computing device runs the application, the attribute information and the running stage information of the target data can be directly obtained from the application. Compared with obtaining the attribute information and the running stage information of the target data from its file, this helps to improve the convenience and efficiency of obtaining the attribute information and the running stage information of the target attributes.

[0205] It can be understood that the attribute information and operation stage information of the target data in the embodiments of the present application can also be stored in the computing device or the storage system.

[0206] In addition, the embodiments of the present application may also write the attribute information and runtime information of the target data into the application by other means besides the data description interface. That is to say, the embodiments of the present application do not limit the manner in which the attribute information and runtime information of the target data are written into the application, and the above is merely an exemplary description.

[0207] Exemplarily, the processor of the computing device may write the attribute information and the running stage information into the application program based on the data description interface.

[0208] In an embodiment of the present application, the running stage information can be written into different locations of the application, wherein the running stage information at different locations of the application is used to indicate different running stages of the application.

[0209] In an embodiment of the present application, data sets, groups, and data items can all have their attribute information described through a data description interface. That is, the computing device can write the attribute information of the data set, the attribute information of the group, and the attribute information of the data item into the application through the data description interface.

[0210] In the embodiment of the present application, the data description interface includes at least one of a DDI_Init (DDI initialization) interface, a DDI_Create_Droup (DDI creation group) interface, a DDI_Add_Data_Entity (DDI adds attribute information of data) interface, a DDI_Set_Stage (DDI sets the running stage) interface, and a DDI_Finalize (DDI ends) interface. Based on this, the data description interface can be considered as a collection of interfaces.

[0211] Among them, DDI_Init is used to initialize DDI, or to initialize resources for session / DataSet. DDI_Create_Droup is used to create a group, or to initialize resources for Drop. DDI_Add_Data_Entity is used to add data attribute information. DDI_Set_Stage is used to set the application's running stage information. DDI_Finalize is used by DDI to indicate the end of the task.

[0212] In this embodiment, the data description interface is set to include a DDI end interface, and the DDI end interface is added after " / / application running ends". In this way, when the processor runs the DDI end interface, the DDI end interface can send task end information to the data orchestration engine to notify the end of the application running. Based on this, the data orchestration engine can delete the target data in the first storage area, thereby avoiding useless data occupying the first storage area, which helps to improve the utilization rate of the fast storage area.

[0213] Exemplarily, the data description interface in the application, the attribute information of the target data, and the running stage information of the application may be as follows:

[0214] / / 1.session(DataSet) initialization

[0215] DDI Init Para para;

[0216] para.proc_init = false;

[0217] std::unique_ptr <ddi>test_ddi=std::make_unioque <ddi>();

[0218] test_ddi->DDI_Init(para);

[0219] / / 2.Group creation

[0220] std::string groupName_1="group1";

[0221] GroupAttr test_groupAttr_1;

[0222] test_groupAttr_1.ioPattern=IOPattern::RANDOM;

[0223] ...(additional Group attributes added)

[0224] test_ddi->DDI_Creat_Group(groupName_1,test_groupAttr_1);

[0225] / / 3. Add Data Entity1 information

[0226] EntityAttr entity1;

[0227] entity1.entityName=" / home / entity1.txt";

[0228] entity1.operation=Operation::READ;

[0229] entity1.scope=ScopeType::PROCESS;

[0230] entity1.ioStage=Stage::MIDDLE;

[0231] ...(other properties added)

[0232] test_ddi->DDI_Add_Data_Entity(entity1);

[0233] / / Add other Data Entity information

[0234] / / 4. Task running stage settings

[0235] test_ddi->DDI_Set_Stage(xxx)

[0236] / / Calculation process of the application ......

[0238] Open / read / write / close...

[0239] / / End of application execution

[0240] / / 5.DDI task ends

[0241] test_ddi->DDI_Finalize();

[0242] In the above content, / / 1.session (DataSet) is initialized as DDI_Init interface, / / 2.Group is created as DDI_Create_Droup interface, / / 3.Data Entity1 information is added as DDI_Add_Data_Entity interface, / / 4.Task running stage is set as DDI_Set_Stage interface, / / 5.DDI task ends as DDI_Finalize interface. / / Application calculation process, / / Application running end is the application code.

[0243] It should be noted that the application may include both / / 2.Group creation and / / 3.Data Entity1 information addition, or may only include / / 2.Group creation or / / 3.Data Entity1.

[0244] During the running of the application, when the " / / 1.session(DataSet) initialization" part of the code is run, the DDI_Init interface initializes the DDI session (dataset). When the " / / 2.Group creation" part of the code is run, the DDI_Create_Droup interface creates a group, and the attribute information of the group is sent to the data orchestration engine based on the DDI_Create_Droup interface. When the " / / 3.Data Entity1 information addition" part of the code is run, the attribute information of the data item is sent to the data orchestration engine based on the DDI_Add_Data_Entity interface. When the " / / 4.Task running stage setting" part of the code is run, the running stage information is sent to the data orchestration engine based on the DDI_Set_Stage interface. When the " / / 5.DDI task end" part of the code is run, the task end information is sent to the data orchestration engine based on the DDI_Finalize interface to indicate the end of the task.

[0245] Exemplarily, when the DDI_Create_Droup interface creates a group, it includes: creating an identifier for the group, wherein the identifier of the group corresponds to the group one by one, that is, different groups have different identifiers.

[0246] In an embodiment of the present application, during the process of running an application, a computing device reads and writes target data and non-target data of the application, wherein the storage space occupied by the target data is greater than the storage space occupied by the non-target data.

[0247] In this embodiment, by receiving the attribute information of the target data and writing the attribute information of the target data into the application, a management strategy can be determined for the target data, and the target data can be managed, thereby achieving management of large data (i.e., data occupying a large storage space) read and written during the running of the application. Compared with writing the attribute information of non-target data into the application, a management strategy can be formulated and managed for large data (i.e., data occupying a large storage space) read and written during the running of the application, which helps to more effectively improve the I / O performance of the computing device. Compared with writing the attribute information of both the target data and the non-target data into the application, it helps to reduce the amount of data that needs to be managed.

[0248] The following, combined Figures 7 to 11 , introduces the second part of the embodiment of the present application.

[0249] Figure 7 1 is a flow chart of a data management method according to an exemplary embodiment. Exemplarily, the data management method includes S701-S703.

[0250] S701: Acquire attribute information of target data.

[0251] The data type of the target data may be file, audio, video, image or binary data, etc. It should be noted that the above is only an exemplary description, and the embodiment of the present application does not limit the specific form of the target data.

[0252] In an embodiment of the present application, during the process of running an application program, before performing read and write operations on target data, the computing device obtains attribute information of the target data based on a data description interface.

[0253] Exemplarily, during the process of the processor of the computing device running the application, the attribute information of the target data is sent to the data orchestration engine through the data description interface in the application. For example, the attribute information of the group can be sent to the data orchestration engine through the DDI_Create_Droup interface in the data description interface, or the attribute information of the data item can be sent to the data orchestration engine through the DDI_Add_Data_Entity interface in the data description interface.

[0254] Among them, sending the attribute information of the target data to the data orchestration engine through the data description interface can be considered as the processor calling the data description interface to send the attribute information of the target data to the data orchestration engine, or it can also be considered as the processor sending the attribute information of the target data to the data orchestration engine when running the data description interface.

[0255] In this embodiment, the user's prior knowledge of the application (i.e., the attribute information of the target data and the running stage information) is written into the application through the data description interface, thereby providing the attribute information of the target data and the running stage information to the computing device. In addition, in the process of running the application, the attribute information of the target data and the running stage information of the application are sent to the data orchestration engine through the data description interface, thereby establishing a bridge between the user's prior knowledge and the data orchestration engine, so that the data orchestration engine can formulate a management strategy for the target data according to the attribute information of the target data and the running stage information of the application, and manage the target data of the application according to the formulated management strategy.

[0256] like Figure 8 , which is a schematic diagram of a data management method provided in an embodiment of the present application.

[0257] See also Figure 8 In the process of running the application, the attribute information of the target data is sent to the data orchestration engine through the data description interface and the RPC service, for example, to the data orchestration policy library module of the data orchestration engine, so that the data orchestration engine can manage the target data through the file system.

[0258] In the embodiment of the present application, the attribute information includes multiple situations, which are explained below through situations 1 to 6.

[0259] Case 1: The attribute information includes an operation type of the target data, and the operation type includes a write operation and / or a read operation.

[0260] Exemplarily, the operation type of the target data is a write operation, which means that the target data is generated during the running of the application program, and after the target data is generated, the processor writes the target data into the storage system.

[0261] It is understandable that the target data generated during the running of an application can be temporary files, non-temporary files, and permanent files. Among them, temporary files refer to files used once during the running of an application, and non-temporary files refer to files that are used throughout the life cycle of the application (i.e., used multiple times), usually configuration files. Permanent files refer to files output during the running of an application, and are usually used to record the running results obtained by running the application.

[0262] Exemplarily, the operation type of the target data is a read operation, which means that the target data is data stored in a storage system, and the target data will be read from the storage system during the running of the application program.

[0263] Exemplarily, the operation type of the target data is a write operation and a read operation, which can also be called a write-before-read type, or a write-and-read type, which means that the target data will be generated during the running of the application program, and after the target data is generated, the processor writes the target data into the storage system. Afterwards, during the running of the application program, the processor also reads the previously written target data from the storage system.

[0264] Case 2: The attribute information includes a target count, which indicates the number of times the target data is read during the running of the application.

[0265] Exemplarily, the target number of times may be 1, or may be N, where N is a positive integer greater than 1.

[0266] Case 3: The attribute information includes a target running phase, which is used to indicate a phase of performing an I / O operation on target data during the running of an application.

[0267] It can be understood that performing an I / O operation on target data can also be considered as performing a write operation / read operation on the target data.

[0268] When the operation type of the target data is a write operation, the first operation stage is an operation stage for executing the write operation on the target data. When the operation type is a read operation, the first operation stage can be considered as an operation stage for executing the read operation on the target data.

[0269] Exemplarily, the running phase of the application may include an initial phase, an intermediate phase, and an end phase. The target running phase may be any one of the initial phase, the intermediate phase, and the end phase.

[0270] It should be noted that the division of the running stages of the above-mentioned application is only for illustrative purposes and is not limited to this in the embodiments of the present application.

[0271] By setting the attribute information to include the first operating stage, the application's use requirements for the target data can be sent to the data orchestration engine, so that the data orchestration engine can determine the application's use requirements for the target data based on the attribute information of the target data, that is, at which specific stage of the application the target data is used, and then determine the management strategy of the target data and the timing of executing the management strategy on the target data based on the application's use requirements for the target data, which helps to improve the accuracy and diversity of the management strategy of the target data.

[0272] Case 4: The attribute information includes a life cycle, which is used to indicate whether the data item is a permanent file or a temporary file.

[0273] Among them, permanent files refer to files that need persistent storage, while temporary files refer to files that do not need persistent storage.

[0274] Case 5: The attribute information also includes a sharing attribute, which is used to indicate whether the target data is a shared file or a non-shared file.

[0275] In the embodiment of the present application, when the application is run by multiple computing devices, the sharing attribute of the target data is used to indicate whether the target data is a shared file or a non-shared file of the multiple computing devices.

[0276] Exemplarily, the multiple computing devices include a first computing device and a second computing device, and the target data generated during the first computing device running an application is data required to be used during the second computing device running the application, or the first computing device and the second computing device in the multiple computing devices both use the target data during the running of the application, then the sharing attribute of the target data item indicates that the target data is a shared file, that is, the target data is a shared file of the first computing device and the second computing device.

[0277] Case 6: The attribute information includes the storage space occupied by the target data.

[0278] The storage space occupied by the target data may also be referred to as the size of the target data.

[0279] It should be noted that the attribute information of the target data may include one or more of the above-mentioned situations 1 to 6, and the embodiment of the present application does not limit this.

[0280] In the embodiment of the present application, the attribute information of the target data may also include the storage path of the target data. Thus, when the pre-read management strategy is executed on the target data, the initial storage location of the target data may be determined based on the storage path of the target data, thereby pre-reading the target data from the initial storage location to the storage location specified in the management strategy.

[0281] In the embodiment of the present application, the attribute information of the target data may include attribute information of the data item and / or attribute information of the group.

[0282] Therein, a group includes multiple data items with corresponding attribute information.

[0283] When the attribute information of the target data includes the attribute information of the data item, the target data includes the data item. Since the minimum unit of reading and writing data during the running of the application is the data item, setting the attribute information of the target data to include the attribute information of the data item helps to improve the richness and completeness of the attribute information of the target data.

[0284] When the attribute information of the target data includes the attribute information of the group, the target data includes multiple data items with corresponding attribute information. Since the group includes multiple data items with the same attribute information, the attribute information of the target data is set to include the attribute information of the group. The attribute information of the group can be used to simultaneously describe the attribute information of multiple data items. In this way, it is not only possible to avoid repeated storage of the same attribute information, thereby reducing the storage space occupied by the attribute information, but also possible to reduce the workload when setting the attribute information of multiple data items. In addition, when it is necessary to manage multiple data items, the attribute information of multiple data items can be obtained at one time, thereby helping to improve the efficiency of obtaining the attribute information.

[0285] like Fig. 9 The figure is a schematic diagram of the attribute information of the group provided in the embodiment of the present application.

[0286] See also Fig. 9 The attributes of the group may include path, operation, open count, life cycle, stage, etc.

[0287] Among them, the path is used to represent the native path of the target data (that is, the storage path), the business is used to represent the operation type of the target data, the number of opens is used to represent the number of times the target data is read and written during the running of the application, and the stage is used to represent the running stage of the application when reading and writing the target data.

[0288] Need to explain, Fig. 9 The attribute information of the group shown is only an example. For the attribute information of the group, reference may be made to the relevant description of the attribute information of the target data mentioned above, which will not be repeated here.

[0289] For example, see Fig. 9 , the path is " / input / input.txt", that is, the target data (i.e., input.txt file) is stored in the input directory. The operation is "read operation", that is, the operation type of the target data is a read operation. The number of opens is "read only once", that is, the target data is read once during the running of the application. The life cycle is "temporary file generated during the running of the application", that is, the target data is a temporary file, not a permanent file. The stage is "I / O request is made in the first running stage of the running application", that is, the target data is read in the first running stage of the application.

[0290] Understandably, Fig. 9 The target data in the group shown has a native path of / input / input.txt. A read operation is performed on it during the running of the application. The number of reads is 1. The life cycle is short. It is a temporary file and is read during a specific running phase (i.e., the first running phase) of the running application.

[0291] In the embodiment of the present application, the attribute information and the value of the attribute information can be represented by a key-value pair, wherein the attribute information of a group and the value of the attribute information, or the attribute information of a data item and the value of the attribute information can be represented by a key-value pair.

[0292] Exemplarily, the attribute information is the key in the key-value pair, and the value of the attribute information is the value in the key-value pair. Fig. 9 Take the attribute information shown as an example, where the attribute information refers to path, business, number of opens, life cycle, stage, etc., and the value of the attribute information refers to / input / input.txt, read operation, read only once, temporary files generated during the running of the application, and I / O requests made during the first running stage of the running application.

[0293] S702: Determine a management strategy for the target data based on the attribute information of the target data.

[0294] In the embodiment of the present application, after the computing device obtains the attribute information of the target data, it determines the management strategy of the target data based on the attribute information of the target data.

[0295] Exemplarily, after receiving the target data attribute information sent through the data description interface, the data orchestration engine determines the management policy of the target data based on the attribute information of the target data. For example, the data orchestration policy library module of the data orchestration engine receives the target data attribute information, and the data orchestration policy library module determines the management policy of the target data, or the data orchestration policy library module and the data placement engine determine the management policy of the target data.

[0296] In the embodiment of the present application, a management strategy is formulated for the target data according to the attribute information of the target data, which not only helps to improve the matching degree between the management strategy and the target data, but also helps to improve the richness of the management strategy. In addition, the data management method provided in the embodiment of the present application can manage the target data, that is, the data corresponding to the attribute information in the application program can be managed. Compared with managing only a small amount of data that is accessed multiple times, the attribute information of the data that has a greater impact on the I / O performance can be written into the application program, so that more targeted management of improving the I / O performance can be achieved, thereby helping to improve the I / O performance, and further helping to improve the operating efficiency of the application program.

[0297] There are multiple implementation methods for determining the management strategy of target data. Two implementation methods are exemplified below.

[0298] In a first implementation, when a read or write instruction of target data is received, a management strategy of the target data is determined based on attribute information of the target data.

[0299] In the second implementation, the management strategy of the target data is determined based on the operation stage information and the attribute information of the target data. This implementation is exemplarily described below through the following steps 1 and 2.

[0300] The running stage information includes first stage information, which is used to indicate that the application is in the first running stage.

[0301] Step 1: Get the first stage information of the application.

[0302] In the embodiment of the present application, during the process of running the application, the computing device obtains the first stage information of the application based on the data description interface.

[0303] Exemplarily, during the process of the processor of the computing device running the application, the first stage information of the application is sent to the data arrangement engine through the data description interface. For example, the first stage information of the application program can be sent to the data arrangement engine through the DDI_Set_Stage interface in the data description interface.

[0304] Step 2: If the first operation stage is earlier than the target operation stage, determine the management strategy of the target data based on the attribute information.

[0305] The target operation phase is the operation phase of the application when the target data is read and written.

[0306] In an embodiment of the present application, if the first operation stage is earlier than the target operation stage, the computing device determines a management strategy for the target data based on attribute information of the target data.

[0307] Exemplarily, the data orchestration engine may determine the management policy of the target data based on the attribute information of the target data.

[0308] In an embodiment of the present application, it is possible to determine whether the current running stage of the application is earlier than the running stage of the application when the target data is read and written based on the target running stage indicated by the attribute information.

[0309] In this embodiment, the running stage information configured in the application includes the first stage information, indicating that the application is in the first running stage. Based on this, when the first stage information is obtained, it can be determined that the application has run to the first running stage, and when the first running stage is earlier than the target running stage, the management policy is determined, and the management policy of the target data can be determined before the target data is read and written, thereby helping to ensure that the management policy is executed on the target data when the computing device reads and writes the target data or before the target data is read and written, thereby helping to avoid the inability to read and write based on the management policy when reading and writing the target data.

[0310] In an embodiment of the present application, different attribute information of the target data can be used to determine different management strategies for the target data. The following, in combination with situations 1 to 6 corresponding to the attribute information of the target data in S701 above, various situations of management strategies for the target data are exemplified through the following situations a to f.

[0311] In case a, the management strategy includes: when a write operation is performed on target data, the target data is written into the first storage area.

[0312] Based on the fact that the operation type of the target data in the above situation 1 includes a write operation, the management strategy of the target data may include: when a write operation is performed on the target data, writing the target data into the first storage area.

[0313] In this way, when a write operation is performed on the target data, the target data can be written to the first storage area. Compared with writing the target data to a non-first storage area, this helps to increase the write speed of the target data, thereby helping to reduce the time of I / O operations when running applications, and improving the I / O performance of the computing device when running applications.

[0314] In case b, the management strategy includes: flushing the target data written in the first storage area to the non-first storage area.

[0315] Based on the fact that the operation type of the target data in the above situation 1 includes a write operation and the life cycle in situation 4 indicates that the target data is a permanent file, the management strategy may include: flushing the target data written in the first storage area to a non-first storage area, and deleting the target data in the first storage area. This helps to avoid permanent files written in the first storage area occupying the storage space of the first storage area, thereby helping to improve the utilization rate of the first storage area.

[0316] In case c, the management strategy includes: pre-reading the target data into the first storage area before performing a read operation on the target data.

[0317] Based on the fact that the operation type of the target data in the above situation 1 includes a read operation, the management strategy may include: pre-reading the target data into the first storage area before performing the read operation on the target data.

[0318] In this way, the target data can be pre-read into the first storage area before performing a read operation on the target data, and then when a read operation is performed on the target data, the target data can be read from the first storage area. Compared with reading the target data from a non-first storage area, this helps to increase the reading speed of the target data, thereby helping to reduce the time of I / O operations when running applications, and improving the I / O performance of the computing device when running applications.

[0319] In addition, based on the fact that the operation type of the target data in the above situation 1 includes a read operation and the stage in which the read operation is performed on the target data during the running application in the above situation 3 is the target running stage, the management strategy may include: pre-reading the target data to the first storage area in the running stage before the target running stage.

[0320] For example, when running to the second running stage of the application, if the second running stage is earlier than the target running stage, the target data that needs to be read in the running stage after the second running stage (i.e., the target running stage) is pre-read into the first storage area, so that when running to the target running stage of the application, the target data can be read directly from the first storage area.

[0321] The operation phase before the target operation phase may be any operation phase before the target operation phase.

[0322] Exemplarily, the running stage of the application may include an initial stage, an intermediate stage, and an end stage. If the target running stage is the end stage, the target data may be pre-read into the first storage area in the initial stage or the intermediate stage. If the target running stage is the intermediate stage, the target data may be pre-read into the first storage area in the initial stage.

[0323] In this embodiment, based on the attribute information indicating the running stage when the target data is read (i.e., the target running stage), when formulating the management strategy for pre-reading the target data, it is clearly stated in which running stages of the application the pre-reading is to be performed (i.e., the running stage before the target running stage). This helps to ensure the timely execution of the pre-reading management strategy, that is, pre-reading the target data into the first storage area before the target data is read.

[0324] Since the target data is pre-read into the first storage area in the operation stage before the target operation stage, compared with pre-reading the target data into the first access area in the operation stage in which the read operation is performed on the target data (i.e., the first operation stage), it helps to ensure that the target data can be read from the first storage area when the read operation is performed on the target data at any time point in the first operation stage.

[0325] In case d, the management strategy includes: deleting target data in the first storage area in a running phase after a target running phase of the application program.

[0326] Based on the fact that the operation type of the target data in the above situation 1 is a read operation and that the stage in which the read operation is performed on the target data during the running of the application in situation 3 is the target running stage, the management strategy of the target data may include: after the target running stage of the application is over, deleting the target data in the first storage area. This helps to avoid useless target data occupying the storage space of the first storage area, thereby helping to improve the utilization rate of the first storage area.

[0327] In case e, the management strategy includes: when the number of times the target data is read is equal to the target number of times, deleting the target data in the first storage area.

[0328] Based on the fact that the operation type of the target data in the above situation 1 is a read operation and that the target data is read a target number of times during the running of the application in situation 2, the management strategy of the target data may include: when the target data is read a target number of times, deleting the data item in the first storage area. This helps to avoid useless target data occupying the storage space of the first storage area, thereby helping to improve the utilization rate of the first storage area.

[0329] Case f: the management policy includes: writing or pre-reading the target data into the shared storage device in the first storage area.

[0330] Based on the sharing attribute in the above case 1 and case 5 indicating that the target file is a shared file, the management strategy of the target data includes: writing or pre-reading the target data to the shared storage device.

[0331] Since the shared storage device is a storage layer shared by multiple computing devices, that is, any computing device can access the shared storage device, only one target file needs to be stored, and multiple computing devices can read it, avoiding different computing devices from storing shared files separately, thereby avoiding shared files from occupying too much storage space.

[0332] It should be noted that the management strategy of the target data may include one or more of the above-mentioned situations a to f, and the embodiment of the present application does not limit the number of management strategies for the target data.

[0333] In the embodiment of the present application, the policy pool of the data orchestration policy library module includes multiple management policies. Therefore, the processor can select a management policy for the target data from the policy pool of the data orchestration policy library module based on the attribute information of the target data.

[0334] In the embodiment of the present application, for the above situation a and situation c, S702 may include: determining a management strategy for the target data based on the attribute information of the target data and the storage capacity of each tier storage device in the first storage area.

[0335] It should be noted that this solution can be used in combination with step 1 and step 2 included in the above S702, or can be used alone, and the embodiments of the present application do not limit this.

[0336] Exemplarily, when a management policy is determined based on the attribute information and the storage capacity of each level of storage devices, the determined management policy may specifically include writing or pre-reading the target data into a target level storage device in the first storage area.

[0337] The target-level storage device may be the storage device with the largest available storage space in the first storage area, or the storage device with the second largest available storage space in the first storage area, or any storage device with non-smallest available storage space in the first storage area.

[0338] In this embodiment, when determining the management strategy corresponding to a read operation or a write operation, the storage capacity of each hierarchical storage device in the first storage area can be comprehensively considered in combination with the storage capacity of each hierarchical storage device, so as to select a suitable placement location for the target data, thereby avoiding the situation where some hierarchical storage devices are overused while other hierarchical storage devices are not fully utilized.

[0339] Optionally, the management policy of the target data includes a first management policy and a second management policy, and the management policy of the target data is determined based on the attribute information of the target data and the storage capacity of each level storage device in the first storage area, including:

[0340] Step 3: Determine a management strategy for the target data based on the attribute information and the first storage capacity of each level storage device in the first storage area, wherein the first storage capacity is the storage capacity of each level storage device at the first moment.

[0341] In the embodiment of the present application, the moment before reading and writing the target data includes a first moment and a second moment, and the second moment is later than the first moment. Based on this, the computing device first determines the management strategy of the target data at the first moment based on the attribute information and the first storage capacity of each level storage device in the first storage area.

[0342] Step 4: Update the target data management policy based on the second storage capacity of each level of storage device, wherein the second storage capacity is the storage capacity of each level of storage device at the second moment.

[0343] Based on the above, at the second moment, the computing device updates the management policy of the target data based on the second storage capacity of each level storage device to obtain an updated management policy.

[0344] It should be noted that step 4 can be executed after the management policy determined in step 3 is executed, or it can be executed before the management policy determined in step 3 is executed. The embodiment of the present application does not limit this.

[0345] It should be noted that the above steps three and four can be used in combination with the above steps one and two, or can be used alone, and the embodiments of the present application do not limit this.

[0346] In this embodiment, the management policy of the target data is first determined based on the first storage capacity of each hierarchical storage device, and then the management policy of the target data is updated based on the second storage capacity of each hierarchical storage device. Since the second storage capacity is the storage capacity of each hierarchical storage device at the latest moment, updating the management policy of the target data based on the second storage capacity helps to ensure that the management policy of the target data is determined based on the latest storage capacity of each hierarchical storage device, which helps to ensure the accuracy and rationality of the management policy of the target data.

[0347] Optionally, the above step 4 may include: determining a management policy for the target data based on the storage space occupied by the target data and the available storage space of each tier storage device in the first storage area.

[0348] The management strategy includes writing or pre-reading the target data to a target storage device in the first storage area; and the available storage space of the target storage device is larger than the available storage space of the non-target storage device.

[0349] It can be understood that the available storage space of the storage device can also be considered as the remaining storage space of the storage device.

[0350] That is to say, when the management policy of the target data is determined to be written or pre-read into the first storage area, the target storage device to which the target data is written / pre-read into the first storage area can be determined based on the size of the target data and the available storage space of each level of storage devices in the first storage area.

[0351] In the embodiment of the present application, the data orchestration policy library module and the data placement engine may determine the management policy of the target data based on the storage space occupied by the target data and the available storage space of each level storage device in the first storage area.

[0352] Exemplarily, when the data orchestration policy library module determines that the initial management policy of the target data is to write or pre-read the target data into the first storage area, the data orchestration policy library module sends the initial management policy of the target data to the data placement engine, and the data placement engine determines the target storage device for writing or pre-reading the target data into the first storage area based on the size of the target data and the available storage space of each level storage device, that is, determines the placement position of the target data in the first storage area, and returns the identifier of the target storage device to the data orchestration policy library module, so that the data orchestration policy library module determines the management policy of the target data based on the identifier of the target storage device.

[0353] It should be noted that the implementation method of the above step three can refer to the implementation method of step four, which will not be repeated here.

[0354] In the embodiment of the present application, the target storage device includes multiple implementation methods, which are described below through methods a to c.

[0355] Hereinafter, the first storage area includes a first-tier storage device, a second-tier storage device, and a third-tier storage device as an example to illustrate methods a to c.

[0356] Method a: The available storage space of the target storage device is larger than the available storage space of the non-target storage device.

[0357] Exemplarily, when the first-tier storage device is the target storage device, the available storage space of the first-tier storage device is greater than the available storage space of the second-tier storage device and the third-tier storage device.

[0358] Method b: The available storage space of the target storage device is greater than or equal to a preset threshold.

[0359] Exemplarily, when the available storage space of the first-tier storage device is greater than or equal to a preset threshold, and the available storage space of the second-tier storage device and the third-tier storage device is less than the predicted threshold, the first-tier storage device is the target storage device.

[0360] Exemplarily, when the available storage space of the second-tier storage device and the third-tier storage device is greater than or equal to the preset threshold, and the available storage space of the first-tier storage device is less than the preset threshold, the second-tier storage device or the third-tier storage device can be the target storage device. In other words, the target storage device can be a storage device of any tier whose available storage space is greater than or equal to the preset universe.

[0361] It should be noted that in the embodiments of the present application, there is no restriction on the specific value of the preset threshold value. The preset threshold value can be determined based on parameters such as the size of the target data (i.e., the storage space occupied by the target data), the available storage space of each level of storage devices in the first storage area, and the total storage space of each level of storage devices.

[0362] Method c: The available storage space of the target storage device is larger than the storage space occupied by the target data.

[0363] Exemplarily, when the first-tier storage space, the second-tier storage space, and the third-tier storage space are all larger than the storage space occupied by the target data, the target storage device can be a storage device at any level among the first-tier storage space, the second-tier storage space, and the third-tier storage space.

[0364] It should be noted that the above-mentioned methods a to c can be used simultaneously, or can be used separately, and the embodiments of the present application do not limit this.

[0365] It should be noted that the first storage area in the embodiment of the present application may also include two levels of storage devices. In this case, the relevant description of the target storage device can refer to the above-mentioned methods a to c and will not be repeated here.

[0366] It should be noted that, when the first storage area includes multiple shared storage devices, the implementation method of "writing or pre-reading the target data to the shared storage device in the first storage area" can refer to the implementation method of "writing / pre-reading the target data to the first storage area". The principles of the two are the same and will not be repeated here.

[0367] In the embodiment of the present application, after determining the management policy of the target data, the computing device can store the management policy of the target data, so that during the running of the application program, the target data can be managed according to the management policy determined in advance for the target data.

[0368] Exemplarily, the management policy of the target data may establish an association relationship with the target data through an identifier of the target data, so that when the management policy of the target data needs to be used, the management policy of the target data may be determined according to the identifier of the target data.

[0369] It should be noted that the embodiment of the present application does not limit the storage location of the management policy of the target data. For example, it can be stored in a storage system, or it can also be stored in a computing device.

[0370] It is understandable that the target data may have one or more management policies, and the embodiment of the present application does not limit the number of management policies for the target data. In addition, the management policies for different target data may be the same, or may be different, and the embodiment of the present application does not limit this.

[0371] Optionally, S702 may include the following steps five to six.

[0372] Step 5: Create an application namespace; the namespace includes the target data.

[0373] In the embodiment of the present application, the namespace of the application can be created by the processor.

[0374] Exemplarily, when the data orchestration engine receives the attribute information of the target data sent through the data description interface, the data orchestration engine sends notification information to the application session management module, and after receiving the notification information, the application session management module creates a namespace of the application program.

[0375] The name of the namespace includes the name and creation time of the application. For example, if the name of the application is A and the creation time is 15:31 on January 21, 2023, the name of the namespace can be A2023.1.21.15.31.

[0376] It should be noted that the embodiment of the present application does not limit the accuracy of the creation time. For example, the creation time can be accurate to minutes, or it can also be accurate to seconds, etc.

[0377] By setting the name of the namespace to include the creation time, applications that are run at different times have different names, which helps avoid duplication of namespace names of applications that are run at different times. In addition, the namespace name includes the name of the application, so that the application to which the namespace belongs can be accurately known, which helps distinguish the namespaces of different applications when the processor runs multiple applications at the same time.

[0378] Step 6: Based on the attribute information of the target data, determine the management strategy of the target data in the namespace.

[0379] In an embodiment of the present application, the processor may determine a management policy for the target data in a namespace based on the attribute information of the target data, so that the management policy is only applicable to applications in the namespace.

[0380] It should be noted that steps five to six can be used simultaneously with the above steps one to two, and steps three to six, or they can be used separately, and the embodiments of the present application do not limit this.

[0381] Since different namespaces can be set with different management permissions, that is, the management policies of different namespaces are isolated from each other, therefore, by creating a namespace for the application and determining the management policy of the target data in the namespace, the management policy of the target data can be made to apply only to the application to which the namespace belongs. In this way, on the one hand, the management policies of different applications can be isolated from each other, thereby preventing the management policies of different applications from interfering with each other, and further preventing the I / O operations of different applications from interfering with each other when the computing device runs multiple applications at the same time. On the other hand, since the namespace can be automatically created and the management policy of the target data in the namespace can be determined, the user does not need to specify the application to which the target data belongs in the attribute information, which helps to reduce the requirements for users and improve the user experience.

[0382] S703: Execute the management policy on the target data.

[0383] There are multiple ways to implement the management policy on the target data. Two methods are described below.

[0384] Method 1: The computing device applies the management policy to the target data, thereby implementing the management policy on the target data.

[0385] For example, a processor of a computing device may apply the management policy to the target data.

[0386] In method 2, the computing device sends the management policy to the storage system to instruct the storage system to apply the management policy to the target data, thereby implementing the management policy on the target data.

[0387] Exemplarily, the computing device processor may send the management policy to the controller of the storage system, and the controller applies the management policy to the target data in response to the received management policy.

[0388] In the embodiment of the present application, the management policy is executed on the target data before the read and write operations are performed on the target data, or during the read and write operations are performed on the target data, or after the read and write operations are performed on the target data.

[0389] Exemplarily, the management policies in the above cases a and f are executed during the process of performing read and write operations on the target data. The management policies in the above cases b, d, and e are executed after performing read and write operations on the target data. The management policy in the above case c is executed before performing read and write operations on the target data.

[0390] There are multiple implementation methods for executing management policies. Three implementation methods are exemplified below.

[0391] The first implementation method is to execute the management policy on the target data based on the management policy of the target data, for example, the management policies in case a and case b.

[0392] The second implementation method is to execute the management policy on the target data based on the management policy of the target data and the running stage information of the application program, for example, the management policies in case c, case d, and case f.

[0393] In a third implementation, a management policy is executed on the target data based on the management policy of the target data and the I / O operation information of the target data, for example, the management policy in case e.

[0394] It should be noted that the first to third implementation modes mentioned above are applicable to the modes 1 to 2 in the above S703.

[0395] The second implementation is exemplarily described below through steps one and two.

[0396] In the embodiment of the present application, the running stage information in the application also includes second stage information, indicating that the application is in the second running stage. The second running stage can be any one of the multiple running stages of the application.

[0397] Step 1: Obtain the second stage information.

[0398] In the embodiment of the present application, the second stage information can be obtained by the processor. Exemplarily, during the running of the application, the second stage information in the application is sent to the data arrangement engine through the data description interface.

[0399] It can be understood that when the second stage information is obtained, it indicates that the application has entered the second running stage.

[0400] Step 2: Based on the information from the second stage, execute management strategies on the target data.

[0401] In this embodiment, the second-stage information of the application is obtained so that the relationship between the current running stage of the application and the target running stage can be determined. Therefore, based on the second-stage information of the application, executing management policies on target data helps to ensure the timeliness and reliability of the execution of management policies for target data.

[0402] In the embodiment of the present application, step two may include: if the second operation stage is earlier than the target operation stage, pre-reading the target data into the first storage area.

[0403] In this embodiment, when the second operation stage is earlier than the target operation stage, the target data is pre-read into the first storage area, thereby realizing the operation stage before reading the target data, and pre-reading the target data into the first storage area, which helps to read the target data from the first storage area at any time point in the target operation stage.

[0404] Alternatively, in the embodiment of the present application, step two may include: the second operation stage is later than the target operation stage, and the target data pre-fetched into the first storage area is deleted.

[0405] In this embodiment, when the second operation stage is later than the target operation stage, the target data pre-fetched into the first storage area is deleted, thereby achieving timely deletion of the target data in the first storage area after the use cycle of the target data ends, which helps to improve the utilization rate of the first storage area.

[0406] In the embodiment of the present application, the processor sends the management policy of the target data to the controller of the storage system based on the second stage information of the application program to instruct the controller to apply the management policy to the target data. For example, when the second operation stage is earlier than the target operation stage, the management policy of pre-reading the target data to the first storage area is sent to the controller, and when the second operation stage is later than the target operation stage, the management policy of deleting the target data in the first storage area is sent to the controller.

[0407] Exemplarily, the data orchestration policy library module may send the management policy of the target data to the data movement engine to instruct the data movement engine to apply the management policy to the target data.

[0408] The third implementation method is exemplarily described below through steps three and four.

[0409] Step 3: Obtain I / O operation information of target data.

[0410] The I / O operation information indicates one or more of an open operation, a close operation, and an unlink operation performed on target data during the running of the application.

[0411] In the embodiment of the present application, during the process of running the application, the processor of the computing device obtains the I / O operation information of the target data. Exemplarily, the I / O operation information of the target data can be sent to the data orchestration engine through the I / O perception module.

[0412] See also Figure 8 ,During the running of the application, the I / O perception module obtains the I / O operation ,information of the target data based on the native I / O operation ,behavior of the target data during the running of the ,application, and sends the I / O operation information to the data ,orchestration engine through the RPC service.

[0413] Step 4: Execute a management policy on the target data based on the I / O operation information of the target data.

[0414] In the embodiment of the present application, the processor may send the management policy of the target data to the controller of the storage system to instruct the controller to execute the management policy on the target data.

[0415] Exemplarily, the data orchestration policy library module sends the management policy of the target data to the data movement engine, and the data movement engine executes a predetermined management policy on the target data, for example, pre-reading the target data into the first storage area.

[0416] In this implementation, since the I / O operation information indicates the reading and writing status of the computing device on the target data, such as opening the target data, closing the target data, unlinking the target data, etc., executing management policies on the target data based on the I / O operation information of the target data helps to improve the timeliness and effectiveness of the execution of the management policies.

[0417] Based on situation 3 in S701, the above step 4 may include: if the I / O operation information indicates that the current number of read and write times of the target data is equal to the target number of times, deleting the target data in the first storage area.

[0418] Here, one opening operation and one closing operation are performed on the target data, and it is determined that the target data is read once.

[0419] In an embodiment of the present application, the processor determines the number of times the target data is read based on the I / O operation information. If the number of times the target data is read is equal to the target number, the processor sends a delete instruction to the controller of the storage system to instruct the controller to delete the target data in the first storage area.

[0420] Exemplarily, the data orchestration policy library module sends a deletion instruction to the data move engine to instruct the data move engine to delete the target data in the first storage area.

[0421] In this embodiment, since the I / O operation information indicates the I / O operation status of the computing device on the target data, determining the current read and write times of the target data based on the I / O operation information helps to improve the accuracy of the current read and write times. In addition, when the current read and write times are equal to the target times, deleting the target data in the first storage area can determine that the computing device is no longer reading the target data, which helps to improve the accuracy of the management policy execution.

[0422] The following, combined Fig.10 , the process of data management in the embodiments of the present application is exemplarily introduced.

[0423] Fig.10 Another schematic diagram of data management provided in an embodiment of the present application.

[0424] See also Fig.10 On the application side, the data description interface is called to send the attribute information (attribute information of the group and / or attribute information of the data item) and the running stage information to the data orchestration strategy library module through the RPC service (server). The I / O perception (Observer) module determines the I / O operation information of the target data based on the perceived native I / O operation behavior of the target data during the running application, such as open, close, unlink, etc., and sends the I / O operation information of the target data to the data orchestration strategy library module through the RPC service.

[0425] It should be noted that attribute information can also be called data information (data msg), which mainly includes static information of data. Runtime information can also be called control information (ctl msg), which mainly includes dynamic information of data.

[0426] On the daemon process side, the data orchestration policy library module determines the management policy of the target data based on the attribute information of the target data, or based on the attribute information of the target data and the operation stage information. Exemplarily, after the data orchestration policy library module preprocesses the attribute information, it determines the management policy of the target data from the policy pool based on the preprocessed attribute information. Afterwards, the data orchestration policy library module sends the management policy of the target data to the data movement engine, and the data movement engine moves or deletes the target data to complete the management policy of the target data.

[0427] In addition, the application session management module saves metadata information during the application program execution (eg, attribute information of target data, current storage path of target data, target data read and written during each application program execution, etc.), for example, in a database (DB).

[0428] In addition, the data arrangement strategy library module can also read the metadata in the database through the application session management module. The HSM module is used to manage storage devices at all levels of the storage system.

[0429] In the embodiment of the present application, there are multiple implementation methods for how to execute the management policy on the target data. The following is an illustrative description of the execution process of the management policy in the above-mentioned situations a to f through methods 1 to 5.

[0430] Mode 1: Based on the management policy in the above situation a, S703 may include: writing the target data into the first storage area.

[0431] In the embodiment of the present application, when the processor writes target data to the storage system, the management policy of the target data is determined. If the management policy of the target data is the management policy in the above case a, the processor writes the target data to the first storage area of ​​the storage system. For example, the processor may send the target data and the management policy of the target data to the controller of the storage system, instructing the controller to write the target data to the first storage area.

[0432] Exemplarily, the processor may determine the management policy of the target data from a storage device storing the management policy according to an identifier of the target data.

[0433] like Fig.11 As shown, it is a schematic diagram of a data management method provided by this embodiment.

[0434] See also Fig.11 , the first-level storage device of the storage system is MEM, the second-level storage device is SSD, the third-level storage device is HDD / BB, and the fourth-level storage device is PFS. Among them, the first storage area of ​​the storage system includes the first-level storage device to the third-level storage device, and the second storage area of ​​the storage system includes the fourth-level storage device.

[0435] The data of the application includes write.txt, write.txt is the target data, and the operation type of write.txt is a write operation. The management strategy of write.txt may include: writing write.txt into the first storage area. Exemplarily, writing write.txt into the first-tier storage device in the first storage area.

[0436] Based on this, the process of executing the management policy on write.txt may include: writing write.txt into the first-tier storage device.

[0437] Mode 2: Based on the management policy in the above situation b, S703 may include: flushing the target data written in the first storage area to the second storage area.

[0438] In the embodiment of the present application, after the target data is written into the first storage area, if the target data is a permanent file, the target data in the first storage area is flushed to the non-first storage area, and the target data in the first storage area is deleted. For example, the processor may send a management policy for deleting the target data in the first storage area to the controller, and the management policy received by the controller is applied to the target data.

[0439] Exemplarily, based on method 1, if write.txt is a permanent file, the management strategy of write.txt may further include: flushing write.txt in the first storage area to the fourth-tier storage device.

[0440] Based on this, the process of executing the management policy on write.txt may further include: flashing write.txt in the first-tier storage device to the fourth-tier storage device, and deleting write.txt in the first-tier storage device.

[0441] Exemplarily, the data moving engine may flush write.txt from the first-tier storage device to the fourth-tier storage device.

[0442] Mode 3, based on the management policy in the above situation c, S703 includes: before performing a read operation on the target data, pre-reading the target data into the first storage area.

[0443] The data of the application also includes 1.txt, 2.txt, and 3.txt, wherein 1.txt, 2.txt, and 3.txt are target data, and the operation types of 1.txt, 2.txt, and 3.txt are all read operations, and are all stored in the fourth-level storage device (PFS).

[0444] The management strategy of 1.txt includes a first management strategy and a second management strategy. The first management strategy of 1.txt may include pre-reading 1.txt to a second-tier storage device (i.e., SSD) in the first storage area, and the second management strategy of 1.txt may include pre-reading 1.txt to a first-tier storage device (i.e., MEM). The management strategy of 2.txt may include pre-reading 2.txt to a second-tier storage device in the first storage area, and the management strategy of 3.txt may include pre-reading 3.txt to a third-tier storage device (HDD / BB) in the first storage area.

[0445] It can be understood that the second management policy can be considered as a management policy obtained by updating the first management policy based on the available storage space of each level of storage devices in the first storage device.

[0446] Based on this, see Fig.11 The process of executing the management policy on 1.txt is as follows: before reading 1.txt, based on the first management policy, 1.txt is pre-read to the second-tier storage device; based on the second management policy, 1.txt is pre-read to the first-tier storage device. In this way, when a read operation is performed on 1.txt, 1.txt can be read from the first-tier storage device, which can significantly improve the reading speed compared to reading 1.txt from the fourth-tier storage device or the second-tier storage device.

[0447] It should be noted that the process of executing the management policy on 2.txt and 3.txt can refer to the above-mentioned process of executing the management policy on 1.txt, which will not be repeated here.

[0448] Exemplarily, the data moving engine may pre-read 1.txt to the second-tier storage device and the first-tier storage device, pre-read 2.txt to the second-tier storage device, and pre-read 3.txt to the third-tier storage device.

[0449] There are multiple implementations for how to pre-read the target data into the first storage area before performing a read operation on the target data. The following is an exemplary description using Method A and Method B.

[0450] Mode A: When the attribute information of the target data includes the target operation stage, before performing a read operation on the target data, pre-reading the target data into the first storage area may include: the following steps 1 to 2.

[0451] In the embodiment of the present application, the running stage information includes the third stage information, wherein the third stage information is used to indicate the running stage of the application.

[0452] Step 1: Get the third stage information in the application.

[0453] Based on this, when the processor obtains the second stage information in the application, it can determine the current running stage of the application according to the third stage information.

[0454] In one example, the third stage information includes a third stage start instruction, which is used to instruct to run to the third running stage of the application, and the third running stage is the running stage before the target running stage. When the processor executes the third stage start instruction, when the processor obtains the third running stage information, based on the third running stage information, it can be determined that the current running stage of the application is the third running stage.

[0455] In another example, the third operation stage information includes a target stage start instruction, which is used to instruct the application to run to the target operation stage, that is, the application runs to the stage of performing a read operation on the target data. Based on this, when the processor executes the target stage start instruction, when the processor obtains the second stage information, it can determine that the current operation stage of the application is the target operation stage based on the second stage information.

[0456] In another example, the third running stage is the running stage before the target running stage, and the third stage information includes a third stage end instruction, which is used to indicate that the third running stage of the application program has ended, that is, the processor is about to run the target running stage of the application program. Based on this, when the processor executes the second stage end instruction, when the processor obtains the second stage information, it can be determined that the third running stage of the application program has ended based on the second stage information.

[0457] Step 2: Based on the second-stage information of the application program, pre-read the target data into the first storage area.

[0458] In an exemplary embodiment, if the second stage information indicates that the current running stage of the application is the third running stage, and the third running stage is a running stage before the target running stage, the target data is pre-read into the first storage area.

[0459] Since the attribute information indicates that the running stage for performing the read operation on the target data is the target running stage, when it is determined that the current running stage of the application is the third running stage, the target data is pre-read into the first storage area. In this way, it can be ensured that when the target data is read at any time in the target running stage, the target data can be read from the first storage area of ​​the storage system.

[0460] In another example, if the second stage information indicates that the application has been executed to a target execution stage, the target data is pre-read into the first storage area.

[0461] In yet another example, if the second stage information indicates that the third running stage of the application program ends, the third running stage is a running stage before the target running stage, and the target data is pre-read into the first storage area.

[0462] Since the attribute information of the target data indicates that the running stage for performing the read operation on the target data is the target running stage, when it is determined that the running has reached the target running stage of the application, or the third running stage of the application has ended (that is, the processor is about to run the target running stage of the application), the target data is pre-read into the first storage area, which helps to avoid occupying the storage space of the first storage area too early, thereby helping to improve the utilization rate of the first storage area.

[0463] Mode B: Before executing a read operation on the target data, the processor pre-reads the target data into the first storage area, including: the following steps three to four.

[0464] Step 3: Obtain a to-be-read instruction, which is used to indicate that target data is to be read.

[0465] Exemplarily, the application may include a target data to be read instruction. When the processor executes the target data to be read instruction, the processor may determine that a read operation is to be performed on the target data.

[0466] Step 4: In response to the to-be-read instruction, the processor pre-reads the target data into the first storage area.

[0467] Exemplarily, after the processor obtains the instruction to read the target data, the processor pre-reads the target data from the non-first storage area of ​​the storage system to the first storage area of ​​the storage system, so that when the processor performs a read operation on the target data, the target data can be read from the first storage area of ​​the storage system. For example, the processor sends a management policy for the target data to the controller (i.e., pre-reading the target data to the first storage area), and the controller applies the management policy to the target data in response to the received management policy.

[0468] Exemplarily, the instruction to be read establishes an association relationship with the target data through the identifier of the target data. After the processor obtains the instruction to be read, it determines the target data that should be pre-read into the first storage area based on the identifier of the target data.

[0469] It can be understood that one instruction to be read is used to indicate one target data, that is, there is a one-to-one correspondence between the instruction to be read and the target data.

[0470] Mode 4, based on the management policy in the above situation d, S703 may include: after the target running phase of the application program ends, deleting the target data in the first storage area.

[0471] After the target running phase of the application program ends, deleting the target data in the first storage area may include steps five and six.

[0472] In the embodiment of the present application, the running stage information includes fourth stage information, wherein the fourth stage information is used to indicate the running stage of the application.

[0473] It should be noted that in the embodiment of the present application, the running stage information may include multiple stages of information, and the multiple stages are respectively used to indicate the stage in which the application is located.

[0474] In addition, the first stage information, second stage information, third stage information and fourth stage information mentioned in the embodiments of the present application are for illustrative purposes of different schemes of the embodiments of the present application, and are not for distinguishing different stage information. Therefore, the first stage information, second stage information, third stage information and fourth stage information can indicate the same running stage of the application, or can also indicate different running stages of the application. The embodiments of the present application do not limit this.

[0475] Step 5: Get the fourth stage information in the application.

[0476] Based on this, the processor obtains the fourth stage information in the application program and can determine the current running stage of the application program according to the fourth stage information.

[0477] In one example, the fourth stage information may include a fourth stage start instruction for indicating that the application is running to the fourth running stage, where the fourth running stage is a stage after the target running stage. When the processor executes the fourth stage start instruction, when the processor obtains the fourth stage information, it can determine that the current running stage of the application is the fourth running stage based on the fourth stage information.

[0478] In another example, the fourth current operation stage information includes a target stage end instruction, which is used to indicate that the target operation stage of the application program has ended. Based on this, when the processor executes the target stage end instruction, when the processor obtains the fourth stage information, it can determine that the target operation stage of the application program has ended based on the fourth stage information.

[0479] It should be noted that for other related instructions about step five, please refer to the related instructions of step one in the above method A, which will not be repeated here.

[0480] Step six: based on the fourth stage information of the application, delete the target data in the first storage area.

[0481] In one example, if the fourth stage information indicates that the running stage of the application is the fourth running stage, since the fourth running stage is a stage after the target running stage, the target data in the first storage area is deleted.

[0482] In another example, if the fourth stage information indicates that the target running stage of the application program ends, the target data in the first storage area is deleted.

[0483] Since the attribute information of the data item indicates that the stage for performing the read operation on the target data is the target operation stage, that is, the stage for using the target data is the target operation stage, therefore, when the current operation stage is the fourth operation stage or the target operation stage of the application ends, the target data in the first storage area is deleted. In this way, it is possible to avoid storing useless target data in the first storage area, thereby avoiding useless target data occupying the storage space of the first storage area, and improving the storage space utilization rate of the first storage area.

[0484] Mode 5, based on the management policy in the above situation e, S703 may include: if the number of times the data item is read is equal to the target number of times, deleting the target data in the first storage area.

[0485] In the embodiment of the present application, the processor obtains I / O operation information of the target data, and determines the number of times the target data is read according to the I / O operation information. If the number of times the data item is read is equal to the target number, the target data in the first storage area is deleted.

[0486] Exemplarily, the processor may obtain the I / O operation information of the target data in real time, or the processor may obtain the I / O operation information of the target data at a preset time interval. If the target data is opened once and closed once, it is considered that the target data is read once.

[0487] It should be noted that the embodiments of the present application do not limit the specific value of the preset time interval, and can be determined based on the frequency of the processor performing I / O operations. If the frequency of the processor performing I / O operations is relatively high, the preset time interval should be set to be smaller so that the actual number of reads of the data item can be accurately recorded. If the frequency of the processor performing I / O operations is relatively low, the preset time interval should be set to be larger, which helps to avoid too frequent execution of operations when the actual number of reads is accurate.

[0488] Mode 6, based on the management policy in the above situation f, S503 includes: writing or pre-reading the target data into the shared storage device in the first storage area.

[0489] In one example, when the processor writes target data to the storage system, if the management policy corresponding to the target data is the management policy in the above situation f, the processor writes the target data to the shared storage device. For example, the processor sends the target data and the management policy of the target data (i.e., writing the target data to the shared storage device) to the controller, and the controller applies the management policy to the target data in response to the received target data and management policy, that is, writing the target data to the shared storage device.

[0490] In another example, when the processor pre-reads the target data, if the management policy of the pre-read target data is the management policy in the above situation f, the processor pre-reads the target data to the shared storage device. For example, the processor sends the management policy of the target data to the controller (i.e., pre-reading to the shared storage device), and the controller pre-reads the target data to the shared storage device in response to the received management policy.

[0491] Exemplarily, when the third-tier storage system is a shared storage device in the first storage area, the processor writes or pre-reads the target data into the third-tier storage system in the first storage area.

[0492] Optionally, performing a read operation or a write operation on the target data includes: performing a read operation or a write operation on the target data based on a native I / O interface in the application.

[0493] Referring to the application shown in S701, the " / / application computing process portion includes Open / read / write / close..." of the application, wherein "Open / read / write / close..." is the native I / O interface. Therefore, when running the computing process of the application, the embodiment of the present application performs read and write operations on the target data based on the native I / O interface (i.e., Open / read / write / close...).

[0494] Exemplarily, performing a read operation or a write operation on target data based on a native I / O interface may be that the computing device performs a read operation or a write operation on target data based on a native I / O interface of a storage system.

[0495] Based on this, the interface for managing data (i.e., the data description interface) and the interface for reading and writing data (i.e., the native I / O interface) are independent of each other, so that the embodiment of the present application can perform a read operation or a write operation on the target data based on the native I / O interface, that is, the computing device can not only manage the data of the application based on the data description interface, but also read and write the target data based on the native I / O interface. In other words, there is no need to convert the native I / O interface into an interface that matches the data description interface.

[0496] Since the data management method provided by the embodiment of the present application does not need to modify the native I / O interface, the data of the application can be managed. In this way, on the one hand, the user does not need to learn and use a new I / O interface, nor does the native I / O interface need to be modified. On the other hand, compared with the need to modify the native I / O interface to read and write the data of the application, it avoids the need to perform a large number of operations to modify the native I / O interface when there are many I / O operations, resulting in a sharp increase in the time cost of modifying the I / O interface, thereby effectively improving the speed of I / O operations. In addition, since there are many types of I / O interfaces, using a universal native I / O interface for reading and writing helps to improve read and write performance and ensure the extensibility of the I / O interface.

[0497] The following, combined Figure 12 to Figure 13 , introduces the third part of the embodiment of this application.

[0498] The target application is an HPC application or an E-class application. HPC applications, also known as HPC applications, refer to applications that use high-performance computing. E-class applications refer to applications that can perform up to 100 trillion calculations per second.

[0499] For example, on an E-class computing device, the amount of I / O data of an HPC application can reach the TB level. Among them, the proportion of I / O operations of the computing device can reach about 86%, and the I / O operations of the cache layer, the reading and writing of variable data, etc. account for about 14%. Since the proportion of I / O operations of computing devices is relatively large, the I / O performance of computing devices has a particularly large impact on computing performance. In addition, users who use HPC applications have certain prior knowledge of the HPC application. Based on the prior knowledge, users can provide attribute information and operation stage information of the HPC application data.

[0500] Exemplarily, the user's prior knowledge of the HPC application includes input files, output files, and the size of the output files, etc. For example, the input file includes in_1.vtk, which is stored in the second-tier storage device (non volatile memory express, NVME) of the storage system. The output files include out_1.dat, ..., out_n.dat, and the size of the output file is 10 gigabytes (GB), which are stored in the fourth-tier storage device (PFS).

[0501] like Fig.12 , which is a diagram of the operation process of the HPC application in the related technology provided in the embodiment of the present application.

[0502] See also Fig.12 , the running process of the HPC application is as follows: after starting the running of the HPC application, the processor initialization and communication initialization are performed. After that, the processor runs the HPC application. During the running of the HPC application, the processor reads the input file (i.e., in_1.vtk) from the second-tier storage device. After that, the calculation process of the HPC application is performed according to in_1.vtk, and the output file (i.e., out_1.dat, ..., out_n.da) is obtained. Since the total amount of the output file is relatively large, the storage capacity of each level of storage devices in the first storage area is insufficient, and the computing device writes the output file to the fourth-tier storage device in the second storage area.

[0503] Processor initialization refers to the check and start of the availability and completeness of system resources required by HPC applications, as well as the check of the system operating environment. Communication initialization refers to some pre-operations performed by HPC applications that need to use communication services before information transmission, including setting network parameters, checking communication addresses, and checking network status.

[0504] like Fig.13 , which is a flow chart of data management for HPC applications provided in an embodiment of the present application.

[0505] Based on the data management method provided in this implementation, the user adds attribute information of the target data of the HPC application in the initial part of the HPC application, and adds operation stage information in different operation stages of the HPC application.

[0506] See also Fig.13 After starting the HPC application, the processor initialization and communication initialization are performed. During the running of the HPC application, the processor loads the data description interface library (libddi.so) and sends the attribute information of the input file and the attribute information of the output file in the HPC application to the RPC service on the daemon side. The data orchestration engine on the daemon side obtains the attribute information of the input file and the attribute information of the output file through the RPC service, and determines the management strategy of the input file and the management strategy of the output file according to the attribute information of the input file and the attribute information of the output file.

[0507] In addition, the processor loads the data description interface library (libddi.so) and can also send the running stage information in the HPC application to the RPC service on the daemon side. The data orchestration engine on the daemon side obtains the running stage information of the HPC application through the RPC service, and determines the management policy of the input file and the management policy of the output file based on the running stage information of the HPC application, so that the management policy can be determined for the input file before the read operation is performed on the input file, and the management policy can be determined for the output file before the write operation is performed on the output file.

[0508] Exemplarily, the attribute information of the input file includes the operation type, which is a read operation. The attribute information of the output file includes the operation type, the size of the output file, and the life cycle of the output file, which is a write operation, the size of the output file is 10GB, and the life cycle indicates that the output file is a permanent file.

[0509] The data orchestration engine determines, based on the attribute information of the input file, that the management strategy of the input file includes pre-reading the input file to the first-tier storage device (MEM). The data orchestration engine determines, based on the attribute information of the output file, that the management strategy of the output file includes writing the output file to the first-tier storage device and the second-tier storage device of the first storage area, and flushing the output file in the first storage device and the second storage device to the fourth-tier storage device in the second storage area.

[0510] In addition, when running HPC applications, the processor passes the running stage information in the HPC application to the RPC service on the daemon side by loading the data description interface library (libddi.so). After the data orchestration engine on the daemon side obtains the running stage information of the HPC application through the RPC service, it can execute management policies on input files and / or output files based on the running stage information of the HPC application.

[0511] In addition, when running HPC applications, the processor can also load the I / O perception library (libobserver.so) to send the I / O operation information determined by the I / O perception module to the RPC service on the daemon side. The data orchestration engine on the daemon side obtains the input and output file I / O operation information through the RPC service, and executes management policies on the input and output files based on the input and output file I / O operation information.

[0512] In the process of running the HPC application, the process of executing the management policy on the input file includes: before performing a read operation on the input file (in_1.vtk), pre-reading the input file from the second-tier storage device to the first-tier storage device. In this way, when performing a read operation on the input file, the input file can be read from the first-tier storage device.

[0513] Since the reading speed of the first-tier storage device is higher than that of the second-tier storage device, after managing the input files of the HPC application based on the data management method provided in the embodiment of the present application, the speed of reading the input files can be increased, thereby effectively improving the I / O performance of the computing device and reducing the time of I / O operations when running the HPC application, thereby helping to improve the operating efficiency of the HPC application and avoid I / O performance becoming a performance bottleneck for the HPC application.

[0514] After the calculation results are obtained, the process of executing the management policy on the output files includes: when performing a write operation on the output files (out_1.dat, ..., out_n.dat), the output files are first written to the first-tier storage device and the second-tier storage device. After that, the output files in the first-tier storage device and the second-tier storage device are flushed to the fourth-tier storage device, and the output files in the first-tier storage device and the second-tier storage device are deleted.

[0515] Since the writing speed of the first-tier storage device and the second-tier storage device is higher than that of the fourth-tier storage device, after managing the output files of the HPC application based on the data management method provided in this embodiment, the speed of writing the output files can be increased, thereby effectively improving the I / O performance of the computing device and reducing the time of I / O operations when running the HPC application, thereby helping to improve the operating efficiency of the HPC application and avoid I / O performance becoming a performance bottleneck for the HPC application.

[0516] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. In order to realize the above functions, the data management device includes a hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.

[0517] The embodiment of the present application can exemplarily divide the functional modules of the data management device according to the above method. For example, the data management device can include various functional modules corresponding to the various functional divisions, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0518] For example, Fig.14 A possible structural diagram of the data management device (referred to as data management device 1400) involved in the above embodiment is shown. For example, the data management device may be the data arrangement engine 500 in the above embodiment. The data management device 1400 includes an acquisition unit 1401, a determination unit 1402, and an execution unit 1403. The acquisition unit 1401 is used to acquire the attribute information of the target data. For example, Figure 7 S701 shown. The determination unit 1402 is used to determine the management strategy of the target data based on the attribute information. For example, Figure 7 S702 shown. The execution unit 1403 is used to execute the management policy on the target data. For example, Figure 7 S703 shown.

[0519] It can be understood that the acquisition unit 1401 and the determination unit 1402 can be the data orchestration policy library module 510 in the data orchestration engine 500. The execution unit 1403 can be the data movement engine 540 in the data orchestration engine 500.

[0520] Optionally, the application further includes a data description interface; the acquisition unit 1401 is specifically used to: acquire attribute information of the target data based on the data description interface.

[0521] Optionally, the application further includes running stage information, and the running stage information is used to indicate the running stage of the application; the determination unit 1402 is specifically used to determine the management strategy of the target data based on the attribute information of the target data and the running stage information.

[0522] Optionally, the running stage information includes first stage information; the determination unit 1402 is specifically used to: obtain the first stage information; the first stage information is used to indicate that the application is in the first running stage; if the first running stage is earlier than the target running stage, based on the attribute information of the target data, determine the management strategy of the target data; the target running stage is the running stage of the application when the target data is read and written.

[0523] Optionally, the computing device is configured with a first storage area, and the read and write speed of the first storage area is greater than the read and write speed of the non-first storage area configured with the computing device; the attribute information indicates that the operation type of the target data includes a write operation, and the management policy includes writing the target data to the first storage area when performing a write operation on the target data.

[0524] Optionally, the attribute information indicates that the target data is a permanent file, and the management strategy includes flushing the target data written in the first storage area to a non-first storage area.

[0525] Optionally, the attribute information indicates that the operation type of the target data includes a read operation and the stage in which the target data is read is a target operation stage, and the management strategy includes pre-reading the target data to a first storage area configured in the computing device in an operation stage before the target operation stage.

[0526] Optionally, the management strategy includes deleting the target data in the first storage area after the life cycle of the target data ends.

[0527] Optionally, the management strategy further includes deleting the target data pre-read into the first storage area in an operation phase after the target operation phase.

[0528] Optionally, the execution unit 1403 is specifically used to: execute a management policy on the target data based on the operation stage information.

[0529] Optionally, the running stage information also includes second stage information; the execution unit 1403 is specifically used to: obtain the second stage information; the second stage information is used to indicate that the application is in the second running stage; if the second running stage is earlier than the target running stage, pre-read the target data into the first storage area; if the second running stage is later than the target running stage, delete the target data pre-fetched into the first storage area.

[0530] Optionally, the determining unit 1402 is specifically configured to determine a management policy for the target data based on the attribute information of the target data and the storage capacity of each level storage device in the first storage area.

[0531] Optionally, the determination unit 1402 is specifically used to: determine the management policy of the target data based on the attribute information and the first storage capacity of each hierarchical storage device in the first storage area; the first storage capacity is the storage capacity of each hierarchical storage device at the first moment; based on the second storage capacity of each hierarchical storage device, update the management policy of the target data; the second storage capacity is the storage capacity of each hierarchical storage device at the second moment, and the second moment is later than the first moment.

[0532] Optionally, the attribute information indicates a target number of times, which is used to indicate the number of times the target data is read during the running of the application; the management strategy also includes deleting the target data pre-read into the first storage area when the number of times the target data is read is equal to the target number of times.

[0533] Optionally, the execution unit 1403 is specifically used to: execute a management policy on the target data based on I / O operation information of the target data; the I / O operation information is used to indicate one or more of an open operation, a close operation and a link cancellation operation performed on the target data.

[0534] Optionally, the execution unit 1403 is specifically used to: obtain I / O operation information of the target data; determine the number of times the target is read based on the I / O operation information; if the number of times the target data is read is equal to the target number, delete the target data pre-read into the first storage area.

[0535] Optionally, the target data includes multiple data items, and the attribute information of the target data includes attribute information of the data items and / or attribute information of a group, wherein the group includes multiple data items with the same attribute.

[0536] Optionally, the determining unit 1402 is specifically configured to: create a namespace of the application; the namespace includes target data; and determine a management strategy for the target data in the namespace of the application based on the attribute information.

[0537] Optionally, the execution unit 1403 is further configured to: perform a read operation or a write operation on the target data based on a native I / O interface of the application program.

[0538] For the detailed description of the above optional methods, please refer to the above method embodiments, which will not be repeated here. In addition, the explanation and description of the beneficial effects of any of the above data management devices 1400 can refer to the above corresponding method embodiments, which will not be repeated here.

[0539] The present application also provides a computing device, for example, Figure 1A , Figure 1B The computing device 110 shown in the figure includes a processor and a memory, the processor is connected to the memory, the memory stores computer-executable instructions, and the processor implements the data processing method in the above embodiment when executing the computer-executable instructions. The embodiment of the present application does not impose any restrictions on the specific form of the computing device. For example, the computing device can be a terminal device or a network device. Among them, the terminal device can be called: terminal, user equipment (UE), terminal equipment, access terminal, user unit, user station, mobile station, remote station, remote terminal, mobile device, user terminal, wireless communication device, user agent or user device, etc. The terminal device can be a mobile phone, augmented reality (AR) device, virtual reality (VR) device, tablet computer, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), etc. The network device can be a server, etc. Among them, the server can be a physical or logical server, or it can be two or more physical or logical servers that share different responsibilities and cooperate with each other to realize the functions of the server.

[0540] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed on a computer, the computer is enabled to execute any of the methods executed by the computing devices provided above.

[0541] For the explanation of the relevant contents and description of the beneficial effects of any of the above-mentioned computer-readable storage media, reference may be made to the above-mentioned corresponding embodiments, which will not be repeated here.

[0542] The embodiment of the present application also provides a chip. The chip integrates a control circuit and one or more ports for implementing the functions of the above-mentioned computing device. Optionally, the functions supported by the chip can be referred to above and will not be repeated here. A person of ordinary skill in the art can understand that all or part of the steps of implementing the above-mentioned embodiment can be completed by instructing the relevant hardware through a program. The program can be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a random access memory, etc. The above-mentioned processing unit or processor can be a central processing unit, a general-purpose processor, an application specific integrated circuit (ASIC), a microprocessor (digital signal processor, DSP), a field programmable gate array (field programmable gatearray, FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof.

[0543] The embodiment of the present application also provides a computer program product including instructions, when the instructions are run on a computer, the computer executes any one of the methods in the above embodiments. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more servers that can be integrated with the medium. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., an SSD), etc.

[0544] It should be noted that the above-mentioned devices for storing computer instructions or computer programs provided in the embodiments of the present application, such as but not limited to the above-mentioned memories, computer-readable storage media and communication chips, etc., are all non-transitory.

[0545] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing computer program instructions on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. Computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, data center, etc. that contains one or more servers that can be integrated with a medium. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (eg, a solid state disk (SSD)).

[0546] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the present application for which protection is claimed, those skilled in the art may understand and implement other variations of the disclosed embodiments by viewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "one" or "an" does not exclude multiple situations. A single processor or other unit may implement several functions listed in a claim. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0547] Although the present application has been described in conjunction with specific features and embodiments thereof, it is obvious that various modifications and combinations may be made thereto without departing from the spirit and scope of the present application. Accordingly, this specification and the drawings are merely exemplary illustrations of the present application as defined by the appended claims, and are deemed to have covered any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, those skilled in the art may make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.< / ddi> < / ddi>

Claims

1. A data management method, characterized in that: Applied to a computing device, the computing device reads and writes target data during the process of running an application program, and the application program includes attribute information of the target data; the method comprises: Acquiring attribute information of the target data; Determining a management strategy for the target data based on the attribute information of the target data; The management policy is executed on the target data.

2. The method according to claim 1, characterized in that The application also includes a data description interface; the acquisition of the attribute information of the target data includes: Based on the data description interface, attribute information of the target data is acquired.

3. The method according to claim 1 or 2, characterized in that: The application further includes running stage information, and the running stage information is used to indicate the running stage of the application; the determining the management strategy of the target data based on the attribute information of the target data includes: A management strategy for the target data is determined based on the attribute information of the target data and the operation stage information.

4. The method according to claim 3, characterized in that The operation stage information includes first stage information; the determining the management strategy of the target data based on the attribute information of the target data and the operation stage information includes: Acquire the first stage information; the first stage information is used to indicate that the application is in the first running stage; If the first running stage is earlier than the target running stage, a management strategy for the target data is determined based on the attribute information of the target data; the target running stage is the running stage of the application when the target data is read and written.

5. The method according to any one of claims 1 to 4, characterized in that The computing device is configured with a first storage area, and the read and write speed of the first storage area is greater than the read and write speed of the non-first storage area configured on the computing device; the attribute information indicates that the operation type of the target data includes a write operation, and the management policy includes writing the target data to the first storage area configured on the computing device when performing a write operation on the target data.

6. The method according to claim 5, characterized in that The attribute information indicates that the target data is a permanent file, and the management strategy includes flushing the target data written in the first storage area to the non-first storage area.

7. The method according to any one of claims 1 to 4, characterized in that The attribute information indicates that the operation type of the target data includes a read operation and the stage in which the target data is read is a target operation stage, and the management strategy includes pre-reading the target data to a first storage area of ​​the computing device in an operation stage before the target operation stage.

8. The method according to claim 7, characterized in that The management strategy further includes deleting the target data pre-read into the first storage area in an operation phase after the target operation phase.

9. The method according to claim 7 or 8, characterized in that: The executing the management policy on the target data includes: Based on the operation stage information, the management policy is executed on the target data.

10. The method according to claim 5 or 7, characterized in that: The determining the management strategy of the target data based on the attribute information of the target data includes: A management policy for the target data is determined based on the attribute information of the target data and the storage capacity of each tier storage device in the first storage area.

11. The method according to claim 10, characterized in that The determining the management strategy of the target data based on the attribute information of the target data and the storage capacity of each level storage device in the first storage includes: Determine a management policy for the target data based on the attribute information and a first storage capacity of each hierarchical storage device in the first storage area; the first storage capacity is the storage capacity of each hierarchical storage device at a first moment; Based on the second storage capacity of each of the hierarchical storage devices, the management policy of the target data is updated; the second storage capacity is the storage capacity of each of the hierarchical storage devices at a second moment, and the second moment is later than the first moment.

12. The method according to any one of claims 7 to 11, characterized in that: The attribute information indicates a target number, which is used to indicate the number of times the target data is read during the running of the application; the management strategy also includes deleting the target data pre-read into the first storage area when the number of times the target data is read is equal to the target number.

13. The method according to claim 12, characterized in that The executing the management policy on the target data includes: The management policy is executed on the target data based on I / O operation information of the target data; the I / O operation information is used to indicate one or more of an open operation, a close operation and a link cancellation operation executed on the target data.

14. The method according to any one of claims 1 to 13, characterized in that The target data includes a plurality of data items, and the attribute information of the target data includes attribute information of the data items and / or attribute information of a group, wherein the group includes a plurality of data items with the same attribute.

15. The method according to any one of claims 1 to 14, characterized in that The determining, based on the attribute information, a management strategy for the target data includes: Creating a namespace of the application; the namespace including the target data; Based on the attribute information, a management policy for the target data in the namespace of the application is determined.

16. A data management device, characterized in that: include: An acquisition unit, used for acquiring attribute information of target data; a determining unit, configured to determine a management strategy for the target data based on the attribute information of the target data; An execution unit is used to execute the management policy on the target data.

17. A data management device, characterized in that: include: Processor and memory; The processor is connected to a memory, the memory is used to store computer-executable instructions, and the processor executes the computer-executable instructions to enable the computing device to implement the method according to any one of claims 1 to 15.

18. A computing device, characterized in that Comprising the data management device as claimed in claim 16 or claim 17.

19. A computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium, characterized in that: When the instructions are executed on a computing device, the computing device is caused to execute the method according to any one of claims 1 to 15.

Images