Data access method and apparatus, and storage medium

Abstract

A data access method: according to the user submitted cache strategy for the first application, the cache resource is scheduled for the first application, and according to the user submitted mapping directory information, the data is pre-fetched into the cache resource of the first application. Subsequently, in the process of running the first application, the cache resource of the first application is accessed according to the cache strategy. Thus, the user's demand is perceived, and then the resource usage of the application is controlled according to the user's demand, thereby improving the application performance.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data access method, apparatus and storage medium. Background Technology

[0002] In data centers, large cluster systems are typically used to provide a shared application execution environment for multiple users. This cluster system usually includes a management node and multiple compute nodes. For any application to be run, the management node can allocate a corresponding compute node, which then runs the application. However, this means users have no control over how their applications use the compute node's resources, leading to application performance failing to meet user needs. Summary of the Invention

[0003] This application provides a data access method, apparatus, and storage medium, which can control the resource usage of an application according to user needs, so that the application performance meets user requirements. The technical solution is as follows:

[0004] In a first aspect, a data access method is provided, the method comprising: receiving a cache configuration request submitted by a user for a first application, the cache configuration request including a cache policy and mapping directory information, the cache policy indicating the cache requirements of the first application, and the mapping directory information being information of a first directory where the application data of the first application is stored in a storage system; scheduling cache resources for the first application according to the cache policy; prefetching the application data of the first application into the cache resources of the first application according to the mapping directory information; and accessing the cache resources of the first application according to the cache policy during the operation of the first application.

[0005] As described above, the method schedules cache resources for the first application based on the user-submitted caching strategy and prefetches data into the application's cache resources according to the user-submitted mapping directory information. Subsequently, during the operation of the first application, the cache resources are accessed according to the caching strategy. Therefore, the data access method provided in this application can sense user needs and control application resource usage accordingly, thereby improving application performance.

[0006] In one possible implementation, the process of scheduling cache resources for the first application according to the caching strategy includes: determining the resource requirement information of each of the multiple tasks of the first application according to the caching strategy, wherein the resource requirement information includes the size of the cache space required by each task and the type of storage medium included; and allocating cache space for each task according to the resource requirement information of each task.

[0007] In this application, the first application can be divided into multiple tasks. This allows the user to specify the resource requirements of each task within the first application, enabling the cluster system to allocate cache space to the corresponding tasks based on these requirements. The resource requirements of different tasks can be the same or different.

[0008] In one possible implementation, the mapping directory information includes the directory path of the first directory, and the process of prefetching the application data of the first application into the cache resources of the first application according to the mapping directory information includes: determining the directory identifier of the subdirectory corresponding to each task in the multiple tasks of the first application; obtaining the data in the subdirectory corresponding to each task stored in the first directory from the storage system according to the directory path of the first directory and the directory identifier of the subdirectory corresponding to each task; and storing the data in the subdirectory corresponding to each task into the cache resources of the first application.

[0009] By using the mapping directory information specified by the user, the data of the first application is prefetched into the cache resources allocated to the first application. This eliminates the need for the user to perform explicit data copying, reducing the complexity of the user experience.

[0010] In one possible implementation, the process of accessing the cache resources of the first application according to the caching strategy includes: when the caching strategy includes a hierarchical caching strategy, according to the hierarchical caching strategy, caching different types of task data of each task of the first application to the corresponding type of storage medium in the cache resources of the first application; when the caching strategy includes a data consistency strategy, when accessing task data in any cache space, performing a locking operation on the accessed task data.

[0011] In this application, data can be cached and accessed in the cache resources of the first application according to the data caching and access strategies in the user-specified caching strategy. For example, data can be cached according to a tiered caching strategy to improve data access performance and save cache space resources. Data can be accessed according to a data consistency strategy to ensure data accuracy during data access. In addition, users can flexibly customize other strategies to achieve flexible settings for data caching and access methods.

[0012] In one possible implementation, before accessing the cached resources of the first application according to the caching policy, the method further includes: obtaining an input / output (IO) request; if the data accessed by the IO request is data under the first directory indicated by the mapping directory information, then the step of accessing the cached resources of the first application according to the caching policy is performed.

[0013] In this application, by setting mapping directory information, access to the first directory indicated by the mapping directory information can be directly intercepted, and then data access can be achieved by accessing the cache resources allocated to the first application, which improves access efficiency and the whole process is imperceptible to the user.

[0014] In one possible implementation, the method further includes: obtaining the bandwidth requirements of data to be migrated to the storage system from the cache resources of each of a plurality of applications, the plurality of applications including the first application; allocating IO bandwidth to the data to be migrated from the cache resources of the first application according to the bandwidth requirements; and storing the data to be migrated from the cache resources of the first application to the storage system according to the IO bandwidth.

[0015] In this application, when each compute node detects that the amount of data in its allocated cache resources for applications has reached a second threshold, it can request bandwidth from the management node to allocate the bandwidth required for migrating the data of the applications it runs. The management node can allocate IO bandwidth for migrating data to the applications running on each compute node based on the collected bandwidth requirements of each compute node. This controls the amount of data migrated from each compute node to the storage system, thereby preventing the application data access volume of different compute nodes from exceeding the available bandwidth of the storage system and thus avoiding I / O bandwidth contention. This achieves orderly access to the storage system by each application in a global view, thereby reducing application performance issues caused by IO contention. Furthermore, in this embodiment, the cluster system can automatically complete data copying based on the user-specified mapping directory information, eliminating the need for manual data copying by the user and reducing operational complexity.

[0016] Secondly, a data access method is provided, the method comprising: a management node receiving a cache configuration request for a first application submitted by a user, the cache configuration request including a cache policy and mapping directory information, the cache policy indicating the cache requirements of the first application, and the mapping directory information being information of a first directory where the application data of the first application is stored in a storage system; the management node scheduling cache resources for the first application from a target computing node according to the cache policy; the management node controlling the target computing node to prefetch the application data of the first application into the cache resources of the first application according to the mapping directory information, and controlling the target computing node to access the cache resources of the first application during the operation of the first application through the cache policy.

[0017] In this application, the management node schedules cache resources for the first application based on the caching policy submitted by the user, and controls the computing node to prefetch data into the cache resources of the first application based on the mapping directory information submitted by the user. Subsequently, during the operation of the first application, the management node controls the computing node to access the cache resources of the first application according to the caching policy. Therefore, this embodiment can sense user needs and control the application's resource usage accordingly, thereby improving application performance.

[0018] In one possible implementation, the management node schedules cache resources for the first application from target computing nodes according to the caching policy, including: the management node obtaining resource requirement information for each of the multiple tasks of the first application from the caching policy, the resource requirement information including the size of the cache space required by each task and the type of storage medium included; the management node allocating target computing nodes to execute each task of the first application according to the resource requirement information; and the management node sending the caching policy to the target computing nodes to instruct the target computing nodes to allocate cache space for the corresponding tasks from their own cache space according to the resource requirement information in the caching policy.

[0019] In this application, the management node can control the resource usage of the computing node by the first application according to the caching policy specified by the user, thereby enabling the user to control the resource usage of the computing node and thus enabling the application performance of the first application to better meet the user's needs.

[0020] In one possible implementation, the mapping directory information includes the directory path of the first directory. The process by which the management node controls the target computing node to prefetch the application data of the first application into the cache resources of the first application based on the mapping directory information can be as follows: the management node sends the directory path of the first directory to the target computing node to instruct the target computing node to prefetch the data from the subdirectories of each task stored in the first directory from the storage system based on the directory path of the first directory, and store the acquired data into the cache resources of the first application.

[0021] In this application, the management node controls the computing node to prefetch the data of the first application into the cache resources allocated to the first application through the mapping directory information specified by the user. In this way, the user does not need to perform explicit data copying, which reduces the complexity of the user.

[0022] In one possible implementation, the method further includes: the management node receiving bandwidth requirements from multiple computing nodes, including the target computing node, for data to be migrated from the cache resources of each application to the storage system; allocating IO bandwidth to the data to be migrated from the cache resources of the first application according to the bandwidth requirements; and sending the allocated IO bandwidth to the target computing node to instruct the target computing node to store the data to be migrated from the cache resources of the first application to the storage system according to the allocated IO bandwidth.

[0023] In this application, the management node can allocate IO bandwidth for migrating data to the applications running on each computing node based on the collected bandwidth requirements of each computing node. This controls the amount of data migrated from each computing node to the storage system, thereby preventing the application data access volume of different computing nodes from exceeding the available bandwidth of the storage system and thus avoiding I / O bandwidth contention. This enables orderly access to the storage system by each application in a global view, thereby reducing application performance problems caused by IO contention.

[0024] Thirdly, a data access method is provided, the method comprising: a computing node receiving a user-specified caching strategy and mapping directory information for a first application, wherein the caching strategy indicates the caching requirements of the first application, and the mapping directory information is information about a first directory where the application data of the first application is stored in a storage system; allocating cache resources for the first application according to the caching strategy; prefetching the application data of the first application into the cache resources of the first application according to the mapping directory information; and accessing the cache resources of the first application according to the caching strategy during the operation of the first application.

[0025] In this application, the compute node can allocate corresponding cache resources to the first application according to the user-specified caching strategy, thereby ensuring that the first application's resource usage on the compute node meets user requirements and thus its performance satisfies user needs. Furthermore, during the execution of the first application, data access can be directly performed within the cache resources allocated to it by the compute node, reducing access to the storage system and thus minimizing contention between compute nodes. Moreover, after allocating cache resources to the first application, the compute node can prefetch the application data of the first application from the first directory in the storage system into the first cache space based on the mapping directory information, eliminating the need for manual data copying by the user and reducing operational complexity.

[0026] In one possible implementation, when allocating cache resources for the first application according to the caching strategy, the computing node obtains resource requirement information for each of the multiple tasks of the first application from the caching strategy. The resource requirement information includes the size of the cache space required by each task and the type of storage medium included. Based on the resource requirement information of each task, the computing node allocates cache space for the first task it is running in its own cache resources. The first task is any one of the multiple tasks running on the computing node.

[0027] In one possible implementation, the process of prefetching the application data of the first application into the cache resources of the first application according to the mapping directory information includes: determining the directory identifier of the subdirectory corresponding to the first task; obtaining the data stored in the subdirectory corresponding to the first task under the first directory from the storage system according to the directory path of the first directory and the directory identifier of the subdirectory corresponding to the first task; and storing the data in the subdirectory corresponding to the first task into the cache space of the first task.

[0028] In one possible implementation, if the caching strategy includes a hierarchical caching strategy, the process of storing the data in the subdirectory corresponding to the first task into the cache space of the first task includes: storing different data into different types of storage media according to the data type of the data in the subdirectory corresponding to the first task.

[0029] In this application, caching data according to a hierarchical caching strategy enables different types of data to be stored in appropriate storage media, thereby improving data access performance and saving cache space resources.

[0030] In one possible implementation, the process of accessing the cached resources of the first application according to the caching policy during the operation of the first application may include: obtaining an IO request; if the data accessed by the IO request is data under the first directory indicated by the mapping directory information, then accessing the cached resources of the first application according to the IO request and the caching policy.

[0031] In this application, by setting mapping directory information, access to the first directory indicated by the mapping directory information can be directly intercepted, and then data access can be achieved by accessing the cache resources allocated to the first application, which improves access efficiency and the whole process is imperceptible to the user.

[0032] In one possible implementation, if the caching strategy includes a data consistency strategy, then when the computing node accesses data in the cached resources of the first application, a locking operation is performed on the accessed data to ensure the accuracy of the data during the data access process.

[0033] In one possible implementation, when the computing node detects that the amount of data in the cache resources allocated to the first application has reached a reference threshold, it sends the bandwidth requirement of the first application to the management node. The bandwidth requirement is used to indicate the bandwidth required to migrate the data to be migrated in the cache resources of the first application on the computing node to the storage system. The computing node receives the IO bandwidth allocated by the management node for the data to be migrated in the cache resources of the first application, and migrates the data to be migrated in the cache resources of the first application to the storage system according to the allocated IO bandwidth.

[0034] In this application, a compute node can request the management node to allocate I / O bandwidth for the first application by sending the bandwidth requirements of the first application to the management node. Since the management node can collect the bandwidth requirements of each compute node at the same time, I / O bandwidth competition can be generated among the compute nodes when migrating data according to the I / O bandwidth allocated by the management node. This enables the orderly access of each application to the storage system in a global view, thereby reducing application performance problems caused by I / O competition.

[0035] Fourthly, a data access device is provided, which has the function of implementing the data access method behavior described in the first aspect. The data access device includes at least one module for implementing the data access method provided in the first aspect.

[0036] Fifthly, a data access device is provided, which has the function of implementing the data access method behavior in the second aspect above. The data access device includes at least one module for implementing the data access method provided in the second aspect above.

[0037] In a sixth aspect, a data access device is provided, the data access device having the function of implementing the data access method behavior in the third aspect above, the data access device including at least one module, the at least one module being used to implement the data access method provided in the third aspect above.

[0038] In a seventh aspect, a cluster system is provided, comprising a management node and computing nodes. Each management node and computing node includes a processor and a memory. The memory stores programs that support the cluster system in executing the data access method provided in the first aspect, and stores data related to implementing the data access method provided in the first aspect. The processor is configured to execute the programs stored in the memory.

[0039] Eighthly, a management node is provided, the management node comprising a processor and a memory, the memory storing programs that support the management node in executing the data access method provided in the second aspect, and storing data related to implementing the data access method provided in the second aspect. The processor is configured to execute the programs stored in the memory.

[0040] A ninth aspect provides a computing node, the computing node comprising a processor and a memory, the memory being used to store programs supporting the computing node in executing the data access method provided in the third aspect, and to store data related to implementing the data access method provided in the third aspect. The processor is configured to execute the programs stored in the memory.

[0041] In a tenth aspect, a computer-readable storage medium is provided, wherein instructions are stored therein, which, when executed on a computer, cause the computer to perform the data access method described in the first, second, or third aspect above.

[0042] In an eleventh aspect, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to perform the data access method described in the first, second, or third aspect above. Attached Figure Description

[0043] Figure 1 This is a system architecture diagram of a data center provided in an embodiment of this application;

[0044] Figure 2 This is a flowchart illustrating a method for using a computer device according to an embodiment of this application;

[0045] Figure 3 This is a flowchart of a data access method provided in an embodiment of this application;

[0046] Figure 4 This is a schematic diagram of the structure of a data access device provided in an embodiment of this application;

[0047] Figure 5 This is a schematic diagram of another data access device provided in an embodiment of this application;

[0048] Figure 6 This is a schematic diagram of the structure of another data access device provided in the embodiments of this application. Detailed Implementation

[0049] To facilitate understanding, the system architecture involved in the embodiments of this application will be introduced first.

[0050] The data access method provided in this application can be applied to data centers that can provide a shared application execution environment for multiple users. The applications running in this data center can be data-intensive applications such as high-performance computing applications and big data applications.

[0051] For example, see Figure 1 The data center includes a cluster system 10 and a storage system 11, and the cluster system 10 and the storage system 11 are connected by a communication link. The cluster system 10 is used to provide an execution environment for multiple applications, and the storage system 11 is used to store the application data of these multiple applications.

[0052] See Figure 1 The cluster system 10 may include a management node 101 and multiple computing nodes 102. The management node 101 and each computing node 102 can communicate via a wired network or a wireless network, and the computing nodes 102 can also communicate with each other via a wired network or a wireless network. In this embodiment, the management node 101 is used to allocate computing nodes 102 for executing an application according to a caching policy specified by the user, and to send the caching policy and the user-specified mapping directory information to the computing nodes 102.

[0053] After receiving the user-specified caching policy and mapping directory information from the management node 101, the compute node 102 allocates cache resources for the application from its own cache resources according to the caching policy, and prefetches the application data stored in the storage system 11 into the application's cache resources according to the mapping directory information. Then, the application is run, and during the application's operation, the application's cache resources are accessed according to the caching policy. Here, the compute node 102's own cache resources refer to the storage media included in the compute node 102. For example, the compute node 102's own cache resources may include its own dynamic random access memory (DRAM), storage class memory (SCM), solid-state drive (SSD), or other types of storage media; this embodiment does not limit this.

[0054] It should be noted that the number of computing nodes 102 allocated by the management node 102 to execute the application can be multiple, so that each computing node 102 can be used to run one or more tasks of the application.

[0055] Using the above method, the management node 101 can schedule cached resources in the computing node 102 according to user needs for the various applications to be run by different users, thereby controlling the corresponding computing node 102 to run the corresponding application.

[0056] During the operation of an application on compute node 102, when it is detected that the amount of data cached in the cache resources of a certain application has reached the second threshold, compute node 102 can send the bandwidth requirements of the data in the cache resources of the application to be migrated to storage system 11 to management node 101, so as to request management node 101 to allocate IO bandwidth for the data to be migrated from the application.

[0057] After receiving the bandwidth requirements of each application from one or more compute nodes 102, management node 101 can allocate IO bandwidth to the data to be migrated for each application according to the bandwidth requirements, and then distribute the allocated IO bandwidth to the corresponding compute node 102. Accordingly, after receiving the IO bandwidth allocated by management node 101 for the data to be migrated for the application, compute node 102 can send the data to be migrated for the application to the storage system 11 for storage according to the IO bandwidth.

[0058] The storage system 11 includes multiple storage nodes 111. Each storage node 111 can communicate with each compute node 102 via wired or wireless communication. Each storage node 111 receives I / O requests from the compute node 102. When the I / O request is a read request sent by the compute node 102 based on mapped directory information, the storage node 111 retrieves the application data according to the read request and returns it to the compute node 102, so that the compute node 102 caches the application data in the cache resources allocated to the application. When the I / O request is a write request carrying application data to be migrated, the storage node 111 can persistently store the data to be migrated according to the write request.

[0059] It should be noted that, in one possible implementation, storage node 111 may include a control unit, a network interface card (NIC), and multiple storage devices. The control unit communicates with computing node 102 via the NIC and accesses multiple storage devices based on I / O requests from computing node 102. These multiple storage devices may include storage-class memory (SCM), solid-state drives (SSDs), or other similar types of storage devices; this embodiment does not limit the specific type.

[0060] Optionally, in this embodiment, the data center may also provide users with a login node for submitting caching policies and mapping directory information. Users submit the caching policies and mapping directory information of the application they wish to run to the management node 101 through this login node, so that the management node 101 can schedule resources for the application based on the application's caching policies and mapping directory information.

[0061] Each of the aforementioned management node 101, compute node 102, storage node 111, and login node can be a separate computer device. The login node can be a terminal device, such as a laptop, desktop computer, tablet, or smartphone. Management node 101 and compute node 102 can be terminal devices or servers. Storage node 111 can be a server.

[0062] Figure 2 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 1 The management node and computing node in the system architecture shown can both be implemented using this computer device. See also Figure 2 The computer device may include one or more processors 201, a communication bus 202, a main memory 203, and one or more communication interfaces 204.

[0063] The processor 201 can be a general-purpose central processing unit (CPU), a network processor (NP), a microprocessor, or one or more integrated circuits for implementing the solutions of this application, such as application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or combinations thereof. The aforementioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0064] The communication bus 202 is used to transmit information between the aforementioned components. The communication bus 202 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, it is represented by only one thick line in the figure, but this does not indicate that there is only one bus or one type of bus.

[0065] Main memory 203 can be read-only memory (ROM), random access memory (RAM), or any other medium capable of carrying or storing desired program code in the form of instructions or data structures, accessible by a computer, but not limited to these. When main memory 203 is RAM, it can be dynamic random access memory (DRAM), or SCM, etc. Main memory 203 can exist independently and be connected to processor 201 via communication bus 202. Main memory 203 can also be integrated with processor 201.

[0066] Communication interface 204 uses any transceiver-like device for communicating with other devices or communication networks. Communication interface 204 includes a wired communication interface and may also include a wireless communication interface. The wired communication interface may be, for example, an Ethernet interface. The Ethernet interface may be an optical interface, an electrical interface, or a combination thereof. The wireless communication interface may be a wireless local area network (WLAN) interface, a cellular network communication interface, or a combination thereof.

[0067] In some embodiments, the computer device may also include other storage media 205, such as hard disk drives, solid-state drives, etc.

[0068] In some embodiments, a computer device may include multiple processors, such as Figure 2 The processors 201 and 206 are shown. Each of these processors can be a single-core processor or a multi-core processor. A processor here can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0069] In a specific implementation, as one embodiment, the computer device may further include an output device 207 and an input device 208. The output device 207 communicates with the processor 201 and can display information in various ways. For example, the output device 207 may be a liquid crystal display (LCD), a light-emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector, etc. The input device 208 communicates with the processor 201 and can receive user input in various ways. For example, the input device 208 may be a mouse, keyboard, touchscreen device, or sensing device, etc.

[0070] In some embodiments, main memory 203 stores a kernel, program code for executing the scheme of this application, and other instructions and data. Processor 201 can execute the program code stored in main memory 203. The program code may include one or more software modules. The computer device can implement the following by using processor 201 and the program code in main memory 203. Figure 3 The data access method provided in the embodiment.

[0071] In the data access method provided in this application, users can flexibly customize caching strategies for applications, and the cluster system can schedule cache resources for applications according to the caching strategies submitted by users. That is, the cluster system can perceive user needs and control the application's resource usage accordingly, thereby improving application performance. Furthermore, the cluster system can prefetch data into the application's cache resources based on the mapping directory information submitted by users, thereby improving data access speed and reducing user complexity. In addition, the management node in the cluster system can allocate IO bandwidth to applications running on the compute nodes by collecting the bandwidth requirements of each compute node, reducing application performance problems caused by IO contention. The embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0072] Figure 3 This is a flowchart illustrating a data access method provided in an embodiment of this application. This method can be applied to... Figure 1 In the cluster system shown in the data center, see Figure 3 The method includes the following steps:

[0073] Step 301: The management node receives a cache configuration request submitted by the user for the first application. The cache configuration request includes a cache policy and mapping directory information. The cache policy is used to indicate the cache requirements of the first application, and the mapping directory information is the information of the first directory where the application data of the first application is stored in the storage system.

[0074] In this embodiment, the user inputs a caching policy and mapping directory information for a first application at the login node. Based on the user-input caching policy and mapping directory information, the login node generates a caching configuration request for the first application and sends this request to the management node. The caching configuration request carries the caching policy and mapping directory information, and the first application refers to the application the user intends to run. Correspondingly, the management node receives the caching configuration request for the first application sent by the login node.

[0075] For example, a command-line tool is deployed on the login node, allowing users to input the caching strategy and mapping directory information for the first application in the command-line interface displayed on the login node. The login node then retrieves the caching strategy and mapping directory information for the first application input by the user in the command-line interface.

[0076] Optionally, a service configuration client can also be deployed on the login node. Users can enter the caching strategy and mapping directory information of the first application in the interface of the service configuration client displayed on the login node. Correspondingly, the login node can obtain the caching strategy and mapping directory information of the first application through this service configuration client, and then generate a caching configuration request for the first application.

[0077] It should be noted that the caching strategy of the first application may include the resource requirements information of the first application and data caching and access strategies.

[0078] In this embodiment, the first application can be divided into multiple tasks executed by multiple computing nodes. In this case, the resource requirement information of the first application specified by the user can include the resource requirement information of each task of the first application. Furthermore, the resource requirement information of each task of the first application can be the same or different. The resource requirement information can include computing resource requirement information and cache resource requirement information. The computing resource requirement information indicates the computing resources required by each task of the first application, such as the number of processor cores and clock speed required to run each task of the first application. The cache resource requirement information includes the size of the cache space required by each task of the first application. In addition, it may also include the type of storage medium included in the cache space required by each task of the first application. For example, the cache space required by each task of the first application may include two different storage media: DRAM and SCM. Optionally, the cache resource requirement information may also include the topology of the cache space required by each task of the first application, that is, the topology of each level of storage medium constituting the cache space on the corresponding computing node.

[0079] Optionally, the resource requirement information of the first application can also be used directly to indicate the resource requirements of the first application. That is, the resource requirement information is not the task-level resource requirement information mentioned above, but the application-level resource requirement information.

[0080] Data caching and access policies can be used to indicate the caching method and access strategy for application data of a first application. For example, the data caching and access policy may include a hierarchical caching policy, used to indicate that different types of application data of the first application are cached in different types of storage media. As another example, the data caching and access policy may include a data consistency policy, used to indicate that when accessing any data in the cached resources of the first application, a locking operation is performed on the accessed data to ensure data consistency. As yet another example, the data caching and access policy may also include a security level policy, used to indicate the access permissions for data in the application's cached resources. The above are only some of the strategies that may be included in the data caching and access policies given in the embodiments of this application. In addition, the data caching and access policy may also include other strategies that can be flexibly customized by the user to better meet user needs and improve application performance.

[0081] Additionally, the mapping directory information refers to the information of the first directory where the application data of the first application is stored in the storage system. For example, this mapping directory information can be the directory path of the first directory in the storage system. Alternatively, the mapping directory information can also be other information that can be used to indicate the storage location of the application data of the first application in the storage system; this application embodiment does not limit this.

[0082] Step 302: The management node allocates the target computing node for executing the first application according to the caching strategy.

[0083] After receiving the cache configuration request for the first application, the management node allocates a target computing node from multiple computing nodes to execute the first application based on the resource requirement information included in the cache policy in the cache configuration request.

[0084] For example, if the caching strategy includes resource requirement information at the task granularity level, the management node can obtain the resource requirement information of each task in the multiple tasks of the first application from the caching strategy, and allocate target computing nodes to execute each task of the first application according to the resource requirement information.

[0085] As described in step 301, the resource requirements for each task can include both computational and cache resource requirements. Based on this, the management node can collect and update the usage of computational and cache resources on each computing node in real time. Then, based on the computational resource requirements of each task and the most recently updated usage of each computing node, it determines candidate computing nodes from a pool of computing nodes that can meet the computational resource requirements for running the first application's tasks. Next, based on the cache resource requirements of each task and the most recently updated usage of cache resources on each candidate computing node, it further determines computing nodes that can meet the cache resource requirements of the first application's tasks from the candidate computing nodes, and the finally determined computing nodes are used as the target computing nodes.

[0086] For example, the management node can determine the remaining computing resources on each computing node based on the resources consumed by the applications running on each computing node in the most recent update. Then, it can identify candidate computing nodes from among the multiple computing nodes whose remaining computing resources meet the computing resource requirements of the first application's task. Next, based on the size of the remaining cache space on each candidate computing node in the most recent update and the type of storage medium comprising the remaining cache space, it can determine computing nodes from among the candidate computing nodes whose remaining cache space is larger than the cache space required by the first application's task, and whose remaining cache space includes the storage medium required by the first application's task, thus obtaining the target computing node.

[0087] Optionally, the management node may also determine candidate computing nodes from multiple computing nodes based on the cache resource requirements of each task, and then determine the target computing node from the candidate computing nodes based on the computing resource requirements of each task of the first application. This will not be described in detail in the embodiments of this application.

[0088] It is worth noting that, using the method described above, the management node can determine the compute node for each task running the first application. The compute node running each task can be different, resulting in multiple target compute nodes. Optionally, the compute node running each task may also be the same target compute node, resulting in only one target compute node. Alternatively, some tasks may be executed through one target compute node, while others may be executed through another, again resulting in multiple target compute nodes.

[0089] In another implementation, if the caching strategy includes application-level resource requirement information, the management node can directly allocate target computing nodes to the first application based on the resource requirement information of the first application. The implementation method can refer to the implementation method of allocating target computing nodes for each task mentioned above, and will not be repeated here in the embodiments of this application.

[0090] Optionally, the management node can also determine the resource requirements of each task in the multiple tasks of the first application based on the resource requirements of the first application and the task division principle of the first application, and allocate target computing nodes to each task using the method of allocating target computing nodes described above.

[0091] Step 303: The management node sends the caching policy and mapping directory information to the target compute node.

[0092] After determining the target computing nodes for executing each task of the first application, the management node can send the caching strategy and mapping directory information of each task of the first application to the target computing nodes, so as to control the target computing nodes to allocate cache resources for each task of the first application according to the caching strategy, and access the cache resources of the first application according to the mapping directory information and caching strategy.

[0093] Optionally, when there are multiple target compute nodes and the caching strategy for each task is the same, the management node can distribute the caching strategy and mapping directory information to each target compute node. When there are multiple target compute nodes and the resource requirement information for each task contained in the caching strategy is different, the management node can use the data caching and access strategies contained in the caching strategy and the resource requirement information for each task as the caching strategy for the corresponding task, and then distribute the mapping directory information and the caching strategy for each task to the target compute node corresponding to the task. The target compute node corresponding to the task runs the target compute node for that task.

[0094] Optionally, while distributing caching policies and mapping directory information to the target compute nodes, the management node can also distribute the identifier of the task to be run to each target compute node, indicating which task of the first application the target compute node is to run. The task identifier uniquely identifies the task.

[0095] Once each target computing node receives the caching policy and mapping directory information sent by the management node, it can run the first application through the following steps 304 to 306.

[0096] Step 304: The target compute node allocates cache resources to the first application according to the caching strategy.

[0097] After receiving the caching policy and mapping directory information from the management node, the target compute node first allocates cache resources to the first application according to the caching policy.

[0098] The target compute node can obtain cache resource requirement information from the received caching policy, and then allocate cache resources to the task of the first application it is to execute based on this cache resource requirement information. The following explanation uses a single target compute node as an example; for ease of explanation, this target compute node will be referred to as the first target compute node.

[0099] For example, if a first task is running on a first target computing node, the first target computing node obtains the cache resource requirement information of the first task from the received caching policy, and then allocates cache space in its own cache resources to meet the cache resource requirement of the first task according to the cache resource requirement information of the first task.

[0100] Optionally, when the cache resource requirements of each task are the same and a task of the first application is running on a target computing node, the first target computing node can allocate a cache space of the same size as indicated by the cache resource requirements from its own cache resources as the cache space of the first application. In this case, the allocated cache space of the first application may be the cache space of the first task of the first application running on the first target computing node, that is, used to store the task data of the first task, or it may be the cache space of other tasks of the first application running on other target computing nodes, that is, used to store the task data of other tasks.

[0101] Each target computing node allocates cache space to each task of the first application according to the caching policy issued by the management node. In this way, the cache space of each task of the first application located on each target computing node constitutes the cache resource of the first application.

[0102] Optionally, if the resource requirement information in the caching strategy is application-level, and the management node directly distributes the resource requirement information of the first application to the target compute node, then there will be one target compute node. In this case, after receiving the cache resource requirement information of the first application, the target compute node can allocate cache space for the first application from its own cache resources according to the cache resource requirement information. In this way, the cache resources of the first application will reside on a single compute node.

[0103] Step 305: The target computing node prefetches the application data of the first application into the cache resources of the first application according to the mapping directory information.

[0104] After allocating corresponding cache resources for the first application, the target compute node can retrieve the application data of the first application from the storage system based on the mapping directory information, and then cache the application data in the cache resources of the first application. The following explanation will continue using the first target compute node as an example.

[0105] In the first possible scenario, if the first target computing node allocates cache space for the first task of the first application it executes, and the mapping directory information is the directory path of the first directory in the storage system, then the first target computing node can determine the directory identifier of the subdirectory corresponding to the first task. Based on the directory path of the first directory and the directory identifier of the subdirectory corresponding to the first task, it can obtain the data stored in the subdirectory corresponding to the first task in the first directory from the storage system, and then store the obtained data in the subdirectory corresponding to the first task in the cache space allocated for the first task.

[0106] The first target computing node can obtain the directory identifier of the subdirectory corresponding to the first task from a preset set of task identifiers and subdirectory directory identifiers, based on the task identifier of the first task. Alternatively, the first target computing node can also generate the directory identifier of the subdirectory corresponding to the first task using a preset rule based on the task identifier of the first task. For example, if the task number of the first task is 1, and the preset rule for generating the directory identifier of the subdirectory corresponding to the task is processor + task number, then according to this preset rule, the directory identifier of the subdirectory corresponding to the first task can be obtained as processor 1.

[0107] Then, the first target computing node can retrieve the data from the first directory stored in the storage system, which is the same as the directory identifier of the subdirectory corresponding to the first task, according to the directory path of the first directory. That is, the task data of the first task. Then, the task data of the first task is stored in the cache space of the first task.

[0108] It should be noted that when storing the task data of the first task in the cache space of the first task, if the caching strategy also includes data caching and access strategies, and the data caching and access strategies include hierarchical caching strategies, the first target computing node can also store different data in different types of storage media according to the data type of the task data of the first task.

[0109] For example, frequently accessed data from the first task can be stored in a high-performance memory (also known as a storage medium), while less frequently accessed data can be stored in a relatively low-performance storage medium. For instance, metadata and other data can be stored in different types of storage media. The first threshold can be set according to business needs, the processing efficiency of the task data, an empirical value, or the system's processing capacity.

[0110] In addition, the first application task executed by the first target computing node may be one or multiple. When the first target computing node executes multiple tasks of the first application, the data in the subdirectory corresponding to each task to be executed can be prefetched into the corresponding cache space of the task in the manner described above.

[0111] In the second possible scenario, if the cache resource requirements of each task are identical, and each target compute node allocates cache space from its own cache resources that is the same size as indicated by the cache resource requirement information, then the first target compute node can retrieve the data from the first directory indicated by the mapping directory information in the storage system. It then performs a hash operation on the directory path of the retrieved data to obtain the hash value corresponding to the data. The target compute node whose node identifier matches the hash value is determined from among multiple target compute nodes. If the target compute node whose node identifier matches the hash value is itself, then the data is stored in the cache space allocated by itself for the first application. If the target compute node whose node identifier matches the hash value is another target compute node, for example, the second target compute node, then the first target compute node can send the data to the second target compute node. After receiving the data, the second target compute node stores the data in the cache space allocated by itself for the first application.

[0112] When storing data in the cache space allocated for the first application, the method described above can also be used to store the data in the corresponding type of storage medium according to the hierarchical caching strategy included in the data caching and access strategy.

[0113] In the third possible scenario, if the target compute node allocates cache space for the first application based on the cache resource information of the first application at the application granularity level, the target compute node can directly obtain the data under the first directory indicated by the mapping directory information from the storage system and store the data in the cache space allocated for the first application.

[0114] The above are several possible implementation methods for prefetching application data of the first application as given in the embodiments of this application. Optionally, the target computing node may also use a multi-replica mechanism to prefetch task data of each task of the first application, or use other implementation methods to prefetch data of the first application, which are not limited in this embodiment.

[0115] Additionally, it is worth noting that the data prefetched from the first directory of the storage system can be all the data in the first directory or only a portion of the data. For each task, the prefetched data may be all or only a portion of the task's data, and this embodiment does not limit this. When only a portion of the data is prefetched, more important data can be prefetched based on the data's access frequency or other information indicating its importance.

[0116] Step 306: During the execution of the first application, the target compute node accesses the cached resources of the first application according to the caching strategy.

[0117] After allocating cache resources for the first application through steps 304 and 305, and prefetching the application data of the first application into the cache resources of the first application, the target computing node starts the running script of the first application, thereby starting to run the first application.

[0118] The first target computing node will be used as an example for explanation. The first target computing node starts the running script of the first application and executes the first task of the first application assigned to it.

[0119] During the execution of the first task, the first target computing node may need to read application data from the first application or write data generated during task execution to the cache resources of the first application. Based on this, the first target computing node can generate an I / O request according to the operation to be performed. This I / O request may be a read request or a write request. Furthermore, the I / O request may include the directory path of the directory containing the accessed target data.

[0120] Upon receiving an IO request, the first target compute node first compares the directory path of the target data to be accessed with the mapping directory information. If the directory path of the target data contains the mapping directory information, it can be determined that the target data to be accessed is data under the first directory. In this case, since the data under the first directory was prefetched into the cache resources of the first application in step 305, the first target compute node can directly access the cache resources of the first application.

[0121] If each target computing node prefetches data using the method described in the first possible case in step 305, then after determining that the data to be accessed by the IO request is data under the first directory, the first target computing node accesses the cached resources of the first application according to the IO request.

[0122] It should be noted that if the IO request is a read request, the first target compute node can first search for the target data in the cache space of the first task. If the target data is found in the cache space of the first task, it retrieves the target data. If the target data is not found in the cache space of the first task, the IO request is sent to other target compute nodes. After receiving the IO request, other target compute nodes search for the target data in the cache space of the task they allocated to the first application. If the target data is found, it is returned to the first target compute node. If the target data is not found, a notification message is returned to the first target compute node to notify it of the data retrieval failure. If all other target compute nodes fail to find the target data, the first target compute node can retrieve the target data from the storage system. Optionally, if the IO request is a write request, the first target compute node can write the target data to the cache space of the first task.

[0123] As described above, when the first target computing node fails to find the target data in the cache space of the first task based on the IO request, it can send the IO request to other target computing nodes. Similarly, after generating an IO request, if other target computing nodes fail to find the data to be accessed in the cache space allocated to the first application, they can also send an IO request to the first target computing node. In this case, the first target computing node can also receive IO requests sent by other target computing nodes and access the cache space of the first task based on the IO request. Optionally, computing nodes can send IO requests to each other's cache space through remote direct memory access (RDMA) technology.

[0124] Optionally, if each target computing node prefetches data using the method described in the second possible case in step 305, the first target computing node can perform a hash operation on the directory path of the target data to be accessed by the IO request to obtain the hash value corresponding to the target data, determine the target computing node whose node identifier matches the hash value corresponding to the target data, and if the determined target computing node is itself, the first target computing node accesses the cache space allocated by itself for the first application to realize the reading or writing of the target data.

[0125] Optionally, if the determined target computing node is another target computing node, the first target computing node sends the IO request to the determined target computing node, and the determined target computing node reads or writes the target data by accessing the cache space allocated by itself for the first application.

[0126] In cases where the target data is not found in the cache space of the first application, the corresponding target computing node can still obtain the target data from the storage system.

[0127] Optionally, if the target computing node prefetches data using the method described in the third possible case in step 305, then since there is only one target computing node, it can access the cache space allocated to the first application based on the IO request. The access method is the same as described in the preceding implementation, and will not be repeated here.

[0128] Optionally, if the data caching and access strategies in the caching strategy also include a data consistency strategy, then in this step, when a target computing node modifies, deletes, or writes target data in the cache resources of the first application according to an IO request, the target computing node can also perform a locking operation on the target data to prevent other target computing nodes from accessing the target data and ensure data consistency.

[0129] Step 307: Multiple compute nodes send the bandwidth requirements of the data to be migrated to the storage system from the cache resources of their running applications to the management node. The multiple compute nodes include the target compute node, and the multiple applications include the first application.

[0130] In this embodiment, when a compute node detects that the amount of data cached in the cache space allocated to a certain application has reached a second threshold, it can send the bandwidth requirement of that application to the management node. Correspondingly, the management node can receive the bandwidth requirements of each application sent by each compute node in real time. This bandwidth requirement indicates the bandwidth needed to migrate the data to be migrated from the cache resources of the corresponding application on the corresponding compute node to the storage system. For example, the bandwidth requirement may include the amount of data to be migrated in the cache resources of the corresponding application on the compute node. Optionally, it may also include other information such as the application identifier; this embodiment does not limit this. Furthermore, the second threshold can be preset based on the size of the cache space allocated to the application on the compute node. For example, the second threshold may be a preset proportion of the total capacity of the cache space allocated to the application, such as 80% of the cache space allocated to the application, or other values; this embodiment does not limit this.

[0131] Among these multiple computing nodes is a target computing node. That is, when the target computing node detects that the amount of data cached in the cache space allocated to the first application has reached a second threshold, it can send the bandwidth requirement of the first application to the management node. At this time, the bandwidth requirement of the first application is used to indicate the bandwidth required to migrate the data to be migrated in the cache space of the first application on the target computing node to the storage system.

[0132] Step 308: The management node allocates IO bandwidth to the data to be migrated in the cache resources of the first application based on the bandwidth requirement.

[0133] After receiving the bandwidth requirements of various applications from multiple compute nodes, including the target compute node, the management node can allocate the corresponding IO bandwidth to the data to be migrated for each application based on the bandwidth required by the data to be migrated for each application and the current remaining bandwidth of the storage system.

[0134] For example, the management node can calculate the proportion of bandwidth required by each application, and then allocate IO bandwidth to each application based on this proportion and the current remaining bandwidth of the storage system. Specifically, if the current remaining bandwidth of the storage system is not greater than the total bandwidth required by each application, the IO bandwidth allocated to each application will be less than its required bandwidth; conversely, if the current remaining bandwidth of the storage system is greater than the total bandwidth required by each application, the IO bandwidth allocated to each application can be equal to its required bandwidth.

[0135] The management node may also use other principles to allocate IO bandwidth to the data to be migrated for each application, and this application embodiment does not limit this.

[0136] In addition, the IO bandwidth allocated to the data to be migrated for each application indicates the maximum amount of data that each application is allowed to migrate per unit of time. For example, when the IO bandwidth allocated to the data to be migrated for the first application is 30MB / s, it means that the target compute node is allowed to migrate a maximum of 30MB of the first application's cached data to the storage system per second.

[0137] Since the first application is included among multiple applications, the management node can allocate IO bandwidth to the data to be migrated in the cache resources of the first application using the method described above.

[0138] Step 309: The management node sends the IO bandwidth allocated for the first application to the target compute node.

[0139] After the management node allocates IO bandwidth to the data to be migrated in the cache resources of each application, it can send the IO bandwidth allocated to the corresponding computing node to the corresponding application.

[0140] For example, the management node can send the IO bandwidth allocated to the target compute node for the data to be migrated in the cache resources of the first application.

[0141] Step 310: The target compute node stores the data to be migrated from the cache resources of the first application to the storage system according to the IO bandwidth allocated to the first application.

[0142] The IO bandwidth allocated to the first application indicates the amount of cached data for the first application that the target compute node is allowed to migrate. Based on this, the target compute node retrieves data no more than the IO bandwidth allocated to the first application from its own allocated cache space as the data to be migrated. Then, according to the user-specified mapping directory information, the data to be migrated is migrated to the storage system for persistent storage. The operation of migrating data to the storage system according to the mapping directory information is the reverse operation of prefetching data from the storage system according to the mapping directory information. The specific implementation method can be referred to the previous description, and will not be repeated here.

[0143] After the target computing node starts running the first application, whenever it detects that the amount of data in the cache space allocated to the first application has reached the second threshold, it can apply for IO bandwidth from the management node through steps 307-310 above, so as to migrate the data in the cache space of the first application to the storage system according to the IO bandwidth, until the first application finishes running and all the data in the cache space of the first application has been migrated to the storage system, the target computing node can release the cache space allocated to the first application.

[0144] In this embodiment, cache resources are scheduled for the first application based on the caching strategy submitted by the user, and data is prefetched into the cache resources of the first application based on the mapping directory information submitted by the user. Subsequently, during the operation of the first application, the cache resources of the first application are accessed according to the caching strategy. Therefore, this embodiment can sense user needs and control the application's resource usage accordingly, thereby improving application performance.

[0145] Secondly, in this embodiment, data of the first application stored in the storage system can be prefetched into the cache resources allocated to the first application based on the user-specified mapping directory information. This way, during subsequent operation of the first application, if the data to be accessed is data in the directory indicated by the mapping directory information, the cache resources allocated to the first application can be accessed directly, improving data access speed. Furthermore, prefetching data from the storage system by specifying the mapping directory information reduces the complexity for users.

[0146] Third, in this embodiment, data can be cached and accessed in the cache resources of the first application according to the data caching and access strategies in the user-specified caching strategy. For example, data can be cached according to a tiered caching strategy to improve data access performance and save cache space resources. Data can be accessed according to a data consistency strategy to ensure data accuracy during data access. In addition, users can flexibly customize other strategies to achieve flexible settings for data caching and access methods.

[0147] Finally, in this embodiment, when each compute node detects that the amount of data in its allocated cache resources for applications has reached a second threshold, it can request the management node to allocate bandwidth for the data to be migrated from the applications it runs. The management node can allocate IO bandwidth for migrating data to the applications running on each compute node based on the collected bandwidth requirements of each compute node. This controls the amount of data migrated from each compute node to the storage system, preventing application data access from different compute nodes from exceeding the available bandwidth of the storage system and thus avoiding I / O bandwidth contention. This achieves orderly access to the storage system by applications in a global view, thereby reducing application performance issues caused by IO contention. Furthermore, in this embodiment, the cluster system can automatically complete data copying based on the user-specified mapping directory information, eliminating the need for manual data copying by the user and reducing operational complexity.

[0148] It should be noted that the steps related to the management node in the above embodiments can be implemented separately as a data access method on the management node side, and the steps related to the computing node side can be implemented separately as a data access method on the computing node side.

[0149] The above text combines Figures 1 to 3 The data access method provided according to the embodiments of this application is described in detail below, in conjunction with... Figures 4 to 6 This application describes a data access apparatus provided according to embodiments thereof.

[0150] See Figure 4 This application provides a data access device 400, which can be applied to a cluster system. The device 400 includes:

[0151] The receiving module 401 is used to execute step 301 in the above embodiments;

[0152] The scheduling module 402 is used to execute steps 302-304 in the above embodiments;

[0153] The prefetch module 403 is used to execute step 305 in the above embodiments;

[0154] Access module 404 is used to execute step 306 in the above embodiment.

[0155] It should be understood that the data access device 400 of this embodiment of the present invention can be implemented by a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. It can also be implemented in software. Figure 3 In the data access method shown, the data access device 400 and its various modules can also be software modules.

[0156] Optionally, the scheduling module 402 is mainly used for:

[0157] Based on the caching strategy, the resource requirement information for each task in the multiple tasks of the first application is determined. The resource requirement information includes the size of the cache space required by each task and the type of storage medium included.

[0158] Allocate cache space for each task based on its resource requirements.

[0159] Optionally, the mapped directory information includes the directory path of the first directory, and the prefetch module 403 is mainly used for:

[0160] Determine the directory identifier of the subdirectory corresponding to each task in the multiple tasks of the first application;

[0161] Based on the directory path of the first directory and the directory identifier of the subdirectory corresponding to each task, retrieve the data in the subdirectory corresponding to each task stored in the first directory from the storage system;

[0162] The data in the subdirectory corresponding to each task is stored in the cache resources of the first application.

[0163] Optionally, access module 404 is primarily used for:

[0164] When the caching strategy includes a tiered caching strategy, according to the tiered caching strategy, different types of task data for each task of the first application are cached in the corresponding type of storage medium in the cache resources of the first application.

[0165] When the caching strategy includes a data consistency strategy, a locking operation is performed on the accessed task data when accessing task data in any cache space.

[0166] Optionally, the device 400 is also used for:

[0167] Obtain input / output I / O requests;

[0168] If the data accessed by the IO request is data in the first directory indicated by the mapping directory information, then the step of accessing the cached resources of the first application according to the caching policy is executed.

[0169] Optionally, the device 400 is also used for:

[0170] Obtain the bandwidth requirements of the data to be migrated to the storage system from the cache resources of each of the multiple applications, including the first application;

[0171] Based on bandwidth requirements, allocate IO bandwidth to the data to be migrated in the cache resources of the first application;

[0172] Based on the IO bandwidth, the data to be migrated from the cache resources of the first application is stored in the storage system.

[0173] The data access device 400 according to embodiments of this application can correspond to executing the method described in the embodiments of this application, and the above and other operations and / or functions of each unit in the data access device 400 are respectively for implementing Figure 3 The corresponding processes executed by each node in each method are not described here for the sake of brevity.

[0174] In summary, in this embodiment, cache resources are scheduled for the first application based on the user-submitted caching strategy, and data is prefetched into the cache resources of the first application based on the user-submitted mapping directory information. Subsequently, during the operation of the first application, the cache resources of the first application are accessed according to the caching strategy. Therefore, this embodiment can sense user needs and control the application's resource usage accordingly, thereby improving application performance.

[0175] See Figure 5 This application provides a data access device 500, which can be applied in a management node. The device 500 includes:

[0176] The receiving module 501 is used to execute step 301 in the above embodiments;

[0177] The scheduling module 502 is used to execute the operation of sending the caching policy to the target computing node in steps 302 and 303 of the above embodiment, so as to control the target computing node to execute steps 304 to 306.

[0178] It should be understood that the apparatus 500 of this embodiment of the present invention can be implemented by a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. It can also be implemented in software. Figure 3 In the data access method shown, the device 500 and its various modules can also be software modules.

[0179] Optionally, the scheduling module 502 is mainly used for:

[0180] Obtain resource requirement information for each task in the first application from the caching strategy. The resource requirement information includes the size of the cache space required by each task and the type of storage medium included.

[0181] Based on resource requirement information, target computing nodes are allocated to execute each task of the first application;

[0182] Send a caching policy to the target computing node to instruct the target computing node to allocate cache space for the corresponding task from its own cache space according to the resource requirement information in the caching policy.

[0183] Optionally, the mapped directory information includes the directory path of the first directory, and the scheduling module 502 is mainly used for:

[0184] Send the directory path of the first directory to the target computing node to instruct the target computing node to prefetch data from the subdirectories of each task stored in the first directory from the storage system according to the directory path of the first directory, and store the retrieved data in the cache resources of the first application.

[0185] Optionally, the device 500 is also used for:

[0186] Receives bandwidth requirements from multiple computing nodes, including the target computing node, for data to be migrated to the storage system from the cache resources of various applications.

[0187] Based on bandwidth requirements, allocate IO bandwidth to the data to be migrated in the cache resources of the first application;

[0188] Send the IO bandwidth allocated for the first application to the target compute node to instruct the target compute node to store the data to be migrated from the cache resources of the first application to the storage system according to the IO bandwidth allocated for the first application.

[0189] The apparatus 500 according to embodiments of the present application of the present invention can correspond to performing the method described in the embodiments of the present application of the present invention, and the above and other operations and / or functions of each unit in the apparatus 500 are respectively for implementing Figure 3 The corresponding processes executed by each node in each method are not described here for the sake of brevity.

[0190] In summary, in this embodiment, the management node schedules cache resources for the first application based on the caching strategy submitted by the user, and controls the computing node to prefetch data into the cache resources of the first application based on the mapping directory information submitted by the user. Subsequently, during the operation of the first application, the computing node is controlled to access the cache resources of the first application according to the caching strategy. Therefore, this embodiment can sense user needs and control the application's resource usage accordingly, thereby improving application performance.

[0191] See Figure 6 This application also provides a data access device 600, such as... Figure 6 As shown, the data access device 600 can be applied in a computing node, and the data access device 600 includes:

[0192] The receiving module 601 is used to receive the caching strategy and mapping directory information of the first application specified by the user. The caching strategy is used to indicate the caching requirements of the first application, and the mapping directory information is the information of the first directory where the application data of the first application is stored in the storage system.

[0193] The allocation module 602 is used to execute step 304 in the foregoing embodiment;

[0194] The prefetch module 603 is used to execute step 305 in the foregoing embodiments;

[0195] Access module 604 is used to execute step 306 in the foregoing embodiments.

[0196] It should be understood that the data access device 600 of this embodiment of the present invention can be implemented by a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. It can also be implemented in software. Figure 3 In the data access method shown, the data access device 600 and its various modules can also be software modules.

[0197] Optionally, the allocation module 602 is mainly used for:

[0198] Obtain resource requirement information for each task in the first application from the caching strategy. The resource requirement information includes the size of the cache space required by each task and the type of storage medium included.

[0199] Based on the resource requirements of each task, a cache space is allocated in its own cache resources for the first task it runs. The first task is any one of the multiple tasks running on the compute node.

[0200] Optionally, the prefetch module 603 is mainly used for:

[0201] Determine the directory identifier of the subdirectory corresponding to the first task;

[0202] Based on the directory path of the first directory and the directory identifier of the subdirectory corresponding to the first task, retrieve the data stored in the first directory and the subdirectory corresponding to the first task from the storage system;

[0203] Store the data in the subdirectory corresponding to the first task in the cache space of the first task.

[0204] Optionally, if the caching strategy includes a tiered caching strategy, the prefetch module is further configured to: store different data in different types of storage media according to the data type of the data in the subdirectory corresponding to the first task.

[0205] Optionally, access module 604 is primarily used for:

[0206] Get I / O requests;

[0207] If the data accessed by the IO request is data in the first directory indicated by the mapping directory information, then the cached resources of the first application are accessed according to the IO request and the caching policy.

[0208] Optionally, if the caching strategy includes a data consistency strategy, the access module is primarily used for:

[0209] When accessing data in the cached resources of the first application, a locking operation is performed on the accessed data.

[0210] Optionally, the device 600 is further configured to: when it detects that the amount of data in the cache resources allocated by itself for the first application has reached a reference threshold, send the bandwidth requirement of the first application to the management node, the bandwidth requirement being used to indicate the bandwidth required to migrate the data to be migrated in the cache resources of the first application on the computing node to the storage system; receive the IO bandwidth allocated by the management node for the data to be migrated in the cache resources of the first application, and migrate the data to be migrated in the cache resources of the first application to the storage system according to the allocated IO bandwidth.

[0211] The apparatus 600 according to embodiments of this application can correspond to performing the method described in the embodiments of this application, and the above and other operations and / or functions of each unit in the apparatus 600 are respectively for implementing Figure 3 The corresponding processes executed by each node in each method are not described here for the sake of brevity.

[0212] In this embodiment, the compute node can allocate corresponding cache resources to the first application according to the user-specified caching strategy, thereby ensuring that the first application's resource usage on the compute node meets user requirements and its performance satisfies user needs. Furthermore, during the execution of the first application, data access can be directly performed within the cache resources allocated to it by the compute node, reducing access to the storage system and thus minimizing contention between compute nodes. Moreover, after allocating cache resources to the first application, the compute node can prefetch the application data of the first application from the first directory in the storage system into the first cache space based on the mapping directory information, eliminating the need for manual data copying by the user and reducing operational complexity.

[0213] This application also provides a data access system, which includes a management node and a computing node, wherein the connection method between the management node and the computing node can be referred to Figure 1 The connection method between the management node and the compute node in the system shown is illustrated. The structure of the management node and the compute node can be referenced from [reference needed]. Figure 2 The diagram shows the structure of the computer device. In this data access system, the management node is used to implement... Figure 3The data access method shown includes the functionality of the management node, while the compute node is used to implement... Figure 3 The functions of the computing nodes in the data access method shown are not described in detail in this embodiment of the application.

[0214] It should be noted that the data access device provided in the above embodiments is only illustrated by the division of the above functional modules when performing data reading and writing. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the data access device and the data access method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0215] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line, DSL) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)).

[0216] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0217] The above description is not intended to limit the embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of the embodiments of this application.

Claims

1. A data access method, characterized in that, The method includes: The system receives a cache configuration request submitted by a user for a first application. The cache configuration request includes a cache policy and mapping directory information. The cache policy is used to indicate the cache requirements of the first application, and the mapping directory information is the information of the first directory where the application data of the first application is stored in the storage system. The resource requirement information for each task in the multiple tasks of the first application is determined according to the caching strategy, wherein the caching strategy includes resource requirement information and data caching and access strategies; the resource requirement information includes the size of the cache space required by each task and the type of storage medium included; the data caching and access strategies include: hierarchical caching strategy, data consistency strategy, and security level strategy; Allocate cache space for each task based on its resource requirements. Based on the mapping directory information, the application data of the first application is prefetched into the cache resources of the first application; During the operation of the first application, the cached resources of the first application are accessed according to the caching strategy.

2. The method according to claim 1, characterized in that, The mapping directory information includes the directory path of the first directory. The step of prefetching the application data of the first application into the cache resources of the first application based on the mapping directory information includes: Determine the directory identifier of the subdirectory corresponding to each task in the multiple tasks of the first application; Based on the directory path of the first directory and the directory identifier of the subdirectory corresponding to each task, retrieve the data stored in the subdirectory corresponding to each task under the first directory from the storage system; The data in the subdirectory corresponding to each task is stored in the cache resources of the first application.

3. The method according to claim 1 or 2, characterized in that, Accessing the cached resources of the first application according to the caching policy includes: When the caching strategy includes a hierarchical caching strategy, according to the hierarchical caching strategy, different types of task data for each task of the first application are cached in the corresponding type of storage medium in the cache resources of the first application; When the caching strategy includes a data consistency strategy, a locking operation is performed on the accessed task data when accessing data in any cache space.

4. The method according to any one of claims 1-2, characterized in that, Before accessing the cached resources of the first application according to the caching policy, the method further includes: Obtain input / output I / O requests; If the data accessed by the IO request is data under the first directory indicated by the mapping directory information, then the step of accessing the cached resources of the first application according to the caching policy is executed.

5. The method according to claim 3, characterized in that, Before accessing the cached resources of the first application according to the caching policy, the method further includes: Obtain input / output I / O requests; If the data accessed by the IO request is data under the first directory indicated by the mapping directory information, then the step of accessing the cached resources of the first application according to the caching policy is executed.

6. The method according to any one of claims 1-2 or 5, wherein the method further comprises: The bandwidth requirements of the data to be migrated to the storage system in the cache resources of each of the multiple applications, including the first application, are obtained. Based on the bandwidth requirements, allocate IO bandwidth to the data to be migrated in the cache resources of the first application; Based on the IO bandwidth, the data to be migrated from the cache resources of the first application is stored in the storage system.

7. The method according to claim 3, further comprising: The bandwidth requirements of the data to be migrated to the storage system in the cache resources of each of the multiple applications, including the first application, are obtained. Based on the bandwidth requirements, allocate IO bandwidth to the data to be migrated in the cache resources of the first application; Based on the IO bandwidth, the data to be migrated from the cache resources of the first application is stored in the storage system.

8. The method according to claim 4, further comprising: The bandwidth requirements of the data to be migrated to the storage system in the cache resources of each of the multiple applications, including the first application, are obtained. Based on the bandwidth requirements, allocate IO bandwidth to the data to be migrated in the cache resources of the first application; Based on the IO bandwidth, the data to be migrated from the cache resources of the first application is stored in the storage system.

9. A data access device, characterized in that, The device includes: The receiving module is used to receive a cache configuration request submitted by a user for a first application. The cache configuration request includes a cache policy and mapping directory information. The cache policy is used to indicate the cache requirements of the first application. The mapping directory information is the information of the first directory where the application data of the first application is stored in the storage system. The scheduling module is used to schedule cache resources for the first application according to the caching strategy; The prefetch module is used to prefetch the application data of the first application into the cache resources of the first application according to the mapping directory information. An access module is used to access the cached resources of the first application according to the caching strategy during the operation of the first application. The scheduling module is mainly used for: The resource requirement information for each task in the multiple tasks of the first application is determined according to the caching strategy, wherein the caching strategy includes resource requirement information and data caching and access strategies; the resource requirement information includes the size of the cache space required by each task and the type of storage medium included; the data caching and access strategies include: hierarchical caching strategy, data consistency strategy, and security level strategy; Allocate cache space for each task based on its resource requirements.

10. The apparatus according to claim 9, characterized in that, The mapped directory information includes the directory path of the first directory, and the prefetch module is mainly used for: Determine the directory identifier of the subdirectory corresponding to each task in the multiple tasks of the first application; Based on the directory path of the first directory and the directory identifier of the subdirectory corresponding to each task, retrieve the data stored in the subdirectory corresponding to each task under the first directory from the storage system; The data in the subdirectory corresponding to each task is stored in the cache resources of the first application.

11. The apparatus according to claim 9 or 10, characterized in that, The access module is mainly used for: When the caching strategy includes a hierarchical caching strategy, according to the hierarchical caching strategy, different types of task data for each task of the first application are cached in the corresponding type of storage medium in the cache resources of the first application; When the caching strategy includes a data consistency strategy, a locking operation is performed on the accessed task data when accessing task data in any cache space.

12. The apparatus according to any one of claims 9-10, characterized in that, The device is also used for: Obtain input / output I / O requests; If the data accessed by the IO request is data under the first directory indicated by the mapping directory information, then the step of accessing the cached resources of the first application according to the caching policy is executed.

13. The apparatus according to claim 11, characterized in that, The device is also used for: Obtain input / output I / O requests; If the data accessed by the IO request is data under the first directory indicated by the mapping directory information, then the step of accessing the cached resources of the first application according to the caching policy is executed.

14. The apparatus according to any one of claims 9-10 or 13, wherein the apparatus is further configured to: The bandwidth requirements of the data to be migrated to the storage system in the cache resources of each of the multiple applications, including the first application, are obtained. Based on the bandwidth requirements, allocate IO bandwidth to the data to be migrated in the cache resources of the first application; Based on the IO bandwidth, the data to be migrated from the cache resources of the first application is stored in the storage system.

15. The apparatus according to claim 11, further comprising: The bandwidth requirements of the data to be migrated to the storage system in the cache resources of each of the multiple applications, including the first application, are obtained. Based on the bandwidth requirements, allocate IO bandwidth to the data to be migrated in the cache resources of the first application; Based on the IO bandwidth, the data to be migrated from the cache resources of the first application is stored in the storage system.

16. The apparatus of claim 12, further comprising: The bandwidth requirements of the data to be migrated to the storage system in the cache resources of each of the multiple applications, including the first application, are obtained. Based on the bandwidth requirements, allocate IO bandwidth to the data to be migrated in the cache resources of the first application; Based on the IO bandwidth, the data to be migrated from the cache resources of the first application is stored in the storage system.

17. A computer-readable storage medium, characterized in that, The storage medium stores instructions that, when executed on a computer, cause the computer to perform the data access method according to any one of claims 1-8.

18. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program is used to implement the data access method as described in any one of claims 1-8.

Images