Data processing method, device and system

Abstract

The invention provides a data processing method, device and system, the method is applied to a computing cluster, the computing cluster comprises a computing resource, a first-layer storage resource and a second-layer storage resource, the computing resource comprises a plurality of sub-computing resources, the first-layer storage resource comprises a plurality of sub-storage resources, and the second-layer storage resource comprises a plurality of sub-storage resources. A first sub-computing resource in the plurality of sub-computing resources corresponds to a first sub-storage resource in the plurality of sub-storage resources, and the performance of accessing the first sub-storage resource by the first sub-computing resource is superior to the performance of accessing other storage resources in the first layer of storage resources by the first sub-computing resource. The first sub-storage resource determines a task executed by the first sub-computing resource, the first sub-storage resource determines information of data corresponding to the task executed by the first sub-computing resource, and the first sub-storage resource loads data from the second-layer storage resource to the first sub-storage resource according to the determined information of the data, the first computing resource retrieves data from the first storage resource to perform the task.

Description

Technical Field

[0001] The present application relates to the field of high performance computing, and in particular to a data processing method, device and system. Background Art

[0002] At present, in large-scale computing systems such as high performance computing (HPC) and supercomputing (SC), multi-layer and multi-level storage is generally used to meet the needs of IO performance and data capacity. It can be generally divided into performance layer and capacity layer storage. Since the data that applications depend on and generate may be stored in the capacity layer, burst buffer is a common technology for performance layer storage in supercomputing systems. The common method is to transfer the data to the performance layer in advance before the application runs (preparation stage) and to the capacity layer during or after the operation. This operation is called data loading. According to the direction of data transmission, it can be divided into staging in and staging out, which represent the data transmission to the performance layer or from the performance layer to the capacity layer. The existing data loading strategy mostly uses hashing to evenly map data from the capacity layer to the performance layer. In some cases, this cannot give full play to the performance of burst buffer and affects the data access efficiency of the application. Summary of the invention

[0003] The embodiments of the present application provide a data processing method, device, and system for improving the locality of application access to data and the data access efficiency of the application.

[0004] In a first aspect, the present application provides a data processing method, which can be applied to a computing cluster, which includes computing resources and multi-layer storage resources, wherein the performance of each layer of storage resources is different, for example, the multi-layer storage resources include at least a first layer of storage resources and a second layer of storage resources, the performance of the second layer of storage resources is higher than that of the first layer of storage resources, the computing resources include multiple sub-computing resources, the first layer of storage resources include multiple sub-storage resources, the first sub-computing resource among the multiple sub-computing resources corresponds to the first sub-storage resource among the multiple sub-storage resources, and the performance of the first sub-computing resource accessing the first sub-storage resource is better than the first sub-computing resource accessing other storage resources in the first layer of storage resources. In one example, the multiple sub-computing resources are used to execute a job, which includes multiple tasks. Each sub-computing resource can be used to execute one or more tasks in the job. The second-tier storage resource can be used to store data for the job, and the data for the job includes data required to execute the tasks. Taking the first sub-computing resource and the first sub-storage resource as an example, the method includes: the first sub-storage resource determines the task assigned to the first sub-computing resource, the first sub-storage resource determines information about the data corresponding to the task of the first sub-computing resource, and the first sub-storage resource loads data from the second-tier storage resource to the first sub-storage resource based on the information about the data; the first computing resource obtains data from the first storage resource to execute the task.

[0005] Through the above method, before the task runs, the first sub-storage resource loads the corresponding data from the second-tier storage resource to the first sub-storage resource according to the information of the data corresponding to the task assigned to the first sub-computing resource. Since the performance of the first sub-computing resource accessing the first sub-storage resource is better than the first sub-computing resource accessing other storage resources in the first-tier storage resource, when the computing resource runs the task of the job, the task data can be obtained nearby from the corresponding first sub-storage resource of the computing node, thereby reducing or even avoiding cross-node or group access to the job data, and improving the locality of application access.

[0006] In a possible design, the first sub-storage resource includes a correspondence between task information and data information, and the first sub-storage resource can obtain data information corresponding to the task assigned to the first sub-computing resource based on the correspondence.

[0007] Through the above method, the first sub-storage resource can efficiently determine the information of the data corresponding to the task based on the corresponding relationship.

[0008] In one possible design, the first sub-computing resource includes at least one computing node, the first sub-storage resource includes at least one storage node, and the at least one computing node is connected to the at least one storage node.

[0009] In one possible design, the first sub-computing resource and the first sub-storage resource exist in the same computing node.

[0010] Through the above method, the first sub-computing resource and the first sub-storage resource are located in the same computing node, so that the first sub-computing resource can quickly read and write data from the local first sub-storage resource when executing tasks, thereby improving the read and write performance of the application and the processing efficiency of the task while realizing flexible expansion of the computing cluster scale.

[0011] In one possible design, the computing cluster also includes a scheduling node, which is used to generate scheduling information for the job, and the scheduling information includes information about the task performed by the first sub-computing resource; the method also includes: the first sub-storage resource obtains the scheduling information generated by the scheduling node, and determines the task performed by the first sub-computing resource based on the scheduling information.

[0012] In the second aspect, the present application also provides a data processing system, which includes a first sub-storage resource and a first sub-computing resource. The first sub-storage resource has the function of implementing the behavior of the first sub-storage resource in the method described in the first aspect above, and the first sub-computing resource has the function of implementing the behavior of the first sub-computing resource in the method described in the first aspect above. The beneficial effects can be found in the description of the first aspect and will not be repeated here.

[0013] In the third aspect, the present application also provides a storage device, which has the function of implementing the first sub-storage resource in the method described in the first aspect above. The beneficial effects can be found in the description of the first aspect and will not be repeated here. The function can be implemented by hardware or by executing corresponding software through hardware. The hardware or software includes one or more modules corresponding to the above functions. In one possible design, the structure of the device includes a determination module and a data loading module. Among them, the determination module is used to determine the task performed by the first sub-computing resource, and determine the information of the data corresponding to the task according to the task. The data loading module is used to load the data from the second-layer storage resource to the first sub-storage resource according to the information of the data; these modules can implement the function of the first sub-storage resource behavior in the method of the first aspect above. Please refer to the detailed description in the method example for details, which will not be repeated here.

[0014] In a fourth aspect, the present application also provides a computing device. The structure of the computing device includes a processor and a memory, and the memory is used to store instructions and / or data. The memory is coupled to the processor, and when the processor executes the program instructions stored in the memory, the function of the first sub-storage resource behavior in the method described in the first aspect above can be implemented. The structure of the computing device also includes a communication interface for communicating with other devices, such as obtaining data corresponding to a task from the first-tier storage resource.

[0015] In a fifth aspect, the present application also provides a computer-readable storage medium, in which instructions are stored, and when the computer-readable storage medium is run on a computer, the computer executes the method in the above-mentioned first aspect and various possible designs of the first aspect.

[0016] In a sixth aspect, the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method in the above-mentioned first aspect and various possible designs of the first aspect.

[0017] In a seventh aspect, the present application also provides a computing chip, which is connected to a memory, and the chip is used to read and execute a software program stored in the memory, and to execute the methods in the above-mentioned first aspect and various possible implementation methods of the first aspect.

[0018] For the description of the beneficial effects of any of the above-mentioned second to sixty-seven aspects, please refer to the description of the technical effects in any possible implementation method of the above-mentioned first aspect, and this application will not repeat them here. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A schematic diagram of a system architecture that may be applicable to an embodiment of the present application;

[0020] Figure 2 Schematic diagram of two Burst Buffer architectures provided for embodiments of the present application;

[0021] Figure 3 A flowchart corresponding to a data processing method provided in an embodiment of the present application;

[0022] Figure 4 A schematic diagram of the execution process of a job provided in an embodiment of the present application;

[0023] Figure 5 A schematic diagram of a data loading process provided in an embodiment of the present application;

[0024] Figure 6 A schematic diagram of various IO modes of an application provided in an embodiment of the present application;

[0025] Figure 7 Another schematic diagram of a data loading process provided in an embodiment of the present application;

[0026] Figure 8 A schematic diagram of the structure of a storage device provided in an embodiment of the present application;

[0027] Fig. 9 A schematic diagram of the structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] High performance computing (HPC) is a cross-industry and cross-application computing discipline that usually uses the most cutting-edge computer technology for the most complex and cutting-edge scientific computing and solutions. It is widely used in the calculation of large-scale scientific problems and the processing of massive data, such as weather forecasting, earthquake prediction, automobile simulation, biopharmaceuticals, gene sequencing, nuclear explosion simulation, rocket launch simulation, and chip design and manufacturing. A computer cluster that is capable of providing HPC services is called an "HPC cluster."

[0029] A computer cluster is a group of loosely or tightly connected computing nodes that work together, usually to perform large jobs. Deploying a cluster is usually more cost-effective than deploying a single computing node of comparable speed or availability to improve overall performance through concurrency. The computing nodes are connected to each other through a network, and each computing node runs its own operating system instance. In most cases, each computing node uses the same hardware and the same operating system, and in some cases, different operating systems can be used on different hardware.

[0030] Figure 1 A schematic diagram of a system architecture that may be applicable to an embodiment of the present application is provided. It should be understood that the system architecture is only an exemplary description, and only provides components related to the embodiment of the present invention, and the number of nodes shown in each component is also only an example, but Figure 1 This is not intended to limit the application of the present invention.

[0031] like Figure 1As shown, the computing node cluster 10 includes multiple computing nodes, such as 100A, 100B, 100C, 100D and 100E. These computing nodes are used to provide computing resources. For a computing node, at the software level, applications can also be installed and run on the operating system. Applications are short for a series of application programs, including applications for executing jobs, such as HPC applications, artificial intelligence (AI) applications, and big data processing applications. HPC applications specifically include WRF (for weather prediction), OpenFOAM (applied in the field of aerodynamics) and other applications, and different applications can be used to process different types of jobs. At the hardware level, it can include multiple processors or processor cores, each processor can include one or more processor cores, and each processor or processor core may be a computing resource, so a physical computing node can provide multiple computing resources. The computing nodes 100A, 100B, 100C, 100D and 100E are interconnected through a network 112. In addition, the computing node 160 is also connected to the network 112 as a scheduler. In operation, scheduler 160 may control the execution of jobs submitted to computing node cluster 10 .

[0032] Jobs can be submitted to the computing node cluster 10 from any suitable source. The present application embodiment does not limit the location where jobs are submitted, nor does it limit the specific mechanism for users to submit jobs. Figure 1 In the example, user 132 can submit job 136 from enterprise 130 to computing node cluster 10. Specifically, in this example, user 132 operates client computer 134 to submit job 136 to computing node cluster 10. In this example, enterprise 130 is connected to computing node cluster 10 via network 120, and network 120 generally represents any telecommunication or computer network, including, for example, an intranet, a wide area network (WAN), a local area network (LAN), a personal area network (PAN), or the Internet. Therefore, user 132 can submit job 136 to computing node cluster 10 from a remote location. Job 136 here is generally a large job that requires more computing resources to be processed in parallel, and the nature and quantity of the job are not limited in this embodiment.

[0033] In the present application, when submitting a job 136, the user 132 can also operate the client computer 134 to submit additional information of the job 136, such as data and configuration parameters of the job 136. Among them, the configuration parameters include but are not limited to the loading rule configuration file of the job 136 (which will be introduced below and will not be repeated here), the number of computing nodes used to execute the job 136, the number of processor cores, the number of processes, etc. Correspondingly, the scheduler 160 can provide a user interface or command line access interface for submitting jobs and additional information of the jobs. The scheduler 160 can also be used to manage computing resources, such as monitoring the busy or idle state of computing resources. The scheduler 160 can also be used to control the resource allocation and life cycle management of application operation, such as scheduling the job 136 to multiple idle computing resources, controlling computing resources to start applications to run the job 136, controlling computing resources to stop applications to end the job 136, and so on.

[0034] The storage system 20 may be used to store data of the job 136. The data of the job 136 includes the initial data of the job 136 submitted by the user. For example, if the job 136 is to predict the weather conditions of a certain area in the future, the initial data of the job may include the historical meteorological data of the area before the time period. The data of the job 136 also includes data such as the intermediate processing results generated by the computing node in the process of executing the job 136. Among them, the storage system 20 may include multiple storage nodes, and the storage device may be a storage server, a storage array, etc., which are usually used to store data persistently. Each storage node may include one or more hard disks, and the hard disk may be a disk or other types of storage media, such as a solid state disk (SSD), a mechanical hard disk (HDD), a storage class memory (SCM), a persistent memory (PM), a shingled magnetic recording hard disk or a volatile memory, etc., without specific limitation, and all memories that can store data are applicable to this embodiment.

[0035] In order to meet the requirements of large-scale supercomputing systems such as HPC or SC for IO performance and data capacity, the storage system 20 usually adopts multi-layer and multi-level storage, such as Figure 1The storage system 20 includes two layers, namely, the first layer storage resources and the second layer storage resources. The difference between the two layers includes that the read and write performance of the first layer storage resources is higher than the read and write performance of the second layer storage resources. For example, the first layer storage resources include multiple SSD hard disks, and the second layer storage resources include multiple HDD hard disks. It is known to those skilled in the art that the read and write rate of SSD hard disks is higher than that of HDD hard disks. For ease of understanding, the first layer storage resources can be recorded as the performance layer, and the second layer storage resources can be recorded as the capacity layer. Among them, the burst buffer is a common technology for performance layer storage of supercomputing systems. By configuring high-speed storage media and accessing the computing node cluster 10 nearby or even deploying it in the computing cabinet or computing node, it provides high-bandwidth, low-latency and fast storage capabilities for applications. The application here refers to the application used to run jobs in the computing node. Since the data that the application depends on and generates may be stored in the capacity layer, the common method is to transfer the job data to the performance layer in the preparation stage (that is, before the application runs the job). And during or after the job is running, the job data is transferred to the capacity layer. This operation is called data staging. According to the different directions of data transmission, data loading can be divided into staging in and staging out. Staging in means that data is transferred from the capacity layer to the performance layer, and staging out means that data is transferred from the performance layer to the capacity layer. Staging in can also be subdivided into data transmission from the capacity layer to the performance layer and between nodes within the performance layer. It can be understood that data loading efficiency has a direct impact on the end-to-end (from start to end) running time of the application.

[0036] Currently, commonly used data loading strategies include data offset hashing, file attribution, data block metadata hashing and other strategies, the purpose of which is to evenly store the job data in the storage resources of the performance layer. For example, the performance layer includes multiple SSD hard disks, and the capacity layer includes multiple HDD hard disks. When any of the above data loading strategies is used to load the data of job 136 into the performance layer, the loaded data will be evenly stored in multiple SSD hard disks, thereby achieving load balancing of the storage device. After the data containing job 136 is loaded into the performance layer, the computing resources can obtain data from the performance layer at a faster read rate when executing the job (relative to obtaining data from the capacity layer), thereby shortening the end-to-end running time of the application.

[0037] These data loading strategies can achieve good results in shared Burst Buffer type storage systems (such as HPE / Cray's DataWrap and DDN's IME, DELL DAC) because the network distance (which can be measured by the number of hops between nodes interconnected by switches, and can be a static physical distance) for all computing nodes in the shared Burst Buffer type storage system to access the performance layer is consistent. However, due to the limitation of the number of network connections under the physical architecture, the shared Burst Buffer type storage system is difficult to expand to a larger cluster scale. Therefore, this embodiment provides two new Burst Buffer architectures and data loading methods suitable for these two new Burst Buffer architectures.

[0038] Next, the technical solution provided by this embodiment is introduced in conjunction with specific drawings.

[0039] Figure 2 The following is an exemplary diagram of two Burst Buffer architectures applicable to the storage system 20 provided in the embodiment of the present application. It can be understood that the performance layer and the capacity layer in the storage system 20 in the present application both include multiple storage nodes. For the sake of distinction, the storage nodes of the performance layer are recorded as burst buffer nodes (recorded as BB nodes) and the storage nodes of the capacity layer are still recorded as storage nodes.

[0040] like Figure 2 In the Burst Buffer architecture shown in (a), the Burst Buffer creates groups, such as BB groups. Each BB group logically includes multiple computing nodes and multiple BB nodes. The BB nodes in a BB group are connected to the nearest computing nodes in the BB group. When the cluster is expanded, it can be expanded in the manner of BB groups, such as expanding exponentially with BB groups as the granularity, thereby solving the memory, performance and IO stability problems caused by too many network connections, so as to achieve logical IO performance that can be expanded to tens of thousands of computing nodes.

[0041] like Figure 2 In the Burst Buffer architecture shown in (b), the performance layer consists of local hard disks of computing nodes (such as Figure 1 In other words, a BB node includes multiple hard disks local to a computing node. It can be seen that this architecture also has no restrictions on the cluster expansion scale, and the cluster scale can reach any number of computing nodes.

[0042] It can be seen that in the two Burst Buffer architectures provided by this application, the network distances from the computing nodes to the performance layer are different. If the aforementioned data loading strategy is adopted in the new architecture, new technical problems will arise. This is because in these two new architectures, application access to BB nodes has local characteristics (that is, some computing nodes can access BB nodes in the group at a shorter network distance, while other computing nodes require a longer network distance to access the same BB nodes), and the applications used to execute jobs are usually large-scale parallel applications. Figure 2 In (a), the application may span multiple computing nodes or Figure 2 In (b), the application may span multiple BB groups. Therefore, when executing a job, the computing nodes under these two architectures may frequently access data across computing nodes or across BB groups, reducing the performance of the application accessing the BB node and increasing the end-to-end running time of the application, thereby affecting the computing efficiency of the cluster and the execution efficiency of the job.

[0043] In view of this, an embodiment of the present application also provides a data processing method, which can be applied to a computing cluster, wherein the computing cluster includes computing resources, a first-layer storage resource, and a second-layer storage resource, wherein the computing resource includes multiple sub-computing resources, the first-layer storage resource includes multiple sub-storage resources, and the first sub-computing resource among the multiple sub-computing resources corresponds to the first sub-storage resource among the multiple sub-storage resources. Among them, the performance of the first sub-computing resource accessing the first sub-storage resource is better than the first sub-computing resource accessing other storage resources in the first-layer storage resources. The method includes: the first sub-storage resource determines the task assigned to the first sub-computing resource, the first sub-storage resource determines the information of the data corresponding to the task of the first sub-computing resource, the first sub-storage resource loads data from the second-layer storage resource to the first sub-storage resource according to the information of the data; the first computing resource obtains data from the first storage resource to perform the task.

[0044] For example, in this method, the computing cluster can be Figure 1 In the system shown, the first-tier storage resources may be storage resources included in the capacity tier, and the second-tier storage resources may be storage resources included in the performance tier. Optionally, Figure 1 The storage system in Figure 2 Any of the structures shown, for example, can be used Figure 2 In the case of the architecture of (a), the first computing sub-resource and the first storage sub-resource may exist in a BB group, the first computing sub-resource includes at least one computing node in the BB group, and the first storage sub-resource includes at least one BB node in the BB group. Figure 2In the case of the architecture of (b), the first sub-computing resource and the first sub-storage resource exist in the same computing node, and the first sub-computing resource may be a computing resource in the computing node, such as a processor core, etc. The first sub-storage resource may be a BB node in the computing node.

[0045] Next, combine Figure 3 The data processing method provided in the embodiment of the present application is applied to Figure 1 The system shown ( Figure 2 Taking any storage system architecture shown in the figure as an example, the data processing method is described in detail. It is worth noting that the method is described from the perspective of a sub-computing resource (i.e., the first sub-computing resource) and a sub-storage resource (i.e., the first sub-storage resource). In fact, in a computing cluster, there may be multiple sub-computing resources for executing tasks of the same job, and each sub-computing resource has its own corresponding sub-storage resource. At this time, each sub-storage resource can execute the operation executed by the first sub-storage resource, thereby completing the loading of data corresponding to the task assigned to the corresponding sub-computing resource.

[0046] Figure 3 A flowchart corresponding to a data processing method provided in an embodiment of the present application. Figure 3 There are two data loading processes, one of which describes the data loading process in the preparation phase of the job, that is, before the job is run (please refer to Figure 5 understand), and the other describes the data loading process during the job running (please combine Figure 7 understand). Figure 3 As shown, the method may include the following steps:

[0047] Step 301 : the scheduler 160 obtains a job to be executed.

[0048] There are many ways for the scheduler 160 to obtain jobs to be executed, such as the scheduler 160 receiving jobs submitted by users. For another example, when the scheduler 160 receives multiple jobs, the multiple jobs can be temporarily stored in a job queue and queued for execution. At this time, the scheduler 160 can select a job to be executed from the job queue based on factors such as the priority of the job, the number of computing resources required for the job, and the number of idle computing resources in the computing node cluster. Usually, the job is executed in the queue order, that is, the first submitted job is executed first.

[0049] Step 302: the scheduler 160 generates a scheduling plan for the job to be executed.

[0050] A job usually includes multiple computing tasks (referred to as tasks), which can be assigned to multiple computing resources for execution. Most tasks are executed concurrently or in parallel.

[0051] For example, job 136 requires predicting the weather conditions of city A within the next hour. Assume that city A includes multiple regions, which are respectively recorded as region 1, region 2, ..., region n (n is a positive integer). For example, job 136 can be divided into multiple tasks according to the granularity of region, such as Figure 4 As shown, the task 136 - predicting the weather conditions of city A in 24 hours is divided into 24 tasks, which are respectively recorded as task 1 to task n, wherein each task is used to predict the weather conditions of an area included in city A within the next hour. For example, task 1 is used to predict the weather conditions of area 1 in the future from 0:00 to 1:00, task 2 is used to predict the weather conditions of area 2 in the future from 0:00 to 1:00, task 3 is used to predict the weather conditions of area 3 in the future from 0:00 to 1:00, and so on.

[0052] The scheduling plan can be used to indicate which computing nodes the multiple tasks included in the job are scheduled to. In one embodiment, the scheduler can use the processor core as the resource allocation unit of the job to implement the scheduling and operation of the job. Specifically, the scheduling plan can be used to indicate the computing resources (such as processor cores) of the tasks that execute the job, and the number of processes to be run by each computing resource. Usually, one processor core is used to run one process, and one process is used to execute one task. In one example, the number of computing nodes and / or the number of computing resources on which the job runs, and the number of processes to be run by the job can be specified by the user, and the scheduler 160 decides which idle computing nodes and / or computing resources in the computing node cluster 10 to use to execute the job based on the status of the monitored computing nodes.

[0053] Exemplarily, the scheduling plan may include one or more of the following: an identifier of a computing node for executing a job, an identifier of a computing resource, an identifier of a task assigned to each computing node or computing resource, and an identifier of a process deployed on each computing resource (such as a process number). In one embodiment, the identifier of a task may be an identifier of a process, that is, the task to be executed is distinguished by a process number, wherein the process number may be assigned by the scheduler 160.

[0054] The subsequent scheduler 160 will schedule the tasks of the job to the corresponding computing resources according to the scheduling plan. The computing node uses the corresponding computing resources to run the application process to execute the assigned tasks, wherein the process distinguishes the tasks to be executed by obtaining its own process number. Those skilled in the art will know that a process is an independent unit for resource allocation and scheduling by the operating system, and is the carrier for the operation of the application. When a program enters the memory to run, it becomes a process. A process is a program in the running process and has certain independent functions.

[0055] Taking homework 136 as an example, suppose Figure 4In the example, n=4 (i.e., city A includes region 1, region 2, region 3, and region 4, and one round of iteration includes 4 tasks), the number of processes is specified to be 4, and a scheduling plan for job 136 indicates that task 1 and task 2 are assigned to two idle processor cores of computing node 100A, and task 3 and task 4 are assigned to two idle processor cores of computing node 100B. Each processor core is used to run a parallel process of the application, which are respectively recorded as process 1-process 4 (or process 0-process 3, and the encoding method of the process number is not limited). Among them, process 1 will be used to execute task 1, process 2 will be used to execute task 2, process 3 will be used to execute task 3, and process 4 will be used to execute task 4. In this way, multiple tasks are executed in parallel by multiple parallel processes.

[0056] It should be noted that the above operations and Figure 4 The decomposition of the work shown and Figure 5 The task scheduling method shown is only an example, and this embodiment does not limit this. For example, the task of the job in this embodiment may also be a task that is executed serially or includes multiple rounds of iterations. Any job decomposition method and scheduling method are applicable to the embodiment of this application.

[0057] It is worth noting that Figure 5 The method is applied to the exact stage of the job. Although the scheduler 160 generates a scheduling plan for the job in step 302, Figure 5 In the method example, the computing resource running application has not been started, that is, the task has not been executed. In addition, the scheduling plan and the method for generating the scheduling plan described in step 302 are an example. All scheduling plans and corresponding generation methods for jobs that can be executed by high-performance computing applications such as HPC applications, whether existing methods or possible future implementation methods, are applicable to the embodiments of this application.

[0058] Step 303: the scheduler 160 obtains the loading rule of the job to be executed.

[0059] The loading rules of the job are used to determine the data to be loaded for the job and the storage location of the data to be loaded in the performance layer. In one embodiment, the loading rules of the job are used to indicate the correspondence between the task information and the data information of the job to be executed, wherein the task information can be used to indicate one or more tasks included in the job, such as the task information can include a task identifier, and a task identifier is used to uniquely identify a task. The data information is used to indicate the data of the job, for example, the data information includes a data identifier, and a data identifier can be used to uniquely identify a piece of data. The BB node can determine the data information corresponding to the task based on the task and loading rules to which the computing resources are assigned.

[0060] This embodiment proposes a method for determining a job loading rule, that is, determining a job loading rule based on an application's IO load characteristics. The IO load characteristics can be understood as an abstraction of the IO behavior generated by the application, and the IO load characteristics may include one or more characteristics such as IO mode, read operation size, write operation size, read and write order / randomness, and concurrent read and write consistency requirements.

[0061] Combine as follows Figure 6 Three IO modes are listed. The IO mode can be distinguished by the number of processes and files that perform IO operations. For example, the IO mode includes rank 0 mode (N:1), independent IO mode (1:1), aggregate IO mode (M:N), etc.

[0062] (1) Rank 0 mode (N:1): Multiple processes of the application read and write a single file concurrently, for example, N processes can read and write a file in parallel.

[0063] (2) Independent IO mode (1:1): Each application process interacts with the storage system 20 independently. Generally, each process accesses its own unique file, and there is no data exchange between processes.

[0064] (3) Aggregate IO mode (M:N): N processes are grouped into m groups, one of which includes multiple processes. One process in the same group is responsible for the IO requests of the other processes in the group (the difference from rank 0 is that in the aggregation mode, the process responsible for read and write IO will aggregate the received IO requests to reduce the number of IO requests to the disk, while rank 0 does not perform aggregation processing and simply passively processes one IO at a time).

[0065] The following describes how to determine the loading rules of a job based on the IO load characteristics of the application:

[0066] by Figure 4 Taking the application shown as an example, some high performance computing applications (such as HPC applications, etc.) are usually run repeatedly and maintain the same IO load characteristics.

[0067] Coarse-grained can be reflected in the fact that when an application executes jobs of the same nature, the processing process for each job may be the same. For example, continue to see Figure 4 , Job 137 is to predict the weather conditions of city A from 1:00 to 2:00 in the future, then Job 137 can be divided into Task 1', Task 2', ..., Task n'. Job 138 is to predict the weather conditions of city A from 2:00 to 3:00 in the future, then Job 138 can be divided into Task 1'', Task 2'', ..., Task n''. It can be seen that the way the application executes Job 136, Job 137, and Job 138 is the same and repeated.

[0068] Fine-grainedness can be reflected in that in a job, the process of executing tasks in each parallel process is usually the same. Simply put, the process is to calculate the output result of a task based on the input data of the task. Taking job 136 and task 1 as an example, the execution process of task 1 specifically includes: process 1 obtains the input data of task 1 from the storage system 20 through read IO. The input data of task 1 can be the meteorological data of the area in a period of time, such as temperature, wind speed, wind direction, air humidity, rainfall, etc. Calculation is performed based on the input data (for weather prediction tasks, equations can be constructed by mathematical methods, and the meteorological data of the area in a period of time can be imported into the same equation for calculation and solution, so as to predict the atmospheric changes and states of the area in the future period of time) to obtain the output result of task 1, such as the meteorological data of area 1 in the future 0:00-1:00. Afterwards, process 1 can write the output result of task 1 to the storage system 20 through write IO. In short, the execution process of each task includes read IO-calculation-write IO. The difference is that the input data of each task is different, and the output result may also be different, but the calculation process is usually the same.

[0069] The initial data of the job (such as the input data of each task of the job) may be submitted by the user or may be generated by the application depending on the previous job, and is persistently stored in the capacity layer of the storage system 20. When performing data loading, the BB node loads the initial data of the job from the capacity layer to the performance layer, and when executing the job, the application (each parallel process) obtains the input data of the task from the performance layer.

[0070] In one access mode, applications access data in the form of files, whether from the capacity layer or the performance layer. That is, the job data is also stored and accessed in the form of files. The multiple files used to store the initial data of the job can be called initial data set files. For example, Figure 4 The initial data set files of job 136 include file 1 (including input data of task 1), file 2 (including input data of task 2), ..., file n (including input data of task n). Specifically, when executing a task, the read IO initiated by the parallel process requests to read a file, which contains the input data of the task executed by the process. Similarly, the write IO initiated by the parallel process is used to request to write a file, which contains the data to be written in the file, that is, the output result of the task executed by the process.

[0071] Above Figure 4 The IO load characteristics of the application shown may include an independent IO mode, such as each parallel process generates a read IO, which is used to read the file containing the input data of the corresponding task, and a write IO, which is used to write the output result of the corresponding task to the corresponding file.

[0072] Based on this, this embodiment provides a specific loading rule, which is used to indicate the mapping relationship between the process and the read-write file. For example, in a loading rule, the task information indicates the process used to execute the task, and the task information may include a process identifier (such as a rank number). A process identifier is used to uniquely identify a process, and the data identifier included in the data information may be the file name, file path, file handle, and other information used to identify the file corresponding to the data. That is, the loading rule of the job is used to indicate the corresponding relationship between the application process and the file of the job (or application).

[0073] The load rule profile for a job can be determined based on the following example:

[0074] Example 1: Describe the correspondence between a file name and the process number of an application. For example, a correspondence is filename->P0, where filename is the file name, indicating that the file corresponding to process P0 (process number is P0) is filename.

[0075] For example, Figure 5 For example, in a possible situation, the IO mode of the application is independent mode. When executing job 136, process 1 will read file 1 containing input data of task 1 by reading IO1, process 2 will read file 2 containing input data of task 2 by reading IO2, process 3 will read file 3 containing input data of task 3 by reading IO3, and process 4 will read file 4 containing input data of task 4 by reading IO4. Assuming that the process number of process 1 is P1, the process number of process 2 is P2, the process number of process 3 is P3, and the process number of process 4 is P4, the file name of file 1 is P1_input.nc, the file name of file 2 is P2_input.nc, the file name of file 3 is P3_input.nc, and the file name of file 4 is P4_input.nc. The job 136 loading rule configuration file may include: P1_input.nc->P1; P2_input.nc2->P2; P3_input.nc->P3; P4_input.nc->P4.

[0076] Example 2: Describe the correspondence between a set of file names and the process number of an application. For example, a correspondence is file1, file2->P0, where file1 and file2 are file names, indicating that the files corresponding to process P0 include file1 and file2.

[0077] For example, if the IO mode of the application is the aggregation mode, process 1 is responsible for the IO requests of process 1 and process 2, and process 3 is responsible for the IO requests of process 3 and process 4. Combined with the process number and file name in the above example, the loading rule configuration file of job 136 may include: P1_input.nc, P2_input.nc->P1; P3_input.nc, P4_input.nc->P3 (process 2).

[0078] Example 3: Use wildcards to describe the correspondence between a set of file names and the process ID of an application. For example, a correspondence is file_P*->P0, which means that the files corresponding to process P0 include all files with the prefix file_P, such as file_PA, file_PB, file_PC, etc.

[0079] For example, when the IO mode of the application is the aggregation mode, and process 1 is responsible for the IO requests of all other processes, combined with the process number and file name in the above example, at this time, the loading rule configuration file of job 136 may include: P*_input.nc->P1. Based on this correspondence, it can be determined that the files corresponding to process P1 include P1_input.nc, P2_input.nc, P3_input.nc, and P4_input.nc.

[0080] Example 4: Use wildcards to describe the correspondence between process numbers and file names of different applications. For example, a correspondence is file_P*->P*, which means that the contents of * on both sides are the same, such as the files corresponding to process P0 include file_P0, the files corresponding to process P1 include file_P1, and so on.

[0081] It is worth noting that the above-mentioned file name naming method is only an example, and the actual file name is the file name of the file storing the job data in the file system (storage system 20). In addition, the contents of the loading rule configuration file listed above are only examples. In this embodiment, the loading rule configuration file may also include a description of the correspondence between the file name and a set of process numbers of the application. For example, a correspondence is file1->P0, P1, indicating that the files corresponding to process P0 and process P1 are both file1. Any rule that can specify the correspondence between processes and files is applicable to the embodiments of the present application.

[0082] In this embodiment, there are multiple ways for the scheduler 160 to obtain the loading rules of the job. In one way, the user submits the loading rule configuration file of the job, such as Figure 5In step 301, the scheduler 160 receives a loading rule configuration file submitted by a user, and the loading rule configuration file includes the loading rules of the job. Optionally, in step 301, the loading rule configuration file of the job and the job may be submitted together. Accordingly, the scheduler 160 may obtain the job submitted by the user and the loading rules of the job in step 301. In one embodiment, the scheduler provides a user interface or a command line to the user. Taking the user interface as an example, the user interface is used by the user to input the loading rules of the job. Optionally, the user interface includes a data loading description mechanism, and the data loading description mechanism is used to prompt the user to input the loading rules of the job. For example, the data loading description mechanism may include but is not limited to any one of the above examples 1 to 4. The user defines the loading rules of the job based on the data loading description mechanism and the IO load characteristics of the application, and writes the defined loading rules into the loading rule configuration file of the job operation.

[0083] In another way, the scheduler 160 determines the loading rules of the job according to the preset system parameters, wherein the preset system parameters may include the loading rule configuration files of commonly used applications. In one possible way, the loading rule configuration files directly include the loading rules of the job. In this case, the scheduler 160 can directly obtain the loading rules of the job from the preset system parameters. In another possible way, the loading rule configuration files do not directly include the loading rules, but are used to indicate how to generate the loading rules of the job. The system can preset the rule relationship between the initial data set file of the commonly used application and the process number of the application, and dynamically generate the loading rule configuration files according to the number of processes applied for when the actual application is running and the file name of the calculation directory. Among them, the computing power directory refers to the file directory where the initial data, intermediate result data and some configuration files of the application are stored. For example, when the loading rule configuration files of the application are as in the above example three or example four, the scheduler 160 also needs to generate the loading rules of the job 136 according to the preset loading rule configuration files of the application and the process name and the file name of the computing power directory actually applied when the application runs the job 136.

[0084] The above is a way for the scheduler to obtain the loading rules of the job by applying the data processing method to the preparation stage of the job. This method can also be applied to other stages of the job. In other stages, in addition to the methods listed above, the scheduler can also generate the loading rules of the job in other ways, which will be introduced below and will not be repeated here.

[0085] Step 304: the scheduler 160 sends a data loading instruction to the first sub-storage resource.

[0086] The data loading instruction is used to trigger the sub-storage resource to perform a data loading operation, and the data loading instruction may include a loading rule of the job to be executed. Optionally, the data loading instruction also includes a scheduling plan for the job to be executed.

[0087] In one example, the first sub-storage resource is a BB node for performing data loading, and a scenario in which the scheduler 160 sends a data loading instruction to the first sub-storage resource may include: in the preparation stage of the job, that is, before the job is run, the scheduler 160 calls the data loading interface of the Burst Buffer system to pass the loading rules of the job to be executed. Specifically, the way in which the scheduler 160 passes the loading rules includes: adding options or environment variables to the job submission syntax to specify the access path of the loading rule configuration file. The scheduler 160 can pass the access path of the loading rule configuration file to the Burst Buffer system through the data loading interface. Alternatively, add options or environment variables to the job submission syntax to specify the string of the content of the loading rule configuration file. The scheduler 160 passes the string to the Burst Buffer system through the data loading interface. Similarly, the scheduling plan of the job is passed. In this way, each BB node used to perform data loading in the Burst Buffer system can obtain the loading rule configuration file of the job, and the optional scheduling plan of the job. The BB nodes used to execute data loading may be part of or all of the BB nodes corresponding to the computing nodes to which the tasks are assigned, and there is no specific limitation.

[0088] In an alternative embodiment, step 304 may also be that the scheduler 160 sends the loading rules of the job to be executed and the scheduling plan of the job to the first sub-storage resource. There is no strict time sequence limitation on the order of sending the loading rules of the job and the scheduling plan of the job. The two may be sent simultaneously or one after another. The sending method may adopt the transmission method described above or other methods, and there is no specific limitation.

[0089] It should be noted that, in an optional manner, the scheduler 160 may also send only the information of the task assigned to the first sub-computing resource in the scheduling plan to the first sub-storage resource.

[0090] Step 305: The first sub-storage resource determines the files to be loaded in the job and the target storage location of each file to be loaded.

[0091] Taking operation 136 as an example, in a specific implementation, step 305 may include the following steps:

[0092] Step 305 - 1 , the first sub-storage resource determines information about the task to which the first sub-computing resource is assigned according to the scheduling plan of the job 136 , such as the process ID of the process running on the first sub-computing resource.

[0093] Assume that the first sub-storage resource is Figure 5The BB node corresponding to the computing node 100A (the first sub-computing resource) in Figure 2 In (a), the BB node is one of the BB nodes in the BB group where the computing node 100A is located. Figure 2 In (b), the BB node is the local hard disk of the computing node 100A. The scheduling plan of job 136 indicates that process 1 (P1) and process 2 (P2) are deployed on the computing node 100A. The BB node determines that the processes corresponding to the BB node include process 1 and process 2 based on the scheduling plan of job 136.

[0094] It is worth noting that in Figure 2 In the architecture shown in (a), since a BB group usually includes multiple BB nodes, the BB node that performs data loading can be one of the BB nodes in the BB group, and the process corresponding to the BB node includes the processes deployed on all computing resources assigned tasks in the BB group. Of course, multiple BB nodes in the BB group can also perform data loading together. In order to avoid repeated loading, the multiple BB nodes are responsible for data loading of some computing resources in the BB group respectively. Figure 2 In the architecture of (b), in order to improve the data loading efficiency, the BB nodes included in each computing node assigned with the task can perform data loading in parallel. Of course, data loading can also be performed by some BB nodes.

[0095] Step 305-2: The first sub-storage resource determines information of the file corresponding to each process (ie, the file to be loaded), such as the name of the file, according to the loading rule of the job.

[0096] Continue with the first sub-storage resource as Figure 5 Taking the BB node corresponding to the computing node 100A as an example, the files to be loaded of the job 136 determined by the BB node include the files corresponding to the processes corresponding to the BB node determined based on the loading rules of the job 136. For example, the processes corresponding to the BB node include process 1 (P1) and process 2 (P2), and the job 136 loading rule configuration file may include: P1_input.nc->P1; P2_input.nc2->P2; P3_input.nc->P3; P4_input.nc->P4. Then the files to be loaded determined by the BB node include: the file P1_input.nc corresponding to process 1 and the file P2_input.nc corresponding to process 2.

[0097] Step 305-3: The first sub-storage resource determines a target storage location of the file to be loaded.

[0098] The target storage location of the file to be loaded determined by the first sub-storage resource is the first sub-storage resource, that is, the storage space formed by the BB node corresponding to the first sub-computing resource in the performance layer of the storage system 20. Figure 2 In the architecture shown in (a), any computing node in a BB group corresponds to any BB node in the BB group. Figure 2 In the architecture shown in (b), the computing resources in a computing node correspond to the BB nodes in the computing node (including the local hard disk of the computing node).

[0099] Optionally, the target storage location of the file to be loaded can be further determined as follows: (1) In the independent IO mode, taking a file to be loaded as an example, the target storage location of the file to be loaded is included in the local BB node or BB group corresponding to the computing node where the process corresponding to the file to be loaded is located in the loading rule. Figure 5 In the example, the target storage location of P1_input.nc and P2_input.nc is the local BB node of the computing node 100A ( Figure 2 (b)) or in the BB group where the computing node 100A is located ( Figure 2 Or, (2) in the aggregate IO mode, taking a file to be loaded as an example, the target storage location of the file to be loaded includes the local BB node corresponding to the computing node where the aggregate process (the aggregate process corresponding to the process corresponding to the file to be loaded in the loading rule) is located ( Figure 2 (b)) or BB grouping ( Figure 2 (a)).

[0100] Step 306: The first sub-storage resource stores each file to be loaded into a corresponding target storage location.

[0101] Taking the first sub-storage resource as a BB node for performing data loading as an example, in one example, the BB node includes a data loading component, and the data loading component is used to load the files included in the job to the performance layer. In the preparation stage of the job, when the BB node runs the data loading component, it writes the files to be loaded of the job (such as the input data containing the task) to the corresponding target storage location to complete the data loading of the initial data. In this way, when the subsequent process executes the task, it can obtain the input data of the task from the local BB node of the computing node or the corresponding BB group nearby, that is, the first sub-computing resource obtains the data corresponding to the task from the target storage location to execute the task, which can improve the locality of application access.

[0102] It should be noted that there is no strict time sequence limitation between step 303 and step 301-step 302. For example, step 301 and step 303 can be executed simultaneously, or step 301 and step 303 can be combined into one step. Figure 1 and Figure 2 The number of storage layers, the number of levels included in each storage layer, and the type of memory included in each storage layer are merely examples of the "multi-layer multi-level architecture" and do not constitute a limitation on the Burst Buffer architecture applicable to the storage system 20 in this embodiment. Figure 1 In the storage system 20, the performance layer and capacity layer may also include other types of memory. For example, the performance layer may also include SCM memory, and the capacity layer may include SSD hard disks and HDD hard disks, or the performance layer may include SSD hard disks, and the capacity layer may include SSD hard disks and HDD hard disks, and so on. For another example, the storage system 20 in this embodiment may include more than two storage layers, such as three-layer or four-layer storage layers, and so on. For another example, each layer in the storage system 20 may be further divided into multiple levels according to the performance of the storage resources. For example, the performance layer may also include two levels, assuming that the second level is composed of SSD hard disks and the first level is composed of SCM memory, wherein the read and write performance of the second-level storage resources is higher than the read and write performance of the first-level storage resources. At this time, Staging in can be further subdivided into loading data from the capacity layer to the performance layer, and loading data from the second-level memory of the performance layer to the first-level memory, and so on. It should also be noted that in order to maintain simplicity, Figure 1 The number of nodes included in each part shown is only an example, and the computing system in practice may have relatively Figure 1 More or fewer nodes as shown, e.g. Figure 1 The computing system in may include at least two schedulers 160, one of which is the main scheduler and the other is the backup scheduler. The main scheduler and the backup scheduler are backups for each other. When the main scheduler fails, one of the backup schedulers takes over the business of the failed main scheduler and continues to provide services to the outside world, thereby avoiding the unavailability of the entire computing system due to hardware failure. For another example, Figure 1 The scheduler in the present application can also be deployed in the form of a scheduler server and a scheduler client. In some scenarios, the scheduler client can also be called a login node. A user can use a client device 134 to log in to the login node to submit a job and the attachment information of the job, and then the login node sends the information submitted by the user to the scheduler server. In addition, in some descriptions, each node or device in the present application may have a different name.

[0103] Through the above design, before the job runs, the BB node determines the job data corresponding to the computing resources used to execute the job according to the loading rules of the job, and loads the job data corresponding to the computing resources into the target storage location of the performance layer. The target storage location is the BB node corresponding to the computing resource, such as the local hard disk of the computing node or the BB group where the computing node is located. In this way, when the computing resource runs the task of the job, the input data of the task can be obtained from the local hard disk of the computing node or the corresponding BB group, reducing or even avoiding access to the job data across nodes or groups, which can improve the locality of application access.

[0104] It is worth noting that the Figure 3 The method instance is combined Figure 5 The method is described as being applied to the preparation stage of a job. The data processing method provided by the present application can also be applied to other stages, such as during the operation of a job, as follows: Figure 7 Come introduce.

[0105] like Figure 7 As shown in the figure, the scheduler obtains the job and schedules the job's tasks to idle computing resources. The computing resources start the application to run the job. During the operation of the job, the checkpoint operation can be used to save the staged state of the application operation, so that it can be restarted from the intermediate state when the operation is resumed, saving computing resources and machine time for completely repeated operation of the application. This method is widely used in high-performance computing, artificial intelligence and other applications that require long-term operation. When executing the checkpoint operation, the checkpoint file (referring to the file of the job generated by the checkpoint operation) will be forced to be removed from the disk to be stored from the performance layer to the capacity layer.

[0106] The scheduler 160 records the BB node used to store the checkpoint file during the application running process. When the application fails and needs to be rescheduled, the relationship between the checkpoint file and the storage location in the BB node is established based on the data loading affinity description mechanism defined in this application according to the storage location of the recorded checkpoint file and the computing node allocated for repeated application running, and a loading rule configuration file is generated.

[0107] During the running of the application, a computing resource failure or a process crash may occur. In this case, the scheduler 160 will reschedule the application process to a new computing node. Based on the data processing method provided in this embodiment, before restarting the application, the scheduler can pass the affinity rules of the checkpoint file and the process information on the reallocated computing node to the BB node, and the BB node reloads the checkpoint file required by the process to the local BB node of the newly allocated computing node or the BB group to which it belongs, so as to complete the data loading of the checkpoint file. For the specific process, please refer to the description of the first embodiment, which will not be repeated here.

[0108] Through the above design, the BB node can pre-load the checkpoint file to the BB node corresponding to the reallocated computing resource, such as the local hard disk of the computing node or the BB group where the computing node is located. In this way, the computing node can obtain the checkpoint file from the local hard disk or the corresponding BB group nearby, reducing or even avoiding cross-node or group data access, which can improve the locality of application access.

[0109] In addition, in the present application, in addition to obtaining the loading rules of the job from the outside, the BB node can also determine the loading rules of the job through the BB node itself. For example, in this example, the BB node monitors the local loss of access data of the application process. For the job data that lacks the loading rules generated during the execution of the job, the BB node can determine the loading rules of these job data by itself, and update the determined loading rules of these job data to the loading rules of the job. For example, the job data that lacks the loading rules may include the checkpoint files generated during the operation of the job. It can be understood that the loading rules of the checkpoint files may not exist in the initial loading rule configuration file. At this time, the BB node can determine the loading rules of the checkpoint files and update the loading rules of the checkpoint files to the loading rules of the job. For another example, the job data that lacks the loading rules includes the data that the computing node needs to access across nodes or groups during the operation of the job. Optionally, it also includes pre-fetched data determined based on factors such as the size, read-only attribute, and access frequency of these data accessed across nodes or groups. For this type of job data, the BB node can determine the loading rules of this type of job data through self-learning (such as based on the experience accumulated from repeated operations of the application). For example, with Figure 2 In the architecture shown in (b), process P0 reads file5 across nodes during multiple rounds of iterations. The storage node (local hard disk) in the computing node 100A where process P0 is located can add the correspondence between file5 and process P0 in the loading rule, and subsequently file5 can be loaded in advance to the local hard disk of the computing node 100A based on the loading rule.

[0110] In the above design, BB nodes monitor and analyze the locality loss of application access data, dynamically generate data loading affinity description rules at runtime, and improve the locality of application access data.

[0111] The method provided by the embodiment of the present application is introduced above in combination with the accompanying drawings, and the device provided by the embodiment of the present application is introduced below in combination with the accompanying drawings.

[0112] Based on the same inventive concept as the method embodiment, the present application embodiment also provides a storage device, which is used to execute the above Figure 3 The method implemented by the first sub-storage resource in the method embodiment. Figure 8As shown, the device 800 includes a determination module 801, a loading module 802, and optionally, an interface module 803. Specifically, in the device 800, each module is connected via a communication path.

[0113] Determination module 801 is used to determine the task to be performed by the first sub-computing resource; determine the information of the data corresponding to the task according to the task; for details, see Figure 3 The description of step 305 in the method embodiment will not be repeated here.

[0114] The loading module 802 is used to load the data from the second-tier storage resource to the first sub-storage resource according to the information of the data; Figure 3 The description of step 306 in the method embodiment will not be repeated here.

[0115] In a possible implementation, the interface module 803 is used to obtain scheduling information generated by the scheduling node, where the scheduling information includes information about the task executed by the first sub-computing resource. Figure 3 The description of step 304 in the method embodiment will not be repeated here.

[0116] The present application also provides a computing device 900. Fig. 9 As shown, the computing device 900 includes: a bus 902, a processor 904, a memory 906, and a communication interface 908. The processor 904, the memory 906, and the communication interface 908 communicate through the bus 902. The computing device 900 can be a server or a storage array or a hard disk enclosure, etc. It should be understood that the present application does not limit the number of processors and memories in the computing device 900.

[0117] The bus 902 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 The bus 902 may include a path for transmitting information between various components of the computing device 900 (eg, the memory 906, the processor 904, and the communication interface 908).

[0118] The processor 904 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0119] The memory 906 may include a volatile memory, such as a random access memory (RAM). The processor 904 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid state drive (SSD), etc., without limitation.

[0120] In one embodiment, the computing device 900 is used to execute the functions of the storage device 800, wherein the memory 906 stores executable program codes, and the processor 904 executes the executable program codes to respectively implement the functions of the aforementioned determination module 801, loading module 802, and interface module 803, thereby implementing the data processing method. That is, the memory 906 stores instructions for the computing device 900 to execute the data processing method provided by the present application.

[0121] The communication interface 908 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 900 and other devices or communication networks.

[0122] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the data processing method.

[0123] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the data processing method.

[0124] Optionally, the computer-executable instructions in the embodiments of the present application may also be referred to as application code, which is not specifically limited in the embodiments of the present application.

[0125] It should be understood that the processor mentioned in the embodiments of the present application can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented by software, the processor can be a general-purpose processor implemented by reading software code stored in a memory.

[0126] Exemplarily, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0127] It should be understood that the memory mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DirectRambus RAM, DR RAM).

[0128] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, the memory (storage module) can be integrated into the processor.

[0129] It should be noted that the memory described herein is intended to include, without being limited to, these and any other suitable types of memory.

[0130] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0131] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0132] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0133] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0134] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A data processing method, characterized in that: Applied to a computing cluster, the computing cluster includes computing resources, first-layer storage resources, and second-layer storage resources, the computing resources include multiple sub-computing resources, the first-layer storage resources include multiple sub-storage resources, a first sub-computing resource in the multiple sub-computing resources corresponds to a first sub-storage resource in the multiple sub-storage resources, and the performance of the first sub-computing resource accessing the first sub-storage resource is better than the first sub-computing resource accessing other storage resources in the first-layer storage resources, the method includes: The first sub-storage resource determines a task performed by the first sub-computing resource; The first sub-storage resource determines information of data corresponding to the task according to the task; The first sub-storage resource loads the data from the second-tier storage resource to the first sub-storage resource according to information of the data; The first computing resource obtains the data from the first storage resource to perform the task.

2. The method according to claim 1, characterized in that The first sub-storage resource includes a correspondence between the information of the task and the information of the data, and the first sub-storage resource determines, according to the task, the information of the data corresponding to the task, including: The first sub-storage resource obtains information about the data according to the corresponding relationship.

3. The method according to claim 1 or 2, characterized in that The first sub-computing resource includes at least one computing node, the first sub-storage resource includes at least one storage node, and the at least one computing node is connected to the at least one storage node.

4. The method according to claim 1 or 2, characterized in that: The first sub-computing resource and the first sub-storage resource exist on the same computing node.

5. The method according to any one of claims 1 to 4, characterized in that: The computing cluster further includes a scheduling node, the scheduling node is used to generate scheduling information, the scheduling information includes information of tasks executed by the first sub-computing resource; The first sub-storage resource determines a task to be performed by the first sub-computing resource, including: The first sub-storage resource determines the task to be performed by the first sub-computing resource according to the scheduling information obtained from the scheduling node.

6. A data processing system, characterized in that: The system includes a computing resource, a first-tier storage resource, and a second-tier storage resource, the computing resource includes a plurality of sub-computing resources, the first-tier storage resource includes a plurality of sub-storage resources, a first sub-computing resource among the plurality of sub-computing resources corresponds to a first sub-storage resource among the plurality of sub-storage resources, and the performance of the first sub-computing resource accessing the first sub-storage resource is better than the performance of the first sub-computing resource accessing other storage resources in the first-tier storage resources, and the method includes: The first sub-storage resource is used to determine the task performed by the first sub-computing resource; The first sub-storage resource is used to determine information of data corresponding to the task according to the task; The first sub-storage resource is used to load the data from the second-tier storage resource to the first sub-storage resource according to information of the data; The first computing resource is used to obtain the data from the first storage resource to perform the task.

7. The system according to claim 6, characterized in that The first sub-storage resource includes a correspondence between the information of the task and the information of the data; The first sub-storage resource is specifically used to obtain information about the data according to the corresponding relationship.

8. The system according to claim 6 or 7, characterized in that The first sub-computing resource includes at least one computing node, the first sub-storage resource includes at least one storage node, and the at least one computing node is connected to the at least one storage node.

9. The system according to claim 6 or 7, characterized in that The first sub-computing resource and the first sub-storage resource exist on the same computing node.

10. The system according to any one of claims 6 to 9, characterized in that: The computing cluster further includes a scheduling node, the scheduling node is used to generate scheduling information, the scheduling information includes information of tasks executed by the first sub-computing resource; The first sub-storage resource is specifically used to determine the task executed by the first sub-computing resource according to the scheduling information obtained from the scheduling node.

11. A computing device, characterized in that: including a processor and a memory; The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device executes the method executed by the first sub-storage resource as described in any one of claims 1-5.

12. A computer-readable storage medium, characterized in that: When the computer-readable storage medium is executed by a storage device, the storage device executes the method executed by the first sub-storage resource as described in any one of claims 1 to 5.

Images