Data processing method and apparatus, and distributed parallel processing system

Abstract

The present application provides a data processing method and apparatus, and a distributed parallel processing system. After receiving a job request for instructing to execute a computing job on data in a target storage space in a first storage medium, a data processing apparatus decomposes the computing job into N computing tasks, and decomposes a task for loading the data in the target storage space into a second storage medium into N loading tasks. Then, the N computing tasks and the N loading tasks are separately executed in a consistent sequence to implement uninterrupted streaming of the N computing tasks, thereby improving the processing efficiency of task nodes and saving processing resources of the task nodes.

Description

Data processing method and device and distributed parallel processing system

[0001] The present application claims priority from the Chinese patent application No. 202410841124.6 filed on June 26, 2024, and entitled "Data processing method and device and distributed parallel processing system", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of communication technology, in particular to a data processing method, device and distributed parallel processing system. BACKGROUND

[0003] The distributed parallel processing system generally includes a management node and a task node cluster. The management node is used to manage the task node cluster, and distribute a computing job to each task node in the task node cluster according to a user request. The task node in the task node cluster is used to execute the computing job in parallel.

[0004] Under a multi-layer storage architecture, the task node can preferentially read data from a storage medium with the fastest access speed (for example, a cache), and if a hit does not occur, the task node needs to read data from other storage media with slower access speeds (for example, a hard disk or even a network storage device) layer by layer until the data is read, and then use the read data to execute the computing job. How to improve the processing efficiency of the task node becomes a problem to be solved. SUMMARY

[0005] In view of this, the present application provides a data processing method, device and distributed parallel processing system for improving the processing efficiency of the task node.

[0006] In a first aspect, the present application provides a data processing method, which can be applied to a data processing device. In the process of executing the data processing method, the data processing device can obtain a job request, which is used to indicate that a data processing task is to be performed on data in a target storage space in a first storage medium. Then, the data processing device can determine a calculation task sequence and a loading task sequence according to the job request. The calculation task sequence includes N calculation tasks arranged in order, and the N calculation tasks are used to implement the data processing task. The loading task sequence includes N loading tasks arranged in order, and the i-th loading task in the loading task sequence is used to load data in a part of the target storage space to a second storage medium, and the i-th calculation task in the calculation task sequence is used to process the data in the part of the target storage space loaded to the second storage medium, where the access speed of the second storage medium is higher than that of the first storage medium, N is a positive integer greater than 1, and i is a positive integer less than or equal to N. Then, the data processing device can execute the N calculation tasks in the order of the arrangement of the calculation tasks in the calculation task sequence, and execute the N loading tasks in the order of the arrangement of the loading tasks in the loading task sequence.

[0007] The data processing device replaces the data to be processed by the data processing task from the first storage medium with a lower access speed to the second storage medium with a higher access speed, and uses the data in the second storage medium to execute the data processing task, which is beneficial to improve the processing efficiency of the data processing device for the data processing task.

[0008] As the amount of data in the target storage space to be processed by the data processing task increases, the data processing device decomposes the data processing task into N calculation tasks, and each calculation task is used to process the data in a part of the target storage space. This is beneficial to fully utilize the processing resources of the data processing device to execute the N calculation tasks, for example, using multiple processing cores of the data processing device to execute different calculation tasks in the N calculation tasks, thereby further improving the processing efficiency of the data processing device for the data processing task.

[0009] Further, the data processing apparatus determines, for each computing task, the storage space in which all data used for processing by the computing task is located, and then determines N loading tasks. The N computing tasks and the N loading tasks are executed in a consistent order, which avoids a computing task that is executed earlier from obtaining data required by the computing task from the second storage medium after all data in the target storage space is loaded, thereby improving the processing efficiency of the data processing apparatus for the computing job and improving the utilization of computing resources. In addition, this also avoids the data processed by a computing task that is executed later from being loaded into the second storage medium first, thereby wasting storage resources of the second storage medium. It should be noted that "earlier" and "later" are used to represent the relative time sequence relationship between different tasks.

[0010] The single loading task mentioned in the present application can include a plurality of sub-loading tasks, and each sub-loading task is used to load a part of all data required by the corresponding loading task. The data processing apparatus can load all data required by the corresponding loading task into the second storage medium by executing the plurality of sub-loading tasks in series or in parallel.

[0011] Optionally, in the process of executing the N loading tasks in the order of the arrangement of the loading tasks in the loading task sequence, the data processing apparatus can execute the (j+1)th loading task in the loading task sequence after the jth loading task in the loading task sequence is executed, where j is a positive integer less than N. The data processing apparatus can determine the start execution time of the (j+1)th computing task before the (j+1)th computing task in the computing task sequence is executed, and determine the start loading time of the (j+1)th loading task according to the start execution time of the (j+1)th computing task. In this way, on the one hand, the data processing apparatus can load data to be processed by the (j+1)th computing task (referred to as (j+1)th data) into the second storage medium before the (j+1)th computing task is executed, thereby improving the processing efficiency of the (j+1)th computing task. On the other hand, this also shortens the time length during which the (j+1)th data occupies the second storage medium and waits for the (j+1)th computing task to be executed, thereby saving storage resources of the second storage medium.

[0012] Optionally, the data processing apparatus determines the start running time of the (j+1)th computing task according to the amount of data processed and the operation speed of the preceding computing tasks before executing the (j+1)th computing task. The preceding computing tasks include the computing task being executed and the computing tasks between the (j+1)th computing task and the computing task. Assuming that the computing task being executed by the data processing apparatus is the (j+1-k)th computing task, the preceding computing tasks of the (j+1)th computing task can include the (j+1-k)th computing task, the (j+1-k-1)th computing task,..., and the jth computing task. k is a positive integer less than or equal to j.

[0013] In this way, the accuracy of the data processing apparatus in determining the start running time of the (j+1)th computing task is improved, thereby ensuring that the (j+1)th data is loaded into the second storage medium before the (j+1)th computing task starts running, improving the processing efficiency of the (j+1)th computing task, and saving the storage resources of the second storage medium.

[0014] Optionally, the data processing apparatus determines the start loading time of the (j+1)th loading task according to the start running time of the (j+1)th computing task and the loading duration of the (j+1)th loading task after determining the start running time of the (j+1)th computing task. The application does not limit the specific determination method, as long as the time interval between the start loading time of the (j+1)th loading task and the start running time of the (j+1)th computing task is greater than or equal to the loading duration of the (j+1)th loading task. In this way, the data required by the (j+1)th computing task is loaded into the second storage medium when the (j+1)th computing task starts running, thereby improving the processing efficiency of the (j+1)th computing task and improving the utilization rate of computing resources.

[0015] The loading duration of the (j+1)th loading task can refer to the duration required to load all data to be processed by the (j+1)th computing task into the second storage medium. Optionally, the data processing apparatus can determine the loading duration of the (j+1)th loading task according to the size of the amount of data loaded by the (j+1)th loading task and the size of the bandwidth of the data processing apparatus loading data from the first storage medium to the second storage medium. In this way, the loading duration of the (j+1)th loading task can be more accurately determined, thereby ensuring that the data required by the (j+1)th computing task is loaded into the second storage medium when the (j+1)th computing task starts running, thereby improving the processing efficiency of the (j+1)th computing task and improving the utilization rate of computing resources.

[0016] Optionally, the method further comprises: deleting, by the data processing apparatus, data processed by a completed computing task in the sequence of computing tasks from the second storage medium. In this way, the storage resources of the second storage medium are saved.

[0017] Optionally, the data processing apparatus decomposes the computing job into the N computing tasks according to the job request, the size of the data in the target storage space, and the size of the processing resources. The processing resources include the available storage space of the second storage medium and / or computing resources for executing the N computing tasks. The application does not limit the specific splitting method of the computing job. For example, the union of the data processed by the N computing tasks is the data in the target storage space. For example, the amount of data processed by a single computing task obtained by decomposition is less than or equal to the size of the available storage space of the second storage medium. For example, the amount of data processed by a single computing task obtained by decomposition is related to the operation speed of the computing resources. The greater the operation speed, the greater the amount of data processed by a single computing task, and vice versa. For example, the difference between the amounts of data processed by different computing tasks is less than a certain threshold (denoted as a first threshold). For example, the difference between the runtimes required by different computing tasks to process the corresponding data is less than a certain threshold (denoted as a second threshold).

[0018] Optionally, the data processing apparatus can determine the storage space (denoted as sub-storage space i) where the data processed by the i-th computing task is located according to the metadata of the data in the target storage space, and further determine the storage location of the data to be loaded by the i-th loading task in the first storage medium. The metadata of the data in the target storage space is used to describe the data in the target storage space, for example, information describing the data in each sub-storage space in the target storage space, such as the size of the amount of data and / or the type of data.

[0019] Optionally, the i-th computing task includes a plurality of sub-computing tasks. In this way, when the data processing apparatus completes the i-th loading task, it loads the data processed by the plurality of sub-computing tasks into the second storage medium, thereby reducing the complexity of the loading task.

[0020] Optionally, the data processing apparatus can be installed on a single task node in a distributed parallel processing system. The distributed parallel processing system includes a task node cluster, and a plurality of task nodes in the task node cluster are used to execute the computing job in parallel.

[0021] The target storage space mentioned in the present application can be a continuous or discontinuous storage space on the first storage medium. Similarly, the sub-storage space i mentioned in the present application can be a continuous or discontinuous storage space on the first storage medium.

[0022] The first storage medium mentioned in the present application can be a single storage medium, or include multiple storage media. When the first storage medium includes multiple storage media, the types of different storage media can be the same or different.

[0023] Similarly, the second storage medium mentioned in the present application can be a single storage medium, or include multiple storage media. When the second storage medium includes multiple storage media, the types of different storage media can be the same or different.

[0024] The data processing device can directly load data from the first storage medium to the second storage medium, i.e., after reading data from the first storage medium, the data is directly written into the second storage medium without passing through other storage media. Alternatively, the data processing device reads data from the first storage medium, stores the data in a third storage medium, and then reads the data from the third storage medium and writes the data into the second storage medium.

[0025] The job request can be used to instruct the data processing device to run the target code, and also to instruct the input data of the target code to be the data of the target storage space. The running process of the target code is used to perform the computing job on the data of the target storage space.

[0026] Optionally, the data processing device performs at least two computing tasks in the N computing tasks by running the same target code, but the input data of the target code is different when the data processing device performs different computing tasks in the at least two computing tasks.

[0027] Optionally, the data processing device performs multiple computing tasks in the N computing tasks by running different code segments in the target code. The input data of the code segments can be the same or different when the data processing device performs different computing tasks in the multiple computing tasks.

[0028] The data used for processing by the i-th computing task can refer to the data input to the target code or the code segment corresponding to the i-th computing task when the data processing device performs the i-th computing task.

[0029] In a second aspect, the present application provides a data processing method in a distributed parallel processing system, the distributed parallel processing system comprising a management node and a cluster of task nodes, the method comprising: receiving, by the management node, a user request, the user request being indicative of performing a computation job on a data set in a first storage medium; and sending, by the management node, a job request to each of a plurality of task nodes in the cluster of task nodes according to the user request, the job request being indicative of performing the computation job on a portion of data in the data set. Each of the plurality of task nodes performs the computation job according to the indication of the received job request, and at least one of the plurality of task nodes performs the computation job according to the method as described in the first aspect or any possible implementation of the first aspect.

[0030] Suppose the at least one of the plurality of task nodes comprises a task node k, the task node k can obtain a job request, the job request being indicative of performing a computation job on data in a target storage space in the first storage medium (i.e. the portion of data in the data set), and then determine a computation task sequence and a load task sequence according to the job request, the computation task sequence comprising N computation tasks arranged in an order, the N computation tasks being used to implement the computation job, the load task sequence comprising N load tasks arranged in an order, an i-th load task in the load task sequence being used to load data in a portion of the target storage space to a second storage medium, and an i-th computation task in the computation task sequence being used to process the data in the portion of the target storage space loaded to the second storage medium, the second storage medium having a higher access speed than the first storage medium, where N is a positive integer greater than 1, and i is a positive integer less than or equal to N. Then, the task node k performs the N computation tasks in the computation task sequence in the order and performs the N load tasks in the load task sequence in the order.

[0031] Since the job request sent by the management node to the task node is indicative of processing a portion of data indicated by the user request, the computation job indicated by the job request can also be referred to as a sub-job of the computation job indicated by the user request.

[0032] Optionally, the job requests sent by the management node to different task nodes are indicative of performing the computation job on different data in the data set. The union of the data indicated by the plurality of job requests sent by the management node to the plurality of task nodes can be the data set.

[0033] The user request can be used to indicate running of the target code, and also used to indicate that input data of the target code is the data set. The running process of the target code is used to perform the computing job on the data set.

[0034] Optionally, the job request sent by the management node to different task nodes is used to indicate that the computing job is performed by running the same target code respectively, but the data processed by the job request sent by the management node to different task nodes can be different.

[0035] Optionally, the job request sent by the management node to different task nodes is used to indicate that the computing job is performed by running different code segments in the target code respectively, and the input data of the code segments can be the same or different.

[0036] Therefore, when the task node runs the method as described in the first aspect or any possible implementation manner of the first aspect, the target code mentioned in the first aspect or any possible implementation manner of the first aspect can be the same as or different from the target code indicated by the user request to run. For example, the target code mentioned in the first aspect or any possible implementation manner of the first aspect can be a part of code (i.e., a code segment) in the target code indicated by the user request to run.

[0037] The third aspect of the embodiment of the present application provides a computer storage medium, which stores one or more instructions, and the instructions cause one or more computers to implement the method as described in the first aspect or any possible implementation manner of the first aspect when the instructions are executed by the one or more computers.

[0038] The fourth aspect of the embodiment of the present application provides a computer program product, which stores instructions, and the instructions cause a computer to implement the method as described in the first aspect or any possible implementation manner of the first aspect when the instructions are executed by the computer.

[0039] The fifth aspect of the embodiment of the present application provides a computer device, which includes a memory and a processor, the memory stores code, and the processor is configured to execute the code, and when the code is executed, the computer device executes the method as described in the first aspect or any possible implementation manner of the first aspect.

[0040] The sixth aspect of the embodiment of the present application further provides a distributed parallel processing system, which includes a management node and a task node cluster; the management node is used to execute any method executed by the management node in the second aspect, and at least one task node in the task node cluster is used to execute the method as described in the first aspect or any possible implementation manner of the first aspect.

[0041] The seventh aspect of the embodiments of the present application further provides a data processing apparatus, comprising: an obtaining module, configured to obtain a job request, the job request being used to indicate performing a calculation job on data in a target storage space in a first storage medium; a management module, configured to determine a calculation task sequence and a loading task sequence according to the job request, the calculation task sequence comprising N calculation tasks arranged in sequence, the N calculation tasks being used to implement the calculation job, the loading task sequence comprising N loading tasks arranged in sequence, an i-th loading task in the loading task sequence being used to load data in a part of the storage space in the target storage space to a second storage medium, and an i-th calculation task in the calculation task sequence being used to process the data in the part of the storage space loaded to the second storage medium, the access speed of the second storage medium being higher than the access speed of the first storage medium, wherein N is a positive integer greater than 1, and i is a positive integer less than or equal to N; a calculation module, configured to execute the N calculation tasks in sequence according to the arrangement sequence of the calculation tasks in the calculation task sequence; and a storage module, configured to execute the N loading tasks in sequence according to the arrangement sequence of the loading tasks in the loading task sequence.

[0042] Optionally, the storage module is specifically configured to execute the (j+1)-th loading task according to the start loading time of the (j+1)-th loading task in the loading task sequence after executing the j-th loading task in the loading task sequence, wherein the start loading time of the (j+1)-th loading task is determined according to the start running time of the (j+1)-th calculation task in the calculation task sequence, and j is a positive integer less than N.

[0043] Optionally, the start running time of the (j+1)-th calculation task is determined according to the amount of data processed by a previous calculation task and the operation speed, the previous calculation task comprising a calculation task being executed in the calculation task sequence and a calculation task between the (j+1)-th calculation task and the calculation task.

[0044] Optionally, the time interval between the start loading time of the (j+1)-th loading task and the start running time of the (j+1)-th calculation task is greater than or equal to the loading duration of the (j+1)-th loading task.

[0045] Optionally, the storage module is further configured to delete, in the second storage medium, data processed by a calculation task executed in the calculation task sequence.

[0046] Optionally, the N calculation tasks are determined according to the job request, the size of the amount of data in the target storage space, and the size of processing resources, the processing resources comprising available storage space of the second storage medium and / or calculation resources used to execute the N calculation tasks.

[0047] Optionally, the part of the storage space is determined according to metadata of the data of the target storage space for the i-th computing task in the computing task sequence.

[0048] Optionally, the i-th computing task comprises a plurality of sub-computing tasks.

[0049] Optionally, the data processing apparatus is deployed or installed on at least one task node in a distributed parallel processing system, the distributed parallel processing system comprising a task node cluster, and a plurality of task nodes in the task node cluster are used to perform the computing job in parallel.

[0050] The apparatus provided in the present application is used to perform the method as described in the first aspect or any one of the possible implementation manners of the first aspect, thus the beneficial effects of the apparatus provided in the present application can be understood with reference to the beneficial effects of the corresponding method described above. BRIEF DESCRIPTION OF DRAWINGS

[0051] FIG. 1-1 schematically shows the structure of a communication system;

[0052] FIG. 1-2 schematically shows a possible flow of a data processing method in the communication system;

[0053] FIG. 1-3 schematically shows data used for processing by different computing jobs in data set;

[0054] FIG. 2 schematically shows the flow of a task node performing a computing job;

[0055] FIG. 3 schematically shows one method of a task node splitting a computing job;

[0056] FIG. 4-1 and FIG. 4-2 respectively schematically show another method of a task node splitting a computing job;

[0057] FIG. 5 schematically shows a possible structure of a data processing apparatus;

[0058] FIG. 6 schematically shows another possible structure of a data processing apparatus;

[0059] FIG. 7 schematically shows a possible flow of a data processing method performed by the data processing apparatus shown in FIG. 6;

[0060] FIG. 8 schematically shows the structure of a computer device. DETAILED DESCRIPTION

[0061] First, the possible application scenarios of the present application are introduced.

[0062] Fig. 1-1 schematically shows a structure of a communication system to which the solution provided in the present application can be applied. As shown in Fig. 1-1, the communication system includes a client and a distributed parallel processing system, the distributed parallel processing system including a management node and a cluster of task nodes. The cluster of task nodes includes a plurality of task nodes, and Fig. 1-1 schematically shows two of them, i.e., task node 1 and task node 2, and the cluster of task nodes can include a larger number of task nodes. The management node is configured to manage the cluster of task nodes, e.g., detect the running state of each task node and / or the size of processing resources, etc., and can have the execution information of a global job. The client can be configured to send a user request to the management node through an interface, and the management node can be configured to distribute a computing job to each task node in the cluster of task nodes according to the user request. Alternatively, the client can be configured to send a job request of a computing job to each task node through an interface. Alternatively, the client and the management node can cooperate to distribute a computing job to each task node. Each task node in the cluster of task nodes is configured to execute the computing job in parallel. After the task node completes the computing job, it can return a processing result to the client through the management node, or directly to the client.

[0063] The communication system shown in Fig. 1-1 can be a distributed processing engine (e.g., spark or apache spark), or a parallel processing architecture, or a distributed cache system (e.g., alluxio or alluxio cluster). Among them, apache spark is an open-source batch processing engine of apache, and alluxio is an open-source in-memory distributed storage system. The communication system shown in Fig. 1-1 is only an example, and the method provided in the present application can run in a single computer device.

[0064] Hereinafter, a possible flow of a data processing method in the communication system shown in Fig. 1-1 is introduced by way of example. As shown in Fig. 1-2, the data processing method can include S101-S104.

[0065] S101, the client sends a user request to the management node through an interface, and correspondingly, the management node receives the user request, the user request being configured to instruct to execute a computing job on a data set in a storage cluster;

[0066] The management node can receive a user request of a single tenant, or receive user requests of multiple tenants. When the user request received by the management node comes from multiple tenants, different tenants can request to execute a computing job on the same or different data in the data set.

[0067] The user request can be configured to instruct to run a target code, and also to instruct that the input data of the target code is the data set. The running process of the target code is configured to execute a computing job on the data set.

[0068] The application does not limit the data set to be stored in the storage cluster. In some examples, the storage cluster in the application example can be replaced by one or more storage media of other types.

[0069] S102, the management node sends a job request 1 and a job request 2 to the task node 1 and the task node 2 respectively according to the user request;

[0070] After receiving the user request, the management node can send the job request 1 to the task node 1 and the job request 2 to the task node 2 according to the user request. Wherein, a single job request is used to indicate to perform a computing job on a part of data in the data set. For example, the job request 1 is used to indicate to perform a computing job on a data subset 1 in the data set, and the job request 2 is used to indicate to perform a computing job on a data subset 2 in the data set. As shown in FIG. 1-3, the union of the data processed by the job request 1 and the job request 2 can be the data set. The task node 1 and the task node 2 can perform the computing job on the data set by performing the computing job on the data subset 1 and the data subset 2 respectively.

[0071] Since the job request sent by the management node to the task node is used to indicate to process a part of data indicated by the user request, the computing job indicated by the job request can also be called a sub-job of the computing job indicated by the user request.

[0072] Under the distributed processing architecture, since the data volume of a single job is large, for example, it takes 1 hour to perform a computing job on a data set, it is generally impossible to put the full amount of data in the memory of a single task node for computing, so a distributed processing method can be used to decompose the computing job into multiple sub-computing jobs (for example, a five-minute task), and then distribute the multiple sub-computing jobs to multiple task nodes to be executed independently, and each task node only processes a part of the data set.

[0073] The data set shown in FIG. 1-3 is only illustrative, and the data set can occupy continuous or discontinuous storage space in the storage cluster. The storage space occupied by the data subset 1 in the storage cluster is referred to as storage space 1, and the storage space occupied by the data subset 2 in the storage cluster is referred to as storage space 2. The storage space 1 and the storage space 2 can be continuous or discontinuous storage spaces respectively.

[0074] S103, the task node 1 acquires the job request 1 and performs a computing job on the data subset 1 according to the indication of the job request 1;

[0075] After receiving the job request 1, the task node 1 can perform a computing job on the data subset 1 according to the indication of the job request 1.

[0076] The job request 1 can be used to instruct the task node 1 to run the target code, and also used to instruct input data of the target code as data in the storage space 1. The running process of the target code is used to perform the computing job on data in the storage space 1.

[0077] As shown in FIG. 1-1, as the data amount of the data subset 1 is increasing, the task node 1 can decompose the computing job into N computing tasks, each of which is used to process a part of data in the data subset 1, and then perform the N computing tasks respectively, which is beneficial to fully utilize processing resources of the task node to perform the N computing tasks, for example, use multiple processing cores of the task node to perform different computing tasks in the N computing tasks, thereby further improving the processing efficiency of the task node on the computing job. FIG. 1-1 takes an example that the task node 1 decomposes the computing job into two computing tasks, and the present application does not limit the specific value of N, as long as N is a positive integer greater than 1.

[0078] In S104, the task node 2 acquires the job request 2 and performs the computing job on the data subset 2 according to the instruction of the job request 2.

[0079] After receiving the job request 2, the task node 2 can perform the computing job on the data subset 2 according to the instruction of the job request 2.

[0080] The job request 2 can be used to instruct the task node 2 to run the target code, and also used to instruct input data of the target code as data in the storage space 2. The running process of the target code is used to perform the computing job on data in the storage space 2 (i.e., the data subset 2).

[0081] Optionally, the job request 1 and the job request 2 are used to instruct to perform the computing job by running the same target code, but the data processed by the job request 1 and the job request 2 (i.e., the data subset 1 and the data subset 2) can be different. As shown in FIG. 1-3, the data subset 1 and the data subset 2 are different data in the data set. In some examples, the data subset 1 and the data subset 2 can have an intersection.

[0082] Optionally, the job request 1 and the job request 2 are used to instruct to perform the computing job by running different code segments in the target code, and the input data of the code segments (i.e., the data subset 1 and the data subset 2) can be the same or different. Therefore, the code run by the task node in the process of performing the computing job according to the job request can be the same as the target code instructed to run by the user request, or the code run by the task node in the process of performing the computing job according to the job request can be different from the target code instructed to run by the user request, for example, the code run by the task node can be a part of the target code (i.e., a code segment) instructed to run by the user request.

[0083] As shown in FIG. 1-1, as the data amount of the data subset 2 is increasing, the task node 2 can decompose the computing job into N computing tasks, each of which is used to process a part of the data in the data subset 2, and then execute the N computing tasks respectively, which is conducive to fully utilizing the processing resources of the task node to execute the N computing tasks, for example, using multiple processing cores of the task node to execute different computing tasks in the N computing tasks, thereby further improving the processing efficiency of the task node for the computing job. FIG. 1-1 takes an example that the task node 2 decomposes the computing job into two computing tasks, and the present application does not limit the specific value of N, as long as N is a positive integer greater than 1.

[0084] It should be noted that different task nodes can decompose the computing job to obtain the same or different number of computing tasks.

[0085] In the communication field, with the increase of data amount and the increase of data types, the data warehouse faces problems such as high investment, low expansion flexibility and difficult operation and maintenance. The cloud native adopts a multi-layer storage architecture to provide customers with flexible expansion, on-demand billing and other experiences. Therefore, the data warehouse is transforming to cloud native, but the multi-layer cloud native storage architecture also brings performance degradation problems. To solve this problem, the industry usually uses cache acceleration technology to improve execution efficiency. Cache acceleration technology is to load data from low-performance media to high-performance cache media and process data in high-performance media to improve processing performance. However, due to the high cost of high-performance media, the size of the cache space is limited, so how to improve the utilization rate and access hit rate of the cache space is the main technical measure to realize the value of the cache.

[0086] FIG. 1-1 schematically shows the cache on the task node. In order to execute the computing job on the data subset in the storage cluster, the task node can load the data subset from the storage cluster to the local cache, and then execute the computing job using the data in the cache.

[0087] As shown in FIG. 2, the task node can also have a hard disk. Correspondingly, the cache on the task node and the hard disk and the storage cluster form a multi-layer storage architecture. The cache on the task node can be referred to as the first storage layer, the hard disk on the task node can be referred to as the second storage layer, and the storage cluster can be referred to as the third storage layer. The cache performance of each storage layer is quite different. For example, the first storage layer has the fastest access speed, but the smallest capacity and the highest cost, and can generally only store a small amount of hot query data. The third storage layer has the slowest access speed, but has a large capacity and the smallest cost per unit of storage space, and generally stores all data. The performance of the second storage layer is generally between that of the first storage layer and that of the third storage layer.

[0088] Under the multi-layer storage architecture, the task node can preferentially read data from a high layer (i.e., a first storage layer), if a hit is not made, read data from a second storage layer, if a hit is not made, read data from a third storage layer, and so on until data is read.

[0089] To improve processing efficiency, an active eviction strategy can be used to load data required by a computing job into the cache in advance or in advance before the computing job is executed. For example, a user can be provided with a loading interface. The loading interface is an interface capability, and the application does not limit the implementation manner of calling the loading interface. For example, alluxio provides a "load" command, such as "$. / bin / alluxio fs load${PATH_TO_FILE}", and the user can customize the task of loading data (for example, an entire file) required by the computing job into the alluxio cache and the loading time of the task. For example, the loading interface can be called by means of a script form, and the loading time of the file is defined.

[0090] However, the data volume of all data (for example, a single or multiple files) required for processing by the computing job is generally large. As introduced previously, the task node can divide the computing job into N computing tasks, each of which is used to process a part of data in a data subset, and then execute the N computing tasks respectively. Since the data volume of the data subset is large, the task node can only load a part of data in the data subset into the cache in each loading process. During execution of a certain computing task (denoted as computing task i), the task node first queries data (denoted as data i) required for processing from the cache, where i is a positive integer less than or equal to N. If data i is hit in the cache, the data i is processed. If data i is not hit, data i is searched in the hard disk. After data i is found in the hard disk, if the remaining storage space of the cache can accommodate the data i, the data i is directly read from the hard disk and placed in the cache as shown by arrow 1 in FIG. 2, if the remaining space of the cache is insufficient, the existing data in the cache is evicted first to release the cache space, and then the data i is loaded into the cache as shown by arrow 3 in FIG. 2. Similarly, if data i is not in the hard disk, the task node can load data i from the storage system to the cache as shown by arrow 3 in FIG. 2. The difference is that the task node generally obtains data from the storage system through network transmission, and obtains data from the hard disk through local hard disk loading. After the task node completes each computing task, the processing result can be returned to the user as shown by arrow 4 in FIG. 2.

[0091] The task node loads data i from the storage system to the cache, which can refer to writing data i in the storage system to the hard disk first, and then loading data i from the hard disk to the cache. The user mentioned in the present application can refer to the client shown in FIG. 1-1. The task node returns the processing result to the user, which can refer to that the task node returns the processing result to the client, or the task node sends the processing result to the management node shown in FIG. 1-1, and the management node returns the processing result of the user's request to the client according to the processing results sent by each task node.

[0092] The present application does not limit the type and quantity of the storage medium of the second storage layer in the task node. For example, the above-mentioned hard disk can be replaced by a magnetic disk and / or a magnetic tape, etc., or the second storage layer can include a magnetic disk and / or a magnetic tape, etc. in addition to the hard disk.

[0093] As shown in FIG. 3, it is assumed that the N computing tasks include computing task 1, computing task 2, computing task 3 and computing task 4, and the data used for processing by each computing task is as shown by the arrows in FIG. 3, for example, computing task 4 is used to process data d in the data subset. Through analysis, it is found that the execution order of the task node for the N computing tasks is generally random, and the loading order of the task node for the data subset is also random. Then, when the task node executes computing task 4 first, the task node can execute multiple loads to load data d into the cache. For example, the task node loads data a first, since data a does not match computing task 4, and the cache space is limited, data a needs to be deleted from the cache, then data b is loaded, since data b does not match computing task 4, and the cache space is limited, data b needs to be deleted from the cache, then data c is loaded, and data c is deleted due to a miss and limited cache space, and finally data d is loaded, and data d is used to execute computing task 4. In this way, it leads to that the task node can execute computing task 4 or start executing computing task 4 after multiple loads, the time delay of the computing task is large, and the running time of the entire computing job is increased. In addition, during the running of computing task 4, multiple data loads need to be performed, so that the full amount of data (i.e. the data subset) occupies the cache in time to hit data d, wasting cache resources. Even if the size of the available storage space of the cache is greater than the data amount of the data subset, since the execution order of the N computing tasks has disorder in the process of phased execution, the full amount of data (i.e. the data subset) required for processing by the N computing tasks needs to be persisted in the cache, which leads to that a large amount of data unrelated to the current stage occupies the cache, and the cache occupancy rate is low.

[0094] To solve the above technical problems, the present application proposes that N computing tasks and N loading tasks are respectively and uniformly sequenced, and then each computing task and each loading task is executed in sequence. The i-th loading task is used to load data (denoted as data i) used for processing the i-th computing task in the storage cluster into the cache. As shown in FIG. 4-1, the computing tasks 1-4 are sequentially computing task 1, computing task 2, computing task 3 and computing task 4 (denoted as computing task sequence 1) in the order of starting running from early to late, the loading tasks 1-4 are sequentially loading task 1, loading task 2, loading task 3 and loading task 4 (denoted as loading task sequence 1) in the order of starting loading from early to late, and the data processed by each computing task and the data loaded by each loading task are shown by arrows in FIG. 4-1. In this way, the task node first executes the computing task 1 in the plurality of computing tasks, and first executes the loading task 1 in the plurality of loading tasks to load the data c into the cache, which is beneficial to the task node to complete the number of loading times required for executing the computing task 1, and is beneficial to reducing the completion delay of the computing task, thereby reducing the running time of the entire computing job. In addition, since the loading task 4 is the last running loading task in the plurality of loading tasks, it is beneficial to avoid the occupation time of the data d required for processing the computing task 4 by the last running computing task 4 on the cache, thereby saving the cache resources.

[0095] In addition, when the cache space is small, in order to avoid deleting the data c required for the computing task 3 after the computing task 1 is completed due to loading the data a and the data b, the order of different computing tasks processing the same data (all the same or part of the same) in the computing task sequence can be set to be adjacent, for example, as shown in FIG. 4-2, the computing tasks 1-4 are sequentially computing task 1, computing task 3, computing task 2 and computing task 4 (denoted as computing task sequence 2) in the order of starting running from early to late, and correspondingly, the loading tasks 1-4 are sequentially loading task 1, loading task 3, loading task 2 and loading task 4 (denoted as loading task sequence 2) in the order of starting loading from early to late. In this way, it is beneficial to reduce the number of loading times in the case that the cache space is smaller than the size of the data subset.

[0096] With reference to FIG. 1-2, the S103 running on the task node 1 can be specifically executed by a data processing device installed on the task node 1. The possible structure of the data processing device and the possible way of executing the S103 are introduced below. The specific implementation of the S104 can be the same as or different from that of the S103.

[0097] FIG. 5 schematically shows a possible structure of the data processing apparatus. As shown in FIG. 5, the data processing apparatus can include an obtaining module, a management module, a computing module and a storage module. In the following, the method executed by each module in the data processing apparatus is introduced by introducing the specific implementation of S103. As shown by the underlined content in FIG. 5, S103 can include S501-S504.

[0098] S501, the obtaining module obtains a job request 1, the job request 1 being used to instruct performing a computing job on data of a storage space 1;

[0099] The obtaining module can obtain the job request 1, the job request 1 being used to instruct performing a computing job on data of the storage space 1. As introduced before, the obtaining module can obtain the job request 1 from a client or a management node.

[0100] S502, the management module determines a computing task sequence and a loading task sequence according to the job request 1;

[0101] After the obtaining module obtains the job request 1, the management module can determine the computing task sequence and the loading task sequence according to the job request 1.

[0102] The computing task sequence includes N computing tasks arranged in order, and the N computing tasks are used to implement the computing job by using data loaded into the cache respectively. For example, the computing task sequence can be the computing task sequence 1 shown in FIG. 4-1 or the computing task sequence 2 shown in FIG. 4-2.

[0103] The loading task sequence includes N loading tasks arranged in order, and the i-th loading task in the loading task sequence is used to load data in a part of the storage space in the target storage space to the second storage medium. For example, the loading task sequence can be the loading task sequence 1 shown in FIG. 4-1 or the loading task sequence 2 shown in FIG. 4-2.

[0104] In addition, the arrangement order of each loading task in the loading task sequence is consistent with the arrangement order of each computing task in the computing task sequence. In other words, the order of the loading task used to load data for the i-th computing task in the loading task sequence is the i-th, and the i-th computing task is used to process the data to be loaded by the i-th loading task. As shown in FIG. 4-1, when the order of the computing task 3 in the computing task sequence is the third, the order of the loading task 3 used to load data c for the computing task 3 in the loading task sequence is also the third. As shown in FIG. 4-2, when the order of the computing task 3 in the computing task sequence is the second, the order of the loading task 3 used to load data c for the computing task 3 in the loading task sequence is also the second.

[0105] The management module can determine, for each computing task, the storage space in which the data used for processing by the computing task is stored in the storage space 1, for example, determining that the data (denoted as data 1) used for processing by the computing task 1 is stored in the sub-storage space 1 in the storage space 1, the data (denoted as data 2) used for processing by the computing task 2 is stored in the sub-storage space 2 in the storage space 1, …, the data (denoted as data i) used for processing by the computing task i is stored in the sub-storage space i in the storage space 1, …, the data (denoted as data N) used for processing by the computing task N is stored in the sub-storage space N in the storage space 1. Then, the management module can determine N loading tasks according to the sub-storage spaces corresponding to the computing tasks (i.e., the storage locations of the data used for processing by the computing tasks in the storage cluster), for example, the i-th loading task is used to load the data in the sub-storage space i in the storage cluster into the cache. Then, the management module can sort the N loading tasks in the order consistent with the computing task sequence to obtain a loading task sequence, for example, so that the i-th loading task is used to load the data i to be processed by the i-th computing task.

[0106] S503, the computing module executes the N computing tasks in the order of the arrangement of the computing tasks in the computing task sequence;

[0107] After the management module determines the computing task sequence, the computing module can execute the N computing tasks in the order of the arrangement of the computing tasks in the computing task sequence. The storage module can include a cache, and the computing module can execute the computing tasks by accessing the cache in the storage module. The computing module can be an entity that runs data and executes computing tasks.

[0108] The computing module can execute the N computing tasks in series in the order of the arrangement of the computing tasks in the computing task sequence by using a single processing unit. Taking the computing task sequence shown in FIG. 4-1 as an example, the processing unit can first execute the computing task 1, then execute the computing task 2 after the computing task 1 is completed, then execute the computing task 3 after the computing task 2 is completed, then execute the computing task 4 after the computing task 3 is completed, and then complete the computing job indicated by the job request 1 after the computing task 4 is completed.

[0109] Alternatively, the computing module can execute the N computing tasks in the order of the arrangement of the computing tasks in the computing task sequence by multiple processing units in parallel. Taking the computing task sequence 1 shown in FIG. 4-1 as the computing task sequence and the multiple processing units including a processing unit 1 and a processing unit 2 as an example, the processing unit 1 can first acquire the computing task 1 from the computing task sequence 1 and start to execute the computing task 1, and then the processing unit 2 acquires the computing task 2 from the computing task sequence 1 and starts to execute the computing task 2, that is, the computing task 1 is executed before the computing task 2, but the execution processes of the two are performed in parallel. That is, the computing task 1 can be completed before or after the computing task 2. Assuming that the computing task 2 is completed before the computing task 1, the processing unit 2 can acquire the computing task 3 from the computing task sequence 1 and start to execute the computing task 3, and then the processing unit 1 acquires the computing task 4 from the computing task sequence 1 and starts to execute the computing task 4. Similarly, the computing task 3 is started before the computing task 4, but the execution processes of the two are performed in parallel by different processing units, so the computing task 4 can be completed before or after the computing task 3. When the computing task 3 and the computing task 4 are both completed, the computing module completes the computing job indicated by the job request 1.

[0110] The application does not limit the type of processing unit, for example, the processing unit can be a processor or a processor core or a thread, etc.

[0111] S504, the storage module executes the N loading tasks in the order of the arrangement of the loading tasks in the loading task sequence.

[0112] After the management module determines the computing task sequence and the loading task sequence, the storage module can execute the N loading tasks in the order of the arrangement of the loading tasks in the loading task sequence.

[0113] Similarly, the storage module can execute the N loading tasks in the order of the arrangement of the loading tasks in the loading task sequence by a single or multiple loading modules. The application also does not limit the type of loading module.

[0114] The data processing apparatus replaces the data to be processed by the computing job from the storage cluster with a lower memory access speed to the cache with a higher memory access speed, and uses the data in the cache to execute the computing job, which is beneficial to improve the processing efficiency of the data processing apparatus for the computing job.

[0115] As the amount of data to be processed by the computing job in the storage space 1 increases, the data processing apparatus divides the computing job into N computing tasks, each of which is used to process the data in a part of the storage space 1, so as to make full use of the processing resources of the data processing apparatus to execute the N computing tasks, for example, using multiple processing cores of the data processing apparatus to execute different computing tasks in the N computing tasks, thereby further improving the processing efficiency of the data processing apparatus for the computing job.

[0116] Further, the data processing apparatus determines the storage space in which the data to be processed by each of the N computing tasks is located in the storage space 1, and then determines N loading tasks, and then executes the N computing tasks and the N loading tasks in a consistent order, so as to avoid the computing task executed first from obtaining the data required by itself from the second storage medium after all the data in the target storage space is loaded, which not only improves the processing efficiency of the data processing apparatus for the computing job, but also improves the utilization of computing resources. In addition, this also helps to avoid the data to be processed by the computing task executed later from being loaded into the second storage medium first, thereby wasting the storage resources of the second storage medium. It should be noted that the "first" and "later" mentioned herein are used to reflect the relative time sequence relationship between different tasks.

[0117] The present application does not limit the order between S503 and S504. The present application also does not limit the storage module to execute the next loading task (for example, the j+1th loading task) in the loading task sequence immediately after executing a certain loading task (for example, the jth loading task) in the loading task sequence. The storage module can execute the j+1th loading task according to the start loading time of the j+1th loading task. Wherein, j is a positive integer less than N. Optionally, the arrangement order between the loading tasks in the loading task sequence refers to the order between the start loading times of the loading tasks.

[0118] The management module can determine the start execution time of the j+1th computing task before the j+1th computing task in the computing task sequence starts to execute, and determine the start loading time of the j+1th loading task according to the start execution time of the j+1th computing task. In this way, on the one hand, it is beneficial for the storage module to load the data (referred to as data j+1) to be processed by the j+1th computing task into the cache before the j+1th computing task is executed by the computing module, thereby improving the processing efficiency of the j+1th computing task. On the other hand, it is beneficial to shorten the time length for which the j+1th data occupies the cache waiting for the j+1th computing task to execute, thereby saving the storage resources of the cache.

[0119] FIG. 6 schematically shows another possible structure of the data processing apparatus, compared with the structure shown in FIG. 5, the functions of the management module are further logically divided, i.e., the management module can include a calculation order planning unit, a loading order planning unit and a running state detection unit. The calculation order planning unit can be used to decompose a calculation job into calculation tasks, determine the execution order of the calculation tasks and the data to be processed by each calculation task, the running state detection unit can be used to track and query the information of the execution state (or running state) of the calculation tasks, the loading order planning unit can be used to determine the loading task and / or the elimination task to be executed according to the information of the execution state obtained by the running state detection unit and the execution sequence of the calculation tasks determined by the calculation order planning unit and the data information of each calculation task, and trigger the storage module providing the data cache service to execute the corresponding task. The running state detection unit is optional. The method performed by each module and unit shown in FIG. 6 will be introduced below by introducing the flow shown in FIG. 7. As shown in FIG. 7, during the execution of S103, the data processing apparatus can specifically perform S701-S710.

[0120] S701, the acquisition module sends the job request 1 to the calculation order planning unit;

[0121] As introduced above, the management node can send the job request 1 to the acquisition module of the task node 1 according to the user request. Correspondingly, the acquisition module can acquire the job request 1 and send the job request 1 to the calculation order planning unit. The job request 1 can include the basic job information requested by the user, such as job ID, job name, user information, submission time, etc., and can also include the encapsulated original program of the job.

[0122] S702, the calculation order planning unit decomposes the calculation job indicated to be executed by the job request 1 into N calculation tasks;

[0123] After receiving the job request 1, the calculation order planning unit can decompose the calculation job into N sub-tasks (i.e., calculation tasks). The application does not limit the way of decomposing the calculation job. For example, the calculation job can be decomposed according to the execution time, and N calculation tasks with similar execution time are obtained to realize the balance of the execution time of each calculation task. For example, the calculation order planning unit can realize the decomposition of the job by the size of the data volume, or by the form of the Partition function, such as by the form of the remainder of the data partition key (key).

[0124] The decomposition of the calculation job can be understood as decomposing the whole set of data into independent sub-data sets, each sub-data set corresponds to a sub-task (i.e., a calculation task), and each sub-task corresponds to a part of the storage space 1. The information of the calculation task can include task ID, task name, data set of the task, etc.

[0125] S703, the computing order planning unit sorts N computing tasks;

[0126] After the computing order planning unit generates N computing tasks (or subtask sets), the N computing tasks need to be sorted to generate an ordered execution sequence of the computing tasks. The sorting method can be implemented in various ways, such as according to the self-incremented number of the subtask, the lexicographic order of the data set path, etc.

[0127] The computing order planning unit sorts the subtasks, which is beneficial to the ordered control of the tasks and clear execution order of the subtasks, to support better calculation of the loading time of the tasks.

[0128] S704, the computing order planning unit sends task information to the computing module;

[0129] After the computing order planning unit determines the execution sequence of the N computing tasks, the task information can be submitted to the computing module for execution. The time of submitting the task information by the computing order planning unit is located in the T0 period (or time period).

[0130] The task information can include the information of the N computing tasks and the execution order of the N computing tasks.

[0131] S705, the computing module executes the N computing tasks in sequence according to the execution order of the N computing tasks;

[0132] After receiving the task information, the computing module can execute the N computing tasks in sequence according to the execution order of the N computing tasks according to the task information.

[0133] S706, the running state detection unit obtains the running information of the computing tasks from the computing module;

[0134] The application does not limit the way in which the running state detection unit obtains the running state of the computing tasks. For example, the computing module can collect the running states of all computing tasks on the node and report the running information of the computing tasks to the running state detection unit. The running information can include the task information and the running state of each computing task.

[0135] The running state of the computing task can include at least one of the following information: task submission time, task start time, and task running status description (to be run, running, and ended).

[0136] S707, the running state detection unit sends the running information of each computing task to the loading order planning unit;

[0137] The running state detection unit obtains the running information of the computing task on the node, and can transmit the running information to the loading order planning unit.

[0138] Optionally, the running state detection unit actively transmits the running information of the computing task to the loading order planning unit only after the information is reported, i.e., at least one state of the computing task changes. Optionally, after the running state detection unit transmits the running information of the computing task to the loading order planning unit for the first time, when the information changes (e.g., the state of a certain computing task changes), the running state detection unit can transmit the updated information to the loading order planning unit, and no longer repeatedly transmit the unchanged information (e.g., the task information).

[0139] S708, the loading order planning unit determines the cache strategy according to the running information of the computing task;

[0140] After the loading order planning unit receives the running information of the computing task, the loading order planning unit can determine the cache strategy according to the running information of the computing task. For example, the loading order planning unit can determine the N loading tasks and the execution order of the N loading tasks according to the task information. The loading order planning unit can also indicate the loading time of at least one loading task. Optionally, the loading order planning unit can also indicate the evicting task and the evicting time of the evicting task, and the evicting task is used to delete the data processed by one or more computing tasks in the cache.

[0141] For example, for the loading task corresponding to the computing task that is running, the loading order planning unit can not update the start loading time in the cache strategy, but can maintain the current cache state in the cache strategy. For the computing task that has ended, if the data of the computing task is still in the cache, the evicting time is marked in the cache strategy, and the evicting time can be the current time or a time after the current time. If the data of the loading task is not in the cache, the evicting time of the loading task can not be marked in the cache strategy. For the loading task corresponding to the computing task to be run, the amount of data processed by the previous computing task and the operation speed can be calculated, and then the start running time of the computing task is determined, and the start loading time of the corresponding loading task is marked in the cache strategy according to the start running time of the computing task. Optionally, the loading order planning unit can also determine the loading duration of the corresponding loading task according to the amount of data processed by the computing task and the loading bandwidth of the current system (hardware attribute), and then determine the start loading time of the corresponding loading task according to the start running time of the computing task and the loading duration of the corresponding loading task.

[0142] When the loading order planning unit receives the updated information of the running information, the loading order planning unit can update the cache strategy.

[0143] S709, the loading order planning unit sends storage control information to the storage module according to the cache strategy;

[0144] The loading order planning unit can send the storage control information to the storage module after determining the cache strategy for the first time or updating the cache strategy. The storage control information is used to indicate information of loading tasks and / or eviction tasks to be executed by the storage module.

[0145] The information of the loading tasks can indicate source addresses of data to be loaded by the loading tasks and loading start times. The information of the eviction tasks can indicate addresses of data to be evicted by the eviction tasks and eviction times.

[0146] Alternatively, the information of the loading tasks can indicate source addresses of data to be loaded by the loading tasks. The loading order planning unit sends the information of the loading tasks to the storage module according to the loading start times. The storage module executes the loading tasks immediately after receiving the information of the loading tasks. Similarly, the information of the eviction tasks can indicate addresses of data to be evicted by the eviction tasks. The loading order planning unit sends the information of the eviction tasks to the storage module according to the eviction times. The storage module executes the eviction tasks immediately after receiving the information of the eviction tasks.

[0147] S710, the storage module executes the N loading tasks in sequence according to the indication of the storage control information.

[0148] After receiving the storage control information, the storage module can execute the loading tasks according to the indication of the storage control information, so as to execute the N loading tasks in sequence according to the order consistent with the N computing tasks. The storage module can also execute the N eviction tasks in sequence according to the indication of the storage control information, so as to execute the N eviction tasks in sequence according to the order consistent with the N computing tasks. The eviction task i is used to evict data processed by the computing task i from the cache.

[0149] The above S705 is used to execute the computing tasks, and the above S706-S710 are used to execute the loading tasks and / or the eviction tasks. The two processes are two parallel processes. The loading task i is completed before the computing task i starts, and i is a positive integer less than or equal to N.

[0150] For example, the storage module executes the loading task 1 at T0 period according to the indication of the storage control information. The computing module executes the computing task 1 at T1 period after T0 period. Since the data processed by the computing task 1 has been preloaded, the computing module can directly read the data processed by the computing task 1 from the cache during the execution of the computing task 1, avoiding blocking or interruption of the computing task 1 during the execution.

[0151] For example, the computing module can complete the computing task 1 in the T1 period, and then continue to execute the computing task 2 (the time period of execution is referred to as a T2 period), the storage module can execute the loading task 2 before the T2 period (for example, the T1 period), and discard the data used by the computing task 1 after the T1 period. In this way, the N computing tasks are completed.

[0152] By sequentially executing the N loading tasks based on the running state of the computing task, the data required for processing of the computing task to be run is preloaded, which is beneficial to realize the similar flow-like non-blocking or interruption-free execution (or flow-like execution) of the N computing tasks, and saves the cache resources.

[0153] The passive replacement strategy is that, after a user initiates a request, if the data in the cache is not hit and the cache is full, the data in the cache is discarded by a specific strategy, so as to give out space to load the data of the latest request. For example, the data discard strategy of the alluxio system supports two methods of least recently used (LRU) and least recently frequently used (LRFU). Among them, LRU releases the block according to the order of least recently used, and LRFU releases according to the order of least recently used and least frequently used with a configured weight. However, the LRU has high new data delay, because after receiving the request, the new data cannot be immediately put into the cache at the first access, but needs to be inserted after deleting the existing data item. This may cause the access delay of the new data to increase. And in the full table scanning scenario, the statistical replacement method will be frequently invalidated. And the LRFU is not adaptive to abnormal traffic, because it needs to consider the access time and access frequency factors, and a fixed value is used to control the proportion of the time factor and the access frequency factor, so when the access mode changes, it cannot be adaptively adjusted, resulting in performance degradation.

[0154] The present application sequentially executes the N discard tasks in the order consistent with the execution order of the N computing tasks, which is beneficial to improve the efficiency of data loading in the limited cache.

[0155] FIGS. 5 and 6 take the storage module including the cache as an example. In some examples, the storage module can not include the cache, and the cache is arranged at other positions in the data processing apparatus or outside the data processing apparatus.

[0156] The above is a detailed description of the method provided by the embodiments of the present application. The data processing apparatus provided by the embodiments of the present application is introduced below.

[0157] The present application does not limit the implementation of each module (and / or unit) shown in FIG. 5 or FIG. 6, nor does it limit the implementation of different modules (and / or units) to be the same. For example, a module can be a part of a computer device (such as a processor, a chip, or a chip system, etc.), or it can be a logic module or software that implements all or part of a computer device, or it can be a computer device.

[0158] The present application does not limit the deployment position of each module (and / or unit) shown in FIG. 5 or FIG. 6. For example, the management node can be deployed on the cluster manager of alluxio, the computation order planning unit can be deployed on the driver program of alluxio, which is responsible for parsing the computation job into computation tasks and determining the execution order of each computation task. The computation module can be deployed on the worker node of alluxio, which specifically computes the running of the task according to the instruction of the computation order planning unit. The running state detection unit can be deployed on the driver program of alluxio, which needs to interact with the information of the running state of each task node. The loading order planning unit can be deployed on the task node of alluxio, which controls the execution of the loading task and / or the eviction task corresponding to each computation task. The storage module can be deployed in the cache component of the task node, which is responsible for executing the loading task and / or the eviction task according to the instruction of the loading order planning unit to cache and / or evict specific data.

[0159] For example, the functions of the data processing apparatus are implemented by a computer device, FIG. 8 schematically shows the structure of the computer device. As shown in FIG. 8, one example of the computer device can include one or more central processors 801 and memories 802, and optionally, at least one of input / output interfaces 803, wired or wireless network interfaces 804, and power supplies 805.

[0160] The memory 802 can be temporary storage or persistent storage. Further, the central processor 801 can be configured to communicate with the memory 802 and execute a series of instruction operations in the memory 802.

[0161] All or part of the functions of the data processing apparatus mentioned above can be implemented by the central processor 801 by executing instructions in the memory 802. Optionally, the data processing apparatus is the computer device shown in FIG. 8.

[0162] The embodiment of the present application also relates to a computer storage medium, which stores a program for data processing, and when the program is executed on a computer, the computer executes the steps performed by the data processing device.

[0163] The embodiment of the present application also relates to a computer program product, which stores instructions, and when the instructions are executed by a computer, the computer executes the steps performed by the data processing device.

[0164] The terms "first", "second", and the like in the description and claims of the present application and the above drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, which is only a distinguishing way adopted in the description of the embodiments of the present application for the objects with the same properties. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, so that the processes, methods, systems, products or devices containing a series of units do not have to be limited to those units, but can include other units not clearly listed or inherent to these processes, methods, products or devices.

[0165] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0166] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the above-described device embodiments are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0167] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. According to actual needs, some or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0168] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0169] When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application, essentially or in the form of a contribution to the prior art, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

Claims

1. A data processing method, characterized by, The method comprises: obtaining a job request, the job request being used to indicate performing a calculation job on data in a target storage space in a first storage medium; determining a calculation task sequence and a loading task sequence according to the job request, the calculation task sequence comprising N calculation tasks arranged in order, the N calculation tasks being used to implement the calculation job, the loading task sequence comprising N loading tasks arranged in order, an i-th loading task in the loading task sequence being used to load data in a part of the storage space in the target storage space to a second storage medium, and an i-th calculation task in the calculation task sequence being used to process the data in the part of the storage space loaded to the second storage medium, the second storage medium having a higher access speed than the first storage medium, wherein N is a positive integer greater than 1, and i is a positive integer less than or equal to N; performing the N calculation tasks in the order of arrangement of the calculation tasks in the calculation task sequence, and performing the N loading tasks in the order of arrangement of the loading tasks in the loading task sequence.

2. The method of claim 1, wherein, The performing the N loading tasks in the order of arrangement of the loading tasks in the loading task sequence comprises: after performing a j-th loading task in the loading task sequence, performing a j+1-th loading task in the loading task sequence according to a start loading time of the j+1-th loading task, wherein the start loading time of the j+1-th loading task is determined according to a start running time of a j+1-th calculation task in the calculation task sequence, and j is a positive integer less than N.

3. The method of claim 2, wherein, The start running time of the j+1-th calculation task is determined according to an amount of data processed by a preceding calculation task and an operation speed of the preceding calculation task, the preceding calculation task comprising a calculation task being executed in the calculation task sequence and a calculation task between the calculation task and the j+1-th calculation task.

4. The method according to claim 2 or 3, characterized in that, A time interval between the start loading time of the j+1-th loading task and the start running time of the j+1-th calculation task is greater than or equal to a loading duration of the j+1-th loading task.

5. The method according to any one of claims 1-4, characterized in that, The method further comprises: deleting, in the second storage medium, data processed by a calculation task in the calculation task sequence that has completed execution.

6. The method according to any one of claims 1-5, characterized in that, The N calculation tasks are determined according to the job request, a size of an amount of data in the target storage space, and a size of a processing resource, the processing resource comprising an available storage space of the second storage medium and / or a calculation resource used to execute the N calculation tasks.

7. The method according to any one of claims 1 to 6, characterized in that, The part of the storage space is determined according to metadata of the data in the target storage space for the i-th calculation task in the calculation task sequence.

8. The method according to any one of claims 1-7, characterized in that, The i-th calculation task comprises a plurality of sub-calculation tasks.

9. The method according to any one of claims 1-8, characterized in that, The method is run on a task node in a distributed parallel processing system, the distributed parallel processing system comprising a task node cluster, a plurality of task nodes in the task node cluster being used to execute the calculation job in parallel.

10. A data processing method, characterized by, The method is applied to a distributed parallel processing system comprising a management node and a cluster of task nodes, and the method comprises: The management node receives a user request for instructing to perform a computing job on a data set in a first storage medium; The management node sends a job request to each of a plurality of task nodes in the cluster of task nodes according to the user request, the job request being for instructing to perform the computing job on a part of data in the data set; The plurality of task nodes perform the computing job according to the received job request respectively, and at least one of the plurality of task nodes performs the computing job according to the method of any one of claims 1-9.

11. A data processing apparatus, characterized by Comprise: An obtaining module, configured to obtain a job request, the job request being for instructing to perform a computing job on data in a target storage space in a first storage medium; A management module, configured to determine a computing task sequence and a loading task sequence according to the job request, the computing task sequence comprising N computing tasks arranged in sequence, the N computing tasks being for implementing the computing job, the loading task sequence comprising N loading tasks arranged in sequence, an i-th loading task in the loading task sequence being for loading data in a part of storage space in the target storage space to a second storage medium, and an i-th computing task in the computing task sequence being for processing the data in the part of storage space loaded to the second storage medium, the second storage medium having a higher access speed than the first storage medium, wherein N is a positive integer greater than 1, and i is a positive integer less than or equal to N; A computing module, configured to sequentially execute the N computing tasks according to the arrangement order of each computing task in the computing task sequence; A storage module, configured to sequentially execute the N loading tasks according to the arrangement order of each loading task in the loading task sequence.

12. A distributed parallel processing system, characterized by The system comprises a management node and a cluster of task nodes; The management node is configured to receive a user request for instructing to perform a computing job on a data set in a first storage medium; The management node is further configured to send a job request to each of a plurality of task nodes in the cluster of task nodes according to the user request, the job request being for instructing to perform the computing job on a part of data in the data set; The plurality of task nodes are configured to perform the computing job according to the received job request respectively, and at least one of the plurality of task nodes is configured to perform the computing job according to the method of any one of claims 1-9.

13. A cluster of computing devices, characterized in that, Comprise at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to run instructions stored in the memory of the at least one computing device, so that the cluster of computing devices runs the method of any one of claims 1-10.

14. A computer program product comprising instructions, characterized in that, When the instructions are run by the cluster of computing devices, the cluster of computing devices runs the method of any one of claims 1-10.

15. A computer-readable storage medium, characterized in that, comprising computer program instructions, which, when executed by a computing device cluster, operate to perform the method of any of claims 1-10.

Images